Researcher Highlights
Using avatars and VR/AR to shape a functional tomorrow

The use of digital user representations and immersive technologies extends beyond the gaming context. By exploring and applying them in serious gaming, Dr Jacqueline Bailey is improving learning outcomes and the lives of human users.

With a PhD in Information Technology (IT), Jacqueline’s core area of research is human-centred computing (HCC), focusing on the development and use of virtual humans, known as avatars, in serious gaming, simulation, and training.

Serious gaming refers to using video games or game-like interactive experiences for purposes beyond entertainment—and Jacqueline’s key industries of interest are healthcare, military, and education where serious gaming, avatars, and simulation training are common.

Her research also focuses on developing and using immersive technologies such as virtual or augmented/mixed reality for the same application.

“My overarching aim is to create novel and interactive learning experiences through role-play scenarios with ‘realistic-enough’ virtual humans,” explains Jacqueline.

“The global virtual reality (VR) market size  was estimated at USD 59.96 billion in 2022 and is expected to grow at a compound annual growth rate (CAGR) of 27.5 per cent from 2023 to 2030. It’s therefore important to understand how best to maximise the use of these technologies for serious purposes.”

The benefits of gamified learning

Jacqueline was drawn to the field of HCC because of its boundless potential to use cutting-edge technology that offers real-world impacts and its ability to shape a better, more functional tomorrow.

“Avatars and immersive realities used in a serious context have the potential to revolutionise the way we train in industry. We can directly engage learners through role-play scenarios to develop skills vital for future career success.”

Developing immersive reality scenarios in the medical, military, and education fields is not only less risky than real-life training, but gamified learning has also been shown to help professionals achieve a deeper understanding.

Some examples include simulations for medical students to practice surgeries, for emergency responders to rehearse disaster scenarios, for soldiers to navigate combat and hostage situations, and for teachers to make abstract concepts tangible through interactive visuals.

Realistic enough virtual humans

However, Jacqueline says there are several ongoing challenges in the field of human-avatar interactions, including realism, uncanniness, expressions, and user acceptance.

This includes how real the avatar looks and how we can prevent a ‘too real’ looking avatar from negatively affecting the end-users.

Because of this, her research focuses on the quantitative analysis of human perceptions of avatars. In particular, the human startle reflex is used as a measure of affective processing.

“Previously, under Professor Karen Blackmore from the School of Design, Communication and IT and i3 Lab, I worked collaboratively with the Australian Defence College Wargaming and Simulation Centre to evaluate how emotional expressions in avatars’ of differing levels of realism and fidelity are perceived by human users.

Jacqueline’s 2016 research, Exploring Avatar Facial Fidelity and Emotional Expressions on Observer Perception of the Uncanny Valley, won her the best paper award at the Australasian Simulation Forum. She was also invited to speak at a ‘Best from around the world’ Session in the USA.

The findings from this research, particularly relating to the role of avatar gender, have helped inform the development of systems for avatar-augmented role play within the Australian Defence College.

More recently, in 2023, she was part of a grant funded by Meta (formerly Facebook) for $93K about highly realistic avatars in a virtual work setting.

Increasing education, jobs, and inclusivity

While her research career is still in its early stages, Jacqueline’s work has the potential to improve people’s lives by offering insights into human-centred technologies, paving the way for more informed decisions and better outcomes for humans using avatar-based and immersive technologies.

At the national level, her findings have been recognised through the publication of several conference papers presented locally, nationally, and internationally, offering a fresh perspective on avatar realism and uncanniness.

Beyond this, she says that the serious gaming industry has opened new job opportunities for game designers, educators, and tech professionals.

“Avatars can also be used in various ways, such as increasing empathy and perspective. Users can step into the shoes of someone from a different background, culture, or even historical period, fostering empathy and cultural understanding.”

“In addition, creating diverse avatars enables representation of various ethnicities, genders, and orientations, fostering inclusivity in the digital realm.”

From avatars to smartphone interfaces

In addition to her work on avatars and immersive realities, Jacqueline has recently branched out as an early career researcher (ECR) to undertake a study on the use of smartphones by older people.

For this, she has put together a team of researchers from all three University of Newcastle colleges to gain insights into the design development of the Older Persons Mode (OPM) — a new, more useable interface for this demographic.

As well as drawing on perspectives from IT, she’s also working with the Creative Industries to help with user experience and design guidelines and the School of Nursing and Midwifery to ensure they consider the physical and cognitive needs of older people who may struggle to use digital technology.

“These insights have been extremely useful as we work towards a solution that works with the older populations in a way that makes them feel safe and confident when using these devices,” she shares.

The work aims to help non-digital natives (60+ years old) feel more connected with the wider world via their phones. This further benefits older people, their families and carers with an easier and more trusted system that allows them to be less dependent on their immediate circle.

Considerations and endless possibilities

Through her work, Jacqueline aims to design solutions with the end-users, not against them.  “No technology solution should ever be developed without the discussion and involvement of its target audience”, she shares, “to do so, wastes the potential the good a tech solution can have”.

Things like cost, use, and ethical and inclusive design principles are also always front of mind. Plus, she’s very aware of the need to balance novelty with usefulness. This is where her partnerships with industry prove invaluable.

Ultimately, her goal is to positively impact the lives of human users who engage with human-computer interaction (HCI) and HCC products.

What keeps her motivated in her work are the possibilities. She shares that there are so many applications in the world of avatars and immersive realities, and she’s excited to continue to find, build, and test tech solutions.

To this end, she has formed an international collaboration team called Interactive Intelligent Virtual Agents (I2VA), comprising ECRs in the human-avatar/agent field from colleges in Australia, Belgium, Japan, and Portugal.

She’s also currently in the early stages of developing a NUGamesTech team (similar to NUBots) for virtual reality competitions. This student-led team will create VR solutions for national and international competitions and run as a part of Professor Karen Blackmore’s i3Lab.