Eye contact

A hand makes a delicate gesture in the air, as if it were turning a dial. With just a flick of the wrist, the hand then swipes to the left. It is the hand of Christina Huber, User Experience Designer at Audi, sitting in a mock-up of the Audi activesphere concept. She takes off her high-tech glasses – and smiles: 

“It’s all about intuitive movement – that’s how Audi dimensions works.” 

Sitting next to her, Jan Pflüger, Co-ordinator Augmented & Virtual Reality and Mixed Reality Expert at Audi, agrees: 

“We equip our customers with sensor technology that works both ways. I deliver the information the users need. At the same time, I get to find out what the users actually want to do with it.”

Christina and Jan developed the innovative operating concept of the Audi activesphere concept, which represents another great leap forward in terms of technology. The graphical user interface (GUI) is no longer permanently displayed in just one place, on screens or projected, but rather is now reproduced in the right place at the right time. It is no longer just two-dimensional either, but three-dimensional, and when you are wearing mixed reality glasses, the operating elements will appear in a three-dimensional space, allowing you to interact with the content using natural gestures within this space.
“With this operating concept, there are no classic screens where you tap on the selected content,” says Christina.  

“The glasses use sensors to detect what is in the focus of the user and move the control element towards them.” 

The fact that a vehicle uses sensors to get an understanding of the surroundings has actually become state of the art, but what’s entirely new is that mixed reality glasses offer the driver the same sensor technology to locate them in space. 

“We want to interact spatially and use the most advanced mixed reality technology currently available,” says Jan.

“Unlike virtual reality glasses that will isolate you from the real world, you can actually see the world around me through the transparent displays of the mixed reality glasses. Any kind of information that is relevant for use is then displayed in your field of vision.” 

The mixed reality glasses, which locate the driver in space, are equipped with numerous sensors and cameras, helping to contextualise the surroundings, programming the experience with the glasses in a way that all the controls appear exactly where they’re needed.

If a user is looking aimlessly around them, nothing will happen and the UI elements, which enable various operations, remain inactive. But once the user focuses their eyes on one of these elements and signals interest, the system displays more detailed information, moving the element towards the user where it can be controlled via gestures. 

As an example, the so-called ‘Audi dimensions anchor point’ located in the door of the Audi activesphere concept is a physical element. When you rest your eyes on it, the system will register your interest and then display the current temperature of the interior. 

But when you focus on it for longer, it will become interactive – the UI element moving towards the user and can be controlled virtually. When you’re done, it disappears after a few seconds. Alternatively, you can also swipe the content away with your hand, thus returning the UI element to its sleep mode. So there are not just physical elements superimposed with virtual 3D content, but an intelligent, context-based user interface which is flexibly displayed and can be controlled seamlessly and conveniently. 

Of course the system also changes to suit the desired operating mode. When you are in the 'driver's' seat of the Audi activesphere concept, the steering wheel is retracted in an invisible position, but when the driver wants to take over, the steering wheel swivels out from its invisible position and the content of the user interface will adjust to manual driving mode.

“Even with navigation, you don’t just click on an icon to access a use case, but you can interact three-dimensionally with the navigation map,” says Christina.

“There are different layers for different levels of information, which build up gradually depending on what interests you.”
Even when you leave the vehicle to explore the outside world, the glasses can continue to be used and supply information on the surrounding terrain. Say you head off on your mountain bike or your skis, for example, the glasses will display a host of information such as trail networks, slope gradients or warnings of limited visibility or hazards ahead. 

“Not only are we making sure that now, for the first time, you can use such a technology to interact with your vehicle and control its functions, but we go far beyond that,” Jan says.

“Ultimately, the vehicle is a digital mixed reality platform that is ready for what we hope to see in the future – I get into the car and it comes naturally to me to control it via the MR glasses. I get out of the car and go about my activities such as skiing or mountain biking, where the glasses will also be of support. The vehicle is my extended companion, enabling this seamless experience.” The car of the future is embedded in an ecosystem, enriching the real world with context-based virtual content.

This technology of the future enables designers to create an unimagined user experience. “The vehicle itself is a holistic feel-good zone without any visible computer technology,” enthuses Christina Huber.

“And yet it is intelligent – but only in the background. Only once I put on the mixed reality glasses, I will bring the controls to life.”
This opens up completely new possibilities for vehicle design in the future, where designers will no longer have to consider where controls should be physically placed. No matter how and where you sit in the vehicle, be it in a lounge or a reclining position, with mixed reality glasses, accessibility to all controls is always guaranteed. This makes completely different design languages conceivable and a completely new interior design becomes possible. The only barrier now is imagination.