Design Engineering
Showcase 2020

Being able to guide the project into areas that I thought were applicable and the I found interesting was one of the highlights of the internship. Having that freedom made me really passionate about the work I was doing. I’m going to present the Design Engineering Thinking used and learnt over the internship by describing the main project I worked on – trying to give teleoperators 360o perception through only a pair of headphones.

The overarching aim of this project was to further Nokia Bell Labs’ understanding of what the best way could be to improve the performance of wheeled robot teleoperators – specifically using multimodal feedback (where information is passed through multiple sensory channels at the same time). My first task was a deep dive into the current state of the art in terms of wheeled robot teleoperation. What information do teleoperators need? What’s the best way to do this now? What weak signals haven’t yet been tried? This research highlighted many areas, from researchers using force feedback joysticks to relay direction to vibrating haptic displays to indicate where collisions were occurring. The area that interested me most was the use of audio feedback, particularly spatial audio, as an incredibly rich way to communicate complex information.

We know that advancements in 5G, robotics and edge computing will quickly allow more and more wheeled robots to enter the workspace. As this happens, we’ll find robots often being placed in chaotic, dynamic environments, where they’ll have to navigate around people and other moving obstacles. Despite the autonomous systems controlling these robots becoming increasingly more advanced, due to the unpredictable nature of these environments, there will always be situations where it will be necessary for teleoperators to step in and take control. When this happens, we need feedback and control systems in place that allow the operators to control the robots as effectively as possible. We need a way to pass the complex information from the robot’s LIDAR and other sensors to the teleoperator in a way that is simple and intuitive to understand.

So, what information actually needs to be passed to teleoperators? We need to package the complex movements of people and other obstacles around the robot in such a way that a teleoperator can understand it immediately. They need to know if someone is coming up behind the robot, or about to step out in front as easily if they themselves were there. A critical factor to achieving this is the need for the operator to understand the information about the dynamic environment at the same time as understanding the visual information through the front facing camera. Current systems use multiple visual displays; one for the camera, and one for the environment information. When doing something quickly becomes really important, overloading the visual field with multiple viewpoints makes things very difficult for the operator! Their performance slows, and the cognitive workload of each task massively increases. What if we could pass the same information through a different sensory channel?

This is where multimodal feedback systems are highly effective. They are one of the few ways in which an operator can be trained to comprehend multiple information streams simultaneously. What if the operator could be trained to understand all the necessary information about the environment around the robot simply by listening to audio? The theory behind neuroplasticity states that given enough training, a person can be taught to associate any sensory stimulus with a piece of information. What if we could associate abstract spatial audio to the entire environment around the robot? What if we could give someone 360o perception of a completely remote environment, through only a pair of headphones?

With this as a goal in mind, I had two main tasks before the end of the internship. First, I had to design an effective way to relay the environment information as audio. Second, I had to build an experiment that would allow me to test this completely remotely (due to COVID-19). I trialled several different approaches to sonify the environment, initially through a more general approach called Panoramic Audio – with tried to express the entire environment as a form of abstract spatial audio. Whilst theoretically interesting, there was far too much complexity for the operator to extract useful information easily. The second approach refined the environment to only the moving obstacles, Focused Audio. Here only the most critical features of the environment were sonified. This was much more effective and promised a real advantage to wheeled robot teleoperators. Next was the task to prove and quantify this.

The experiment was run through a Unity build – a single executable file that participants ran five times to train themselves up. Initially people really struggled to understand the audio, but as they practised more and more their scores improved. By the end of the five sessions, simply having the presence of the audio was enough to increase performance by 45% compared to those who didn’t have it. This improvement was about 50% greater than the next best audio feedback system for wheeled robot teleoperators that I came across in my literature review. Participants were able to intuitively understand the audio and react to moving obstacles they couldn’t see. They could cast their perception ahead of them and listen around corners before they made a turn. They were able to achieve 360o perception, through only a pair of headphones.

My name is Luke Holland, and I’ve just completed a six-month internship at Nokia Bell Labs, Dublin, as part of the Hardware Innovation and Prototyping team. Prior to the COIVD-19 pandemic, my role at Nokia Bell Labs was mainly going to be associated with hardware – through the design and development of wearables. However, when the pandemic struck, and I had to go fully remote (never meeting my team in person) and the nature of my work changed. I shifted into a predominantly software-based role and learnt so much as a result. My contributions were much more research based and having the chance to explore a cutting-edge concept was incredible.

Hopefully the work I was doing will at some point be incorporated into one of the larger industrial application projects at Nokia Bell Labs. My brief was to further the understanding of what was possible in the world of teleoperation. How might we be looking to control wheeled robots in the future? What new means of feedback could we adopt? A method that uses abstract spatial audio could be a viable option - allowing efficient transfer of information, completely intuitively.

One of the things I loved about working for Nokia Bell Labs, was the freedom to push the project in my own direction, whilst also being to draw from the vast experience of those around me. Reaching out to other teams and not tackling problems alone was definitely one of the most useful working styles I adopted over my time there. I’m going to continue investigating the world of neuroplasticity for my Master’s project, and hopefully produce some more exciting work.