Design Engineering
Showcase 2020

Nokia Bell Labs

Hardware Innovation and Prototyping Department
Dr Nan Li
Hardware Innovation and Prototyping Intern
Technology, Media and Telecommunications

For the last six months, I’ve been working for Nokia Bell Labs – a very exciting opportunity. I’ve been tackling the challenges of working completely remotely whilst focusing on researching and developing new methods to provide feedback to wheeled robot teleoperators. There was a huge amount of freedom in the direction I took the project, the result was designing, implementing and experimenting with my own spatial audio feedback system, leveraging theory on neuroplasticity. The variety of expertise across the company means that for almost any problem you encounter, someone, somewhere in the company is an expert in that field.

 — Nokia Bell Labs

Demonstration of Design Engineering Thinking and Skills

Being able to guide the project into areas that I thought were applicable and the I found interesting was one of the highlights of the internship. Having that freedom made me really passionate about the work I was doing. I’m going to present the Design Engineering Thinking used and learnt over the internship by describing the main project I worked on – trying to give teleoperators 360o perception through only a pair of headphones.

The overarching aim of this project was to further Nokia Bell Labs’ understanding of what the best way could be to improve the performance of wheeled robot teleoperators – specifically using multimodal feedback (where information is passed through multiple sensory channels at the same time). My first task was a deep dive into the current state of the art in terms of wheeled robot teleoperation. What information do teleoperators need? What’s the best way to do this now? What weak signals haven’t yet been tried? This research highlighted many areas, from researchers using force feedback joysticks to relay direction to vibrating haptic displays to indicate where collisions were occurring. The area that interested me most was the use of audio feedback, particularly spatial audio, as an incredibly rich way to communicate complex information.

We know that advancements in 5G, robotics and edge computing will quickly allow more and more wheeled robots to enter the workspace. As this happens, we’ll find robots often being placed in chaotic, dynamic environments, where they’ll have to navigate around people and other moving obstacles. Despite the autonomous systems controlling these robots becoming increasingly more advanced, due to the unpredictable nature of these environments, there will always be situations where it will be necessary for teleoperators to step in and take control. When this happens, we need feedback and control systems in place that allow the operators to control the robots as effectively as possible. We need a way to pass the complex information from the robot’s LIDAR and other sensors to the teleoperator in a way that is simple and intuitive to understand.

So, what information actually needs to be passed to teleoperators? We need to package the complex movements of people and other obstacles around the robot in such a way that a teleoperator can understand it immediately. They need to know if someone is coming up behind the robot, or about to step out in front as easily if they themselves were there. A critical factor to achieving this is the need for the operator to understand the information about the dynamic environment at the same time as understanding the visual information through the front facing camera. Current systems use multiple visual displays; one for the camera, and one for the environment information. When doing something quickly becomes really important, overloading the visual field with multiple viewpoints makes things very difficult for the operator! Their performance slows, and the cognitive workload of each task massively increases. What if we could pass the same information through a different sensory channel?

This is where multimodal feedback systems are highly effective. They are one of the few ways in which an operator can be trained to comprehend multiple information streams simultaneously. What if the operator could be trained to understand all the necessary information about the environment around the robot simply by listening to audio? The theory behind neuroplasticity states that given enough training, a person can be taught to associate any sensory stimulus with a piece of information. What if we could associate abstract spatial audio to the entire environment around the robot? What if we could give someone 360o perception of a completely remote environment, through only a pair of headphones?

 — Nokia Bell Labs
The Nokia Bell Labs branded version of the Clearpath Jackal robot. This was the robot I based my control and feedback assumptions on.

With this as a goal in mind, I had two main tasks before the end of the internship. First, I had to design an effective way to relay the environment information as audio. Second, I had to build an experiment that would allow me to test this completely remotely (due to COVID-19). I trialled several different approaches to sonify the environment, initially through a more general approach called Panoramic Audio – with tried to express the entire environment as a form of abstract spatial audio. Whilst theoretically interesting, there was far too much complexity for the operator to extract useful information easily. The second approach refined the environment to only the moving obstacles, Focused Audio. Here only the most critical features of the environment were sonified. This was much more effective and promised a real advantage to wheeled robot teleoperators. Next was the task to prove and quantify this.

The experiment was run through a Unity build – a single executable file that participants ran five times to train themselves up. Initially people really struggled to understand the audio, but as they practised more and more their scores improved. By the end of the five sessions, simply having the presence of the audio was enough to increase performance by 45% compared to those who didn’t have it. This improvement was about 50% greater than the next best audio feedback system for wheeled robot teleoperators that I came across in my literature review. Participants were able to intuitively understand the audio and react to moving obstacles they couldn’t see. They could cast their perception ahead of them and listen around corners before they made a turn. They were able to achieve 360o perception, through only a pair of headphones.

 — Nokia Bell Labs
As more and more wheeled robots enter the workplace, we'll have an increasing demand for efficient teleoperation.

Role and Contributions

My name is Luke Holland, and I’ve just completed a six-month internship at Nokia Bell Labs, Dublin, as part of the Hardware Innovation and Prototyping team. Prior to the COIVD-19 pandemic, my role at Nokia Bell Labs was mainly going to be associated with hardware – through the design and development of wearables. However, when the pandemic struck, and I had to go fully remote (never meeting my team in person) and the nature of my work changed. I shifted into a predominantly software-based role and learnt so much as a result. My contributions were much more research based and having the chance to explore a cutting-edge concept was incredible.

Hopefully the work I was doing will at some point be incorporated into one of the larger industrial application projects at Nokia Bell Labs. My brief was to further the understanding of what was possible in the world of teleoperation. How might we be looking to control wheeled robots in the future? What new means of feedback could we adopt? A method that uses abstract spatial audio could be a viable option - allowing efficient transfer of information, completely intuitively.

 — Nokia Bell Labs
Operators need to be able to simultaneously understand information that is coming from the front facing camera AND the environment around around them.


One of the things I loved about working for Nokia Bell Labs, was the freedom to push the project in my own direction, whilst also being to draw from the vast experience of those around me. Reaching out to other teams and not tackling problems alone was definitely one of the most useful working styles I adopted over my time there. I’m going to continue investigating the world of neuroplasticity for my Master’s project, and hopefully produce some more exciting work.

 — Nokia Bell Labs
The experiment I designed placed the operator in a simulated dynamic environment, populated by avatars that they had to avoid.


No comments have been posted on this project yet.

Outdated Browser

This website has been built using some of the latest web technologies. Unfortunately, your browser doesn't support these technologies. To update your browser, please visit Outdated Browser.