HC#39 AVATAR Xprize Winner - Nimbro team with Christian and Max

Podcast Link: https://open.spotify.com/episode/7swQVfAlCcdU4FAv4pzXiM?si=00e002bcdc584ce7

In this episode of the Haptics Club we met Christian Lenz and Max Schwarz, two PhD students and members of team Nimbro, the winning team of the AVATAR Xprize competition. We talked with them about robotic avatars, their development process and the achievement that won them the competition.

About team Nimbro

In today's blog post, we'll introduce you to Max and Christian, two brilliant Ph.D. students from the University of Bonn who have been working tirelessly on an innovative avatar system as part of team Nimbro. Their dedication and hard work paid off when their robotic system won the prestigious AVATAR Xprize competition, beating out numerous other teams. With their cutting-edge technology, Max and Christian have enabled human operators to remotely interact with spaces through a range of actions, from manipulating objects to communicating with people. Their system is considered the first of its kind to provide a real in-depth connection with haptics, allowing users to feel the objects they interact with. Christian, one of the key members of the team, is passionate about the role of haptics in robotic manipulation for autonomous and teleoperated robots. He firmly believes that the ability to feel objects is essential to complete the tasks involved. Join us as we delve into Max and Christian's inspiring journey of changing the game for remote interactions and robotics.

Inside the Development of a Robotic Avatar: Progress and Challenges

The team embarked on their robotic avatar project about 3-4 years ago, starting with a simple setup of two robotic arms on separate tables to explore their interactions. As the project progressed, they developed more advanced features such as a head-mounted display for immersive visualization and a cave system with image predictors. Despite facing challenges during the integration phase, the team was ultimately successful in getting the system up and running. They emphasized the importance of thorough testing to identify and address any potential issues.

The strengths of team Nimbros Avatar system

One critical aspect of the project was the operator's visibility, which Max emphasized. To address this, the team developed a unique system where the avatar's head is mounted on a robotic arm, allowing it to move freely in response to the operator's head movements. This approach gave them a competitive edge, as the operator could look around in a more realistic way. The team also developed a system for visualizing the operator space, capturing mouth and eye movements from three cameras to create a lifelike image. They focused on haptic technology to measure joint movement and interaction forces accurately, actively lifting fingers to simulate a more natural feel. These features may not have played a significant role in the end, but the team invested significant time and effort in researching and developing them. Overall, their attention to detail and innovative approach helped them stand out in the competition.

Insights on the importance and assistance of sensory feedback

Max shared some of the team's research on providing sensory feedback to operators in human-machine interfaces. They discovered that achieving a 1:1 and sensory correspondence is the most effective way to provide sensory feedback. To achieve this, they fed data directly to the operator and used sensory mechanisms that felt natural to them. However, they recognized that there are situations where assistance functions are necessary. To address this, the team developed a depth camera that could help operators locate hidden stones by creating an overlay that displayed the stones' location without revealing their surface. This technology allowed operators to locate and feel the stones without seeing them, enhancing their performance. The team's research demonstrates that replicating natural human senses is critical for designing effective human-machine interfaces.

Communicating surface textures from avatar to operator

Max and Christian explained their strategy for allowing users to recognize textures without using a high-fidelity tactile device. Max explained that they developed an analogy where the user feels like they're wearing a thick glass and feels the vibration of moving over the texture. They tried different sensors, including force sensors and haptic sensors, but found that combining audio feedback with force feedback was the most effective way to make users feel the texture and help them decide whether it was rough or smooth. The problem was much harder than it appeared at first glance. Eric noted that they used sensory substitution strategies to convey the results instead of a 1:1 rendering or measuring system, which was impressive. The discussion then moved on to the future of haptics.

Nimbro’s Avatar Triumphs Over Unpredictable Challenges

During the competition, team Nimbro’s robotic avatar had to complete ten tasks, and the most important thing to note is that we had no control over who operated the robot, as it was someone from an international expert panel who we only met an hour before the run. We had to train the operator for only one hour to get them comfortable with our robot. The tasks included basic locomotion, communication with a judge, and manipulation of objects with different weights and textures, among others. The last task involved the operator distinguishing between artificial stones with different textures and retrieving the correct one. Despite the challenges, our robot successfully completed all ten tasks, demonstrating its capabilities in various situations.

Christians’ advice on building great robotic avatars

Christian had a great piece of advice for people building their own successful robotic avatar; you should break the rules and focus on the important components necessary to solve the task. Don't assume that the most complex solution is the best one; easy solutions can be more beneficial in terms of maintenance and robustness. Work on multiple things simultaneously, but focus on the weakest link and improve everything one by one. Additionally, it's crucial to pay attention to the haptics problem and increase the sensors' sensitivity so that the operator can feel different modalities.

Thanks Max and Christian

We would like to extend our gratitude to Max Schwarz and Christian Lenz, who generously took the time to share their insights with us. Their expertise and experience in robotics and human-machine interfaces have been valuable, and we appreciate the opportunity to learn from them. For those who are interested in their work, you can visit the website of team Nimbro to find more information about their research and projects. Once again, we thank Max and Christian for their time and contribution to this interview.

Disclaimer: Please note that this blog post has been partially generated by AI and transcribed from an interview. While efforts have been made to ensure accuracy, there may be occasional flaws or missing information in the transcription process. We apologize for any discrepancies and encourage readers to verify details from additional sources or contact the individuals mentioned for precise and up-to-date information.