GARMI: Assistant robot for care and support

Assistant robots with AI
GARMI interacts with doctors and patients via ChatGPT

GARMI is an assistant robot for the care and support of people that understands commands via ChatGPT. It can also already grasp objects. However, it will still take some time before it can be used safely in nursing homes.

The assistance robot GARMI (GERiatric Assistant Munich) is an assistance robot that is being developed by the Technical University of Munich at the Research Center for Geriatronics in Garmisch-Partenkirchen. As the researchers from TUM’s Munich Institute of Robotics and Machine Intelligence (MIRMI) show at the ICRA 2024 robotics trade fair in Yokohama, Japan, the robot not only understands various commands via ChatGPT, but also performs different tasks and skills such as grasping objects, maneuver safely and communicate with patients autonomously.

He also communicates live with doctors who carry out telemedical examinations. “GARMI is now able to carry out various individual skills that we have taught him in recent years, safely and on demand via ChatGPT,” explains the head of the geriatronics project, Dr.-Ing. Abdeldjallil Naceri.

Human-robot interaction

A digital twin is used to test the robot’s movements in a simulation before real interaction with humans, thus avoiding collisions. Artificial intelligence (AI) helps GARMI to grasp cups and glasses and pass them over without spilling liquid. Finally, ChatGPT acts as a link between the robot and patients, physiotherapists and doctors.

Researcher Naceri compares the technology built into GARMI with developments in autonomous driving: “Many development steps are necessary before a new function such as autonomous parking assistance is available in a vehicle,” says the researcher, “similar to the technology in care robotics be absolutely reliable and safe because it is used in an environment with people.”

Grasp and move precisely from a distance

To enable GARMI to grasp in a similar way to humans, the researchers brought together a camera, a robot arm with seven joints and an artificial hand, and artificial intelligence. First, the camera takes a picture of the object that GARMI is supposed to grab and uses neural networks to identify it as a cup, bottle or ball.

Since the camera only sees the object from one side, the system adds invisible areas, such as a cup, by comparing what it sees with other images and reconstructs the complete 3D object. The researchers show the probabilities with which the object is displayed as it actually looks in a color-graded graphic, a heat map. This allows you to determine the ideal position for the hand that should grip a cup, for example. In nine out of ten cases, the complex system is now able to do this. “If it works with one cup, our system can transfer it to all other cup shapes,” says Naceri.

Interact remotely

In order to research whether doctors can work with patients via teleoperation, the researchers came up with a special experimental setup. To do this, they drew simple shapes on a digital graphics tablet. GARMI was also equipped with a pen in one hand and a camera in the other.

A room away, GARMI was supposed to transfer what the researchers had previously drawn onto a screen – i.e. project a simple drawing into a complex robotic system. It turned out that the best circles, squares and triangles were created when GARMI used the camera autonomously.

In a new research paper, the researchers show how tools can be maneuvered past an object. The challenge is to keep an eye on the distances on the one hand and to correctly estimate the mobility of the robot arm with all its joints on the other. If this succeeds, the robot can even dodge balls flying towards it.

Robot responds within a millisecond

GARMI processes information at a rate of one millisecond. This applies equally to perception, interaction and navigation. The force sensors of the robot arms register the slightest touch and react immediately. If a human accidentally bumps into the robot’s arm, it stops within a millisecond for safety reasons.

To prevent accidents, humans and robots first meet as digital twins in a virtual environment. This is essential because the assistant robot can theoretically reach speeds of up to 20 km/h in a nursing home apartment. In the computer simulation, GARMI uses its Safety Motion Unit to register via its sensors when a person comes too close and moves more slowly. If the person moves away, it moves faster again.

ChatGPT uses list of commands

The AI ​​tool ChatGPT works as a translator between technology and people. It has learned various commands such as “Start rehab”, “Show me tomorrow’s weather” or “Call the doctor”, and GARMI uses this tool to communicate with patients. “The researchers currently have a list of 15 to 20 commands that trigger certain actions.

“We can potentially expand it as much as we want,” says robotics researcher Naceri, “which makes us one of the first institutes where robots and people interact with each other using ChatGPT.”

Robotic hands should take on finer tasks

The new universal GARMI is now being used in a model apartment in Garmisch-Partenkirchen. The main field of research will be the further development of hands that can perform increasingly finer tasks. It will be several years before GARMI is used in nursing homes. (heh)

(ID:50035589)