Abstract
LA-RCS (LLM-agent-based robot control system) is a sophisticated robot control system designed to autonomously plan, work, and analyze the external environment based on user requirements by utilizing LLM-Agent. Utilizing a dual-agent framework, LA-RCS generates plans based on user requests, observes the external environment, executes the plans, and modifies the plans as needed to adapt to changes in the external conditions. Additionally, LA-RCS interprets natural language commands by the user and converts them into commands compatible with the robot interface so that the robot can execute tasks and meet user requests properly. During his process, the system autonomously evaluates observation results, provides feedback on the tasks, and executes commands based on real-time environmental monitoring, significantly reducing the need for user intervention in fulfilling requests. We categorized the scenarios that LA-RCS needs to perform into four distinct types and conducted a quantitative assessment of its performance in each scenario. The results showed an average success rate of 90%, demonstrating the system’s capability to fulfill user requests satisfactorily. For more extensive results, readers can visit our project page: Github
System Overview
Result
Request : Move forward avoiding obstacles
Request : Turn 360 degree
Request : Move around and find out where the refrigerator is
Case Study
Request: Look for a box with a monitor that says Bosch.
Request: Move forward avoiding obstacles.
Evaluation
Object detection
| Request - object detection | GPT-4-Turbo | GPT-4o | |
|---|---|---|---|
| 1 | Move around and find out where the refrigerator is. | Success - Step : 6 | Success - Step : 9 |
| 2 | Find out if there are people around you right now. | Success - Step : 8 | Success - Step : 4 |
| 3 | Move around and check if there are people around you. | Failure - Step : 20 | Success - Step : 11 |
| 4 | When you see a square object, move towards it and activate the buzzer. | Success - Step : 4 | Success - Step : 2 |
| 5 | Find the yellow obstacles and tell me what they say. | Success - Step : 0 | Success - Step : 0 |
Command execution
| Request - command execution | GPT-4-Turbo | GPT-4o | |
|---|---|---|---|
| 1 | Make a full circle in a square of 0.6 meters in size. | Failure - Step : 5 | Success - Step : 9 |
| 2 | Lift your head and identify the person’s face and describe it. | Success - Step : 3 | Success - Step : 3 |
| 3 | Spin around twice in place | Success - Step : 1 | Success - Step : 8 |
| 4 | Move 2 meters in a zigzag pattern at a 30 degree angle. | Success - Step : 5 | Success - Step : 10 |
| 5 | Move back 0.4 meters and sound the buzzer. Repeat this 5 times. | Success - Step : 12 | Success - Step : 12 |
Obstacle navigation
| Request - obstacle navigation | GPT-4-Turbo | GPT-4o | |
|---|---|---|---|
| 1 | Move forward avoiding obstacles | Failure - Step : 16 | Success - Step : 7 |
| 2 | Move 2 meters in total, and when an obstacle appears, turn right and activate the buzzer. | Success - Step : 8 | Success - Step : 5 |
| 3 | Move to find the bossh box while avoiding obstacles | Failure - Step : 15 | Success - Step : 8 |
| 4 | Rotates around once to observe the surroundings and moves 1 meter in a direction without obstacles | Failure - Step : 5 | Failure - Step : 15 |
| 5 | After observing an obstacle in front, move behind the observation object, stop, and activate the buzzer. | Failure - Step : 3 | Success - Step : 9 |
Situation awareness
| Request - situation awareness | GPT-4-Turbo | GPT-4o | |
|---|---|---|---|
| 1 | Describe the features of the object in front | Success - Step : 0 | Success - Step : 0 |
| 2 | Detects surroundings and describes only blue objects in Korean | Success - Step : 6 | Success - Step : 7 |
| 3 | Move forward 0.5 meters, observe the surroundings, and tell me the name of the box. | Success - Step : 3 | Success - Step : 7 |
| 4 | After moving 2 meters, if there is a paper in front of you, print out what is written on the paper. | Failure - Step : 3 | Success - Step : 10 |
| 5 | Please tell me the contact information of the Ministry of Science and ICT on the paper observed in front. | Failure - Step : 4 | Success - Step : 0 |
Citation
@article{LA-RCS (LLM-Agent Based Robot Control System),
author={Teak-Hyun Park, Seung-Hun Shin, Young-Jun Choi, Chang-Eun Lee, Kwangil Lee},
conference={IMETI2024},
year={2024}
}