In order to test if people can use ChatGPT to instruct robots without knowing programming languages or understanding robotics systems, Microsoft has conducted research to see if it can think outside the text and think about the physical world to help with robotics tasks.
“The key challenge here is teaching ChatGPT how to solve problems considering the laws of physics, the context of the operating environment, and how the robot’s physical actions can change the state of the world,” Microsoft said in a blog post.
“It turns out that ChatGPT can do a lot by itself, but it still needs some help. Our technical paper describes a series of design principles that can be used to guide language models toward solving robotics tasks. These include, and are not limited to, special prompting structures, high-level APIs, and human feedback via text,” it added.
After ChatGPT was given access to object-detection and object-distance data via application interfaces, the Microsoft researchers investigated its ability to generate code, mostly in Python, for robotics scenarios such as zero-shot planning and code generation.
As the AI chatbot was trained on a large amount of code and written text, it can generate code.
The system can solve coding problems and debug programs, as well as respond to dialogue and seek clarifications, according to the researchers.
Moreover, ChatGPT was tested as a language-based interface between a non-technical user and a drone using these dialogue and clarification capabilities in mind.
The researchers note that while GPT-3, LaMDA, and Codex demonstrated good results when it comes to robotics planning and coding, ChatGPT stands out as “a potentially more versatile robotics tool because it incorporates natural language and code generation models with dialogue flexibility”.
“ChatGPT asked clarification questions when the user’s instructions were ambiguous and wrote complex code structures for the drone such as a zig-zag pattern to visually inspect shelves,” the researchers said.
Microsoft tested ChatGPT to use a robotic arm to move blocks around to form the Microsoft logo, with writing an algorithm for a drone to reach a point without crashing into obstacles, and whether ChatGPT can decide where a robot should go based on sensor feedback in real-time.
