By now you may have either heard or experienced for yourself the amazing capabilities of OpenAI’s ChatGPT and its ability to interact with all sorts of human prompts and questions. In addition to its capacity as a language and research tool, ChatGPT also makes a great coding assistant – able to generate and debug code in programming languages like C++, Python, JavaScript and more. With this in mind, the Autonomous Systems and Robotics Research Group at Microsoft got to work. The team wondered if they could leverage ChatGPT’s innate ability to digest human language and transform it into actionable code to operate robotics such as Drones. The team’s goal was to show how a Layperson, with no code-writing ability, could operate robotics using basic conversational language by tapping ChatGPT to do the heavy lifting of real-time programming.

To test this Human-to-Robot pipeline, the researchers tested a few different robotic scenarios. One of the scenarios was using a Tello Drone to travel and locate a specific can of soda in the Microsoft offices. To achieve this, the team started by equipping ChatGPT with some design principles in order to help it interpret prompts for robotic tasks. These design principles were specific to the robots (drones) functions and physics as well as descriptive enough for the AI of ChatGPT to reason and troubleshoot. In this case, ChatGPT would sometimes ask questions for clarification before creating the code. From there, the code would be inspected or run through a simulator before launching the Drone on its mission. Once the design principles were honed and ChatGPT was familiarized with the functions, the Drone was off finding beverages and taking selfies in the mirror. These tasks were all successfully prompted by a “Non-Technical user” a.k.a. someone who delivered layperson instructions. Phrases such as “I am thirsty – I just came back from the gym and am looking for a healthy option” were fed into ChatGPT and resulted in the Tello Drone locating the carton of Vitamin Water in the room. “Take a selfie using a reflective surface” had ChatGPT provide the code for the Drone to fly to the mirror in the office and snap a photo of itself.

At DroneBlocks, we decided to experiment with this concept using our Tello drone simulator and Python. While our simulator is built around Google’s Blockly, we are working towards an implementation where the simulator can also interface with Python. The goal was to use ChatGPT to fly a simulated Tello drone with Python around our minimal environment. Let’s just say that it worked incredibly well, and you can watch a detailed video overview from Dennis Baldwin, DroneBlocks co-founder, on his YouTube channel below. The code used in the demonstration is completely open source and can be found in this GitHub repo. The wonderful thing is that this same code can be used to control a real Tello drone. We’ll share more details on that later!

The Microsoft research team had several other scenarios in addition to the Tello Drone such as inspecting a wind farm in Microsoft AirSim and using a robotic arm to arrange colored blocks in the design of the Microsoft logo. All in all, the team found that crafting prompts and design principles for ChatGPT in order to ensure accurate code was integral in the transition from language to successful robotic missions. By using tools such as ChatGPT to simplify the command pipeline and remove the manual human coding process, Microsoft believes robotics could be “taken out of the lab and into the real world” and more accessible and useful to the general population. Learn more by visiting Microsoft’s full article on their experiments and findings.