Will large language models change robotics?

Originally posted, 13 November 2024 – Will large language models change robotics?

Key Takeaways

• Covariant uses LLMs to give robots more flexibility in real-time decision-making.
• Traditional robot programming is limited, but Covariant’s approach offers adaptability for complex tasks.
• Amazon hired Covariant’s founders, signalling the potential for AI-driven robotics in global logistics.
• This technology could revolutionise how robots manage item sorting and logistics operations for companies like Amazon.
• The intersection of AI and robotics is set to transform industrial operations on a global scale.

Hans Moravec, an adjunct faculty member at the Robotics Institute of Carnegie Mellon University, wrote in 1988, “It is comparatively easy to make computers exhibit adult-level performance on intelligence tests or playing checkers, and difficult or impossible to give them the skills of a one-year-old when it comes to perception and mobility”.

Another way to think about this regards driving. In the US, it is typical for a high school-aged student around 16 years old to get a ‘learner’s permit’ over the adjacent summer, usually between sophomore year and junior year. Over that summer, the student, alongside a licensed driver, gains a certain number of hours and miles of driving experience. In most cases, this is enough to then take the ‘driving test’ which is administered on a state-by-state basis. 

On the other hand, there are autonomous driving systems that have been trained, in many cases in simulation, on the equivalent of billions of miles driven. These systems – again, with billions of miles under their virtual belts – are not trusted enough to be widely deployed across the world. 

There are many cases like this where it becomes clearer and clearer that the way that humans and computer systems learn is totally different. 

The case of Covariant

Covariant Robotics has a great website where it’s possible to watch different videos of robotic systems undertaking different tasks. We have loved pointing this out to investors because it is a simple way to show what we believe is a very important, central point:

• The ‘old way’ of running a robotic system is that one programs the system on precisely what to do. The positive of this is that the system is completely deterministic – you never see the robot doing something unexpected. The problem with this is that if the task cannot be explicitly programmed, the robot will not be able to do it. 
• The ‘new way’ of running a robotic system is that one points a large language model (LLM) at a task and gives the system an overall goal. One example could be sorting different items in a container. Instead of programming every possible scenario and seeking to account for every possible combination of things in a container, the system could optimise sorting items from a container based on a set of foundational principles. The benefit of this is, of course, flexibility, but the risk is that it could be difficult to understand what the robot will do – thereby creating a possibly dangerous set of circumstances if, for instance, people are positioned near the robot. 

In our opinion, this is the power of what Covariant’s videos show on its website. The system pairs the robot’s control system with a natural language interface, so as the system attempts different things to complete a task, text appears on a screen, lending explainability to what the robot is about to try. 

The video with the gripping system based largely on suction attempting to pick up and sort pairs of socks was particularly powerful. If one can picture how socks tend to be packed, there is usually a smooth, sticky paper-like surface at the centre with the cloth socks extending above and below. 

As people living in the world, we know what is likely to happen if the suction system attempts to pick up the socks connecting to the cloth – it wouldn’t be possible to get a good seal for the suction to do its job against the fabric. The system would be better served if it focused on the package’s smooth paper, often the label. 

Covariant’s system allows you to see the text of the robotic system working this out in real-time. 

Amazon hires the founders of Covariant¹

Amazon is one of the most amazing studies of logistics management we have ever seen in global commerce. It can deliver many different items within a day or less to almost any location worldwide. It is a staggering operation. 

Intuitively, robots would be additive to this effort because they can run tirelessly, 24-7-365. However, a core part of what is needed is the capability to pick up and sort almost any type of item appropriately. Remember what Hans Moravec noted – developing and programming a system that can properly handle any object is vastly more challenging than it may seem. 

When we saw an article in WIRED noting that Amazon was hiring the founders of Covariant and licensing its technology, we could not have been more excited. It was always cool to visit Covariant’s website, but if, in the coming years, we can start telling the story of how the robotics technology that connects to LLMs is powering Amazon’s logistics operation – this story becomes basically self-explanatory in terms of the power of the intersection between robotics and AI. It is still relatively early, but the robotics story is something we are keeping a close eye on. 

Source

¹ Knight, Will. “This Could Be the Start of Amazon’s Next Robot Revolution.” WIRED. 4 September 2024.