The one thing stopping AI taking over the world – or at least everyone’s jobs? Dexterity. Human hands are anatomical marvels that engineers have struggled for years to replicate. Experts are racing to change that
Just outside Luton, in an Amazon warehouse the size of four football pitches, yellow shelving units pirouette in a digitally choreographed dance. Tubs containing a Lego Spider-Man kit, followed by dishwasher tablets, followed by a pouch of muscle building protein powder whiz along conveyor belts, until they arrive at a human being tasked with stacking the shelving unit that has come to a stop in front of their workstation. This, Amazon says, is the “symphony of people and technology” to which the retail giant owes its success.
Following a barrage of negative publicity about working practices, Amazon began to open up its fulfilment centres to carefully controlled free public tours a decade ago. Our guide tells us about the competitive rates of pay Amazon workers receive, their pensions, dental cover and options for flexible working. Her delivery is somewhat robotic.
“You will see robots and humans working in harmony,” she declares as we move between different stations of the warehouse. Everything is designed to maximise what humans are good at, she continues, and what robots are good at – “working without getting tired”.
Almost every judgment call here seems to be made by a computer – not just where to stow items when they arrive at the warehouse (the system shines a light on to the spot on the specific shelf where the worker should place each item) but also which size box to pack something in once it’s been ordered, and even precisely how much tape to use to seal that box. The human beings are here to provide the hands, not the brains.
So far, at least, there is no hardware or software to rival a human hand. Humanoid robots – often called “embodied AI” – have made great strides over the past decade (literally, as evidenced in China’s recent five-day robot Olympics). But producing a robot that can pack a shelf, tighten a bolt and also pick a raspberry – actions that would make them useful rather than simply entertaining, actions most of us perform without any conscious effort – has proved surprisingly difficult. Replicating the breadth of function of the human hand is one of the biggest remaining challenges in robotics. Until robots are dexterous, human workers will always be necessary.
But investors are now gambling on humanoid robots soon becoming as much a part of everyday life as AI-powered large language models are for many of us today. The Chinese government has pledged to invest more than £100bn in robotics and high-tech industries over the next two decades. Earlier this year, Tesla halted production of its Model X and Model S cars so that factories could focus instead on producing Optimus humanoid robots. “Human hands have evolved to be this incredibly sophisticated machine. Your hand is actually a remarkable thing,” Tesla chief executive Elon Musk told a conference last year. “In order to create a robot that can be a generalised humanoid, you must solve the hands problem.”
Robot hands sewing – videoAmazon employs more than 1.5 million people worldwide today. But from what I can see on the warehouse floor, it seems that if a task could be done more efficiently by a machine, Amazon would rather leave it to a robot. What will happen to these workers when the “hands problem” is solved?
And what will happen to the rest of us? Artificial dexterity combined with artificial intelligence will be hugely consequential for humanity. Forget retraining as a plumber in the age of AI – if engineers manage to replicate human hands, then potentially no job will be future-proof.
Dexterous robots that can rival a human hand are not here yet, but they are coming. What makes hands so difficult to reproduce in hardware and software? How close are robotics engineers to getting it right? And what will happen to the world of work when they do?
“The hand is our conduit between the brain and the world,” says Alistair Hunter, a hand and wrist surgeon at University College London hospitals. “The hand is our way of turning thought into action.” Hunter has spent most of his professional life seeking to understand how hands work, and how to return them to normal function when they don’t. “Hands are special because they are very complicated anatomically,” he tells me over coffee in north London. “There are 27 bones in the hand; there are nerves and muscles that control it. But behind all that is the brain, which is interacting millisecond by millisecond to understand what the hand is sensing, adjusting to that, and then providing action.”
We might assume that we owe our evolutionary advantage to our intelligence, but Hunter argues it’s actually the combination of our cognitive power and our dexterity that has given us the edge. Over the millions of years that the frontal cortex of our brains was developing, the hand evolved to differentiate itself from limbs, allowing for the development of fingers that could grasp branches and climb trees. Thumbs came next. All the great apes might have them, but ours are larger. “The joint at the bottom of the thumb – the trapeziometacarpal joint – is one of the main advances in evolution to allow us to do very fine pincer grip, very fine dexterous control between thumb, index and middle finger, but also incredible power, incredible grasp. That allowed us to manipulate things like stones and rocks, so three million years ago we started to develop tools.”
The brain is constantly coordinating the interaction between larger muscles in the forearm, which allow for power and strong grip, and smaller muscles in the hand itself that give us fine control. It also receives huge amounts of data from the hand. “The density of nerve fibres and sensory endings in the hand is enormous,” Hunter continues. As well as the nerve endings, the hand has specialist receptors that can tell our brains about very fine details: different corpuscle cells to detect vibration, roughness and smoothness, or how our fingers and hands stretch and conform around objects. “It’s that interplay between the sensory side, the motor side and the brain that makes the hand wonderful – and so difficult to replicate.”
When you pick a raspberry, your fingers sense how soft it is, whether it’s slippery, how the skin in your fingertips conforms around it. Your brain compares this information against what it expected to experience, and then recalibrates accordingly, slowing down the compression force if it feels as if the raspberry is being crushed. “That feedback loop takes about a tenth of a second,” Hunter tells me. “We’re also able to predict what that raspberry feels like – that’s called feed forward. Even before I’ve grasped it, my fingers and my hand know what pressure to put on it to pick it and not to squash it. That’s partly why you see a baby making a mess before they’ve learned that feed forward mechanism. Their hand is clumsy because it hasn’t got that predictive model.”
From striking a match to throwing a spear to playing piano, the range of tasks hands can perform is enormous. We can use them to catch, or cup them to turn them into a drinking receptacle. And they allow us to communicate – with gestures, with pens, with our thumbs over pocket-sized screens. Our hands are constantly adapting to our ever-changing needs.
It’s this breadth of function combined with adaptability that Hunter says would be such a challenge for any engineer seeking to replicate human dexterity in a robotic hand: being able to feedback and feed forward at the same time, using a model learned from years of experience and interaction with the world. That’s why robot engineers previously focused on creating task-based robots, able to perform a specific function, he adds, rather than trying to emulate everything a human hand can do.
“I think at some point they’ll be able to do it. We’ve got many, many years ahead of us, and you can see that already robots are able to walk and run. But building a computer that is able to predict the outside world and also deliver the right amount of sensory information back – that’s going to be difficult.”
If there ever was a symbol of dystopian tech, it is the robotic hand left clutching at Sarah Connor’s throat in the final scenes of the original Terminator movie. That’s the first thing I think of when I walk into the HQ of Shadow Robot, Europe’s premier manufacturer of dexterous hands, and see one of its prototypes – skeletal and grasping – mounted on a high shelf a few metres from the entrance. Then one of the office dogs bounds up to me, and I remember that I am in a cheery workspace a minute away from London’s Hampstead Heath.
Shadow Robot began in the 1980s as a collective of hobbyists and robot enthusiasts who met up in a north London attic to tinker around with scavenged materials, hoping to one day build a full humanoid. The company was founded after they received their first commercial order – for a leg – in 1997. For the last decade or so, their exclusive focus has been on hands.
In the demo room, the Shadow Dexterous Hands they have set up for me to see look almost friendly. Mounted on two outstretched motorised arms, their palms are open with their fingers curled slightly inwards, as if waiting to give me a hug, but we begin by shaking hands instead. Its right hand moves slowly towards mine, the fingers splayed, the wrist cocked in anticipation. Its grip is gentle – natural, even – but I don’t want to shake it too hard for fear of crushing it: it’s made of MDF and nylon, with textured magnetic sensors on its fingertips, and it feels very delicate. The starting price – without any kind of software or operating system – is £20,000 per hand; the full system costs £100,000. But my instinct to be gentle is mostly because it seems so human: it moves in an expressive way, as if it were animated by some sort of soul.
View image in fullscreen Photograph: David Vintiner/The GuardianAnd it kind of is. It’s being operated by Liem Tran, robotics deployment and support manager at Shadow Robot, who is wearing gloves with sensors that track his movements, which the robot hands copy in real time. This teleoperation is a way of training the robot, showing the hands what to do, giving them a reference point on what constitutes a successful motion for any given task. Through machine learning, the hand will ultimately be able to replicate that movement without human help. The Shadow Robot company builds the hardware and basic systems; their customers – in industry, manufacturing and academic research – typically develop their own software to get what they want from the hands.
It’s a great time to be a hand roboticist, Tran says. “We don’t have anything that’s industry standardised, or anything that’s fully developed and ready for use. It’s exciting, because we’re essentially dictating what happens next with the tech. I like the idea of replicating a human in machine form and seeing where it takes us … Do you want a grape?”
The left robot hand reaches down to a brown paper bag on the table in front of us and pulls out a bunch of white grapes, holding the stalk with a delicate pincer movement between thumb and index finger. The right hand deftly plucks a lone grape and drops it into my palm. Each hand weighs roughly 4.5kg, Tran tells me, but the grape is resolutely unsquashed. I eat it. It passes me another.