OpenAI’s system consists of a humanoid hand, from UK-based Shadow Robot Company, connected to a powerful computer system and an array of cameras and other sensors. Dactyl figures out how to manipulate something using reinforcement learning, which trains a neural network to control the hand based on extensive experimentation.
Reinforcement learning has produced other impressive AI demos. Most famously, DeepMind, an Alphabet subsidiary, used reinforcement learning to train a program called AlphaGo to play the devilishly difficult and subtle board game Go better than the best human players.
The technique has been used with robots as well. In 2008, Andrew Ng, an AI expert who would go on to hold prominent roles at Google and Baidu, used the technique to make drones perform aerobatics. A few years later, one of Ng’s students, Pieter Abbeel, showed that the approach can teach a robot to fold towels, although this never proved commercially viable. (Abbeel also previously worked part time at OpenAI and still serves as an adviser to the company).
Last year, OpenAI showed Dactyl simply rotating a cube in its hand using a motion learned through reinforcement learning. To wrangle the Rubik’s Cube, however, Dactyl didn’t rely entirely on reinforcement learning. It got help from a more conventional algorithm to determine how to solve the puzzle. What’s more, although Dactyl is equipped with several cameras, it cannot see every side of the cube. So it required a special cube equipped with sensors to understand how the squares are oriented.
Success in applying reinforcement learning to robotics have been hard won because the process is prone to failure. In the real world, it’s not practical for a robot to spend years practicing a task, so training is often done in simulation. But it’s often difficult to translate what works in simulation to more complex conditions, where the slightest bit of friction or noise in a robot’s joints can throw things off.