RoboCat: Foundation Agent For Robotic Manipulation.pdf

We propose a foundation agent for robotic manipulation, named RoboCat, which is a visual goal-conditioned decision transformer capable of consuming multi-embodiment action-labelled visual experience. This data spans a large repertoire of motor control skills from simulated and real robotic arms with varying sets of observations and actions. We demonstrate the ability to generalise to new tasks and robots, both zero-shot as well as through adaptation using only 100–1000 examples for the target task. We also show how a trained model itself can be used to generate data for subsequent training iterations, thus providing a basic building block for an autonomous improvement loop. We investigate the agent’s capabilities, with large-scale evaluations both in simulation and on three different real robot embodiments. We find that as we grow and diversify its training data, RoboCat not only shows signs of cross-task transfer, but also becomes more efficient at adapting to new tasks. RoboCat is based on Gato and a VQ-GAN encoder, which is pretrained on a broad set of images and enables fast iteration. We specify tasks via visual goal-conditioning, which has the desirable property that any image in a trajectory can be labelled as a valid “hind-sight goal” for all time steps leading up to it. This means that hind-sight goals in existing data can be extracted without additional human supervision and that even suboptimal data collected by the agent can be incorporated back into the training set for self-improvement. Our main contributions are: (1) we demonstrate that a large transformer sequence model can solve a large set of dexterous tasks on multiple real robotic embodiments with differing observation and action specifications; (2) we investigate RoboCat’s capabilities in adapting to unseen tasks, with just a small dataset of expert demonstrations, lowering the bar of learning a new skill, compared to baselines; (3) we show that it is possible to incorporate these skills back to the generalist with a simple but effective self-improvement process; and (4) we show that by scaling and broadening the training data, RoboCat performs better on training tasks and is more efficient at fine-tuning.