Stop Simulating, Start Experiencing
The empirical and developmental case for sandbox-trained robots
Introduction
Nowadays most if not all research in robotics has been focusing on simulation first, reality second: you take a high-fidelity replica of a robot in a simulator (Newton, MuJoCo, Gazebo, Isaac), you run all your RL magic on it until it can walk, jump or whatever martial art you fancy, and then you apply the learned policy or VLA into the real robot. Landmark examples include OpenAI’s Dactyl (Andrychowicz et al., 2020), which learned dexterous in-hand manipulation entirely in simulation, and ANYmal (Hwangbo et al., 2019), which transferred agile locomotion policies to a physical quadruped. More recently Vision-Language-Action models like RT-2 (Brohan et al., 2023) attempt to go further by grounding language instructions in real-world robot actions, also starting from large-scale simulated or web data. Hopefully you do some testing before deploying it into its intended environment. If the robot doesn’t perform well, you go back, add more randomization (see my other article, “randomization is all you need” — the formal treatment is Tobin et al., 2017), retrain the policy and try again. Maybe if your engineers have some extra time they will apply system identification mechanisms to the motor joints, to better estimate how the joints behave under stress (because the vendor doesn’t give you those details); probabilistic approaches like BayesSim (Ramos et al., 2019) try to automate this by inferring simulator parameters from real robot data, but you will always go back to the simulator first to apply those changes.
Don’t get me wrong, I am all for simulation, and back in the day I used it extensively for FEM analysis in electronic, electric and mechanical engineering: this methodology has been used for designing airplane wings, car parts and so forth. But remember, you then always put the actual car, plane or boat into a controlled environment to make sure your simulation is “good enough” to replicate the metrics you observed in the real world.
Unfortunately this kind of approach is even harder in robotics: plane dynamics is complex but we know the equations of aerodynamics, and the plane is flying through a mostly empty sky. In robotics you are dealing with all sorts of terrain, obstacles, and annoying humans and animals who do not obey any rigid physics (unless you are an X-Man), and the complexity of simulating the real world approaches the famous MATRIX paradox.
Also when you look at those PID controllers or Kalman filters and the like, they don’t have to learn new policies all the time. Once the parameters are tuned you don’t really have to change them, unless the car, boat or plane has changed its physical configuration.
Instead in robotics we want to learn from an infinite world of possibilities. Even if a robot spends its entire life in an IKEA kitchen, there will be variations in the layout, external lighting conditions and the owner’s taste (well, I already have full respect for IKEA buyers).
So I argue that this methodology will not help achieve some form of robotic AGI in the short term. It produces good gains yes, but in the long term it will not be sustainable. So what are the alternatives? Well I have some propositions for the robotics industry: have the robot learn in a sandbox with a curriculum first until we can prove it is safe, then when deployed allow continuous learning with supervision from humans and more experienced robots, and finally transfer that knowledge when new robots are produced.
Now I sound crazy. Most people will argue that as humans we had millions of years of evolution on our side, so this will be a fairly slow process even just to get a robot to walk. Well, let’s consider some counter-arguments:
(a) Robots have fixed hardware. Their bodies don’t grow, they don’t age, and the only problem is wear and tear of mechanical parts, which are easy to factor in.
(b) Robots are not born. You can pretty much clone the entire body and brain when producing a new one, so it can almost instantaneously encode the most efficient learned behaviour and discard the inefficient ones.
(c) Robots can learn in real time from other experienced robots or experienced humans.
(d) Robots don’t sleep, get bored or lose motivation. A robot can run a learning episode continuously for days or weeks at a time, accumulating experience at a rate no biological organism could sustain.
(e) Robots can run in parallel. You can deploy a thousand identical bodies simultaneously in different environments and aggregate what every single one of them learned, something evolution could never do across a single generation.
(f) Robots don’t fear failure the way biological creatures do. A fall or a crash is just a data point, not a survival threat, which means the exploration-exploitation trade-off can be tuned far more aggressively than any animal would tolerate. Obviously this need to have some safety rules you don’t want to learn to crash into humans, instead crashing into a solid rock is less of a problem (last time I checked rocks have no feelings).
(g) The cost of forgetting is close to zero. If a new skill turns out to be harmful or wrong, you can roll back to a previous checkpoint instantly, something no biological brain can do.
Now let’s see in practice how this idea could work in practice, we are assuming that you are a robotic manufacturer and you are making your first robot which you will have to produce, test, certify and sell.
We are going to assume it is a humaoind robot designed for a family house hold, it needs to do basic chores like cleaning, cooking, gardening and something along those lines.
Phase 1: Robot Design
Here you already have certain hard constraints before you write a single line of code. It needs to be humanoid with the right proportions (look up the Cormic index if you want to get nerdy about it), it needs safe torque limits so it cannot accidentally crush a hand, a certain weight so it doesn’t destroy your floors or tip over when reaching for a high shelf, and a certain dexterity in the fingers and wrists based on the tasks you are targeting.
And yes, this is exactly where simulation belongs. Nobody is arguing you should carve the first prototype out of aluminium blind. You absolutely should use a simulator at this stage to:
test direct and inverse kinematics across the full range of motion
perform stress tests on joints and structural parts under realistic load scenarios
do energy consumption estimates so your battery lasts more than twenty minutes
run collision geometry checks so the robot can navigate a doorframe without taking the frame with it
validate actuator sizing: too weak and it can’t lift a pot of water, too powerful and you have a safety liability
The key point is that simulation here is doing what it was always meant to do: inform hardware decisions before you commit to manufacturing. This is completely legitimate. A bad joint design caught in simulation saves you six months and a lot of money.
What simulation cannot do at this stage is tell you how the robot will actually behave once it starts interacting with a messy kitchen, a curious dog, or a toddler who decided the robot’s leg is a climbing frame. That is a different problem entirely, and it belongs to the next phases.
One more thing worth mentioning: at this stage you should also be thinking about your sensor suite carefully. Camera placement, IMU positioning, tactile sensors in the fingertips, proximity sensors on the arms. These are not afterthoughts you bolt on later. They define what the robot can perceive, and what it can perceive defines what it can ever learn. Get this wrong in Phase 1 and no amount of clever training will fix a blind spot in the sensor coverage.
If I was the vendor I would spend a lot of amount of work on making sure the robot has great safety properties especially if it has high torque, make sure the compliance control can stop a powerful servo and that the sensors have good telemetry to report to your home base if they are going to break for example.
The sandbox environment
Now this is the most controversial part for sure. You need the robot to explore and adapt to a real environment. This can be a prefab house where you can vary the items: the robot needs to make as many mistakes as it can, through a combination of self-discovery and assisted learning where a human operator teaches the robot how to stand up, walk and so forth.
Physical sandbox design
What makes a good sandbox is not a spotless lab. It is a modular space with swappable furniture, programmable lighting, different floor surfaces (carpet, tile, wet tiles if you are brave), and objects placed in different positions every session. Think of it as a stage set where the props get reshuffled every morning.
The important concept here is what I’d call a “chaos budget”: you deliberately introduce disorder on a schedule. Early in training the environment is relatively clean and predictable. Over time you start knocking things over, leaving obstacles in unexpected places, varying the lighting conditions, maybe even introducing a toy on wheels that moves unpredictably. The robot is not learning in a clean room, it is learning to tolerate a real one.
And if you can afford it, run several physically different sandboxes in parallel with copies of the same robot. One sandbox has a European-style kitchen, one has an American open-plan layout, one has a narrow Japanese apartment. Each unit learns something different and you aggregate all of it. This directly applies point (e) from earlier: the parallelism advantage is not just theoretical, it is a core part of the training strategy.
Safety architecture inside the sandbox
Just because the robot is allowed to fail doesn’t mean you let it fail dangerously. Early in training the sandbox should have padded furniture edges, soft walls for locomotion practice, and floor mats. The robot’s own control stack should run with tight torque and velocity governors that relax progressively as it earns trust.
Think of it as a quantified safety envelope that widens with demonstrated competence. You define numerical thresholds: fall rate below X per hour, maximum unintended contact force below Y newtons, successful navigation of a standard obstacle course above Z percent. When the robot meets a threshold, the envelope expands slightly and new challenges unlock. The formal framework for this kind of constrained policy optimisation is well established in safe RL: Constrained Policy Optimisation (CPO) by Achiam et al. (2017) and the more recent Safety Gymnasium benchmark (Ji et al., 2023) both provide principled ways to define and enforce safety constraints during learning, which is exactly the kind of machinery you want running inside the sandbox. This is not just a training tool, it is also your paper trail for regulators when you go to certify the product.
A human safety operator should be present with an e-stop, especially in the early stages. But here is the crucial part: every time the operator intervenes, you log it. The timestamp, the sensor state, the robot’s intended action, the reason for the intervention. Every single one of those events is labelled training data. The operator is not just a safety net, they are an involuntary teacher.
Primitive bootstrapping vs. discovery
Imitation transfer learning is already quite good, and the robot does not need to bootstrap everything from scratch. The basic locomotion primitives, standing up, balancing, walking, do not need to be rediscovered by the robot any more than a human baby needs to reinvent crawling from first principles. There is growing research in retargeting motion capture data from humans to humanoid robots — MoCapAct (Wagener et al., 2022) and PhysHOI (Wang et al., 2023) are recent examples — and this is a perfectly sensible way to give the robot a working locomotion baseline on day one.
The distinction worth drawing carefully is between three types of primitives. Hard-coded primitives are things you bake in directly: joint limits, emergency stop responses, basic balance reflexes. Soft primitives are learned by imitation from human demonstration or motion capture: walking gaits, reaching motions, basic grasps. Emergent primitives are what the robot discovers through exploration in the sandbox: shortcuts, adaptations to unusual surfaces, its own preferred way of picking up an oddly-shaped object. The robot should never need to rediscover gravity. But it absolutely should discover the best way to pick up a slippery bottle.
If you want biologically-inspired methods that are also sample-efficient, Central Pattern Generators (CPGs) are worth serious attention. They produce rhythmic locomotion patterns that generalise well and do not require thousands of falls to converge. Ijspeert (2008) gave the definitive robotics treatment of CPGs in his Nature Reviews Neuroscience paper; more recent work like Bellegarda and Ijspeert (2022) has shown how CPG-RL hybrids combine the sample efficiency of oscillator-based gaits with the adaptability of learned policies. Traditional RL can eventually learn these things too, but CPGs get you there in a fraction of the time, which matters a lot when your sandbox time is expensive real-world robot-hours.
The key target is composability. You want the robot to learn primitives and, crucially, learn how to combine them into longer sequences for multi-step tasks. This is where algorithms like Model Predictive Control (MPC) or predictive coding frameworks become useful: they let the robot plan a few steps ahead using its learned model of the world, stringing primitives together rather than treating every new task as an entirely new problem. On the MPC side, MPPI (Williams et al., 2017) and its neural variants have been shown to work well for real-time robot control; on the hierarchical planning side, options frameworks (Sutton et al., 1999) and more recent Hierarchical RL approaches like HIRO (Nachum et al., 2018) provide the formal machinery for composing sub-policies.
Failure as signal
Every fall, every dropped object, every collision should be logged with full sensor state and flagged for review. But not all failures are equal, and confusing them will waste your time.
A productive failure is one where the robot attempted something reasonable, the outcome was bad, and the sensor data tells you something useful about why. These are gold. An unproductive failure is one caused by a hardware fault, a sensor dropout, or a software bug. These need to be filtered out before they corrupt your training signal, otherwise the robot starts learning to compensate for a broken encoder rather than learning to navigate a kitchen.
The sandbox should also function as a living adversarial test harness. Once the robot gets comfortable with a given layout, you change it. You add a new obstacle in a familiar corridor. You replace a light-coloured mug with a dark one. You put the mop in a different corner. The goal is to keep generating situations the robot has not quite seen before, not because you are being cruel, but because a robot that only performs well on its training distribution is not a product, it is a demo. This directly embodies the core insight of educational psychologist John Holt, who in his 1964 book How Children Fail wrote: “The true test of intelligence is not how much we know how to do, but how we behave when we don’t know what to do.” In robotics, our goal should not be to hard-code or simulate every conceivable scenario, but to build architectures that know how to behave gracefully in the face of the unknown.
Knowledge transfer out of the sandbox
At some point you have to decide: when is the robot sandbox-complete? You need a graduation criterion, and it needs to be quantitative. Not “it seems to be doing well” but “fall rate below 0.1 per hour, grasp success above 92%, obstacle navigation success above 95% across all layout variants, zero safety operator interventions in the last 48 hours of continuous operation.” You define these numbers upfront, ideally in consultation with whoever is going to certify the product for sale, because those are ultimately the numbers that matter for liability.
When the robot graduates, the policy does not ship as-is. You compress and audit it. Some parts of what the robot learned get frozen: the locomotion primitives, the basic manipulation skills, the safety reflexes. These do not change in the field. Other parts stay plastic: the adaptation layer that lets the robot adjust to a specific household’s layout, the preference model that learns how a particular owner likes things done. Getting this split right is one of the harder engineering problems in the whole pipeline, but it is also what makes the difference between a robot that improves with use and one that quietly drifts into bad habits.
Arguably this will be a longer phase compared to the current sim2real pipeline, but this investment will save you a lot of hours that you would otherwise spend when your robot fails in the field, or all the time it takes to identify robot parameters after deployment.
Let me give you a concrete example of what that waste looks like in practice. Say you trained your humanoid in simulation for six weeks on a photorealistic IKEA kitchen. Great sim2real transfer, the robot passes your lab tests. You ship it to a hundred beta households. Two weeks in, you start getting reports: the robot is dropping mugs consistently, but only in households where the kitchen has glossy cabinet fronts. Your simulation used matte textures. The specular reflection from the glossy surface throws off the depth estimation, the grasp planner miscalculates the distance to the object, and down goes the mug.
Now what? You go back to simulation. You add glossy surface materials. You retrain, which takes another four weeks on your GPU cluster. You re-run your sim2real pipeline. You push an OTA update. The mug-dropping stops, but now you are getting a new failure report: the robot mishandles cylindrical objects under LED strip lighting, because your simulator’s lighting model doesn’t reproduce the specular highlight pattern that LED strips produce on smooth surfaces. Back to simulation. Retrain. Repeat.
This is not a hypothetical. Variants of this cycle have happened at every serious robotics company that has tried to ship a general-purpose manipulation system. Each iteration costs weeks of engineering time, compute budget, and customer trust. The sim2real gap is not a one-time crossing: it is a leaky pipe you are patching forever, because the real world will always find a new way to be different from your simulator. A sandbox-trained robot that learned on real reflective surfaces, real LED lighting, and real users dropping things in unexpected places does not have this problem, because it never had a sim2real gap to close in the first place.
The empirical case for real-world data
The glossy-cabinet story is not an anecdote — it is a pattern the research literature has begun to quantify. Systematic comparisons of sim-trained versus teleoperation-trained policies show the gap widening sharply with contact complexity:
TaskSim policy on real hardwareTeleoperation policyPick-place (rigid)75–85%90–95%Peg-in-hole30–50%80–90%Deformable manipulation15–30%65–80%Cable routing10–25%70–85%Tool use5–15%60–75%
Two concrete cases illustrate what this looks like in practice. For soap-bar grasping, a policy trained in Isaac Sim exceeded 80% success in simulation but fell below 40% on hardware; switching to 300 teleoperation demonstrations brought success back above 85%. For cable insertion, sim-trained policies reached roughly 20% real-world success; a 500-demonstration teleoperation dataset achieved 70–80%.
The sample-efficiency gap is equally striking. James et al. (2019) showed that 5,000 real-world robot grasps matched the performance of a system trained on 580,000 simulated grasps — a 99% reduction in data required when switching from simulation to real collection. Simulation is not just imprecise for contact tasks: it is extraordinarily wasteful as a data source, requiring orders of magnitude more samples to reach the same real-world outcome.
For manipulation specifically, ACT (Zhao et al., 2023) puts a fine point on this: 50–100 teleoperation demonstrations — roughly ten minutes of collection — achieved 80–90% success on six fine bimanual tasks that sim-to-real approaches have not reliably solved at all.
Co-training ablations confirm the asymmetry directly. With 10 real demonstrations and 2,000 simulated ones, a diffusion policy reaches 70% success; adding just 40 more real demonstrations brings it to 90% (Sim-and-Real Co-training, 2024). The authors are explicit: “sim data can effectively supplement real data, but cannot replace it; real data is still needed to increase the co-training ceiling.” Each additional real demonstration outweighs any further scaling of simulation data once a baseline of simulated coverage is in place.
This is the core quantitative case for the sandbox approach. You are not merely avoiding the patching cycle described above: you are collecting data in the format that is inherently most efficient for learning real-world behaviour.
Bio-inspired learning: curriculum
This is the most important part. The robot should mimic how primates and other animals learn to learn (meta-learning). Even if its body doesn’t grow or change, it should follow the typical developmental phases we observe across the animal kingdom: first learn to control your own body, then explore and understand the environment, then use tools, then cooperate with others. The idea that curriculum structure matters for learning has solid theoretical backing in machine learning too: Bengio et al. (2009) showed that presenting examples in a meaningful order from easy to hard significantly improves convergence, and the subsequent literature on self-paced learning and automatic curriculum generation (Portelas et al., 2020) has extended this considerably.
What developmental biology actually tells us
The sequence is not arbitrary. It is deeply conserved across species, which is a strong signal that it reflects something fundamental about how intelligence has to be built up, not just how it happens to develop in humans.
A human infant spends roughly the first three months doing almost nothing except learning proprioception: where are my limbs, what happens when I move them, what is the relationship between my motor commands and my sensory feedback. There is essentially no goal-directed behaviour at this stage. The nervous system is calibrating itself. This is what Piaget called the sensorimotor stage (Piaget, 1952), and it lasts until around age two. The infant is not trying to achieve anything external yet. It is building an internal model of its own body.
Only once that internal model is sufficiently stable does the infant start exploring objects. Reach, grasp, bang, drop, pick up again. Object permanence develops around eight to twelve months: the understanding that things continue to exist when you cannot see them. This is the beginning of a world model. And here is the key insight: this exploration is not random. Studies by Celeste Kidd and others have shown that infants preferentially attend to stimuli that are moderately surprising, not too familiar and not too novel (Kidd et al., 2012, “The Goldilocks Effect”). They are running their own intrinsic curriculum, seeking out the learning signal that is most informative given their current model of the world.
Tool use comes later, and it has a striking developmental pattern in both humans and other primates. Chimpanzees in the wild learn to crack nuts with stones by observing adults for months before attempting it themselves (Boesch and Boesch, 1983; Matsuzawa, 1994). Young chimps have been filmed sitting next to an experienced adult, watching carefully, occasionally picking up the stone and turning it over, before one day suddenly getting it right. The observation period is not passive. It is building a predictive model of the action that gets tested all at once. In humans, the same pattern shows up in how children learn to use scissors, spoons, and eventually complex tools: extended observation, then imitation, then refinement through practice.
Social learning and cooperation emerge last in the sequence, and they build on everything before. A two-year-old who does not yet have stable object permanence cannot coordinate meaningfully with another person on a shared task. A chimp that has not mastered locomotion and basic manipulation cannot participate in a group hunt. The social layer requires the lower layers to be solid first.
Why this sequence should be your curriculum design
The robot should not be thrown into a multi-step household task on day one. The developmental sequence gives you a principled ordering that you do not have to invent from scratch:
Stage 1: Body schema calibration. Before any task learning, the robot runs in pure proprioceptive exploration mode. It moves each joint through its range of motion, builds an accurate internal model of its kinematic state, learns to detect when its sensors disagree with its motor commands (which is your earliest fault detection signal), and develops stable balance reflexes. No external goals. No rewards except for minimising proprioceptive prediction error. This is the sensorimotor stage, and skipping it is why so many robots have shaky manipulation: they never built a clean body model to build on top of.
Stage 2: Object exploration and world modelling. Now the robot starts interacting with objects, but still without assigned tasks. Pick up, put down, push, observe what happens. The goal here is building object permanence and physical intuition: mass, compliance, friction, how objects behave when stacked, poured, rolled, dropped. Intrinsic motivation methods like curiosity-driven exploration are well-suited here because they naturally implement the “moderately surprising stimuli” preference that Kidd observed in infants. ICM (Intrinsic Curiosity Module, Pathak et al., 2017) rewards the agent for transitions its forward model failed to predict; RND (Random Network Distillation, Burda et al., 2018) rewards novelty by measuring how much the agent’s learned network diverges from a fixed random target. Both drive the agent towards the edges of its current competence, which is precisely what Stage 2 needs. You are not rewarding the robot for achieving anything, you are rewarding it for reducing its prediction error about the physical world.
Stage 3: Tool use and task primitives. Once the robot has stable body schema and a working physical world model, you introduce tools and tasks. Mop, broom, spatula, pot. Each one is first demonstrated by a human operator (the chimp-watching-adult phase), then attempted by the robot under light supervision, then practiced to criterion. The operator intervention log from this phase is particularly valuable because it captures exactly the moments where the robot’s world model broke down.
Stage 4: Multi-step task composition. Combine the primitives into sequences: fetch the mop, fill the bucket, mop the floor, empty the bucket, return the mop. This is where MPC and hierarchical planning earn their place. The robot is not learning a monolithic policy for “clean the floor”, it is learning to sequence already-mastered primitives. Failures here are almost always either a world model error (it thought the bucket was full when it wasn’t) or a missing primitive (nobody taught it how to wring out a mop), both of which are diagnosable and fixable.
Stage 5: Social and cooperative learning. The robot works alongside humans and, eventually, other robots. It learns to interpret intent from gesture and gaze, to hand objects anticipatorily, to get out of the way, to ask for help when its confidence is below threshold. This stage never fully ends: it continues into deployment and is precisely where the continuous learning machinery from the later phases pays off.
The total elapsed time through all five stages will be longer than a sim2real pipeline on first pass. But every stage produces a robot that is more robust, more diagnosable, and more capable of learning new things, because each stage is building on verified foundations rather than hoping the simulation covered the right cases.
Social learning
The robot needs the capability to learn from real humans and also from its more experienced peers. This capability should be tested in the sandbox, but more importantly it should be designed in from the start, not bolted on as an afterthought.
The robot should learn from other robots and humans by imitation, which can take different forms: active guided motion (a human physically moves the robot’s arm through the desired trajectory), passive observation (the robot watches and builds a predictive model), or tele-operation with shadowing (the human operates in a parallel environment while the robot observes and mirrors). They should also be able to exchange information through a compressed communication layer that is efficient and scalable in a distributed network. More on that shortly.
Learning from humans: imitation and beyond
The most straightforward form of human-to-robot knowledge transfer is Learning from Demonstration (LfD), also called imitation learning. The idea has been around for decades but the methods have improved dramatically. The classic formulation, Behavioural Cloning (BC), treats the human demonstrations as supervised data and trains a policy to map observations to actions. It works reasonably well for simple tasks but famously suffers from distribution shift: the robot strays slightly from the demonstrated trajectory, ends up in a state the demonstrator never visited, and has no idea what to do.
DAgger (Dataset Aggregation) by Ross et al. (2011) addressed this directly: you run the trained policy, ask the human expert to label what the correct action would have been at each visited state, and add those labels to the training set. Iterate. This closes the distribution shift gap because the policy is now trained on the states it actually visits, not just the states the human visited.
More recent work has moved towards learning reward functions rather than policies directly. Inverse Reinforcement Learning (IRL) infers the implicit reward function that explains the human’s behaviour, then optimises for that reward. The advantage is generalisation: a learned reward function can guide behaviour in novel situations that no demonstration covered. AIRL (Adversarial IRL) by Fu et al. (2018) and its variants produce reward functions that are more robust to environment changes, which matters a lot in the messy real-world settings we care about.
GAIL (Generative Adversarial Imitation Learning) by Ho and Ermon (2016) takes yet another angle: it trains a discriminator to distinguish robot behaviour from expert behaviour, and uses the discriminator’s signal as a reward. No explicit reward engineering needed. In practice GAIL is sample-hungry but recent variants like VAIL and DAC have improved this considerably.
For manipulation specifically, ACT (Action Chunking with Transformers) by Zhao et al. (2023) has shown impressive results from relatively few demonstrations by predicting chunks of actions rather than single steps, which reduces compounding errors. Similarly, Diffusion Policy by Chi et al. (2023) uses a diffusion model to represent the action distribution, handling multimodal demonstrations gracefully (the robot does not average between two valid ways of grasping an object, it picks one coherently).
One dimension that most imitation learning papers underplay is intent recognition: the robot should not just copy what the human does, it should infer why. A human reaching for a mug might be intending to drink from it, move it, or hand it to someone. The same motion has different implications for what the robot should do next. Theory of Mind models, though still largely in the research phase, attempt to build exactly this: a model of another agent’s goals, not just their actions. Work by Baker et al. on Bayesian ToM and more recent neural approaches like TOM-Net are early steps in this direction.
Learning from other robots: peer-to-peer transfer
Robot-to-robot knowledge transfer is arguably even more interesting, because it scales in ways human demonstration never can. If you have a fleet of a thousand robots deployed in households worldwide, you want a mechanism for a robot that mastered opening a particular style of cabinet handle to share that skill with every other robot in the fleet without requiring a central retraining loop.
Federated Learning is the obvious starting point. Each robot trains locally on its own experience, then periodically uploads gradient updates (not raw data, which is important for privacy) to a central server that aggregates them. Google pioneered this for phones; the same principle applies to robots. The key challenge in the robotics case is domain heterogeneity: robot A lives in a Japanese apartment, robot B in a Texas farmhouse. Their local distributions are very different. FedProx and SCAFFOLD are variants of federated learning specifically designed to handle this kind of heterogeneity by penalising local models that drift too far from the global model while still allowing local specialisation.
Beyond gradient sharing, there is a more direct form of robot-to-robot transfer: policy distillation. A senior robot (one that has been deployed longer and has a more capable policy) can act as a teacher for a junior robot by generating a soft target distribution over actions for any given state. The junior robot trains against those soft targets, which carry more information than hard action labels. DQN distillation (Rusu et al., 2016) and policy distillation (same group) established this pattern; more recent work has extended it to continuous control and multi-task settings.
For skill transfer specifically, MAML (Model-Agnostic Meta-Learning) by Finn et al. (2017) is worth mentioning here. A robot trained with MAML learns not a policy for any specific task, but a parameter initialisation that can be fine-tuned to a new task with very few gradient steps. A robot receiving a policy distilled from a MAML-trained senior can adapt to local conditions quickly, which is exactly what you want when a newly manufactured robot arrives in a household it has never seen.
The communication layer
For robot-to-robot communication to work at scale, you need a shared representation language. Natural language is too verbose and ambiguous. Raw sensor data is too large. What you want is something in between: a compact, structured representation of skills, observations, and intentions that robots can transmit efficiently over a network.
This is an open research problem but there are promising directions. World models (Ha and Schmidhuber, 2018; more recently DreamerV3 by Hafner et al.) provide a natural candidate: a robot’s world model is already a compressed representation of its experience. Two robots that share the same world model architecture can, in principle, share learned representations directly. VQ-VAE and similar discrete bottleneck architectures produce codebook representations that are compact, transmissible, and interpretable.
More practically, work on multi-agent communication in cooperative MARL (multi-agent reinforcement learning) has produced learned communication protocols that are both efficient and task-relevant. DIAL (Differentiable Inter-Agent Learning) by Foerster et al. (2016) and CommNet showed that agents can learn to communicate useful information even when the communication channel is heavily bandwidth-constrained. The key insight is that you do not need the robots to communicate in human-readable language: you need them to communicate in a language that is maximally useful for the shared task, which an end-to-end trained system can discover on its own.
The longer-term vision here is something like a robotic collective memory: a distributed knowledge base that any robot in the fleet can query, contribute to, and learn from. Not a centralised database (single point of failure, privacy nightmare) but a federated graph where skills, failure cases, and environmental knowledge are distributed across the fleet and retrieved on demand. This is closer to how memory works in biological social groups than to how knowledge is stored in current AI systems, and it is the architecture that would actually let point (e) from the introduction deliver on its promise.
What’s next?
This article is a work in progress, and I want to be upfront about that. The ideas here — physical sandboxes, bio-inspired curricula, peer-to-peer robot learning, the empirical case against sim2real as a primary training strategy — are not a finished blueprint. They are a provocation, a set of design principles I believe deserve more serious attention from the robotics community than they currently receive.
The sim-first orthodoxy is understandable. Simulation is cheap, safe, and parallelisable, and for hardware design and kinematic validation it remains the right tool. But as the numbers in this article show, it increasingly struggles to be the right tool for policy learning in contact-rich, visually complex, real-world environments. The gap it leaves behind does not close itself.
What I am arguing for is a shift in where we place our trust: less in photorealistic renderers and domain randomisation, more in real physical experience collected under controlled but genuine conditions, structured by the same developmental logic that has produced robust intelligence in biological systems over millions of years. Whether that takes the form of the sandbox architecture described here, or something else entirely, the direction matters.
If you are a robotics engineer, researcher, or product designer and any of this resonates — or if you think I have got something badly wrong — I would genuinely like to hear from you. The field moves fast and no single perspective has the full picture. Consider this a starting point for a conversation, not the last word on it.
References
Achiam, J. et al. (2017). Constrained Policy Optimization. ICML.
Andrychowicz, M. et al. (2020). Learning dexterous in-hand manipulation. IJRR.
Bellegarda, G. and Ijspeert, A. (2022). CPG-RL: Learning central pattern generators for quadruped locomotion. IEEE RA-L.
Bengio, Y. et al. (2009). Curriculum learning. ICML.
Boesch, C. and Boesch, H. (1983). Optimization of nut-cracking with natural hammers by wild chimpanzees. Behaviour.
Brohan, A. et al. (2023). RT-2: Vision-Language-Action models transfer web knowledge to robotic control. arXiv:2307.15818.
Burda, Y. et al. (2018). Exploration by random network distillation. ICLR.
Chi, C. et al. (2023). Diffusion policy: Visuomotor policy learning via action diffusion. RSS.
Dass, S. et al. (2024). Empirical Analysis of Sim-and-Real Cotraining for Diffusion Policies. arXiv.
DIAL: Foerster, J. et al. (2016). Learning to communicate with deep multi-agent reinforcement learning. NeurIPS.
Finn, C. et al. (2017). Model-agnostic meta-learning for fast adaptation of deep networks. ICML.
Fu, J. et al. (2018). Learning robust rewards with adversarial inverse reinforcement learning. ICLR.
Ha, D. and Schmidhuber, J. (2018). World models. arXiv:1803.10122.
Hafner, D. et al. (2023). Mastering diverse domains through world models (DreamerV3). arXiv:2301.04104.
Ho, J. and Ermon, S. (2016). Generative adversarial imitation learning. NeurIPS.
Holt, J. (1964). How Children Fail. Pitman Publishing.
Hwangbo, J. et al. (2019). Learning agile and dynamic motor skills for legged robots. Science Robotics.
Ijspeert, A. (2008). Central pattern generators for locomotion control in animals and robots. Neural Networks.
James, S. et al. (2019). Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks. CVPR.
Ji, J. et al. (2023). Safety Gymnasium: A unified safe reinforcement learning benchmark. NeurIPS.
Kidd, C. et al. (2012). The Goldilocks effect: Human infants allocate attention to visual sequences that are neither too simple nor too complex. PLOS ONE.
Matsuzawa, T. (1994). Field experiments on use of stone tools by chimpanzees in the wild. In Chimpanzee Cultures, MIT Press.
Nachum, O. et al. (2018). Data-efficient hierarchical reinforcement learning (HIRO). NeurIPS.
Pathak, D. et al. (2017). Curiosity-driven exploration by self-supervised prediction (ICM). ICML.
Piaget, J. (1952). The Origins of Intelligence in Children. International Universities Press.
Portelas, R. et al. (2020). Automatic curriculum learning for deep RL: A short survey. IJCAI.
Ramos, F. et al. (2019). BayesSim: Adaptive domain randomization via probabilistic inference. RSS.
Ross, S. et al. (2011). A reduction of imitation learning and structured prediction to no-regret online learning (DAgger). AISTATS.
Rusu, A. et al. (2016). Policy distillation. ICLR.
Sutton, R. et al. (1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning (Options). Artificial Intelligence.
Tobin, J. et al. (2017). Domain randomization for transferring deep neural networks from simulation to the real world. IROS.
Wagener, N. et al. (2022). MoCapAct: A multi-task dataset for simulated humanoid control. NeurIPS.
Wang, T. et al. (2023). PhysHOI: Physics-based imitation of dynamic human-object interaction. arXiv:2312.04393.
Williams, G. et al. (2017). Information theoretic MPC for model-based reinforcement learning (MPPI). ICRA.
Zhao, T. et al. (2023). Learning fine-grained bimanual manipulation with low-cost hardware (ACT). RSS.