Speed
A simulator can run faster than real time, and thousands of copies at once. A policy can see more episodes before lunch than a physical robot will see in its lifetime.
The catch: Only if the thing you sped up is the thing that mattered.
SIMULATION / FROM PHYSICS TO LEARNING
Build a world.
Then find out where it lies to you.
Run small physical worlds in your browser, then connect them to robot learning. Explore Isaac Sim, Isaac Lab, and MuJoCo; compare physics with learned dynamics; and follow DDPG and A3C from experience to a policy update.
8live simulationsreal physics, every frame
3integrators racedand one of them loses
90lidar rayscast against real walls
240episodes per dragin the reality-gap panel
A simulator is a bet about what matters.
Every simulator is a deliberate simplification: this force counts, that one does not, and the difference is small enough to ignore. Those bets are usually reasonable and occasionally catastrophic, and the whole discipline is knowing which is which.
Safety
A failed grasp costs nothing. A failed grasp on real hardware costs a gripper, and a failed manoeuvre on a real vehicle costs considerably more.
The catch: Safe failure teaches you less urgency than unsafe failure does.
Resets
The single most underrated property. A simulator can be returned to an exact starting state, instantly, forever. Reinforcement learning is largely unworkable without it.
The catch: Real environments have no reset button, which is why deployment is a different problem.
Ground truth
You know every position, velocity, contact force and object identity exactly, with no sensor noise, because you computed them.
The catch: A perception model trained on perfect labels has never met an occluded, motion-blurred frame.
Rare events
You can make the improbable routine: the pedestrian who steps out, the joint that seizes, the sensor that drops a frame.
The catch: You can only generate the failures you already thought of.
A simulator is a loop that guesses the next moment.
Physics gives you accelerations. What you need is positions, one small step of time later, over and over. How you take that step is the single most consequential choice in the engine, and it is usually three lines of code.
Three integrators. One orbit. Watch which one survives.
Each panel runs the identical starting condition: a body in a perfectly circular orbit. Any change in the radius is error the integrator invented. Nothing here is drawn from a recording — the positions come from the equations on every frame.
energy -0.5000 exact −0.5
energy -0.5000 exact −0.5
energy -0.5000 exact −0.5
orbits completed 0.0
The faint ring is the true orbit. Explicit Euler adds energy at every step and spirals off it; semi-implicit Euler stays on it with the same amount of arithmetic. Raise the step size and the difference arrives faster.
Think it through: Why does the cheap method beat the accurate one in a game engine?
RK4 is more accurate per step and costs four force evaluations to get there. Semi-implicit Euler costs one, and although each step carries more error, the errors are structured — it is symplectic, so energy oscillates around the true value instead of creeping away from it. Over a million steps that property matters far more than per-step accuracy.
This is why physics engines built for games and robotics reach for semi-implicit Euler or a velocity-Verlet variant, while an orbital-mechanics package that must stay accurate over decades reaches for something higher order. Different failure you are trying to avoid.
Every simulator has a speed limit.
Step too far and the numbers stop describing anything. Not gradually — a simulation that is slightly too coarse does not look slightly wrong, it looks fine until it detonates.
The same spring, stepped at the same rate, with two different orders of operations.
A mass on a spring, released from the same place. Left uses explicit Euler, right uses semi-implicit. The only difference is whether the position update reads the old velocity or the new one.
x += v·dt ; v += a·dtamplitude 1.000 should stay 1
v += a·dt ; x += v·dtamplitude 1.000 should stay 1
stability limit dt < 2.00
Both look calm here — but only one of them stays that way as you raise the step.
Think it through: What is the actual stability limit?
For an undamped oscillator of frequency ω, semi-implicit Euler is stable while the step stays below 2/ω. Here ω = 1, so it holds together until dt reaches 2 and comes apart above it — which is exactly what the slider shows. Explicit Euler has no stable region at all for this system: it gains energy at every step, and a small dt only slows the inevitable.
That formula is why stiff systems are painful. A stiff contact adds a very high ω, and the whole simulation is then forced down to a step size that suits the fastest mode — even though nothing else in the scene needs it. Substepping, implicit solvers and constraint-based contact all exist to escape that tax.
Contact is where simulators earn their reputation.
Bodies flying through space is solved arithmetic. Bodies touching is a discontinuity: a force that is zero one instant and enormous the next, applied over an area nobody measured, between materials nobody characterised. Nearly every difficulty in robotics simulation is downstream of this.
| Approach | How it works | Why you would | What it costs |
|---|---|---|---|
| Penalty (spring) contact | Let bodies overlap slightly and push them apart in proportion to the overlap. | Simple, smooth, and differentiable — which matters if gradients flow through the simulator. | Stiff. A believable surface needs a very high spring constant, and that forces a tiny timestep on the whole scene. |
| Impulse-based contact | Detect the interpenetration, then apply an instantaneous velocity change to resolve it. | Handles hard surfaces without demanding tiny steps. | Discontinuous, so gradients are unhelpful, and stacked objects can jitter as impulses fight one another. |
| Constraint solvers (LCP) | Treat all contacts as a single system of inequalities and solve them together each step. | Stable stacks and coherent multi-contact behaviour. This is what most robotics engines do. | Solved iteratively with a step budget, so a hard scene returns an approximation and tells you it is done. |
| Soft-body and deformables | Discretise the object itself and simulate its internal forces. | Cloth, rope, tissue, deformable grasps. | Expensive, and the parameters are genuinely hard to identify from a real material. |
Notice the thread running through all four: each is a different compromise between stiffness and step size, which is the stability limit from the previous chapter reappearing in physical clothing. A rigid surface is a very stiff spring, a very stiff spring has a high frequency, and a high frequency demands a small step. Engines that feel solid without crawling have found somewhere clever to hide that tension.
The behaviour you wanted is rarely the behaviour you wrote.
Simulation is at its most useful when it produces something you did not put in. It is at its most dangerous for exactly the same reason.
A flock, from three rules and no leader.
Every agent follows the same three rules, applied only to neighbours it can see: steer away from anyone too close, match the heading of those nearby, and drift toward the middle of the group. That is the whole program. Pull a rule to zero and watch what the flock loses.
alignment of the flock 0.000 1 = one stream
Separation to zero and they collapse into a single point. Alignment to zero and the flock loses its shared heading but keeps its clump. Cohesion to zero and the group dissolves into drifting pairs. None of those behaviours is written down anywhere.
Think it through: What has a flock got to do with training a model?
Two things. First, it is the cleanest demonstration that complex global behaviour does not require a global controller — every bird here reads only its immediate neighbours, and the flock is nowhere in the code. That is the premise of multi-agent RL, swarm robotics and traffic simulation.
Second, it is a warning about evaluation. Turn cohesion up and you get a beautiful tight flock that looks like success; it is also a flock that will fly straight into a wall together. A simulation that produces impressive-looking behaviour has told you nothing until you have defined what you were actually measuring.
An environment is a contract, not a picture.
Strip away the rendering and a reinforcement-learning environment is four things: a way to reset to a known state, a way to apply an action, an observation handed back, and a number saying how that went. Everything else is presentation.
reset()Return to a known starting state and hand back the first observation.
step(action)Advance the physics, and return the next observation and a reward.
observationWhat the agent is allowed to know. The most consequential design choice you will make.
rewardOne number per step. The agent will optimise it exactly, including the parts you did not intend.
CartPole, the real dynamics, with the policy in your hands.
This is the standard benchmark, with the standard constants, integrated at 50 Hz. The four sliders are a linear policy: multiply each state variable by its weight, add them up, push right if the total is positive. An episode ends if the cart leaves the track or the pole passes twelve degrees.
steps alive 0best 0episodes 0
Nothing here is learned — you are the optimiser. Reinforcement learning replaces your hand on the sliders with a search over these same four numbers, guided only by how long the pole stayed up.
Think it through: Why does the obvious policy fail?
Set the gains to Angle only. Pushing right whenever the pole leans right is the intuitive rule, and it collapses in about forty steps — because by the time you react to the angle, the pole already has angular velocity, and you arrive late every single time. Adding the θ̇ term lets the controller respond to where the pole is going, which is the difference between damping an oscillation and amplifying it.
That is also the whole reason the state includes velocities. Give an agent only positions and the problem stops being Markov: the same picture can mean falling left or falling right, and no policy can tell them apart. When people say a state must capture everything needed to act, this is the concrete version of it.
Now put a body on it.
Everything so far has been a point or a cart. A robot is a chain of rigid links with motors at the joints, and that changes the question: you no longer command a position, you command angles and hope the hand ends up where you meant.
| Layer | Its job | On this page |
|---|---|---|
| Perception | Turn raw sensor returns into something about the world: obstacles, surfaces, objects, people. | The lidar panel casts real rays against real walls. |
| State estimation | Work out where the robot is and how it is moving, from sources that each lie in different ways. | The drifting odometry, and why one sensor is never enough. |
| Planning | Choose a path or a sequence of actions that reaches the goal without hitting anything. | Touched only — the waypoint follower is the simplest possible version. |
| Kinematics & dynamics | Relate joint angles to where the hand is, and torques to how things move. | The arm panel: forward, inverse, and the Jacobian. |
| Control | Make the actual joint reach the commanded position despite load, friction and delay. | The PID panel, holding a joint against gravity. |
Forward is arithmetic. Inverse is where robotics starts.
Drag anywhere in the panel to move the target. Forward kinematics — joint angles to hand position — is one line of trigonometry with one answer. Inverse kinematics is the reverse, and it has two answers, sometimes none, and occasionally infinitely many.
joint 1 · θ₁-1.9°
joint 2 · θ₂82.8°
targetreachable
manipulability · |det J|1.084
The ellipse shows which directions the hand can move quickly for a given joint speed. A round ellipse is a well-conditioned pose; a thin one is a pose to avoid.
Think it through: Why does the arm slow down near the edge of its reach?
At full extension the two links are in line, and both joints can only move the hand along the same arc — there is no combination of joint speeds that moves it straight outward. That is a singularity, and the determinant of the Jacobian is exactly zero there. Because the controller inverts the Jacobian to turn a desired hand velocity into joint velocities, approaching that point asks it to divide by something approaching zero, and the commanded joint speeds run away.
Watch the ellipse collapse into a line as you push the target outward. Every real manipulator has these, they are a property of the mechanism rather than the software, and planning around them is a permanent part of the job.
Commanding a joint is not the same as moving it.
Kinematics told you which angles you want. Nothing about that makes a motor produce them. Between the two sits a control loop, and its failures — overshoot, oscillation, quietly settling somewhere near but not at the target — are the failures you will actually spend your time on.
Tell a joint where to go, and it will not simply go there.
One joint with a mass on the end and gravity pulling it down, commanded to hold a position it is not currently at. The three gains are the entire controller.
steady-state error 1.0000peak overshoot 0.0%
Start on P only and watch it settle short of the dashed line — that gap is gravity winning. Add integral and the gap closes; add derivative and it stops ringing on the way.
Think it through: Why does proportional control alone stop short?
A constant load — gravity on the link — is pulling the joint down the whole time. Proportional torque is produced only by error, so the joint settles exactly where the proportional term balances the load, and that is not the target. The gap is a steady-state error, and no amount of extra Kp removes it; it only shrinks it while making the approach more violent, as the fourth preset shows.
Integral action fixes it properly: error that persists keeps accumulating, so the torque keeps rising until the error is actually zero. Derivative action responds to how fast the error is closing, which damps the overshoot the other two create. That is the whole of PID, and it still runs an enormous share of the machines in the world.
There is a thread worth noticing here. The gain that is too large produces overshoot and ringing; the gain that is too small crawls. That is the same trade-off as the learning rate in the descent panel on the mathematics page, and the same one as the timestep in chapter three. A control loop, a training loop and an integrator are all the same shape of problem: how far do you dare move on the strength of local information.
A robot that cannot tell you where it is cannot do anything else.
Every layer above depends on the pose being known. Kinematics puts the hand somewhere relative to the base; that is useless if the base is somewhere other than you think.
The robot can see the room. It still does not know where it is standing.
A differential-drive base follows four waypoints around a room, casting 90 lidar rays every frame against the real walls. The solid robot is the truth. The outline is where the robot believes it is, integrating its own wheel commands.
truth vs belief 0.0 px apart
Set the error to zero and the outline stays locked to the robot forever — perfect wheels, perfect knowledge. Nudge it to 2% and watch the two separate, slowly at first, then without limit. The lidar has been seeing the walls correctly the entire time; nothing is using that information.
Think it through: If odometry always drifts, why is it in every robot?
Because it is fast, always available, and locally excellent. Over a second it is close to perfect; the problem is only that the error accumulates and nothing ever removes it. A wheel a few percent off its nominal radius is a bias, not noise, so it integrates rather than averaging out.
The fix is not a better wheel encoder — it is a second source of information that does not drift. The lidar sees the walls, and walls do not move, so matching this scan against a map pins the pose absolutely. That combination is what SLAM is: dead reckoning for the short term, landmarks for the long term, and a filter deciding how much to trust each. This panel is the argument for it, drawn.
The simulator was never the point.
A policy that works in simulation and fails on hardware is the default outcome, not an unlucky one. The gap has specific, nameable causes, and most of them are things somebody decided not to model.
A policy that is perfect in the simulator, and the physics it never met.
Pick a policy, then widen the gap. Every trial re-rolls the pole mass, pole length, cart mass and motor strength by up to the percentage shown — and, because parameters are rarely what actually bites, also delays the controller's decision by up to six steps and adds noise to the angle it reads. That is 240 complete simulations, recomputed as you drag.
EPISODES SURVIVED100%240 of 240 lasted the full 600 steps
At 0% every trial runs on the simulator's exact numbers, with an instant, perfect reading of the pole — the result you would put in a slide. Widen it and the same controller, unchanged, starts dropping the pole. Compare the two working policies as you drag: Tuned degrades far more slowly than Add damping, which is what robustness actually looks like when you measure it rather than assert it.
Think it through: So how does domain randomisation actually help?
The slider here only shows you the problem: a controller tuned on one set of numbers degrades as the world stops matching them. The fix is to move the randomisation from evaluation into training — sample new physics every episode, so the policy is optimised against the whole distribution rather than one point in it.
What you get is a policy that is slightly worse on nominal physics and dramatically better off it, because it was never allowed to depend on a number it could not trust. The real robot then looks like one more sample from the training distribution rather than an unpleasant surprise. The cost is real: a wide enough distribution can make the task unlearnable, and choosing what to randomise — masses and friction, yes; gravity, rarely — is judgement, not a recipe.
Where the gap actually comes from
Unmodelled friction and compliance
Real joints have stiction, backlash and flex. Most engines give you a friction coefficient and stop there.
Actuator dynamics
A commanded torque is not an achieved torque. Motors saturate, lag and heat up; simulators usually apply the number you asked for.
Sensor and latency realism
Real observations arrive late, noisy, quantised and occasionally not at all. A policy trained on instant perfect state has never had to cope.
Contact parameters nobody measured
Restitution and friction are usually set to whatever looked right. They are rarely identified from the actual materials.
Distribution of situations
The simulator contains the scenarios you wrote. The world contains the ones you did not.
And how people close it
| Technique | What it does | What it costs |
|---|---|---|
| Domain randomisation | Sample physics, textures and latencies afresh each episode so the policy cannot depend on any single value. | Cheap and effective. Too wide a distribution can make the task unlearnable. |
| System identification | Measure the real system and fit the simulator's parameters to match it. | Narrows the gap rather than papering over it, but needs hardware access and careful experiments. |
| Actuator modelling | Learn a small model of what the motors actually do, and put it between the policy and the simulated joint. | Often the single highest-value addition for legged and manipulation work. |
| Real-world fine-tuning | Train in simulation, then adapt with a small amount of real experience. | Powerful, and reintroduces every safety and reset problem simulation was avoiding. |
| Observation grounding | Feed the policy quantities you can measure reliably on the real robot, not privileged simulator state. | A policy that reads a true object pose in simulation has no equivalent input on hardware. |
Pick the kind of tool before the brand.
A physics engine, a robotics platform, an environment API and a domain simulator solve different parts of the problem, and people routinely compare across the categories as though they were competing. Checked against official documentation on September 20, 2026.
Physics engines
MuJoCo
An open-source engine for articulated dynamics and contact, with CPU and accelerated ecosystem options.
Check: Inspect the MJCF model or programmatic model specification, actuator settings, and backend before comparing results.
Bullet / PyBullet
A widely used open-source engine with a direct Python interface and a large body of example environments.
Check: Compare its contact settings with whatever you are reproducing; defaults differ between engines.
Brax
A differentiable engine written in JAX, built to run many environments in parallel on accelerators.
Check: Throughput comes from batching. A single environment is not where it wins.
Robotics platforms
NVIDIA Isaac Sim
A GPU-accelerated robotics simulator with photorealistic rendering and synthetic-data tooling.
Check: Check the hardware requirement and whether you need the rendering fidelity you are paying for.
Gazebo
The long-standing simulator of the ROS ecosystem, with sensor models and plugin support.
Check: Its value is the ROS integration. Evaluate it against your actual middleware, not in isolation.
Webots
An open-source robot simulator with a large library of modelled robots and sensors.
Check: Good for getting a known robot moving quickly; check fidelity for contact-heavy tasks.
Robot-learning frameworks
NVIDIA Isaac Lab
Task definitions, observations, rewards, resets, and batched robot-learning workflows, with integrations for RL libraries.
Check: Distinguish the framework from the physics backend and training algorithm. See the version-specific examples below.
RL environments
Gymnasium
The maintained standard interface for RL environments — reset, step, observation, reward, termination.
Check: It is an API, not a physics engine. What is underneath still decides your results.
Unity ML-Agents
Turns a Unity scene into a trainable environment, with a Python side for the learning.
Check: Strong for custom visual scenes; the physics is a game engine's, with the trade-offs that implies.
Domain simulators
CARLA
An open-source driving simulator with maps, traffic, weather and sensor suites.
Check: Sensor realism and traffic behaviour are the parts that decide whether results transfer.
Habitat
A fast simulator for embodied agents navigating scanned indoor scenes.
Check: Built for throughput on navigation; check what it does and does not model about interaction.
The world, the task, and the learner.
Isaac Sim, Isaac Lab, and MuJoCo occupy different layers. Start by separating the system that advances the world from the system that chooses actions and learns from the results.
EXPLORE THE CONNECTION
state + action → next physical state
sensors → observation · reward · ending
actor chooses actions · optimizer updates weights
Next action returns to the world ↻
Franka reaching with joint-position targets. Isaac Sim supplies the scene. Isaac Lab defines the learning task. The selected RL runner updates the policy.
- Physics rate (fixed)
- 120 Hz
- Policy control rate
- 30.0 Hz
- Action held for
- 33.3 ms
These rates are an illustrative configuration. Changing action decimation changes the control problem; rendering frequency is another independent setting.
Isaac Sim
Build robot scenes, run physics, simulate sensors, render images, and connect robotics software. A cube can fall correctly without any neural network in the application.
Isaac Sim example ↗Isaac Lab
Define observations, actions, rewards, resets, randomization, and batches of robot environments. An integrated RL library supplies the training algorithm.
Isaac Lab training workflows ↗MuJoCo
Model articulated bodies, actuators, and contacts. Use its Python bindings directly or an environment wrapper. Its simulator does not automatically train a policy.
MuJoCo Python API ↗| Your main need | Useful starting point | Why / what to measure |
|---|---|---|
| Learn dynamics and control on a laptop | MuJoCo CPU | Compact headless experiments; inspect equations, control rates, and solver behavior. |
| Camera-rich robotics scenes and sensor integration | Isaac Sim | Scene and sensor workflows; measure rendering cost and the realism your task needs. |
| Train many robot environments together in the Isaac ecosystem | Isaac Lab + an RL library | Task construction and batched rollouts; measure total training throughput and GPU memory. |
| Large batched MuJoCo experiments | MJX-JAX or MJX-Warp, with a learning stack | Select by hardware and feature support; include compilation, transfer, and contact costs. |
| Differentiate through dynamics | A supported differentiable backend, such as MJX-JAX | Validate actual gradients and contact behavior. GPU acceleration alone does not imply differentiation. |
| Compare contact-rich manipulation results | Both, with matched physical and control settings | Compare penetration, slip, grasp success, solver convergence, and sensitivity. There is no task-independent accuracy winner. |
The accelerated MuJoCo options differ: MJX-JAX supports automatic differentiation for supported operations; the current MJX-Warp documentation explicitly excludes autodiff. Check the feature matrix for your model. MJX implementation reference ↗
Moving a policy between engines requires matching joint order, units, observation normalization, action meaning, limits, control frequency, and reset conditions. A shared robot file or a successful import does not guarantee identical dynamics.
Three examples, three different jobs.
First make a physical system move. Then add a task and a learning loop. These examples separate those steps so you can see what each tool contributes.
A carriage follows a target on a rail
A proportional–derivative controller turns position error and velocity into force. MuJoCo advances the constrained body. Physics runs at 400 Hz; the command is held for four steps, giving a 100 Hz controller. There is no neural network in this baseline.
Run in a Python environment
Download MuJoCo servo ↓python -m pip install mujoco
python sim-mujoco-servo.pyRead the complete MuJoCo servo script
sim-mujoco-servo.py
"""A force-controlled slider in MuJoCo. No neural network and no viewer.
python -m pip install mujoco
python sim-mujoco-servo.py
Creates sim-mujoco-servo.csv in the current folder.
"""
import csv
import math
import mujoco
XML = """
<mujoco model="one_axis_servo">
<option timestep="0.0025" gravity="0 0 -9.81"/>
<worldbody>
<geom type="plane" size="2 2 .1"/>
<body name="carriage" pos="0 0 .2">
<joint name="rail" type="slide" axis="1 0 0" damping=".4"/>
<geom type="box" size=".1 .1 .1" mass="1"/>
</body>
</worldbody>
<actuator><motor joint="rail" gear="1" ctrllimited="true" ctrlrange="-2 2"/></actuator>
</mujoco>
"""
def main():
model = mujoco.MjModel.from_xml_string(XML)
data = mujoco.MjData(model)
data.qpos[0] = -.5
mujoco.mj_forward(model, data)
target, decimation, force = .5, 4, 0.
rows = []
for step in range(2400): # six simulated seconds at 400 Hz
if step % decimation == 0: # controller at 100 Hz
error = target-data.qpos[0]
force = max(-2., min(2., 8.*error-4.*data.qvel[0]))
data.ctrl[0] = force
mujoco.mj_step(model, data)
assert math.isfinite(data.qpos[0]) and math.isfinite(data.qvel[0])
rows.append((data.time, data.qpos[0], data.qvel[0], force))
with open('sim-mujoco-servo.csv', 'w', newline='') as f:
writer = csv.writer(f)
writer.writerow(['time_s', 'position_m', 'velocity_m_s', 'force_N'])
writer.writerows(rows)
print('MuJoCo', mujoco.__version__)
print('Physics: 400 Hz; controller: 100 Hz; action is a force in newtons.')
print(f'Final position={data.qpos[0]:.5f} m; target={target}; absolute error={abs(target-data.qpos[0]):.5f} m')
print('Saved sim-mujoco-servo.csv. This rail-constrained example tests control, not contact fidelity.')
if __name__ == '__main__':
main()
The CSV records time, position, velocity, and force. Compare overshoot, settling, and saturation while changing mass or damping. This rail example does not test contact fidelity. API reference ↗
A cube falls and settles on a ground plane
Create a scene, give a cube collision geometry and mass, reset, and advance physics. Inspect its center height as contact brings it to rest. Adding a camera would add an observation source; it would not add a learning objective.
Isaac Sim 5.1 Core API environment
Download Isaac Sim cube ↓# From a compatible Isaac Sim installation:
./python.sh /path/to/sim-isaac-cube.py
# Windows uses python.bat. This is not ordinary system Python.Read the complete Isaac Sim cube script
sim-isaac-cube.py
"""Isaac Sim 5.1 Core API example: drop a cube, inspect its height.
Run in a compatible Isaac Sim environment, NOT an ordinary Python venv:
./python.sh /path/to/sim-isaac-cube.py
On Windows use python.bat. Requires the release's supported OS/GPU/driver.
This example is source-checked, not executed by the website or in the browser.
"""
from isaacsim import SimulationApp
simulation_app = SimulationApp({'headless': True})
# Isaac modules must be imported AFTER SimulationApp starts.
try:
import numpy as np
from isaacsim.core.api import World
from isaacsim.core.api.objects import DynamicCuboid
world = World(stage_units_in_meters=1., physics_dt=1./120., rendering_dt=1./60.)
world.scene.add_default_ground_plane()
cube = world.scene.add(DynamicCuboid(
prim_path='/World/StudyCube', name='study_cube',
position=np.array([0., 0., 1.]), size=.2, mass=1.,
color=np.array([.1, .7, .9]),
))
world.reset()
for step in range(600):
world.step(render=False)
if step % 120 == 0:
position, _ = cube.get_world_pose()
print(f'step={step:03d}, cube center z={position[2]:.4f} m')
print('The cube center should settle near half its side length, subject to contact settings.')
print('There is no policy or reward in this example: it demonstrates the simulation layer.')
finally:
simulation_app.close()
This download targets the documented 5.1 Core API; newer releases may use different APIs. It is source-checked, not executed here. Use a supported OS/GPU or a remote machine; macOS is not listed for the full Isaac Sim application. Versioned example API ↗ · Current hardware requirements ↗
A Franka arm learns to reach a target
The environment supplies robot observations, actions, target-pose error, reward terms, and episode resets. The RL runner collects experience and updates a policy. Start with the official task and inspect its configuration before changing the reward.
Documented Isaac Lab 2.3 / RSL-RL workflow
# In an installed, compatible Isaac Lab checkout:
./isaaclab.sh -i rsl_rl
./isaaclab.sh -p scripts/reinforcement_learning/rsl_rl/train.py \
--task Isaac-Reach-Franka-v0 --num_envs 64 --headless
# List tasks in your checkout before choosing a different one:
./isaaclab.sh -p scripts/environments/list_envs.pyThis is a starting command for the task's configured RSL-RL agent, commonly PPO; it is not a DDPG or A3C command. Increase environment count only after checking memory and throughput. Commands were checked against the versioned documentation, not run on this site. Training reference ↗ · Task catalog ↗ · Compatible installation ↗
| Piece | Example specification | Check before training |
|---|---|---|
| Observation | Joint positions/velocities and target-relative pose | Are all inputs available at deployment? |
| Action | Joint-position offsets, Cartesian targets, or torques | These are different control problems; record the mapping. |
| Reward | Position/orientation error plus chosen effort penalties | Units, weights, and unwanted shortcuts. |
| Reset and ending | Sample targets; distinguish failure from time limits | Bootstrap correctly and use the actual final observation. |
| Evaluation | Success within a pose tolerance on held-out targets | Report multiple seeds, cost, and failure cases. |
Physics and neural networks can share the work.
A physics engine advances a modeled state. A neural runtime, such as PyTorch or JAX, evaluates parameterized functions and computes training gradients. The connection between them can take several forms.
| Arrangement | What the neural network does | What remains physical |
|---|---|---|
| Policy in a simulator | Maps observations to actions | The engine advances the world. DDPG and A3C fit here. |
| Learned dynamics surrogate | Predicts the next state or state change | Physical training data may inform it, but constraints are not automatically preserved. |
| Residual dynamics model | Predicts a correction to a known dynamics model | The analytical model carries the structure it already explains. |
| Residual controller | Adds a learned action correction to a baseline controller | The engine or real plant still defines the dynamics. |
Two distinct hybrid designs
Dynamics: x_next = f_physics(x, u) + residual_theta(x, u)
Control: u = clip(u_baseline(x) + policy_theta(observation))
A policy trained in simulation is not automatically a learned simulator.
Model-free policy training does not require gradients through the physics engine.BROWSER EXPERIMENT / A LEARNED DYNAMICS CORRECTION
Keep the spring model. Learn the missing drag.
The reference law is x″ = −x − cv − 0.3 tanh(3v). The incomplete physics model uses x″ = −x − 0.1v. A small neural readout learns the missing acceleration from 49 synthetic velocity samples, with seven fixed tanh features and a velocity feature.
Current correction trained at c = 0.40
● Reference law┄ Incomplete physics━ Physics + neural correction
- Physics rollout RMSE
- 0.4718
- Hybrid rollout RMSE
- 0.0052
- Training velocity range
- −2 to 2 m/s
Change drag before refitting to expose a distribution shift. Fitting updates eight output weights by gradient descent; it does not train a control policy. All trajectories use the same 0.01 s semi-implicit integrator, so this comparison isolates model error rather than integration accuracy.
Residual learning helps when the known model captures useful structure and the correction sees representative data. Test longer rollouts and new parameter settings: a small one-step error can accumulate, and a learned correction can violate energy or stability properties. A physics-informed loss or differentiable simulator is another design choice, not a synonym for every hybrid model.
For a research example of combining an existing controller with learned corrections, see Johannink et al., Residual Reinforcement Learning for Robot Control. The browser experiment above learns a dynamics correction, which is a different placement of the learned component.
DDPG: learn a continuous action and its value.
Deep Deterministic Policy Gradient uses an actor μ(s) that proposes a continuous action and a critic Q(s,a) that estimates its return. Think of a torque-controlled joint: the action is a number in a range, not a class label.
Actor + exploration noise→Physics environment→Replay transitions→Critic + actor updates
DDPG is off-policy: it learns from stored transitions collected by earlier behavior policies. Exploration noise is added while collecting data; evaluation normally uses the deterministic actor. Slowly updated target networks provide the bootstrap target.
DDPG update equations
y = r + gamma * (1 - terminated) * Q_target(s_next, mu_target(s_next))
critic_loss = mean((Q(s, a) - stop_gradient(y))^2)
actor_loss = -mean(Q(s, mu(s)))
target_parameters = (1 - tau) * target_parameters + tau * online_parametersThe critic trains on the action actually stored in replay. The actor trains on its current proposed action and follows the critic's gradient with respect to that action. During the actor step, keep the critic's input gradient while holding its parameters fixed. Target calculations do not receive gradients. These are the mechanisms introduced by the DDPG paper.
WORKED UPDATE / DDPG
Follow the critic's action gradient
Use a fixed toy critic Q(a) = −(a − a*)² to expose the calculation. The scalar actor parameter is its action. The old target actor outputs 0.1, the discount is 0.95, and actions are bounded to [−1, 1].
- Critic target y
- -0.4375
- Action gradient ∂Q/∂a
- 2.0000
- Actor after one step
- -0.2000
- Target actor after mixing
- 0.0850
○ Current action● Updated action│ Critic optimum
The critic is deliberately fixed here; this is update arithmetic, not a trained DDPG agent. A real run learns the critic from replay and softly updates both actor and critic targets. At true termination, y becomes the immediate reward.
Connect it to MuJoCo or Isaac Lab
A MuJoCo wrapper or an Isaac Lab task can provide observations, rewards, and transitions. The trainer must match the task's action bounds, vectorization, tensor device, and reset semantics. Neither engine forces DDPG. Availability in a particular RL library is a separate compatibility question.
Training-loop sketch / conceptual, not a package API
observe state
action = clamp(actor(state) + exploration_noise, action_limits)
next_state, reward, terminated, truncated = environment.step(action)
replay.store(state, action, reward, actual_next_state, terminated)
for sampled_batch in replay:
update_critic_with_detached_target(sampled_batch)
update_actor_through_critic_action_gradient(sampled_batch.states)
softly_update_both_target_networks()
# Reset on termination OR truncation.
# Remove bootstrap only for true task termination.A3C: several workers, one evolving policy.
Asynchronous Advantage Actor-Critic collects short rollouts in multiple environment workers. Each worker uses a local copy of shared parameters, computes policy and value gradients, and applies updates without waiting for every other worker.
Worker 1 · local rolloutWorker 2 · local rolloutWorker 3 · local rollout
→Updates arrive at different times. Workers refresh their local weights.
The actor represents a stochastic policy π(a|s); the critic estimates V(s). An n-step return combines observed rewards with a final value estimate when the rollout remains nonterminal. Subtracting V(s) produces an advantage estimate. An entropy term can encourage exploration. The original method and its asynchronous design are described in Mnih et al. (2016).
A3C rollout objectives
R_t = r_t + gamma*r_(t+1) + ... + gamma^n * V(s_(t+n))
# Use zero final bootstrap at true termination.
advantage_t = R_t - V(s_t)
actor_loss = -log pi(a_t|s_t) * stop_gradient(advantage_t) - beta * entropy
value_loss = 0.5 * (V(s_t) - stop_gradient(R_t))^2WORKED ROLLOUT / A3C
Turn three rewards into an advantage
One worker observes rewards [r₀, −0.1, 1]. Walk backward from its final value estimate to compute returns. The actor example uses a binary policy and an entropy coefficient of 0.01.
Time 0 · reward 0.20R0 = 1.2845
Time 1 · reward -0.10R1 = 1.2050
Time 2 · reward 1.00R2 = 1.4500
- Starting advantage
- 0.8845
- Actor + entropy loss
- 0.4451
- Value loss at t=0
- 0.3912
A positive advantage supports increasing the selected action's probability in the policy-gradient term.
Returns and advantages are treated as fixed targets in their respective loss gradients. This panel computes one worker's arithmetic; it does not launch asynchronous workers or simulate optimizer races.
Asynchronous is a training schedule, not a physics feature
A3C can use categorical actions or a continuous-action distribution, such as a Gaussian with a properly handled bounded-action transformation. It does not need a differentiable engine. Separate MuJoCo CPU environments are a natural teaching setup for independent workers. A large synchronized GPU batch in Isaac Lab is not automatically A3C.
A3C is conventionally treated as on-policy because updates use recent local rollouts without a large replay buffer. Asynchronous parameter delay still introduces policy lag. A2C synchronizes worker updates; PPO uses a different update objective. Parallel environments alone do not make these algorithms equivalent.
Worker-loop sketch / conceptual, not executable multiprocessing code
repeat in each worker:
local_parameters = copy(shared_parameters)
collect up to n steps using the local policy
bootstrap = 0 if truly_terminal else local_value(final_observation)
walk rewards backward to compute n-step returns
compute actor, value, and entropy losses
apply local gradients to shared parameters
refresh local parameters
# Correct shared optimizer state and update synchronization are implementation work.
# Using fresh independent local optimizers is not automatically equivalent to A3C.Choose the simulator and algorithm separately.
There is no fixed pairing of MuJoCo with DDPG or Isaac Lab with A3C. Choose the environment for its physics, sensors, hardware, and task workflow. Choose the algorithm for its action space, data reuse, and update behavior.
| Dimension | DDPG | A3C |
|---|---|---|
| Actor | Deterministic continuous-action mapping | Stochastic policy; discrete or continuous variants |
| Critic | Action value Q(s,a) | State value V(s) |
| Data | Off-policy replay batches | Recent local rollouts; asynchronous policy lag |
| Target | Bootstrapped Q with slow target networks | n-step return with a final value bootstrap |
| Exploration | Behavior noise around the actor | Policy sampling, often entropy regularization |
| Parallelism | Possible, but not its defining mechanism | Independent workers update shared parameters |
| Typical concern | Critic errors can misdirect the actor | Stale gradients, variance, and update coordination |
A controlled joint
Start with the MuJoCo PD example. Define observations and reward, then compare a DDPG agent with the same action limits and control rate. Report tracking error, effort, and environment steps.
Parallel reaching
Use Isaac Lab's reaching task with its supported baseline. Scale the number of environments while recording end-to-end throughput, GPU memory, and held-out success—not just frames per second.
Independent workers
Use a small CPU control task to compare one rollout worker with several A3C workers. Keep the environment-step budget fixed and report wall time, return variation, and parameter lag.
Why does a learned policy not require differentiable physics?
DDPG differentiates the learned critic with respect to the actor's action. A3C differentiates policy log-probabilities. Both can learn from sampled transitions without differentiating the simulator. Differentiable physics supports a different route: backpropagating through modeled state transitions.
What must match before comparing engines?
Robot mass and inertia, geometry and contact parameters, joint order and limits, actuator model, observation processing, reward, reset distribution, physics step, control rate, and evaluation targets. Report settings that cannot be matched instead of attributing every difference to the engine.
How does a hybrid result earn trust?
Compare the baseline model or controller against the hybrid using unseen trajectories and parameter conditions. Report the training domain, rollout horizon, constraint violations, failure cases, and sensitivity to timestep. A successful fit to training transitions is only the first check.
Platform and backend notes reviewed . Versioned example links identify the APIs used. The browser labs are small mathematical models implemented for this course; they do not embed Isaac Sim, Isaac Lab, or MuJoCo.
Build a small one, properly.
The instinct is to reach for a photorealistic platform. Resist it. A correct environment of twenty lines will teach you more than a beautiful one you cannot reason about, and these steps are in this order for a reason.
- STEP 01Write the reset before anything else
An environment you cannot return to a known state is not an environment, it is an animation. Decide what a starting state is and make it exact.
Done when: Two resets with the same seed produce byte-identical observations.
- STEP 02Choose the integrator and the step deliberately
Semi-implicit Euler at a fixed step is the right default. Write the step size down somewhere visible; it will explain a bug later.
Done when: You can say what your stability limit is and why.
- STEP 03Define the observation and be honest about it
List every quantity the agent sees. Mark the ones you could not measure on real hardware. Those are your future problems.
Done when: No observation is privileged simulator state you cannot obtain outside the simulator.
- STEP 04Write the reward last and distrust it
Reward shaping is where projects quietly go wrong: the agent will optimise exactly what you wrote, including the parts you did not mean.
Done when: You have watched a high-reward episode and agreed it deserved the score.
- STEP 05Randomise before you tune
Add variation to masses, friction, latency and starting states early. Tuning against one fixed world produces a policy that only works there.
Done when: Performance is reported across a distribution, with a spread, not as one number.
- STEP 06Keep a fixed evaluation set
Hold out specific seeds and scenarios that you never train on and never adjust.
Done when: You can compare this week's run with last month's and mean it.