How it works

A [5, 16, 8, 1] tanh network learns to back a tractor-trailer into a loading dock. Nobody shows it how, and no second network imitates the truck: each drive's docking error is sent backward through time, through the truck's own equations, and that gradient alone trains the controller. Every number in the lab is computed in your browser.

The task

A tractor-trailer must back up to a loading dock. The driver can do only two things: reverse at a fixed slow speed and steer the cab's front wheels. The goal is the rear of the trailer at the dock (x = 0) on its centre line (y = 0), with the trailer square to the dock (θt = 0).

This is the classic control benchmark of Nguyen and Widrow (1990). It is hard for three reasons:

  1. Unstable dynamics. A trailer pushed backward is an inverted pendulum in disguise: any angle between cab and trailer grows on its own. Do nothing and the rig jackknifes.
  2. Delayed reward. Whether a steering decision was good becomes clear only hundreds of steps later, when the truck arrives, or does not.
  3. No direct supervision. Nobody provides correct steering angles to imitate, so plain supervised learning on (state → steering) pairs is impossible. The error signal exists only at the end of a drive.

The truck’s kinematics

The state is four numbers, (x, y, θt, θc): the position of the trailer's rear axle, the trailer's heading and the cab's heading. Their difference, θd = θc − θt, is the hitch angle. The control is one number, the steering angle u, limited to |u| ≤ 1.2 rad (68.8°). The hitch sits over the cab's rear axle; the cab's wheelbase is Lc = 3 m and the hitch-to-trailer-axle length Lt = 6 m. Each step the rig reverses r = 0.25 m, a kinematic bicycle model with a trailer:

x′  = x  − r · cos θd · cos θt
y′  = y  − r · cos θd · sin θt
θt′ = θt − (r/Lt) · sin θd
θc′ = θc − (r/Lc) · tan u

The trailer's axle follows the hitch, the hitch's sideways pull turns the trailer, and the steered wheels turn the reversing cab.

The field is 50 × 50 m: x from 0 to 50 and y from −25 to 25, with the dock at the origin on the left edge. An episode ends when the trailer's axle crosses the dock line, x ≤ 0 (docked), when it leaves the field (out of bounds), when the hitch angle passes 90° (jackknifed), or after 600 steps without any of these (timed out). The dock is checked first, then the field's edge, then the jackknife.

The instability is built in. Subtracting the trailer's update from the cab's gives θd′ = θd + (r/Lt) sin θd − (r/Lc) tan u. With the wheel straight (u = 0) the hitch angle feeds itself: near zero, a small hitch error grows by 1 + r/Lt, ×1.042 per step, about 1.8 orders of magnitude every 100 steps. That is why backing a trailer is hard, and why the controller must keep counter-steering.

Concept cards: Hitch barrier and jackknife · Success criteria

The controller

The controller is a small multilayer perceptron, [5, 16, 8, 1]: 5 inputs, hidden layers of 16 and 8 units and one output, with tanh on every layer, the output included: 241 weights and biases in all. Its output a, scaled by the steering limit, is the steering command, u = 1.2 · a.

The pose enters the network roughly normalized to [−1, 1] and free of angle wrap:

  • x/25 − 1, the distance from the dock;
  • y/25, the lateral offset;
  • sin θt and cos θt, the trailer's heading, with no seam at ±180°;
  • θd/(π/2), the hitch angle, used as it is: an episode ends at a jackknife long before the angle could wind past ±π.

Because the output is a tanh scaled to the steering limit, the command is bounded and smooth: there is no clipping kink inside the gradient path. Every unit's backward pass multiplies its gradient by tanh's slope, 1 − a2, which is nearly 0 when the unit saturates near ±1; a saturated unit passes almost no gradient back, so the weights feeding it stop learning. The lab's network view rings each unit by this factor.

The network is written by hand rather than with a library, because backpropagation through time needs every step's activations and the gradient with respect to the network's inputs, not only its weights.

Concept cards: Tanh saturation and gate rings · Receptive-field map

The loss, and why every term exists

Each episode is scored by its terminal docking error plus two small per-step terms:

L =wy yK2+ wθ θt,K2+ wh hK2+ [not docked] wx xK2+ Σk ( wu uk2+ wb max(0, |hk| − π/3)2 )

with h = θc − θt the hitch angle and K the final step. The default weights are wx = 0.05, wy = 0.5, wθ = 2, wh = 0.3, wu = 0.002 (steering effort) and wb = 0.15 (the soft jackknife barrier), with the barrier at π/3 (60°). The trailer heading in the loss is deliberately not wrapped: a trailer that lines up by looping a full 2π is still 2π wrong.

Why every term exists

Circling and escaping the field are not bugs of one kind: they are the two stable failure attractors of this problem. Each of these seven defenses closes off one route into them:

  1. Curriculum and replay. The controller always has a competence zone that includes the dock, so gradients always point somewhere useful. Circling in training mostly appears when every episode is too hard ever to reach the target; the curriculum makes that state unreachable (see Curriculum and replay).
  2. Steering-effort penalty (0.002 · u2 per step). A circle needs sustained steering, so an orbit accrues unbounded cost while a straightening drive's penalty fades. The weight is small enough not to matter for the one genuine 180° swing a turned-around start needs (about 0.03 in total) but ruinous for perpetual turning: at full lock it costs 0.00288 every step.
  3. The unwrapped trailer heading. The terminal loss penalizes θt, not its wrapped value. A trailer that lines up by winding a full 2π is still 2π wrong, which costs wθ·(2π)2 ≈ 79, so a loop is never a shortcut to low loss. The network sees the heading only as its sine and cosine; only the loss counts loops. Judging a finished drive is different: the success test wraps the heading, because there only the trailer's direction matters.
  4. The episode timeout and the progress term. An episode that circles ends after 600 steps and is then charged wxx2 for the distance it failed to cover, as is one that leaves the field or jackknifes, so orbiting far from the dock is strictly worse than approaching it.
  5. A boundary ending with a gradient, not just a penalty. Leaving the field ends the episode and evaluates the same quadratic loss at the escape point, and backpropagation pushes the whole trajectory inward. A flat penalty would give zero gradient; squares in x and y tell the network which way is in.
  6. Jackknife defense in depth. A soft quadratic barrier past 60° of hitch angle, which is differentiable, so the controller learns to counter-steer before the limit; and a hard stop at 90°, so an episode's record never contains physically meaningless folded poses.
  7. Gradient clipping and Adam. Divergence of training itself is the last way trucks end up circling: a destroyed policy steers at random, and saturated tanh units lock the wheel at full deflection. Clipping keeps the unstable backward dynamics from ever destroying the weights (see Gradient clipping and Adam).

The lab's training lab trains a baseline and a variant side by side from the same starting state; most of its presets switch off one of these defenses, so you can see what each one does. Not every defense is load-bearing on every seed: in one measured default-seed run of 3,000 epochs, removing replay, the steering penalty or the progress term left the benchmark result unchanged.

Concept cards: Terminal loss · Steering-effort penalty · Unwrapped trailer heading · Progress term (wX) · Hitch barrier and jackknife · Training lab and break-it presets

Backpropagation through time

One episode from the start state s0 under the current controller is

uk = 1.2 · NW(enc(sk)),sk+1 = F(sk, uk),k = 0 … K − 1,

where NW is the network with weights W, enc the input encoding and F one kinematic step. Nguyen and Widrow's insight was that if the truck can be differentiated, a whole drive is one enormous feed-forward computation:

s₀ → controller → u₀ → truck → s₁
s₁ → controller → u₁ → truck → s₂
  ⋮
s_K → loss L

The same network appears at every step (shared weights), with a copy of the truck's equations between consecutive copies. The terminal error is propagated backward through the whole chain, through every truck step and every controller copy, and adds up to one gradient for the shared weights: that is backpropagation through time.

No emulator network

The original paper treated the truck as unknown, so it first trained a second network, the emulator, to imitate it, purely to have something differentiable to propagate the error through. Here the equations are known and smooth, so the error flows back through the exact derivatives of the real kinematics instead, the per-step Jacobians

Ak = ∂sk+1/∂sk (4 × 4),bk = ∂sk+1/∂uk (4),Ek = ∂enc(sk)/∂sk (5 × 4).

That removes a whole source of error, the emulator's mismatch, and is otherwise the same algorithm.

The backward recursion

Starting from the terminal gradient gK = ∂Lterminal/∂sK, each step walking backward does four things:

  1. The steering gradient, how the loss changes with this step's command: ūk = bk⊤ gk+1 + 2 wu uk.
  2. Through the controller: a standard backward pass through the network with output gradient 1.2 · ūk adds ∂L/∂W into the episode's weight gradient and returns the gradient at the network's inputs, ∇enc.
  3. Into the previous state: gk = Ak⊤ gk+1 + Ek⊤ ∇enc plus the barrier term's gradient, ∂/∂sk [wb max(0, |hk| − π/3)2].
  4. Repeat until g0.

The second term in step 3 is easy to forget and wrong to omit: the state at step k influences the loss twice, through the physics and through the steering decision the controller made when it looked at that state. The same code path also adds up the loss, so the gradient can be checked against exactly the quantity training minimizes: the tests compare every entry of every step's Jacobian, and the end-to-end gradient for all 241 parameters, with central finite differences, in the Swift app and in this port alike.

The gradient horizon

gk is the adjoint: how much the episode's loss would change per metre or radian of a nudge to the pose at step k. Its size against the number of steps back from the end shows how far back in time the docking error still matters. A curve that falls going back means vanishing sensitivity: the closed loop is stable, so an early nudge is corrected before the end. A curve that rises means exploding sensitivity, which is when gradient clipping earns its keep. Carried back through the truck alone, without the controller's reaction, a small hitch error's sensitivity grows ×1.042 per step, and the whole plant-alone curve usually climbs faster still; the gap between it and the recorded curve is the controller's stabilizing work. The lab's backward-pass inspector draws both after Analyze.

Concept cards: Backpropagation through time · Adjoint dL/d(state) · Gradient horizon

Curriculum and replay

Started anywhere with any heading, an untrained network gets its gradients from episodes that end 600 steps later in a wall. They point nowhere useful, and training falls into exactly the failures above: endless circling and repeated escapes. Nguyen and Widrow's answer was lessons: practise easy placements first, then harder ones.

Here the difficulty d ramps linearly from 0 to 1 across each training run, d = epoch ÷ (epochs − 1), reaching 1 on the final epoch. The start box does not just grow: it slides away from the dock as it widens, so the easy lessons are short, nearly aligned approaches rather than a subset of the hard ones. For an aligned trailer the box is x ∈ [8 + 12d, 14 + 26d] m, |y| ≤ min(2 + 18d, 20) m, |θt| ≤ 0.3 + (π − 0.3)d rad and |θd| ≤ 0.4d rad:

The curriculum's start box for an aligned trailer at three difficulties
Difficultyx from the dock (m)Lateral y (m)Trailer headingHitch
0.08–14±2±17°0
0.514–27±11±99°±0.2 rad
1.020–40±20±180°±0.4 rad

Poses facing away from the dock start nearer the middle of the field: the box's far edge comes in by 14·|θt|/π m and its half-width by 8·|θt|/π m, so at d = 1 a fully reversed trailer starts from x 20–26 m with |y| ≤ 12 m. A trailer pointing away from the dock can only drive away from it at first, and needs some 15–20 m of room to swing around; sampling such poses near the far wall would make out-of-bounds endings unavoidable and teach nothing.

Across runs the ramp is floored at the highest difficulty the network has already handled competently: an epoch that docks at least 50 % of its batch raises the floor to its difficulty. Pressing Train again therefore fine-tunes at the level the network reached instead of re-running the easy lessons, and a short tryout on an untrained network cannot pin later runs at full difficulty.

Replay. For each episode, with probability 30 % the trainer draws an easier difficulty instead, uniformly between 0 and the current one. Without it, the large errors of long, hard episodes dominate the updates and the network forgets how to finish precisely (catastrophic forgetting); replay keeps short, exact dockings in the mix. While training, the lab outlines the live lesson's start box with a dashed rectangle and the replay range around it with a dotted one.

Concept cards: Curriculum and the start box · Replay of easier lessons · Curriculum floor

Gradient clipping and Adam

Each epoch is one step of mini-batch gradient descent: roll out 4 episodes, backpropagate each through time, average their gradients, clip the average to a global norm of 1, and apply one Adam update. What is unusual is where the gradient comes from: backpropagation through the analytic kinematics gives the exact gradient of each episode's loss with respect to every weight, with no emulator network and no sampled, reinforcement-learning-style estimate. The only randomness is which starts make up each batch.

Clipping

The clip measures the length of the averaged gradient over all the weights and biases together, its global norm n. If n is above the clip limit c, the gradient is scaled by c/n, keeping its direction; if n is not a finite number, the update is skipped. Clipping is not optional here: gradients flow backward through the same unstable dynamics that make a trailer jackknife forward in time, so over hundreds of steps they can explode. It is also routine: in one measured default run of the macOS app, 2,284 of its 3,000 updates were clipped, and a clipped update still points the same way.

Adam

m ← β1m + (1 − β1)g,v ← β2v + (1 − β2)g2,w ← w − η · m̂ / (√v̂ + ε)

Adam keeps two running averages for every parameter: m, of its gradient (momentum, which smooths noisy batch-to-batch directions), and v, of its squared gradient (a running measure of that parameter's gradient scale). Each step moves the weight by η · m̂ / (√v̂ + ε), so every parameter steps by roughly η whatever the size of its raw gradient. The hats are bias corrections, dividing by 1 − βt, that undo the averages' start from zero. Here η = 0.002, β1 = 0.9, β2 = 0.999 and ε = 10−8. The per-parameter scaling matters because episodes differ wildly: a short, easy docking produces small gradients and a long, hard drive huge ones.

Concept cards: Gradient clipping · Adam optimizer

How a controller is judged

A drive docks when the trailer's rear axle crosses the dock line, x ≤ 0. It succeeds when it docks within 600 steps with a lateral error |y| of at most 2 m and the trailer heading, wrapped to ±180°, within 0.4 rad (22.9°). Crossing the dock line outside either tolerance is a dock plane miss; the other endings are out of bounds, jackknifed and timed out. Simulation, the checkpoint ghosts, policy evaluation, the diagnostic after training and the manual driving challenges all use these same criteria.

The lab reports success on three different sets of starts, because they answer different questions.

The 200-start diagnostic. After each training run, 200 random starts from the full-difficulty training box (x 20–40 m, |y| ≤ 20 m, any trailer heading, a hitch of up to 0.4 rad), drawn from a random stream of their own so they never disturb training. It measures the training distribution, so it never tests a start nearer than 20 m. In the macOS app's reference run every one of these starts docked within tolerance; its tests require at least 90 % docked and 80 % successful, and this port's tests require the same.

The fixed benchmark, stratified-v1: 243 deterministic stress starts in three groups, each group a Cartesian product of the values below (angles in radians). They are balanced stress cases, not a representative sample, and some may be physically unrecoverable in reverse.

The stratified-v1 fixed benchmark's groups
GroupStartsx (m)y (m)Trailer headingHitch
Near dock812, 8, 16−6, 0, 6−0.4, 0, 0.4−0.4, 0, 0.4
Mid-field8120, 30, 40−15, 0, 15−π/2, 0, π/2−0.4, 0, 0.4
Reversed8120, 23, 26−10, 0, 10−π, −3π/4, 3π/4−0.4, 0, 0.4

The success map: one start at the centre of each cell of a 16 × 16 grid over the field (256 starts, x 1.5625–48.4375 m, |y| up to 23.4375 m), all at one chosen trailer heading and hitch angle. It is one slice through the possible starts, a conditional picture rather than an accuracy estimate.

Raw training loss is no yardstick. Each epoch's loss averages its batch, and the curriculum makes those episodes harder as the run goes on, so the loss can rise while the controller improves. It is also a weighted sum, so changing a weight in the training settings re-prices every later epoch. To compare controllers or settings, use the fixed benchmark: identical starts every time.

Concept cards: Success criteria · Fixed benchmark vs. training diagnostic · Why raw loss is not comparable

What the web edition does differently

This edition is a line-by-line port of the macOS and iOS app's core to TypeScript, and every number is computed in your browser. There is no server: the site is a set of prerendered pages.

  • Training runs in a Web Worker. The live trainer stays on the page, as the Swift app keeps it on its main thread. Train hands its complete state (the weights, Adam's moments, the curriculum floor and the random streams' positions) to a background worker, which runs the same epoch loop and sends its progress at most every 50 ms, each time only after the page has applied the previous update. When the run ends or is stopped, the page restores the trainer from the worker's final state, so the session continues exactly as if it had trained in place. Policy evaluation, the hardest-start search and the training lab run in workers of their own, so an evaluation or a lab comparison can run beside training.
  • The same arithmetic, different transcendental functions. The port keeps the Swift core's operations in the same order. Everything that involves no sine, cosine, tangent or tanh (the random streams, the weight initialisation, the start boxes, the benchmark and success-map starts) matches the Swift app bit for bit. Those four functions, though, come from the browser's JavaScript engine rather than Apple's math library, and the two can differ in the last bit or two for some inputs: in the calibration run the first difference is one tanh in the first epoch, one unit in the last place. Over thousands of epochs such differences compound, so a long training run here drifts from the macOS app's run with the same seeds: the same method and the same calibration thresholds, not the same final weights. In a one-off check under Node, with Apple's sine, cosine, tangent and tanh in place of the engine's, the port reproduced the Swift app's 3,000-epoch calibration run bit for bit.
  • Browsers can differ too, since each JavaScript engine brings its own versions of these functions; the port's measurements were made with V8, the engine of Chrome, Edge and Node.
  • Experiment files use the macOS app's format (version 1), so an experiment saved in one edition can be loaded in the other.

References

  • D. Nguyen and B. Widrow, “Neural Networks for Self-Learning Control Systems,” IEEE Control Systems Magazine, 1990: the original truck backer-upper and its emulator and backpropagation-through-time scheme.
  • P. Werbos, “Backpropagation Through Time: What It Does and How to Do It,” Proceedings of the IEEE, 1990.
  • D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” 2014.