A swing-up policy trained offline

The acrobot is like a gymnast on a high bar who can bend only at the hips. It is two links hinged end to end; the shoulder joint turns freely and the elbow has a motor. Its state is $(q_1,q_2,{\dot{q}}_1,{\dot{q}}_2)$ and its input is a single torque, so the shoulder moves only through its coupling to the elbow. A linear balancing controller works only near the upright, and the arm starts hanging straight down, far outside that region.

So it takes two controllers and a rule for when to switch from one to the other.

The balancing controller. Linearize the dynamics about the upright by central differences on the same chain_rate this page simulates, and the balancing gains come from the Riccati equation (the LQR method). Solved by the Schur method, it gives gains $K=(-189.5,-47.5,-89.4,-29.2)$ and a closed loop whose slowest mode decays like $e^{-1.68t}$.

The Riccati solution also gives the switching rule. Write $\delta$ for the state's distance from upright. The Riccati matrix $P$ turns it into one number, ${\delta }^{\top }P\delta$, the cost the balancing controller expects to pay to bring the arm from $\delta$ back to upright (its cost-to-go). When that number is small enough, the controller brings the arm upright within the torque limit; when it is larger, the controller can fail. To find how small is small enough (the switching level), the full nonlinear model with the torque limit was simulated offline from states along random directions in the four-dimensional state space, bisecting on the cost-to-go along each, and the level was set below the smallest one at which the controller failed. The arm switches to the balancing controller when the cost-to-go drops below 20.

The swing-up. It is not derived. The torque is $\tau ={\sum }_i{w}_i{\varphi }_i$ over nine features (a constant, the sines and cosines of both angles, both rates, the energy error, and the energy error times the shoulder rate), limited to 40 N·m. The nine weights come from a cross-entropy search: sample forty policies from a Gaussian, simulate each one, refit the Gaussian to the best six, and repeat forty times. That is 1600 simulated swings, run as Viv compiled to native code. The search runs offline; this page runs only the nine weights it found.

What it does. From hanging, the arm swings up in 1.188 seconds, the cost-to-go drops below 20, and the balancing controller takes over. At the end of the six seconds this page simulates, the two joint angles add up to 0.0045 radians from upright and are still closing; with twelve seconds instead of six they reach $2.8\times 10^{-7}$. The torque hits its 40 N·m limit early in the swing.

What it does not do. The weights were trained from one starting position, and they fail from a nearby one. Start with the shoulder 0.04 radians away from its trained value and the elbow at 0.05 radians instead of 0.08. Two seconds later the arm still has not reached the switching level; it is still turning, and its two joint angles add up to 9.5 radians away from upright. From the trained start the switch happened at 1.188 seconds. A test checks this failure. The search scored each policy on one swing from one start, so nothing in it rewarded working from other starts.