Optimization

BFGS against gradient descent on the Rosenbrock valley

Both methods start at $(-1.2,1)$ on Rosenbrock's function and use the same backtracking line search; they differ only in the direction they search along. BFGS is within $10^{-9}$ of the minimum at $(1,1)$ after 35 iterations and has a zero gradient to $10^{-12}$ after 36. Gradient descent takes 22,962 iterations to get within $10^{-9}$, and near the minimum it gains a factor of only 0.99919 per step, close to $(\kappa -1)/(\kappa +1)$ for the Hessian's condition number $\kappa =2508$.