I forgot an important technique that is somewhere between numeric differentiation and automatic differentiation:

(4) Complex-step differentiation http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.141....

The technique is clever: they add an finite difference along the imaginary axis, and this turns out to give a more accurate derivative than normal forward differences.

We don't support this in Ceres and suggest autodiff instead.