First I added a pair of functions internal::assert_failure and internal::check_failure which get called when assertions and checks fail. You can have the debugger break in these to find failures.

The derivative accumulator checks for NaN.

And finally ConjugateGradients wasn't initializing its threshold_ member variable.