Re: [IMP-dev] change of optimizer performance from v7018 to v9277
On May 18, 2011, at 9:05 AM, Friedrich Foerster wrote:
> I remember painfully writing a test case for the em restraint and the derivatives when I implemented the em score. Is that test not functional anymore? They are still there, but I'm not sure if they pass. It also isn't clear to me what exactly they test as they use some modeller functionality that I am not familiar with to do the actual checking. (Reason I'm not sure about their status: I broke the build last night on all platforms except windows, and for some reason the windows test run didn't terminate. And the em tests have been failing in general for rather a while, and it is not something I've looked in to. And I don't have modeller installed on any of my machines, so that is not a test I run myself).
Keren, do you have more sense of the status of those tests and, if they are broken, when they broke?
Anyway, there is a test I added a bit back to test that the magnitudes of the derivatives make sense, since they didn't (eg that they roughly match the number one gets for a numeric approximation of the derivative). So there are definitely problems that need to be fixed :-)
> > Cheers > > Friedrich > > Dr. Friedrich Foerster > Max-Planck Institute of Biochemistry > Am Klopferspitz 18 > D-81252 Martinsried > > www.tomotronic.org > > > On 18.05.2011, at 16:55, Keren Lasker kerenl@salilab.org wrote: > >> Major changes to the code from my part were in verion 7212 in class CoarseCC, which mainly dealt with rigid bodies support. >> I am not sure if that is the reason for Pia's problem, but as she needs to use EM based optimization relatively soon, i will take a look again. >> In any case - Pia - it would be useful if you can write a test case that I can check things against. >> keren. >> On May 18, 2011, at 7:43 AM, Daniel Russel wrote: >> >>> I suspect it is an issue with em::FitRestraint as the derivatives returned by that restraint are currently broken (the direction is fine, but the magnitude off by quite a bit). As a result derivative-using optimizers don't perform well with it. >>> >>> Keren, Ben, is there a plan for fixing it? >>> --Daniel >>> >>> On May 18, 2011, at 6:31 AM, Pia Unverdorben wrote: >>> >>>> Dear all, >>>> >>>> is it possible, that something fundamental changed in the optimizers, steepest descent as well as conjugate gradients? >>>> I try to fit a protein to an em density, simulated from another conformational state using the em restraint and plenty of harmonic restraints. With an older version (revision 7018) it moves towards the density, whereas with the revision 9277 the score and the positions barely change, irrespective of the number of steps, threshold, stepsize and different weights of restraints. >>>> More importantly, i did compare the optimization results of v 7018 and 9277 of conjugate gradient optimization and steepest descent from identical starting positions with identical restraints and optimizer settings. the initial scores are identical, but then the problem starts: >>>> For example, with v7018 my model improves from a score of 1835 to 1101 after 100 steps of steepest descent optimization. However, with v9277 my score increases (!) to 1836. behavior of conjugate gradients similarly deteriorates for v9277, albeit i did not observe increases of scores. nevertheless, improvements in score were disappointingly low. >>>> Does someone know, why the optimizers behave so differently, for my purposes worse? >>>> >>>> Thanks a lot! >>>> >>>> Regards, >>>> >>>> Pia >>>> _______________________________________________ >>>> IMP-dev mailing list >>>> IMP-dev@salilab.org >>>> https://salilab.org/mailman/listinfo/imp-dev >>> >> >> >> _______________________________________________ >> IMP-dev mailing list >> IMP-dev@salilab.org >> https://salilab.org/mailman/listinfo/imp-dev >>
i think the test is broken. i worked on that a while back with daniel (ie daniel's comment on the magnitude) but i did not manage to resolve the problem. i will look at that again, clearly it is very important. will report back to the list once resolved. keren. On May 18, 2011, at 10:39 AM, Daniel Russel wrote:
> On May 18, 2011, at 9:05 AM, Friedrich Foerster wrote: > >> I remember painfully writing a test case for the em restraint and the derivatives when I implemented the em score. Is that test not functional anymore? > They are still there, but I'm not sure if they pass. It also isn't clear to me what exactly they test as they use some modeller functionality that I am not familiar with to do the actual checking. (Reason I'm not sure about their status: I broke the build last night on all platforms except windows, and for some reason the windows test run didn't terminate. And the em tests have been failing in general for rather a while, and it is not something I've looked in to. And I don't have modeller installed on any of my machines, so that is not a test I run myself). > > Keren, do you have more sense of the status of those tests and, if they are broken, when they broke? > > Anyway, there is a test I added a bit back to test that the magnitudes of the derivatives make sense, since they didn't (eg that they roughly match the number one gets for a numeric approximation of the derivative). So there are definitely problems that need to be fixed :-) > >> >> Cheers >> >> Friedrich >> >> Dr. Friedrich Foerster >> Max-Planck Institute of Biochemistry >> Am Klopferspitz 18 >> D-81252 Martinsried >> >> www.tomotronic.org >> >> >> On 18.05.2011, at 16:55, Keren Lasker kerenl@salilab.org wrote: >> >>> Major changes to the code from my part were in verion 7212 in class CoarseCC, which mainly dealt with rigid bodies support. >>> I am not sure if that is the reason for Pia's problem, but as she needs to use EM based optimization relatively soon, i will take a look again. >>> In any case - Pia - it would be useful if you can write a test case that I can check things against. >>> keren. >>> On May 18, 2011, at 7:43 AM, Daniel Russel wrote: >>> >>>> I suspect it is an issue with em::FitRestraint as the derivatives returned by that restraint are currently broken (the direction is fine, but the magnitude off by quite a bit). As a result derivative-using optimizers don't perform well with it. >>>> >>>> Keren, Ben, is there a plan for fixing it? >>>> --Daniel >>>> >>>> On May 18, 2011, at 6:31 AM, Pia Unverdorben wrote: >>>> >>>>> Dear all, >>>>> >>>>> is it possible, that something fundamental changed in the optimizers, steepest descent as well as conjugate gradients? >>>>> I try to fit a protein to an em density, simulated from another conformational state using the em restraint and plenty of harmonic restraints. With an older version (revision 7018) it moves towards the density, whereas with the revision 9277 the score and the positions barely change, irrespective of the number of steps, threshold, stepsize and different weights of restraints. >>>>> More importantly, i did compare the optimization results of v 7018 and 9277 of conjugate gradient optimization and steepest descent from identical starting positions with identical restraints and optimizer settings. the initial scores are identical, but then the problem starts: >>>>> For example, with v7018 my model improves from a score of 1835 to 1101 after 100 steps of steepest descent optimization. However, with v9277 my score increases (!) to 1836. behavior of conjugate gradients similarly deteriorates for v9277, albeit i did not observe increases of scores. nevertheless, improvements in score were disappointingly low. >>>>> Does someone know, why the optimizers behave so differently, for my purposes worse? >>>>> >>>>> Thanks a lot! >>>>> >>>>> Regards, >>>>> >>>>> Pia >>>>> _______________________________________________ >>>>> IMP-dev mailing list >>>>> IMP-dev@salilab.org >>>>> https://salilab.org/mailman/listinfo/imp-dev >>>> >>> >>> >>> _______________________________________________ >>> IMP-dev mailing list >>> IMP-dev@salilab.org >>> https://salilab.org/mailman/listinfo/imp-dev >>> > > > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev
It appears that Frido's tests were passing (and probably still are), so they don't seem to have been sensitive to the magnitude of the derivative being off.
On May 18, 2011, at 10:44 AM, Keren Lasker wrote:
> i think the test is broken. > i worked on that a while back with daniel (ie daniel's comment on the magnitude) but i did not manage to resolve the problem. > i will look at that again, clearly it is very important. > will report back to the list once resolved. > keren. > On May 18, 2011, at 10:39 AM, Daniel Russel wrote: > >> On May 18, 2011, at 9:05 AM, Friedrich Foerster wrote: >> >>> I remember painfully writing a test case for the em restraint and the derivatives when I implemented the em score. Is that test not functional anymore? >> They are still there, but I'm not sure if they pass. It also isn't clear to me what exactly they test as they use some modeller functionality that I am not familiar with to do the actual checking. (Reason I'm not sure about their status: I broke the build last night on all platforms except windows, and for some reason the windows test run didn't terminate. And the em tests have been failing in general for rather a while, and it is not something I've looked in to. And I don't have modeller installed on any of my machines, so that is not a test I run myself). >> >> Keren, do you have more sense of the status of those tests and, if they are broken, when they broke? >> >> Anyway, there is a test I added a bit back to test that the magnitudes of the derivatives make sense, since they didn't (eg that they roughly match the number one gets for a numeric approximation of the derivative). So there are definitely problems that need to be fixed :-) >> >>> >>> Cheers >>> >>> Friedrich >>> >>> Dr. Friedrich Foerster >>> Max-Planck Institute of Biochemistry >>> Am Klopferspitz 18 >>> D-81252 Martinsried >>> >>> www.tomotronic.org >>> >>> >>> On 18.05.2011, at 16:55, Keren Lasker kerenl@salilab.org wrote: >>> >>>> Major changes to the code from my part were in verion 7212 in class CoarseCC, which mainly dealt with rigid bodies support. >>>> I am not sure if that is the reason for Pia's problem, but as she needs to use EM based optimization relatively soon, i will take a look again. >>>> In any case - Pia - it would be useful if you can write a test case that I can check things against. >>>> keren. >>>> On May 18, 2011, at 7:43 AM, Daniel Russel wrote: >>>> >>>>> I suspect it is an issue with em::FitRestraint as the derivatives returned by that restraint are currently broken (the direction is fine, but the magnitude off by quite a bit). As a result derivative-using optimizers don't perform well with it. >>>>> >>>>> Keren, Ben, is there a plan for fixing it? >>>>> --Daniel >>>>> >>>>> On May 18, 2011, at 6:31 AM, Pia Unverdorben wrote: >>>>> >>>>>> Dear all, >>>>>> >>>>>> is it possible, that something fundamental changed in the optimizers, steepest descent as well as conjugate gradients? >>>>>> I try to fit a protein to an em density, simulated from another conformational state using the em restraint and plenty of harmonic restraints. With an older version (revision 7018) it moves towards the density, whereas with the revision 9277 the score and the positions barely change, irrespective of the number of steps, threshold, stepsize and different weights of restraints. >>>>>> More importantly, i did compare the optimization results of v 7018 and 9277 of conjugate gradient optimization and steepest descent from identical starting positions with identical restraints and optimizer settings. the initial scores are identical, but then the problem starts: >>>>>> For example, with v7018 my model improves from a score of 1835 to 1101 after 100 steps of steepest descent optimization. However, with v9277 my score increases (!) to 1836. behavior of conjugate gradients similarly deteriorates for v9277, albeit i did not observe increases of scores. nevertheless, improvements in score were disappointingly low. >>>>>> Does someone know, why the optimizers behave so differently, for my purposes worse? >>>>>> >>>>>> Thanks a lot! >>>>>> >>>>>> Regards, >>>>>> >>>>>> Pia >>>>>> _______________________________________________ >>>>>> IMP-dev mailing list >>>>>> IMP-dev@salilab.org >>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>> >>>> >>>> >>>> _______________________________________________ >>>> IMP-dev mailing list >>>> IMP-dev@salilab.org >>>> https://salilab.org/mailman/listinfo/imp-dev >>>> >> >> >> _______________________________________________ >> IMP-dev mailing list >> IMP-dev@salilab.org >> https://salilab.org/mailman/listinfo/imp-dev > > > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev
On 05/18/2011 11:00 AM, Daniel Russel wrote: > It appears that Frido's tests were passing (and probably still are), > so they don't seem to have been sensitive to the magnitude of the > derivative being off.
The magnitude of the derivatives is definitely checked by these tests. However, it may be that Keren's CoarseCC code isn't being utilized here, so the derivatives are actually correct in this test.
Ben
On 05/18/2011 10:39 AM, Daniel Russel wrote: > On May 18, 2011, at 9:05 AM, Friedrich Foerster wrote: >> I remember painfully writing a test case for the em restraint and >> the derivatives when I implemented the em score. Is that test not >> functional anymore? > They are still there, but I'm not sure if they pass. It also isn't > clear to me what exactly they test as they use some modeller > functionality that I am not familiar with to do the actual checking.
They use Modeller functionality that essentially does the same thing as IMP.test.TestCase.assertXYZDerivativesInTolerance() - i.e. it calculates the xyz derivatives numerically (basically by moving each atom a small amount along each axis and recalculating the score) and compares with the analytical values. In principle we could rework this test to no longer require Modeller. But I'd prefer to wait until it actually works before we do that. ;)
> and for some reason the windows test run didn't terminate
The tests didn't complete because the IMP.rmf tests took several hours to run, leaving no time for the rest of the tests. Unit tests shouldn't take hours to run...
(The reason this only just started happening is that we didn't build IMP.rmf on Windows until yesterday, when I built the necessary HDF5 and Boost DLL dependencies.)
> And the em tests have been failing in general > for rather a while, and it is not something I've looked in to.
Just like Keren said, she broke the derivatives. That happened around October last year. Every now and again I mention it to Keren again and she says "yes, I will look at it". ;)
Ben
no need to continue this thread too much. its on me and i will solve it. keren. On May 18, 2011, at 11:01 AM, Ben Webb wrote:
> On 05/18/2011 10:39 AM, Daniel Russel wrote: >> On May 18, 2011, at 9:05 AM, Friedrich Foerster wrote: >>> I remember painfully writing a test case for the em restraint and >>> the derivatives when I implemented the em score. Is that test not >>> functional anymore? >> They are still there, but I'm not sure if they pass. It also isn't >> clear to me what exactly they test as they use some modeller >> functionality that I am not familiar with to do the actual checking. > > They use Modeller functionality that essentially does the same thing as > IMP.test.TestCase.assertXYZDerivativesInTolerance() - i.e. it calculates > the xyz derivatives numerically (basically by moving each atom a small > amount along each axis and recalculating the score) and compares with > the analytical values. In principle we could rework this test to no > longer require Modeller. But I'd prefer to wait until it actually works > before we do that. ;) > >> and for some reason the windows test run didn't terminate > > The tests didn't complete because the IMP.rmf tests took several hours > to run, leaving no time for the rest of the tests. Unit tests shouldn't > take hours to run... > > (The reason this only just started happening is that we didn't build > IMP.rmf on Windows until yesterday, when I built the necessary HDF5 and > Boost DLL dependencies.) > >> And the em tests have been failing in general >> for rather a while, and it is not something I've looked in to. > > Just like Keren said, she broke the derivatives. That happened around > October last year. Every now and again I mention it to Keren again and > she says "yes, I will look at it". ;) > > Ben > -- > ben@salilab.org http://salilab.org/~ben/ > "It is a capital mistake to theorize before one has data." > - Sir Arthur Conan Doyle > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev
participants (3)
-
Ben Webb -
Daniel Russel -
Keren Lasker