- IMP-dev - salilab.org

Cleanup of monte carlo
by Daniel Russel 09 Jun '11

09 Jun '11

The core::MonteCarlo code is rather a mess, has bugs and has subject to a lot of churn. This is primarily because it is trying to do too many different things, resulting is very complicated code. I'd like to break it up into several classes each with a single mode of operation. In particular, there would be - core.MonteCarlo, which does simple Monte Carlo (apply some movers and accept or reject) as well as provide some finer grained functionality for implementing less trivial monte carlo protocols. - core.MonteCarloWithLocalOptimization which applies some local optimization scheme after the moves and accepts or rejects the relaxed conformation - core.MonteCarloWithBasinHopping which supports basin hopping (apply a monte carlo move and then accept or reject based on the score after The main noticeably change would be that code that looks like mc= IMP.core.MonteCarlo() mc.set_model(m) cg= IMP.core.ConjugateGradients() mc.set_local_optimizer(cg) mc.set_local_steps(10) would need to be replaced by cg= IMP.core.ConjugateGradients(m) IMP.core.MonteCarloWithLocalOptimization(cg, 10) Comments?

4 8

values and objects
by Daniel Russel 08 Jun '11

08 Jun '11

I found a bug in the swig wrappers which I would like to fix. Fixing it though exposes issues with various things being passed around improperly by functions that are part of the api in various modules. In particular the following have issues (proposed fixes show) - em2d::do_segmentation(): return the opencv matrix instead of pass by ref. As it is internally reference counted there is no reason not to. - em2d::Image::get_min_and_max_values: return a FloatRange - em2d::get_peak() return a pair of values - em2d::ProjectionParameters::get_keys()- return FloatKeys - em::MapReaderWriter all methods: I don't see any need to export them to python anyway - em::ImageHeader::get_{date,time,title}: return std::string - em::KerenParameters and em::RadiusDependentDistanceMask are pretty much a mess in terms of how they are passed and stored, there are plans to revamp it all, so just hiding the methods dealing with them seems ok for now. - saxs::Score::fit_profile: the profile argument can probably be passed by const-ref or value, but it is not entirely clear to me - multifit::FFTFitting::get_wrapped_index and get_wrapped_correlation_map: can return an Ints - multifit::fitting_clustering: should probably just return the solutions + a few others which had obvious, non-breaking fixes which can make. I'd like to commit the wrapper changes. To do this, I'd just hide the above functions from python, they can be fixed and exposed as needed. Thoughts.

6 13

imp on the cluster question
by Dave Barkan 08 Jun '11

08 Jun '11

I am using the nightly build of IMP that gets installed on diva1 to run IMP on the cluster nodes. From what I can tell, this is built in debug or release mode. Is there a way to run it from diva in fast mode instead? thanks, Dave

2 1

[Bug 116] New: Large memory usage on first evaluate with many restraints
by bugzilla-daemon＠salilab.org 08 Jun '11

08 Jun '11

https://salilab.org/imp/bugs/show_bug.cgi?id=116 Summary: Large memory usage on first evaluate with many restraints Product: IMP Version: unspecified Platform: PC OS/Version: Mac OS Status: NEW Severity: enhancement Priority: P5 Component: kernel AssignedTo: drussel(a)SALILAB.ORG ReportedBy: drussel(a)SALILAB.ORG CC: imp-dev(a)salilab.org Estimated Hours: 0.0 'm trying to optimize a set of particles as done for the alpha-globin domain (I'm doing some testing on a larger chromatin domain) using a large set of restraints (over 10^6 restraints). The memory usage is OK until all the restraints are generated (~2GB), and then increase to 100% of memory and swap (16GB + 16GB) when I first call m.evaluate(False) (to check the initial score) causing the script to stop. I understand that a large number of restraints are being evaluated, but I was wondering if this is an expected behavior (i.e. if there is a sort of limitation in the number of restraints that can be implemented per memory GB) or if there is a workaround, before trying to run the script on a (shared) machine with much more memory. -- Configure bugmail: https://salilab.org/imp/bugs/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

1 1

file not found error
by Keren Lasker 04 Jun '11

04 Jun '11

hello - Say a program needs to read a parameter file. having IMP_USAGE_CHECK to check that the file is open is not optimal because it will not show up in fast mode. assuming that i will then just have a simple IF statement, what is the best way to exit the program ? a exit(1); seems bad especially if the program should be run though a webserver and we want to notify users on the exact problem. did someone have similar problems? thanks, keren.

2 1

change of optimizer performance from v7018 to v9277
by Pia Unverdorben 25 May '11

25 May '11

Dear all, is it possible, that something fundamental changed in the optimizers, steepest descent as well as conjugate gradients? I try to fit a protein to an em density, simulated from another conformational state using the em restraint and plenty of harmonic restraints. With an older version (revision 7018) it moves towards the density, whereas with the revision 9277 the score and the positions barely change, irrespective of the number of steps, threshold, stepsize and different weights of restraints. More importantly, i did compare the optimization results of v 7018 and 9277 of conjugate gradient optimization and steepest descent from identical starting positions with identical restraints and optimizer settings. the initial scores are identical, but then the problem starts: For example, with v7018 my model improves from a score of 1835 to 1101 after 100 steps of steepest descent optimization. However, with v9277 my score increases (!) to 1836. behavior of conjugate gradients similarly deteriorates for v9277, albeit i did not observe increases of scores. nevertheless, improvements in score were disappointingly low. Does someone know, why the optimizers behave so differently, for my purposes worse? Thanks a lot! Regards, Pia

3 3

imp-dev meeting Wednesday
by Daniel Russel 24 May '11

24 May '11

On running IMP in parallel, particularly Domino. It will mostly be unstructured discussion about planning for the future, rather than recipes for how to do things. 11am. Room to be decided. --Daniel

1 0

Re: [IMP-dev] change of optimizer performance from v7018 to v9277
by Daniel Russel 18 May '11

18 May '11

On May 18, 2011, at 9:05 AM, Friedrich Foerster wrote: > I remember painfully writing a test case for the em restraint and the derivatives when I implemented the em score. Is that test not functional anymore? They are still there, but I'm not sure if they pass. It also isn't clear to me what exactly they test as they use some modeller functionality that I am not familiar with to do the actual checking. (Reason I'm not sure about their status: I broke the build last night on all platforms except windows, and for some reason the windows test run didn't terminate. And the em tests have been failing in general for rather a while, and it is not something I've looked in to. And I don't have modeller installed on any of my machines, so that is not a test I run myself). Keren, do you have more sense of the status of those tests and, if they are broken, when they broke? Anyway, there is a test I added a bit back to test that the magnitudes of the derivatives make sense, since they didn't (eg that they roughly match the number one gets for a numeric approximation of the derivative). So there are definitely problems that need to be fixed :-) > > Cheers > > Friedrich > > Dr. Friedrich Foerster > Max-Planck Institute of Biochemistry > Am Klopferspitz 18 > D-81252 Martinsried > > www.tomotronic.org > > > On 18.05.2011, at 16:55, Keren Lasker <kerenl(a)salilab.org> wrote: > >> Major changes to the code from my part were in verion 7212 in class CoarseCC, which mainly dealt with rigid bodies support. >> I am not sure if that is the reason for Pia's problem, but as she needs to use EM based optimization relatively soon, i will take a look again. >> In any case - Pia - it would be useful if you can write a test case that I can check things against. >> keren. >> On May 18, 2011, at 7:43 AM, Daniel Russel wrote: >> >>> I suspect it is an issue with em::FitRestraint as the derivatives returned by that restraint are currently broken (the direction is fine, but the magnitude off by quite a bit). As a result derivative-using optimizers don't perform well with it. >>> >>> Keren, Ben, is there a plan for fixing it? >>> --Daniel >>> >>> On May 18, 2011, at 6:31 AM, Pia Unverdorben wrote: >>> >>>> Dear all, >>>> >>>> is it possible, that something fundamental changed in the optimizers, steepest descent as well as conjugate gradients? >>>> I try to fit a protein to an em density, simulated from another conformational state using the em restraint and plenty of harmonic restraints. With an older version (revision 7018) it moves towards the density, whereas with the revision 9277 the score and the positions barely change, irrespective of the number of steps, threshold, stepsize and different weights of restraints. >>>> More importantly, i did compare the optimization results of v 7018 and 9277 of conjugate gradient optimization and steepest descent from identical starting positions with identical restraints and optimizer settings. the initial scores are identical, but then the problem starts: >>>> For example, with v7018 my model improves from a score of 1835 to 1101 after 100 steps of steepest descent optimization. However, with v9277 my score increases (!) to 1836. behavior of conjugate gradients similarly deteriorates for v9277, albeit i did not observe increases of scores. nevertheless, improvements in score were disappointingly low. >>>> Does someone know, why the optimizers behave so differently, for my purposes worse? >>>> >>>> Thanks a lot! >>>> >>>> Regards, >>>> >>>> Pia >>>> _______________________________________________ >>>> IMP-dev mailing list >>>> IMP-dev(a)salilab.org >>>> https://salilab.org/mailman/listinfo/imp-dev >>> >> >> >> _______________________________________________ >> IMP-dev mailing list >> IMP-dev(a)salilab.org >> https://salilab.org/mailman/listinfo/imp-dev >>

3 5

recently in IMP
by Daniel Russel 14 May '11

14 May '11

- An interactive mode has been added to IMP::domino::DominoSampler. The interactive mode is useful for understanding domino sampling runs and distributing domino comptutations across a cluster. - Functions IMP::domino::set_assignments() and IMP::domino::get_assignments() were added which support writing subset assignments to data fields in hdf5 files. These can be used to helping to distribute domino on a cluster as well as debugging the sampling process. - a python function IMP.show_graphviz() has been added that attempts to render a graph with \c dot and then open the pdf. - The python functions IMP.show_altgraph() and IMP.get_networkx_graph() have been added to provide alternatives for quick viewing of \imp graphs. Unfortunately, they still pale compared to graphviz. - IMP.atom.create_cover() was added to facilitate creating a summary particle for part of a hierarchy in order to accelerate restraints. - IMP.hdf5 was renamed to IMP.rmf, the proposed name of the file format that it supports. That also better separates the hdf5 support from the file format support. - The old I/O functions have been marked as deprecated in favor of IMP.hdf5. You may need to compile with deprecated="yes" if you use them. - The python graph interface now has a remove_vertex method.

1 0

atom::setup_as_rigid_body
by Daniel Russel 11 May '11

11 May '11

There is some bug but with setup_as_rigid_body which makes is not work correctly with Selections (the selection will only choose the root particle under certain circumstances when it should choose the leaves as some of the internal particles are given a radius of 0). Since the method is not documented or supported and has other areas where it doesn't work right, I'm not terribly motivated to fix it and would rather simply remove it. Thoughts?

2 2