I don't think we have ever tried nearly that many restraints, so there may be some inefficiency that we haven't noticed. One possibility is building the dependency graph which is used to figure out the relationship between restraints and particles and scores states. Another is the non-bonded list or lists if you have many packed particles. It can be a bit difficult to control memory usage and in fact there was a bug a bit back with their memory usage increasing uncontrollably.
For tracking down problems, if you have a mac where you can run things, there is a wonderful program there called "Instruments" which can track all memory allocations and deallocations and report where they were done. I would also suggest first updating to a more recent version of IMP to see if it is the non-bonded bug.
On Jun 7, 2011, at 7:52 AM, Davide Baù wrote:
> Hi everyone, > > I'm having a problem with the memory usage of the IMP::Model::evaluate(bool) method. > > I'm trying to optimize a set of particles as done for the alpha-globin domain (I'm doing some testing on a larger chromatin domain) using a large set of restraints (over 10^6 restraints). > The memory usage is OK until all the restraints are generated (~2GB), and then increase to 100% of memory and swap (16GB + 16GB) when I first call m.evaluate(False) (to check the initial score) causing the script to stop. I understand that a large number of restraints are being evaluated, but I was wondering if this is an expected behavior (i.e. if there is a sort of limitation in the number of restraints that can be implemented per memory GB) or if there is a workaround, before trying to run the script on a (shared) machine with much more memory. > > I'm using an old version of IMP (r7392) with the following optimizer: >> # Set up optimizer >> o = IMP.core.MonteCarlo() >> o.set_return_best(True) >> o.set_model(m) >> fk = IMP.core.XYZ.get_xyz_keys() >> mov = IMP.core.NormalMover(ps, fk, 0.25) >> o.add_mover(mov) >> lo = IMP.core.ConjugateGradients() >> o.set_local_steps(lsteps) >> o.set_local_optimizer(lo) > > I've tried without the ConjugateGradients after reading this https://salilab.org/imp/bugs/show_bug.cgi?id=106, but I guess it does not depend on CG (I don't have ClosePairContainer in the current code). > > Thanks and best regards, > Davide > > > -- > Davide Baù > Structural Genomics Laboratory > Bioinformatics & Genomics Department, Prince Felipe Research Center > Avda. Autopista del Saler 16, 46012 Valencia, Spain > Tel: +34 96 328 96 80 (ext. 1004) Fax: +34 96 328 97 01 > email: dbau@cipf.es web: http://bioinfo.cipf.es/dbau/ > > *** http://www.saveaswwf.com *** > > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev