I will try to combine restraints and make use of container::PairsRestraint as you suggested. Beside harmonic (harmonic, lowerbound and upperbound) and excluded volume restraints I use the ConnectivityRestraint. While I don't know if this has a strong effect on memory usage, it makes the computation longer (as somehow expected).
Davide
On Jun 8, 2011, at 7:50 PM, Daniel Russel wrote: > I can reproduce the high memory usage. I'll look in to what can be done. For the time being, you may want to look in to whether you can combine various restraints into one as that will help with memory usage a well as make your optimization run faster. For example, if you have many restraints on pairs of particles with similar operations, try to replace them by a container::PairsRestraint. This isn't always possible though. > > > On Jun 8, 2011, at 9:05 AM, Davide Baù wrote: > >> Thanks Daniel, the local module compilation went fine now. >> >> The blow up was at the time of the first evaluate call (it does not get through the first evaluate call). >> >> Davide >> >> >> On Jun 8, 2011, at 5:49 PM, Daniel Russel wrote: >> >>> One thing I forgot to ask yesterday was whether the blow up was at the time of the first evaluate call or was a gradual increase as optimization ran. >>> >>> As for the compilation error, at some point there was a minor change in how external dependencies for modules need to be specified in order to be able to support certain libraries that need more than one lib linked at once. It looks like your module has the dummy external library code that used to be in the example module, and which predates the change. Just delete references to Example in your module's SConscript (modules/mymodule/SConscript). If you have real external dependencies, make the libs specification be a python list ["libname"], instead of "libname". Sorry about that. >>> >>> On Jun 8, 2011, at 7:31 AM, Davide Baù wrote: >>> >>>> I've never used Instruments but I'm giving it a try now. I will also try to reduce the number of restraints using lists if possible. >>>> On a side note, I've updated to the latest trunk version (r9632). While it >>>> compiles fine on both linux and mac, I'm not able to compile local modules. with the scons option "local=True", I get the following error: >>>> >>>>> Checking for ExampleLib with variables...(cached) no >>>>> Checking for ExampleLib with pkg-config...no >>>>> The lib argument must be given as a list. It was not for ExampleLib >>>> >>>> On IMP version r7392 local modules compile with no errors. >>>> Davide >>>> >>>> >>>> >>>> On Jun 7, 2011, at 6:12 PM, Daniel Russel wrote: >>>>> I don't think we have ever tried nearly that many restraints, so there may be some inefficiency that we haven't noticed. One possibility is building the dependency graph which is used to figure out the relationship between restraints and particles and scores states. Another is the non-bonded list or lists if you have many packed particles. It can be a bit difficult to control memory usage and in fact there was a bug a bit back with their memory usage increasing uncontrollably. >>>>> >>>>> For tracking down problems, if you have a mac where you can run things, there is a wonderful program there called "Instruments" which can track all memory allocations and deallocations and report where they were done. I would also suggest first updating to a more recent version of IMP to see if it is the non-bonded bug. >>>>> >>>>> On Jun 7, 2011, at 7:52 AM, Davide Baù wrote: >>>>> >>>>>> Hi everyone, >>>>>> >>>>>> I'm having a problem with the memory usage of the IMP::Model::evaluate(bool) method. >>>>>> >>>>>> I'm trying to optimize a set of particles as done for the alpha-globin domain (I'm doing some testing on a larger chromatin domain) using a large set of restraints (over 10^6 restraints). >>>>>> The memory usage is OK until all the restraints are generated (~2GB), and then increase to 100% of memory and swap (16GB + 16GB) when I first call m.evaluate(False) (to check the initial score) causing the script to stop. I understand that a large number of restraints are being evaluated, but I was wondering if this is an expected behavior (i.e. if there is a sort of limitation in the number of restraints that can be implemented per memory GB) or if there is a workaround, before trying to run the script on a (shared) machine with much more memory. >>>>>> >>>>>> I'm using an old version of IMP (r7392) with the following optimizer: >>>>>>> # Set up optimizer >>>>>>> o = IMP.core.MonteCarlo() >>>>>>> o.set_return_best(True) >>>>>>> o.set_model(m) >>>>>>> fk = IMP.core.XYZ.get_xyz_keys() >>>>>>> mov = IMP.core.NormalMover(ps, fk, 0.25) >>>>>>> o.add_mover(mov) >>>>>>> lo = IMP.core.ConjugateGradients() >>>>>>> o.set_local_steps(lsteps) >>>>>>> o.set_local_optimizer(lo) >>>>>> >>>>>> I've tried without the ConjugateGradients after reading this https://salilab.org/imp/bugs/show_bug.cgi?id=106, but I guess it does not depend on CG (I don't have ClosePairContainer in the current code). >>>>>> >>>>>> Thanks and best regards, >>>>>> Davide >>>>>> >>>>>> >>>>>> -- >>>>>> Davide Baù >>>>>> Structural Genomics Laboratory >>>>>> Bioinformatics & Genomics Department, Prince Felipe Research Center >>>>>> Avda. Autopista del Saler 16, 46012 Valencia, Spain >>>>>> Tel: +34 96 328 96 80 (ext. 1004) Fax: +34 96 328 97 01 >>>>>> email: dbau@cipf.es web: http://bioinfo.cipf.es/dbau/ >>>>>> >>>>>> *** http://www.saveaswwf.com *** >>>>>> >>>>>> _______________________________________________ >>>>>> IMP-dev mailing list >>>>>> IMP-dev@salilab.org >>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>> >>>>> _______________________________________________ >>>>> IMP-dev mailing list >>>>> IMP-dev@salilab.org >>>>> https://salilab.org/mailman/listinfo/imp-dev >>>> >>>> _______________________________________________ >>>> IMP-dev mailing list >>>> IMP-dev@salilab.org >>>> https://salilab.org/mailman/listinfo/imp-dev >>> >>> _______________________________________________ >>> IMP-dev mailing list >>> IMP-dev@salilab.org >>> https://salilab.org/mailman/listinfo/imp-dev >> >> _______________________________________________ >> IMP-dev mailing list >> IMP-dev@salilab.org >> https://salilab.org/mailman/listinfo/imp-dev > > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev