Thanks Daniel, the local module compilation went fine now.

The blow up was at the time of the first evaluate call (it does not get through the first evaluate call).

Davide


On Jun 8, 2011, at 5:49 PM, Daniel Russel wrote:

One thing I forgot to ask yesterday was whether the blow up was at the time of the first evaluate call or was a gradual increase as optimization ran.

As for the compilation error, at some point there was a minor change in how external dependencies for modules need to be specified in order to be able to support certain libraries that need more than one lib linked at once. It looks like your module has the dummy external library code that used to be in the example module, and which predates the change. Just delete references to Example in your module's SConscript (modules/mymodule/SConscript). If you have real external dependencies, make the libs specification be a python list ["libname"], instead of "libname". Sorry about that.

On Jun 8, 2011, at 7:31 AM, Davide Bał wrote:

I've never used Instruments but I'm giving it a try now. I will also try to reduce the number of restraints using lists if possible. 
On a side note, I've updated to the latest trunk version (r9632). While it 
compiles fine on both linux and mac, I'm not able to compile local modules. with the scons option "local=True", I get the following error:

Checking for ExampleLib with variables...(cached) no
Checking for ExampleLib with pkg-config...no
The lib argument must be given as a list. It was not for ExampleLib

On IMP version r7392 local modules compile with no errors.
Davide

  

On Jun 7, 2011, at 6:12 PM, Daniel Russel wrote:
I don't think we have ever tried nearly that many restraints, so there may be some inefficiency that we haven't noticed. One possibility is building the dependency graph which is used to figure out the relationship between restraints and particles and scores states. Another is the non-bonded list or lists if you have many packed particles. It can be a bit difficult to control memory usage and in fact there was a bug a bit back with their memory usage increasing uncontrollably.

For tracking down problems, if you have a mac where you can run things, there is a wonderful program there called "Instruments" which can track all memory allocations and deallocations and report where they were done. I would also suggest first updating to a more recent version of IMP to see if it is the non-bonded bug.

On Jun 7, 2011, at 7:52 AM, Davide Bał wrote:

Hi everyone,

I'm having a problem with the memory usage of the IMP::Model::evaluate(bool) method.

I'm trying to optimize a set of particles as done for the alpha-globin domain (I'm doing some testing on a larger chromatin domain) using a large set of restraints (over 10^6 restraints).
The memory usage is OK until all the restraints are generated (~2GB), and then increase to 100% of memory and swap (16GB + 16GB) when I first call m.evaluate(False) (to check the initial score) causing the script to stop. I understand that a large number of restraints are being evaluated, but I was wondering if this is an expected behavior (i.e. if there is a sort of limitation in the number of restraints that can be implemented per memory GB) or if there is a workaround, before trying to run the script on a (shared) machine with much more memory. 

I'm using an old version of IMP (r7392) with the following optimizer:
# Set up optimizer
o = IMP.core.MonteCarlo()
o.set_return_best(True)
o.set_model(m)
fk = IMP.core.XYZ.get_xyz_keys()
mov = IMP.core.NormalMover(ps, fk, 0.25)
o.add_mover(mov)
lo = IMP.core.ConjugateGradients()
o.set_local_steps(lsteps)
o.set_local_optimizer(lo)

I've tried without the ConjugateGradients after reading this https://salilab.org/imp/bugs/show_bug.cgi?id=106, but I guess it does not depend on CG (I don't have ClosePairContainer in the current code).

Thanks and best regards,
Davide


--
Davide Bał
Structural Genomics Laboratory
Bioinformatics & Genomics Department, Prince Felipe Research Center
Avda. Autopista del Saler 16, 46012 Valencia, Spain
Tel: +34 96 328 96 80 (ext. 1004)  Fax: +34 96 328 97 01
email: dbau@cipf.es  web: http://bioinfo.cipf.es/dbau/


_______________________________________________
IMP-dev mailing list
IMP-dev@salilab.org
https://salilab.org/mailman/listinfo/imp-dev

_______________________________________________
IMP-dev mailing list
IMP-dev@salilab.org
https://salilab.org/mailman/listinfo/imp-dev

_______________________________________________
IMP-dev mailing list
IMP-dev@salilab.org
https://salilab.org/mailman/listinfo/imp-dev

_______________________________________________
IMP-dev mailing list
IMP-dev@salilab.org
https://salilab.org/mailman/listinfo/imp-dev