Dear imp users & developers, I am playing IMP with a large numbers of particles, e.g. > 10^5 and fail to optimize due to memoryError even though I still have enough VMEM and MEM. An example: -------------- Setting up score state for container ListSingletonContainer 0 Model(Model, kernel 1.0)127482 particles 0 restraints 1 score states
Setting up score state for container ListSingletonContainer 3 Setting up score state for container ClosePairContainer Setting up score state for container MovedSingletonContainer Restraint SingletonsRestraint 0 has weight 1 Restraint PairsRestraint 1 has weight 1 Traceback (most recent call last): File "test.py", line 180, in <module> o.optimize(1000) File "/home/htjong/bin/imp-1.0/build/lib/IMP/core/__init__.py", line 665, in optimize def optimize(self, *args): return _IMP_core.ConjugateGradients_optimize(self, *args) MemoryError: std::bad_alloc --------------
I suppose I have to increase the memory usage limit during compilation. How is the proper way to do it? Thanks. Harianto.
On 4/6/10 11:43 AM, Harianto Tjong wrote: > I am playing IMP with a large numbers of particles, e.g. > 10^5 and fail > to optimize due to memoryError even though I still have enough VMEM and MEM.
It is possible that this isn't memory being exhausted, but rather a bug in the system somewhere (maybe because you have 0 restraints - what happens if you add a single distance restraint, for example?). 10^5 particles isn't that huge, and shouldn't exhaust system memory on a typical modern system. You should check your shell limits though ('limit' or 'ulimit') to make sure the process isn't being restricted to a smaller amount of memory. Can you open a bug in our bug tracker at https://salilab.org/imp/bugs/ and attach your input files so we can reproduce the problem here? It should be fixed, of course.
> I suppose I have to increase the memory usage limit during compilation. > How is the proper way to do it?
No, IMP is limited only by system memory. There shouldn't be any hard-coded compile-time limits.
Ben
Hi Ben, Thanks for the respond. What you see 0 restrant is because I did 'm.show()' before adding restraint. With the attached code running on my desktop (8 GB memory with ulimit: unlimited): ------ Setting up score state for container ListSingletonContainer 0 total bead: 100619 Setting up score state for container ListSingletonContainer 3 Setting up score state for container ClosePairContainer Setting up score state for container MovedSingletonContainer Model(Model, kernel 1.0)201230 particles 2 restraints 4 score states
Restraint SingletonsRestraint 0 has weight 1 Restraint PairsRestraint 1 has weight 1 Traceback (most recent call last): File "memtest.py", line 102, in <module> o.optimize(1000) File "/home/htjong/bin/imp-1.0/build/lib/IMP/core/__init__.py", line 665, in optimize def optimize(self, *args): return _IMP_core.ConjugateGradients_optimize(self, *args) MemoryError: std::bad_alloc ------------ When I did 'top' on shell, the mem never went more than 15% before it crashed. Is there any IMP liner that can print out memory consumed during run? I'm not sure what whether it's a bug, but when I reduce the number of particle it's fine. One bug I found: set_name("blabla") of IMP.display.XYZRsGeometry doesn't print 'blabla'.
Harianto.
On Thu, Apr 8, 2010 at 12:32 PM, Ben Webb ben@salilab.org wrote:
> On 4/6/10 11:43 AM, Harianto Tjong wrote: > > I am playing IMP with a large numbers of particles, e.g. > 10^5 and fail > > to optimize due to memoryError even though I still have enough VMEM and > MEM. > > It is possible that this isn't memory being exhausted, but rather a bug > in the system somewhere (maybe because you have 0 restraints - what > happens if you add a single distance restraint, for example?). 10^5 > particles isn't that huge, and shouldn't exhaust system memory on a > typical modern system. You should check your shell limits though > ('limit' or 'ulimit') to make sure the process isn't being restricted to > a smaller amount of memory. Can you open a bug in our bug tracker at > https://salilab.org/imp/bugs/ and attach your input files so we can > reproduce the problem here? It should be fixed, of course. > > > I suppose I have to increase the memory usage limit during compilation. > > How is the proper way to do it? > > No, IMP is limited only by system memory. There shouldn't be any > hard-coded compile-time limits. > > Ben > -- > ben@salilab.org http://salilab.org/~ben/http://salilab.org/%7Eben/ > "It is a capital mistake to theorize before one has data." > - Sir Arthur Conan Doyle > _______________________________________________ > IMP-users mailing list > IMP-users@salilab.org > https://salilab.org/mailman/listinfo/imp-users >
On 04/08/2010 03:55 PM, Harianto Tjong wrote: > Thanks for the respond. > What you see 0 restrant is because I did 'm.show()' before adding > restraint. With the attached code running on my desktop (8 GB memory > with ulimit: unlimited):
Thanks - I was able to reproduce this problem on my machine. Yes, it looks like the system is running out of memory, but it should be able to handle a system of your size without any trouble. I opened a bug report for it: https://salilab.org/imp/bugs/show_bug.cgi?id=106 We'll update that bug report when we address the bug, so you can certainly add yourself to the bug's CC list if you want to be notified of our progress.
Ben
participants (2)
-
Ben Webb -
Harianto Tjong