hi all,
somehow i keep on stealing my computer memory when running imp from python. either my awkward programming style or a bug in imp/swig steals the memory from my system. attached is an example illustrating the problem. if you monitor the memory usage, you'll see the consumed memory gradually increasing - although the same variable is used throughout the loop. thus, apparently new memory is constantly allocated and i have no clue what happened to the old one. needless to say the code could eventually kills every system given the loop is large enough. any help on the whereabouts of my memory would be greatly appreciated...
tx
frido
I haven't had a chance to look into it in great depth, but just some quick comments: - you should never call __del__ on a restraint (or score state, or Particle) directly as they are ref counted within C++. I think swig maps just calls onto a no-op, but I'm not quite sure.
- in general, you avoid inheriting from classes except for those which are explicitly designed to be inherited from (i.e. it is OK to inherit from Restraint, but not so great from Model). You should instead wrap them (i.e. have assembly take a Model as an argument). There are several reasons for this: first, wrapping is much more flexible and less likely to result in you having to go back and change things later; second, in the IMP case, SWIG memory management is generally problematic and inheriting from a C++ object in python makes things more complicated. It appears that this is your problem as removing the assembly class and putting the relevant code inline in the loop makes the problem go away.
- from my understanding of python, there is no reason to have all the __del__ calls all over the place. Python does reference counting, so simply assigning xxx to the next assembly will make assembly go away. I suspect, as in C++, you shouldn't write a destructor unless you have a really good reason and know what you are doing.
On Jan 20, 2009, at 1:56 PM, Friedrich Foerster wrote:
> hi all, > > somehow i keep on stealing my computer memory when running imp from > python. > either my awkward programming style or a bug in imp/swig steals the > memory from my system. attached is an example illustrating the > problem. if you monitor the memory usage, you'll see the consumed > memory gradually increasing - although the same variable is used > throughout the loop. thus, apparently new memory is constantly > allocated and i have no clue what happened to the old one. needless to > say the code could eventually kills every system given the loop is > large enough. > any help on the whereabouts of my memory would be greatly > appreciated... > > tx > > frido > <test_memory_leak.py>_______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev
thanks. removing the inheritance of Model and the __del__ statement for the restraints eventually resolved the issue. i would recommend putting infos like these into a "Dos and Don't Dos" file or so at an intuitive location for keeping others from running into similar trouble.
cheers
frido
On Jan 20, 2009, at 11:36 PM, Daniel Russel wrote:
> I haven't had a chance to look into it in great depth, but just some > quick comments: > - you should never call __del__ on a restraint (or score state, or > Particle) directly as they are ref counted within C++. I think swig > maps just calls onto a no-op, but I'm not quite sure. > > - in general, you avoid inheriting from classes except for those > which are explicitly designed to be inherited from (i.e. it is OK to > inherit from Restraint, but not so great from Model). You should > instead wrap them (i.e. have assembly take a Model as an argument). > There are several reasons for this: first, wrapping is much more > flexible and less likely to result in you having to go back and > change things later; second, in the IMP case, SWIG memory management > is generally problematic and inheriting from a C++ object in python > makes things more complicated. It appears that this is your problem > as removing the assembly class and putting the relevant code inline > in the loop makes the problem go away. > > - from my understanding of python, there is no reason to have all > the __del__ calls all over the place. Python does reference > counting, so simply assigning xxx to the next assembly will make > assembly go away. I suspect, as in C++, you shouldn't write a > destructor unless you have a really good reason and know what you > are doing. > > > On Jan 20, 2009, at 1:56 PM, Friedrich Foerster wrote: > >> hi all, >> >> somehow i keep on stealing my computer memory when running imp from >> python. >> either my awkward programming style or a bug in imp/swig steals the >> memory from my system. attached is an example illustrating the >> problem. if you monitor the memory usage, you'll see the consumed >> memory gradually increasing - although the same variable is used >> throughout the loop. thus, apparently new memory is constantly >> allocated and i have no clue what happened to the old one. needless >> to >> say the code could eventually kills every system given the loop is >> large enough. >> any help on the whereabouts of my memory would be greatly >> appreciated... >> >> tx >> >> frido >> <test_memory_leak.py>_______________________________________________ >> IMP-dev mailing list >> IMP-dev@salilab.org >> https://salilab.org/mailman/listinfo/imp-dev > > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev >
--
Friedrich Foerster Max-Planck Institut fuer Biochemie Am Klopferspitz 18 D-82152 Martinsried
Tel: +49 89 8578 2651 Fax: +49 89 8578 2641
foerster@biochem.mpg.de
www.tomotronic.org
I simplified the memory management section of the FAQ so it now simply defines a relatively safe set of allowed actions.
Friedrich Foerster wrote: > thanks. > removing the inheritance of Model and the __del__ statement for the > restraints eventually resolved the issue. > i would recommend putting infos like these into a "Dos and Don't Dos" > file or so at an intuitive location for keeping others from running > into similar trouble. > > cheers > > frido > > > On Jan 20, 2009, at 11:36 PM, Daniel Russel wrote: > >> I haven't had a chance to look into it in great depth, but just some >> quick comments: >> - you should never call __del__ on a restraint (or score state, or >> Particle) directly as they are ref counted within C++. I think swig >> maps just calls onto a no-op, but I'm not quite sure. >> >> - in general, you avoid inheriting from classes except for those >> which are explicitly designed to be inherited from (i.e. it is OK to >> inherit from Restraint, but not so great from Model). You should >> instead wrap them (i.e. have assembly take a Model as an argument). >> There are several reasons for this: first, wrapping is much more >> flexible and less likely to result in you having to go back and >> change things later; second, in the IMP case, SWIG memory management >> is generally problematic and inheriting from a C++ object in python >> makes things more complicated. It appears that this is your problem >> as removing the assembly class and putting the relevant code inline >> in the loop makes the problem go away. >> >> - from my understanding of python, there is no reason to have all the >> __del__ calls all over the place. Python does reference counting, so >> simply assigning xxx to the next assembly will make assembly go away. >> I suspect, as in C++, you shouldn't write a destructor unless you >> have a really good reason and know what you are doing. >> >> >> On Jan 20, 2009, at 1:56 PM, Friedrich Foerster wrote: >> >>> hi all, >>> >>> somehow i keep on stealing my computer memory when running imp from >>> python. >>> either my awkward programming style or a bug in imp/swig steals the >>> memory from my system. attached is an example illustrating the >>> problem. if you monitor the memory usage, you'll see the consumed >>> memory gradually increasing - although the same variable is used >>> throughout the loop. thus, apparently new memory is constantly >>> allocated and i have no clue what happened to the old one. needless to >>> say the code could eventually kills every system given the loop is >>> large enough. >>> any help on the whereabouts of my memory would be greatly >>> appreciated... >>> >>> tx >>> >>> frido >>> <test_memory_leak.py>_______________________________________________ >>> IMP-dev mailing list >>> IMP-dev@salilab.org >>> https://salilab.org/mailman/listinfo/imp-dev >> >> _______________________________________________ >> IMP-dev mailing list >> IMP-dev@salilab.org >> https://salilab.org/mailman/listinfo/imp-dev >> > > -- > > Friedrich Foerster > Max-Planck Institut fuer Biochemie > Am Klopferspitz 18 > D-82152 Martinsried > > Tel: +49 89 8578 2651 > Fax: +49 89 8578 2641 > > foerster@biochem.mpg.de > > www.tomotronic.org > > > > > > > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev
Friedrich Foerster wrote: > somehow i keep on stealing my computer memory when running imp from > python. either my awkward programming style or a bug in imp/swig > steals the memory from my system.
It is the former. ;) While Daniel is correct that you probably shouldn't be calling those __del__ methods all over the place (to delete an object 'x', say 'del x', not 'x.__del__()') it isn't going to break anything. The only place you need to call __del__ is in your assembly.__del__ method (where you correctly say b.__del__(self) to delete anything held by the base classes). But because of Python's reference counting, you don't need to delete things held by the classes (e.g. both AssemblyRestraints.__del__ and assembly.__del__ are probably unnecessary). You also don't need to explicitly call xxx.__del__() since that should happen automatically when xxx goes out of scope.
It is also true that you have to be a little bit careful when deriving from IMP classes in Python, because C++ references do not map to equivalent Python references. But this isn't an issue with IMP.Model, since the lifetime of the C++ object is the same as the Python one.
Anyway, your problem is that assembly is never deleted because you have a circular reference. It is a pure Python problem - nothing to do with IMP. assembly.restraints is a reference to an AssemblyRestraints object. But AssemblyRestraints.assembs is a reference back to the assembly object. Python can't delete 'assembly' because there's a reference to it from AssemblyRestraints, but it can't delete AssemblyRestraints because assembly has a reference to it... and so on. To delete the object, you need to rethink your design so you don't have a circular reference or (and this is a hack) you need to explicitly break the cycle by saying something like 'del xxx.restraints'.
Ben
participants (3)
-
Ben Webb -
Daniel Russel -
Friedrich Foerster