June 2011 - IMP-dev - salilab.org

Accelerating restraint evaluation
by Daniel Russel 25 Jun '11

25 Jun '11

We have been discussing ideas for accelerating the evaluation of certain restraints, particularly excluded volume and connectivity-based ones, by taking advantage of the fact that you often don't care how bad things are once they are sufficiently bad. To be more concrete, the idea is to add a function Model::evaluate_if_ok(Restraints, Weights) which - returns the score if the score is less than Model::get_maximum_score and each restraint score is less than its maximum - otherwise returns a very large score (something greater than Model::get_maximum_score() The idea is that restraint evaluation in general, and individual restraints in particular can cease their computations if the can prove the score is too high. The usage scenarios envisioned are: - domino: since we reject any state which scores badly, we can use this method directly - monte carlo: any state that score sufficiently badly is vanishingly unlikely to be accepted, and so we could, in principle, save some work in evaluation by dynamically varying the maximum score to be, say, 5kT times the current score. I'm not sure how useful this is in practice. Restraints that could benefit: - excluded volume: there is no need to evaluation all of the pairs once we have found several collisions. How to best combine this optimization with the optimization currently used of only updating the non-bonded list periodically is non-obvious, especially given the often incoherent evaluations performed by domino. - connectivity restraints: any large edge that is required means we don't have to check anything else - pairsrestraint: we can skip evaluating later pairs of the earlier ones score too badly Further directions: - in domino we often only care if each restraint score is below its maximum (assuming there is no model maximum score). In this case, many restraints can do a cheaper computation to simply determine if things are OK. The context of using this is much narrower and the benifit smaller, so I'm not sure we want to implement this now. The impact on existing code: - no changes need to be made to any restraints/pair scores - restraint/pair scores can be updated to take advantage of this functionality by overloading the Restraint::do_evaluate_if_ok() method (a default implementation is provided) My questions are: - are there other areas where these ideas might be applied which should be kept in mind when implementing this functionality to make it more generally useful? - are there other restraints which could easily benefit?

2 2

RMF warning
by Daniel Russel 23 Jun '11

23 Jun '11

I had to make a chance to the rmf file format that is not backwards compatible. If there is someone who has lots of data saved for whom going through another format is a problem, let me know and we can work on a solution. Otherwise, be aware that updating IMP today will render your old files unreadable. In detail, the problem is that I was using an hdf5 attribute to store the list of keys. Apparently there is a maximum size on how much data can be stored in attributes, and I was running in to that limit. As a result, the list of keys has been moved to a data set, which is something that can grow arbitrarily large.

1 0

removing incremental evaluation
by Daniel Russel 20 Jun '11

20 Jun '11

I'd like to remove the support for incremental evaluation from IMP. My reasons for proposing it are: - it complicates various code in the model and Particle - there seems to be a reasonable runtime cost even when it is turned off - we have an alternative mechanism based on decomposing restraints and analyzing the dependency graph which is more efficient and simpler. This method is not super automatic, but can be made so if needed. - I don't think anyone is using it.

3 2

IMP::display::CMMWriter
by Davide Baù 14 Jun '11

14 Jun '11

Hi, I have some problem in understanding how the new writers work. I'm using the CMMwriter like, writing out only the final model (frame): > # Setup a CMM writer > writer = IMP.display.CMMWriter("model.%1%.cmm") > log = IMP.display.WriteOptimizerState(writer) > g = IMP.display.XYZRsGeometry(ps) > log.add_geometry(g) > log.update() This generates the model file with the coordinates (each line is a chimera marker) and a file called "model.%1%.cmm", which only contains the header line: <marker_set name="model.0.cmm"> This header should be in the model file, but since it is missing it causes a problem with chimera. Is there a way to get rid of the model.%1%.cmm file so that to have just one file containing all the informations? I've tried without the %1% string: > writer = IMP.display.CMMWriter("model.1.cmm") but that rises the error: > e = o.do_optimize(steps) > File "/usr/src/imp/trunk/build/lib/IMP/core/__init__.py", line 4338, in do_optimize > return _IMP_core.MonteCarlo_do_optimize(self, *args) > _IMP.InternalException: Cannot set frame on writer without file name pattern. Thanks, Davide -- Davide Baù Structural Genomics Laboratory Bioinformatics & Genomics Department, Prince Felipe Research Center Avda. Autopista del Saler 16, 46012 Valencia, Spain Tel: +34 96 328 96 80 (ext. 1004) Fax: +34 96 328 97 01 email: dbau(a)cipf.es web: http://bioinfo.cipf.es/dbau/ *** http://www.saveaswwf.com ***

3 3

Cleanup of monte carlo
by Daniel Russel 09 Jun '11

09 Jun '11

The core::MonteCarlo code is rather a mess, has bugs and has subject to a lot of churn. This is primarily because it is trying to do too many different things, resulting is very complicated code. I'd like to break it up into several classes each with a single mode of operation. In particular, there would be - core.MonteCarlo, which does simple Monte Carlo (apply some movers and accept or reject) as well as provide some finer grained functionality for implementing less trivial monte carlo protocols. - core.MonteCarloWithLocalOptimization which applies some local optimization scheme after the moves and accepts or rejects the relaxed conformation - core.MonteCarloWithBasinHopping which supports basin hopping (apply a monte carlo move and then accept or reject based on the score after The main noticeably change would be that code that looks like mc= IMP.core.MonteCarlo() mc.set_model(m) cg= IMP.core.ConjugateGradients() mc.set_local_optimizer(cg) mc.set_local_steps(10) would need to be replaced by cg= IMP.core.ConjugateGradients(m) IMP.core.MonteCarloWithLocalOptimization(cg, 10) Comments?

4 8

values and objects
by Daniel Russel 08 Jun '11

08 Jun '11

I found a bug in the swig wrappers which I would like to fix. Fixing it though exposes issues with various things being passed around improperly by functions that are part of the api in various modules. In particular the following have issues (proposed fixes show) - em2d::do_segmentation(): return the opencv matrix instead of pass by ref. As it is internally reference counted there is no reason not to. - em2d::Image::get_min_and_max_values: return a FloatRange - em2d::get_peak() return a pair of values - em2d::ProjectionParameters::get_keys()- return FloatKeys - em::MapReaderWriter all methods: I don't see any need to export them to python anyway - em::ImageHeader::get_{date,time,title}: return std::string - em::KerenParameters and em::RadiusDependentDistanceMask are pretty much a mess in terms of how they are passed and stored, there are plans to revamp it all, so just hiding the methods dealing with them seems ok for now. - saxs::Score::fit_profile: the profile argument can probably be passed by const-ref or value, but it is not entirely clear to me - multifit::FFTFitting::get_wrapped_index and get_wrapped_correlation_map: can return an Ints - multifit::fitting_clustering: should probably just return the solutions + a few others which had obvious, non-breaking fixes which can make. I'd like to commit the wrapper changes. To do this, I'd just hide the above functions from python, they can be fixed and exposed as needed. Thoughts.

6 13

imp on the cluster question
by Dave Barkan 08 Jun '11

08 Jun '11

I am using the nightly build of IMP that gets installed on diva1 to run IMP on the cluster nodes. From what I can tell, this is built in debug or release mode. Is there a way to run it from diva in fast mode instead? thanks, Dave

2 1

[Bug 116] New: Large memory usage on first evaluate with many restraints
by bugzilla-daemon＠salilab.org 08 Jun '11

08 Jun '11

https://salilab.org/imp/bugs/show_bug.cgi?id=116 Summary: Large memory usage on first evaluate with many restraints Product: IMP Version: unspecified Platform: PC OS/Version: Mac OS Status: NEW Severity: enhancement Priority: P5 Component: kernel AssignedTo: drussel(a)SALILAB.ORG ReportedBy: drussel(a)SALILAB.ORG CC: imp-dev(a)salilab.org Estimated Hours: 0.0 'm trying to optimize a set of particles as done for the alpha-globin domain (I'm doing some testing on a larger chromatin domain) using a large set of restraints (over 10^6 restraints). The memory usage is OK until all the restraints are generated (~2GB), and then increase to 100% of memory and swap (16GB + 16GB) when I first call m.evaluate(False) (to check the initial score) causing the script to stop. I understand that a large number of restraints are being evaluated, but I was wondering if this is an expected behavior (i.e. if there is a sort of limitation in the number of restraints that can be implemented per memory GB) or if there is a workaround, before trying to run the script on a (shared) machine with much more memory. -- Configure bugmail: https://salilab.org/imp/bugs/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

1 1

file not found error
by Keren Lasker 04 Jun '11

04 Jun '11

hello - Say a program needs to read a parameter file. having IMP_USAGE_CHECK to check that the file is open is not optimal because it will not show up in fast mode. assuming that i will then just have a simple IF statement, what is the best way to exit the program ? a exit(1); seems bad especially if the program should be run though a webserver and we want to notify users on the exact problem. did someone have similar problems? thanks, keren.

2 1