- IMP-dev - salilab.org

Recently in IMP
by Daniel Russel 15 Oct '10

15 Oct '10

- IMP::core::ClosePairsPairScore has been split into IMP::core::ClosePairsPairScore and IMP::core::KClosePairsPairScore as the previous version encompassed two different, easily confused, behaviors. It now is accelerated and takes advantage of rigid bodies. The constructor signature was changed to force you to revisit code that might be affected by the behavior changes. IMP::core::RigidBodyDistancePairScore is now just a wrapper on IMP::core::KClosePairsPairScore. - IMP::atom::Selection was added to facilitate representation independent setup of restraints and manipulation of hierarchies. Check it out. - a new, more involved example, kernel/examples/nup84_cg was added, blatantly ripping off Elina's restrainer nup84 example

1 0

biological names for things
by Daniel Russel 14 Oct '10

14 Oct '10

Since I keep bringing up that it is good to be able to refer to parts of the representation using biological names, I thought I would throw together some code to do that. The goal is to allow one to set up the restraints without ever explicitly referring to particles, just asking for parts of the hierarchy. The advantage of this is that - one can change the representation pretty much arbitrarily without messing with the restraints code - the restraints code is much easier to read - you don't have to worry about whether the representation happens to contain a Fragment referring to the piece of the molecule you are interested in (and hence avoid the problem of overlapping, but not nested, fragments). For the implementation, there is a class IMP.atom.Named which describes a part of one or more molecules. You create a name with doing various things like nm=IMP.atom.Named(hierarchy=h, molecule="my molecule", residue=10) (assuming there is more than one molecule in h) and then can get the one or more particles referenced by it with nmps=nm.get_particles() eg, if h is an all atom mode, nm.get_particles() will return the atoms in the residue, if it is a coarse grained model, it will return the particle containing the residue. The names can be used to create a distance restraint with IMP.atom.create_harmonic_distance_restraint(IMP.atom.Named(hierarchy=all, molecule="proteina", residue=10), IMP.atom.Named(hierarchy=all, molecule="proteinb", residues=(1,10), 10, 1) which will return a restrain limiting the distance between the 10th residue of proteina and the first 10 residues of proteinb to 10A with a spring constant of 1. The restraint is a SphereDistanceRestraint if each of the two Named refer to just one particle and a ConnectivityRestraint using a ClosePairsPairScore if they refer to more than one (the ClosePairsPairScore internally takes advantage of rigid bodies if the hierarchies are rigid bodies). I haven't wrapped any other restraint types yet. With EM, for example, one might as well just pass nm0.get_particles()+nm1.get_particles() to restrain two molecules with a map. It is just something I threw together on the train, so I wouldn't trust the code much yet (and the name "Named" isn't great, but I want it to be short). The Named objects can be created with any combination of - one or more names of molecules - a single residue index or list of residue indexes (or ranges)-- later we can add tags for the C- and N- terminus - one or more atom types - one or more residue types - one or more chain identifiers - one or more domain names - a target radius (so that you can refer to intermediate levels of the hierarchy) eg Named(hierarchy=h, chains=['a', 'b'], target_radius=10) which gets as close to 10A representation of the two chains as one can. Name(hierarchy=h, atom_types=[CA]) which gets all C alphas This way you can write functions to set up restraints without any direct reference to particles. Thoughts? Does it seem useful? I quite like using such code when setting things up.

1 1

Recently in IMP
by Daniel Russel 11 Oct '10

11 Oct '10

Some recent changes that might be of interest - a wrapper for the Bullet physics simulation system was added to allow it to be used to optimize models. I recommend everyone take a look at it as a possible a replacement for conjugate gradients in certain contexts. It is not general purpose (it works best when excluded volume and harmonics dominate), but it is very stable and can get to a decent configuration from quite bad ones. We can add support for rigid bodies if desired, but currently only XYZR particles are optimized. An example is provided. - various bits of IMP should be faster (at least with fast builds), including non-bonded lists. I added a section on speeding things up in the introduction, http://salilab.org/imp/nightly/doc/html/introduction.html. - modules can be selectively ignored when running scons by using "disabledmodules=helper:restrainer:domino" (or some other list of modules) on the command line or the config.py. This can make scons run much faster when you are working on modifying one module since it won't configure the listed modules. Be aware that modules that require the disabled modules will also be disabled (and so ignored). Also note that you can skip the scons dependency checks by doing, eg fftw3="yes" and fftw3libs="fftw3". or create a fully specified config file configuration file (by doing "scons config.py"). As a note, if you are not doing out of source builds, you probably should be as it makes switching between fast and debug builds painless. See instructions at <http://salilab.org/imp/nightly/doc/html/installation.html>. Likewise, everyone should probably have their own module (see "managing your own module" on <http://salilab.org/imp/nightly/doc/html/devguide.html>), checked in to your own svn repository so you have an easy way to fiddle with things.

2 3

Proposed modification of how code accepts rigid bodies
by Daniel Russel 08 Oct '10

08 Oct '10

Currently, code that handles rigid bodies does it by taking a list consisting of both RigidBody particles and non-rigid body XYZR particles along with a Refiner to turn the rigid bodies into XYZR particles. This has several disadvantages: - it seems to confuse the hell out of everyone - it forces all rigid bodies to use the same refiner - it means that you have to pass a completely different set of particles if some of them happen to be rigid than if none are rigid I propose a new standard type of interface, namely, simply taking all the XYZR particles of interest, whether or not they happen to be part of a rigid body. That is, if you have some rigid proteins and some non-rigid proteins and are representing everything with all atoms, you simply pass all the atoms. I think this has the following advantages: - the user no longer has to worry about whether things are rigid bodies or not, certain code just runs faster when they are - the user doesn't have to change the set of particles passed when moving to rigid bodies It has the following disadvantages - code that takes advantage of rigid bodies is more complicated as it has to group the particles into rigid bodies: this is only a few lines of code and can be factored out, so it doesn't seem too bad - code that takes advantage of rigid body has to do more work. I don't think the extra work will be that large in practice, but am not yet sure Typical things affected would be - ExcludedVolumeRestraint - RigidClosePairFinder - perhaps FitRestraint Thoughts?

2 2

IMP::algebra::get_uniform_cover_rotations_3d
by Keren Lasker 06 Oct '10

06 Oct '10

hi Daniel - Frido and I are using get_uniform_cover_rotations_3d for exhaustive rotational sampling. The code for this function seems to sample random vectors on a 4D sphere. How sensitive is it to poles over sampling? thanks, Keren.

2 3

optimizing restraints for speed
by Daniel Russel 30 Sep '10

30 Sep '10

prompted by work I did breaking up restraints in domino2 in order to improve filtering and restraint evaluation and work special casing restraints in order to handle spring natively in Bullet and openmm, I thought it could be cool to add functionality to IMP to automatically modify the set of restraints used in order to make them faster. The idea would be to have a static list in the kernel of known optimizations. Restraints, or anything else, could add optimizations to the list, and then optimizers would apply the optimizations before performing its optimization steps (and clean up the changes upon exit from the optimizer). optimizations could include things like: - automatically replacing DistancePairScores which use Harmonics with HarmonicDistancePairScore - replacing BondSingletonScores with HarmonicDistancePairScores when appropopriate (the makeup of the bonds can't change) etc - use the C++-only generic restraints in some common cases These can provide a factor of two reduction in evaluation time in certain cases. Disadvantages: - calling optimize with only one iteration would be kind of slow, but why would you do that - work needs to be done by me :-) This would be different from the restraint/container optimization in Domino2--the name of that should probably change to decomposition or something less generic. So, how much do people want more speed when evaluating lots of simple restraints?

2 1

ClosePairsPairScore
by Daniel Russel 27 Sep '10

27 Sep '10

Does anyone use it? It seems awfully complicated as it is. I would propose changing the implementation to - get rid of the recusive application of the Refiner - not require intermediate balls for efficiency - use the algebra nearest neighbor search (build in ANN or CGAL) to make things actually faster

1 0

rigid fitting into EM density maps in IMP
by Keren Lasker 12 Sep '10

12 Sep '10

Dear IMP dev, In the sake of transparency and collaboration I wanted to give everyone an update on the work we have been doing on em fitting. We will try to send these updates to everyone from time to time so the work being done is fully transparent and clear. The recent efforts can be divided to two main categories: 1. scoring We worked with Daniel and Ben on various code optimization that now allow for even faster rigid body scoring. Specifically, the model simulated map is resampled only once and for the rest of the transformations the resampled map is transformed. We are also now able to apply the same idea to multiple rigid bodies, as we found a way to decompose the CC score into rigid terms, while still maintaining the correct combined score. Examples can be found in the test can examples directories on the em module. 2. optimization We have developed 4 rigid fitting procedures: (i) MC/CG local refinement, (ii) principal component based rigid fitting, (iii) fft based rigid fitting and (iv) anchor points based rigid fitting. (i) and (ii) are fully integrated into IMP, while (iii) and (iv) require external libraries (fftw3 and gamb), and are now in the process of being merged into IMP with Ben's help. Examples can be found in the test can examples directories on the em and multifit modules. Current usages of the new code were quite successful. Charles used it to fit 1000 proteins into more than 5000 maps and all went well. In addition, we tested the new code on a large benchmark of protein assemblies and were able to get accurate fits. As we are in the process of making all publicly available in IMP 1.1 and currently writing a tutorial on the em module in IMP, we will be happy to get specific feedback, comments or ideas for improvements. thanks, Keren.

2 1

em2D to em2d rename and possible issues on mac os
by Daniel Russel 31 Aug '10

31 Aug '10

The em2D module has been renamed em2d as certain scripts assume module names are lower case. If you get an error on a mac (which has a case-insensitive file system and so tends to get confused about this sort of thing), delete modules/em2D, do svn update and build again.

3 3

Recent and coming changes in IMP
by Daniel Russel 27 Aug '10

27 Aug '10

I haven't sent such an email in a while. Here is a summary of some of the interesting recent changes: - Javi added his 2D EM scoring functionality as IMP::em2D - some new ways to speed up scons have been added. See IMP::core::HarmonicDistancePairScore, IMP::core::SoftSpherePairScore and, for C++ users, the generic restraint support IMP::container::create_restraint() and IMP::core::create_restraint(). These each offer between 2x and 1.2x speedup when using simple scores (like harmonics on distances). In addition, a function IMP::container::get_slack_estimate() has been added which will estimate the correct slack to use with a IMP::container::ClosePairContainer. This will be extended to support IMP::core::ExcludedVolumeRestraint eventually. - support for dense and sparse grids have been added to IMP::algebra. See the IMP::algebra module page (I'm still working on how to make the docs clearer). - all write functions now have associated IMP::OptimizerState and IMP::FailureHandler classes. For examples, see IMP::WriteOptimizerState, IMP::WriteBinaryOptimizerState, IMP::WriteFailureHandler, IMP::atom::WritePDBOptimizerState etc. - functions IMP::write_binary_model(), IMP::read_binary_model() and IMP::read_configurations() have been added to support space and filesystem efficient storage of model configurations. Multiple configuration can be written to a single file. This functionality requires NetCDF be installed (NetCDF is a standard library for reading and writing binary data). - domino2 has been added. It provides a simpler interface on Keren's domino optimize as well as a number of new optimization and hooks for customizing the optimization process - a better explaination of how to use reference counted pointers has been added to IMP::Pointer. - all simple, planar polygons are now supported for display when CGAL is present Some of the future changes: - The IMP::algebra::Grid3D will be used to store EM grids and many of the computational kernels for EM will be moved to algebra so that density maps can be used in core and statistics. - the excluded volume restraint is likely to get faster (and produce better derivatives) as Keren is working on an implementation using grids rather than zillions of atoms

1 0