- IMP-dev - salilab.org

git flow tutorial + quick start
by Barak Raveh 21 Jun '12

21 Jun '12

This is the git flow tutorial (ask Ben to install git flow for you on your machine). http://yakiloo.com/getting-started-git-flow/ Quick start: 1) Open a github account https://github.com/ 2) Get Daniel to invite you to the github project , 3) create and change to a folder of your choice 4) "git flow init" and enter-enter-enter-enter-enter 5) "git clone <address of git-hub repository provided by Daniel>" You can then start new features like Daniel showed yesterday, as described in the tutorial, etc. (e.g., "git flow feature start <feature-name>")

2 1

self-organizing maps
by Yannick Spill 18 Jun '12

18 Jun '12

Hi list, In the light of my recent discovery of the statistics module in imp, I was wondering if some of you would be interested in an implementation of self-organizing maps (SOM). They are a generalization of principal components analysis to nonlinear models. http://en.wikipedia.org/wiki/Self-organizing_map There's a shortcut for us to have it in IMP. It's called GPLVM, and is a probabilistic SOM based on gaussian processes. Is that something that would interest anyone? Yannick

4 4

clustering - code consolidation
by Barak Raveh 14 Jun '12

14 Jun '12

Now the full version... Daniel and I discussed a little bit consolidation of clustering things in statistics and kmeans modules. Please tell me if it is agreed that things will work with the following interface: * The Embedding family of classes (used to embed data in vector form) will remain as is * There will be a "Clustering" class from which all clustering algorithms will derive, with a constructor that takes either Embedding class, or EmbeddingAdaptor for implicit conversions from e.g., FloatsList * The Clustering classes will also have a *void ::execute()* method and a*::get_clustering results() *method, that will return the clustering results (using the exisiting PartitioningClustering class, perhaps we can change its name to PartitioningClusteringResults). So bottom line, if you will want to cluster some data, you will do something like *FloatsList data; // or create an Embedding object* *KMeansClustering kmeans(data, params);* *kmeans.execute();* *PartitioningClustering clustering_results = kmeans.get_clustering_results() * Makes sense? For backward compatibility, I will add a DEPRECATED warning to existing clustering methods, so they will be removed within a few months completely. Barak

2 4

clustering - consolidation
by Barak Raveh 14 Jun '12

14 Jun '12

Hi, Daniel and I discussed a little bit consolidation of clustering things in statistics. I am not touching anything out of the statistics / kmeans module for now, but please tell me if it is agreed that things will work as follows: * The Embedding family of classes (used to embed data in vector form) will remain as is * There will be a "Clustering" class from which all clustering algorithms will derive, with: * constructor that takes either Embedding class, or EmbeddingAdaptor for implicit conversions from e.g., vector form -- Barak

2 1

distance based scoring
by Daniel Russel 14 Jun '12

14 Jun '12

Following further discussion after the meeting yesterday, Dina and I have an alternative proposal for how to move forward with distance-based scoring. The changes would be to provide a class template <class DistanceScore, class DistanceScoreAndDerivatives> class DistancePairScore{ ... }; The DistanceScore argument would be a functor (or a function) with the following signature double DistanceScore(Model *, ParticleIndexPair, double distance) where the return value is the score for the pair of particles separated by distance. And the DistanceScoreAndDerivatives argument would be the same except that it would return a pair of doubles, on for the score and one for the derivative. We would then (gradually), provide functor versions of the existing distance-based pair scores and implement the corresponding PairScores in terms of the functor and this template class. For python users, we could provide a python implementation of DistancePairScore that would allow experimenting with such pair scores (but which would be rather slow). The advantages of this over the proposed changes are mostly that of flexibility, efficiency and reduced amount of code to right. On the flexibility side, doing things this way makes it easier to create modified versions of existing pair scores (eg by weighting them) as that just requires writing a functor that calls the other functions and adds up the values with the expected weights. On the efficiency side, such an approach computes the distances and then uses them immediately, rather than storing them in memory and passing over the stored values repeatedly. And on the reduced code side, it just requires adding one non-trivial class (whose implementation already exists) and we had wanted to move the scoring computations to functors anyway (and that can be done on-demand).

2 2

Git and IMP and more frequent releases
by Daniel Russel 14 Jun '12

14 Jun '12

At the imp-dev meeting yesterday, the question of switching to git from svn was raised. The hope is that it would facilitate having more frequent mini-releases of IMP (monthly being a target) as it makes it much easier to maintain parallel development threads. The main advantage of git over svn is that it makes it very easy to make and maintain branches since one can merge branches repeatedly without problems. As an illustration, someone who is making a major change to a module that would take a while to implement could create a new branch in the repository for the change, lets call it distance_functions. While distance_functions is being developed, any changes made to the IMP main branch can be easy pulled into the distance_functions branch simply by doing "git rebase" (doing this is very painful with svn) allowing the two to keep in better sync. And when the change is done, it can be brought back into the main development trunk without complication (and done more than once if so desired). This change would also makes dealing with releases much simpler as a release simply means pulling changes from the main development trunk into a release branch and adding a tag. Any fixes we want to make to release can be easily done in a hotfix branch and then directly brought into both the release and the main development trunk, rather than having to be manually pulled into both places as with svn. At the meeting, a reasonably fraction of the people were already using git for other projects (and I'm using it instead of svn for accessing IMP). So learning it does not seem to be an insurmountable hurdlw I would also suggest we check out git flow. It defines a standard branch structure that supports releases, hotfixes to releases, feature development branches etc and provides scripts to aid in doing basic tasks. It is available on home brew, but there does not seem to be a standard rpm for fedora. Boost appears to be moving towards using it, among other projects. We can provide some kind of support for existing svn users (probably read-only). This may simply involve using git-svn to sync to the git head and push it to the existing svn repository.

2 3

Re: [IMP-dev] caching advice
by Daniel Russel 12 Jun '12

12 Jun '12

You can mark member variables "mutable" to mean they aren't const when the class is. This is generally used for caches and stuff like that. In general, declaring a method const is used to imply logical constness instead of physical constness. That is, it means that doing it 2x with the same arguments should give the same answer, rather than that none of the bits should change. On Wed, Jun 6, 2012 at 2:14 PM, Barak Raveh <barak.raveh(a)gmail.com> wrote: > Hi, > > Let me consult with the high IMP society about some IMP scoring and > caching design issues, don't wanna mess things up. > > I've been profiling code, and consequently wanted to cache some > intermediate computations that are made during pair score evaluations > (e.g., the distance between particles). I just need the last intermediate > from each computation, so I thought of simply adding a [cache_] variable to > my scoring class, and store the intermediate computations there. BUT, > ::evaluate() and ::evaluate_index() are const functions, so I can't do > that. Another option is to add an output parameter to ::evaluate(), but > that's also not very clean. Is it absolutely necessary that these functions > are const? Or any other suggestions so that I can cache intermediate > calculations during ::evaluate()? This can speed some things up > considerably. > > Barak

3 9

Re: [IMP-dev] IMP-dev Digest, Vol 55, Issue 2
by Barak Raveh 08 Jun '12

08 Jun '12

ok, I'll stop using digest mode :) Wed@10 (or any hour was suggested) works for me. On Jun 8, 2012, at 12:58 AM, imp-dev-request(a)salilab.org wrote: > Send IMP-dev mailing list submissions to > imp-dev(a)salilab.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://salilab.org/mailman/listinfo/imp-dev > or, via email, send a message with subject or body 'help' to > imp-dev-request(a)salilab.org > > You can reach the person managing the list at > imp-dev-owner(a)salilab.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of IMP-dev digest..." > > > Today's Topics: > > 1. Re: caching advice (Dina Schneidman) > 2. Re: caching advice (Daniel Russel) > 3. Re: caching advice (Yannick SPILL) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 7 Jun 2012 11:06:42 -0700 > From: Dina Schneidman <duhovka(a)gmail.com> > To: List for IMP development <imp-dev(a)salilab.org> > Subject: Re: [IMP-dev] caching advice > Message-ID: > <CAN4WCj=rCxgPss5GGM_kKZs40kBtxFOCvM26KWA9BLbDHyUnvQ(a)mail.gmail.com> > Content-Type: text/plain; charset=ISO-8859-1 > > maybe we should have an IMP "high society" meeting and think how to > improve both software design and performance of scoring functions. I > have been looking into atomic resolution functions recently and have > some suggestions there, and Daniel and Barak know what we need for low > resolution. So we can come up with something that will cover all > resolutions efficiently. > > On Wed, Jun 6, 2012 at 9:29 PM, Yannick Spill <yannick(a)salilab.org> wrote: >> just my 2 cents: >> I const_cast things. Don't know if it's a good thing, people were telling me >> to avoid mutable. >> I'm ok with dina that we need something more standardized. >> >> Y >> >> >> Le 07/06/12 00:19, Daniel Russel a ?crit?: >> >> It might make sense to introduce something a DistancePairScore base. It >> could have an evaluate that takes the distance between the two particles and >> the difference vector (and maybe the distance to avoid dup sqrts). One could >> then have a variant of PairsRestraint that takes a list of such pair scores >> and caches things for each pair passed in. >> >> It would also have the nice advantage of making it clear that certain scores >> are invariant to rigid transforms and so provide a more principled basis for >> ignoring, eg, terms that are within a rigid body or not recomputing things >> when rigid mc moves are performed. >> >> Doing this would involve forking: >> IMP::PairScore (and corresponding macros) >> IMP::core::PairRestraint >> IMP::container::PairsRestraint >> IMP::internal::TupleRestraint >> IMP::internal::ContainerRestraint >> >> On Wed, Jun 6, 2012 at 2:40 PM, Dina Schneidman <duhovka(a)gmail.com> wrote: >>> >>> I think we need more general caching mechanism. For example, if my >>> score includes LennardJones and Coulombic terms, currently each one >>> will compute the same distance, in addition there is external >>> calculation that decides if we should score the pair of particles >>> based on the distance between them. so instead of one distance >>> calculation there are three. It would be great just to be able to pass >>> the distance to evaluate() to avoid additional calculations. >>> >>> On Wed, Jun 6, 2012 at 2:19 PM, Daniel Russel <drussel(a)gmail.com> wrote: >>>> You can mark member variables "mutable" to mean they aren't const when >>>> the >>>> class is. This is generally used for caches and stuff like that. In >>>> general, >>>> declaring a method const is used to imply logical constness instead of >>>> physical constness. That is, it means that doing it 2x with the same >>>> arguments should give the same answer, rather than that none of the bits >>>> should change. >>>> >>>> >>>> On Wed, Jun 6, 2012 at 2:14 PM, Barak Raveh <barak.raveh(a)gmail.com> >>>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> Let me consult with the high IMP society about some IMP scoring and >>>>> caching design issues, don't wanna mess things up. >>>>> >>>>> I've been profiling code, and consequently wanted to cache some >>>>> intermediate computations that are made during pair score evaluations >>>>> (e.g., >>>>> the distance between particles). I just need the last intermediate from >>>>> each >>>>> computation, so I thought of simply adding a [cache_] variable to my >>>>> scoring >>>>> class, and store the intermediate computations there. BUT, ::evaluate() >>>>> and >>>>> ::evaluate_index() are const functions, so I can't do that. Another >>>>> option >>>>> is to add an output parameter to ::evaluate(), but that's also not very >>>>> clean. Is it absolutely necessary that these functions are const? Or >>>>> any >>>>> other suggestions so that I can cache intermediate calculations during >>>>> ::evaluate()? This can speed some things up considerably. >>>>> >>>>> Barak >>>> >>>> >>>> >>>> _______________________________________________ >>>> IMP-dev mailing list >>>> IMP-dev(a)salilab.org >>>> https://salilab.org/mailman/listinfo/imp-dev >>>> >>> _______________________________________________ >>> IMP-dev mailing list >>> IMP-dev(a)salilab.org >>> https://salilab.org/mailman/listinfo/imp-dev >> >> >> >> >> _______________________________________________ >> IMP-dev mailing list >> IMP-dev(a)salilab.org >> https://salilab.org/mailman/listinfo/imp-dev >> >> >> >> _______________________________________________ >> IMP-dev mailing list >> IMP-dev(a)salilab.org >> https://salilab.org/mailman/listinfo/imp-dev >> > > > > ------------------------------ > > Message: 2 > Date: Thu, 7 Jun 2012 12:22:24 -0700 > From: Daniel Russel <drussel(a)gmail.com> > To: List for IMP development <imp-dev(a)salilab.org> > Subject: Re: [IMP-dev] caching advice > Message-ID: > <CADEvLN7wesVz72HK3c9-DQh6DTjcpy8y2hNOZi8kHFZdEyBGbA(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Sounds like a good idea. How about next Wednesday at 11 (PST)? Can everyone > who is interested make it or VC in then? > > On Thu, Jun 7, 2012 at 11:06 AM, Dina Schneidman <duhovka(a)gmail.com> wrote: > >> maybe we should have an IMP "high society" meeting and think how to >> improve both software design and performance of scoring functions. I >> have been looking into atomic resolution functions recently and have >> some suggestions there, and Daniel and Barak know what we need for low >> resolution. So we can come up with something that will cover all >> resolutions efficiently. >> >> On Wed, Jun 6, 2012 at 9:29 PM, Yannick Spill <yannick(a)salilab.org> wrote: >>> just my 2 cents: >>> I const_cast things. Don't know if it's a good thing, people were >> telling me >>> to avoid mutable. >>> I'm ok with dina that we need something more standardized. >>> >>> Y >>> >>> >>> Le 07/06/12 00:19, Daniel Russel a ?crit : >>> >>> It might make sense to introduce something a DistancePairScore base. It >>> could have an evaluate that takes the distance between the two particles >> and >>> the difference vector (and maybe the distance to avoid dup sqrts). One >> could >>> then have a variant of PairsRestraint that takes a list of such pair >> scores >>> and caches things for each pair passed in. >>> >>> It would also have the nice advantage of making it clear that certain >> scores >>> are invariant to rigid transforms and so provide a more principled basis >> for >>> ignoring, eg, terms that are within a rigid body or not recomputing >> things >>> when rigid mc moves are performed. >>> >>> Doing this would involve forking: >>> IMP::PairScore (and corresponding macros) >>> IMP::core::PairRestraint >>> IMP::container::PairsRestraint >>> IMP::internal::TupleRestraint >>> IMP::internal::ContainerRestraint >>> >>> On Wed, Jun 6, 2012 at 2:40 PM, Dina Schneidman <duhovka(a)gmail.com> >> wrote: >>>> >>>> I think we need more general caching mechanism. For example, if my >>>> score includes LennardJones and Coulombic terms, currently each one >>>> will compute the same distance, in addition there is external >>>> calculation that decides if we should score the pair of particles >>>> based on the distance between them. so instead of one distance >>>> calculation there are three. It would be great just to be able to pass >>>> the distance to evaluate() to avoid additional calculations. >>>> >>>> On Wed, Jun 6, 2012 at 2:19 PM, Daniel Russel <drussel(a)gmail.com> >> wrote: >>>>> You can mark member variables "mutable" to mean they aren't const when >>>>> the >>>>> class is. This is generally used for caches and stuff like that. In >>>>> general, >>>>> declaring a method const is used to imply logical constness instead of >>>>> physical constness. That is, it means that doing it 2x with the same >>>>> arguments should give the same answer, rather than that none of the >> bits >>>>> should change. >>>>> >>>>> >>>>> On Wed, Jun 6, 2012 at 2:14 PM, Barak Raveh <barak.raveh(a)gmail.com> >>>>> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> Let me consult with the high IMP society about some IMP scoring and >>>>>> caching design issues, don't wanna mess things up. >>>>>> >>>>>> I've been profiling code, and consequently wanted to cache some >>>>>> intermediate computations that are made during pair score evaluations >>>>>> (e.g., >>>>>> the distance between particles). I just need the last intermediate >> from >>>>>> each >>>>>> computation, so I thought of simply adding a [cache_] variable to my >>>>>> scoring >>>>>> class, and store the intermediate computations there. BUT, >> ::evaluate() >>>>>> and >>>>>> ::evaluate_index() are const functions, so I can't do that. Another >>>>>> option >>>>>> is to add an output parameter to ::evaluate(), but that's also not >> very >>>>>> clean. Is it absolutely necessary that these functions are const? Or >>>>>> any >>>>>> other suggestions so that I can cache intermediate calculations >> during >>>>>> ::evaluate()? This can speed some things up considerably. >>>>>> >>>>>> Barak >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> IMP-dev mailing list >>>>> IMP-dev(a)salilab.org >>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>> >>>> _______________________________________________ >>>> IMP-dev mailing list >>>> IMP-dev(a)salilab.org >>>> https://salilab.org/mailman/listinfo/imp-dev >>> >>> >>> >>> >>> _______________________________________________ >>> IMP-dev mailing list >>> IMP-dev(a)salilab.org >>> https://salilab.org/mailman/listinfo/imp-dev >>> >>> >>> >>> _______________________________________________ >>> IMP-dev mailing list >>> IMP-dev(a)salilab.org >>> https://salilab.org/mailman/listinfo/imp-dev >>> >> >> _______________________________________________ >> IMP-dev mailing list >> IMP-dev(a)salilab.org >> https://salilab.org/mailman/listinfo/imp-dev >> >

1 0

Re: [IMP-dev] hierarchy
by Daniel Russel 21 May '12

21 May '12

The quick answer is "nope" :-) Part of the reason is that we don't want to embed the assumption that the hierarchy has the same depth everywhere in code, and methods like that are hard to use without that assumption. The original idea had been that one would extract layers from the hierarchy by various criteria, such as radius. That is, ask for - give me the finest level representation - give me a representation where the particles all have radius close to 15A etc The atom::Selection class gives that sort of capability (by setting the radius for the selection). On Mon, May 21, 2012 at 1:43 PM, Charles Greenberg <cgreen(a)salilab.org>wrote: > Hey Daniel. Question for you. Kale is doing some very cool domino > experiments, using multiple levels of beads (coarse to fine) with a set of > distance restraints at each level. Currently he's using a simple dictionary > to establish connections between the levels, but he suggested it would be > cool to use the IMP hierarchy format. It's pretty clear how to make a > general hierarchy of particles, and to get children and parents, but is > there code to do simple queries like "How many levels are there?" and "Get > me all particles at level X"? Thanks! > > --Charles

1 0

Re: [IMP-dev] runtime question
by Daniel Russel 21 May '12

21 May '12

Each module has a get_data_path(file_name) function that returns the path to a file named "file_name" located in the data directory (whether you are running it our of the build dir or have installed IMP). For some reason the methods are currently excluded from the docs. I'll fix that. Similarly, there is a get_example_path(file_name) that returns the path to a file in the example directory (to be used in writing examples that load data). On May 20, 2012, at 9:43 PM, Keren Lasker wrote: > daniel - > i have scripts that copy from module/data folder. how can i access that in runtime from python without using $IMP/module/data? > keren.

1 0