Most of the restraints and other useful functionality are moving out of the kernel. Currently the idea is that they go into a module called dsr (for lack of another name) which would be modifiable by everyone. Only the angle restraints and a couple of other non-kernel objects would be left in the kernel. I propose instead we call it "basic" and move all existing restraints into it. That way we have less random split of code between modules. After all, does anyone want to remember that "DistanceRestraint" is in IMP but "DistancePairScore" or "NonbondedListScoreState" is part of IMP.dsr?
In either case, the npc-specific code would stay elsewhere.
Daniel Russel wrote: > Most of the restraints and other useful functionality are moving out of > the kernel. Currently the idea is that they go into a module called dsr > (for lack of another name) which would be modifiable by everyone. Only > the angle restraints and a couple of other non-kernel objects would be > left in the kernel. I propose instead we call it "basic" and move all > existing restraints into it. That way we have less random split of code > between modules. After all, does anyone want to remember that > "DistanceRestraint" is in IMP but "DistancePairScore" or > "NonbondedListScoreState" is part of IMP.dsr?
Let's not confuse the issues here. There are two quite separate issues: 1. Easier access to SVN for people wanting to try new methods. 2. Decision on what should and should not lie in the kernel.
I proposed creating the 'unstable' module to deal with (1). As for (2), the kernel interfaces are comparatively stable, so in the short term at least I don't envisage large changes there. (There is, of course, nothing to stop you simply copying a kernel function into unstable and developing it there. Once it's stable, the kernel version can be updated, or it can be moved into a stable module.)
Before we get too carried away with modules, let's define some ground rules: 1. Modules group classes and functions by similar functionality, not by author, and should be named accordingly. For example, the 'em' module contains only classes for cryo EM fitting. 2. Each module has a responsible owner or group. 3. Modules should still obey IMP coding standards, code review is still encouraged if possible, and major API changes should be advertised on the IMP-dev mailing list. 4. Modules can link with external libraries that the IMP kernel does not link with. However, these libraries must be publicly accessible (so that others can use the module) and not be incompatible with the IMP license (for example, no GPL libraries, but BSD or LGPL is OK). 5. For developmental code that doesn't obviously fit into a neat category, the 'unstable' module is writable by all.
As for the kernel, clearly there is a need for guidelines for what lives in there. For now, however, we could start by moving the experimental stuff that is not relied upon by other projects into the 'unstable' module. (Seems that this would be some or all of CoverBondsScoreState, ParticleRefiner and subclasses, BrownianDynamics, RefineOncePairScore, TunnelSingletonScore, WormLikeChain.)
Ben
> Let's not confuse the issues here. There are two quite separate issues: > 1. Easier access to SVN for people wanting to try new methods. > 2. Decision on what should and should not lie in the kernel. > Agreed. > I proposed creating the 'unstable' module to deal with (1). I really don't see the point of that. Two things: - first, the traditional approach is to have stable and unstable branches in SVN. The advantage of this is that we don't have to change all the calls (by changing or removing the module) if we want to move stuff from one place to another. - second, given the structure of IMP, there is no reason to put unstable and stable classes in different directories. We just mark which restraints and score states are stable and which are unstable, with, for example, a doxygen group. Moving something from one module to another is a pain for anyone who uses it.
> As for the kernel, clearly there is a need for guidelines for what lives > in there. I propose that the kernel is the IMP/*.h only. After all, that is the kernel... Everything else is optional.
> For now, however, we could start by moving the experimental > stuff that is not relied upon by other projects into the 'unstable' > module. (Seems that this would be some or all of CoverBondsScoreState, > ParticleRefiner and subclasses, BrownianDynamics, RefineOncePairScore, > TunnelSingletonScore, WormLikeChain.) > For me, the important thing is to have NonbondedList, HierarchyDecorator (which the pdb reader depends on) and things like that somewhere were I can actually modify them and can commit patches in a reasonable fashion. Weird things like most of the ones in the list have no good reason to go in the IMP svn at all.
Anyway, we should get something done soon, this is taking a bit long given that few people are inclined to comment.
It seems to me the reasonable options are:
1) an IMP kernel which is just the kernel (so the classes in IMP/*.h) and then various modules with other classes including a core module which has most of the current restraints and is open to everyone to commit things.
2) use a more normal revision control system with release and unstable branches. Anyone can commit to the unstable branch and Ben is in charge of moving things from unstable to release.
As always, I prefer the second, especially since IMP is fairly modular and so one can pick and choose which unstable things one uses use.
So, one or two?
Daniel Russel wrote: >> Let's not confuse the issues here. There are two quite separate issues: >> 1. Easier access to SVN for people wanting to try new methods. >> 2. Decision on what should and should not lie in the kernel. >> >> > Agreed. > >> I proposed creating the 'unstable' module to deal with (1). >> > I really don't see the point of that. Two things: > - first, the traditional approach is to have stable and unstable > branches in SVN. The advantage of this is that we don't have to change > all the calls (by changing or removing the module) if we want to move > stuff from one place to another. > - second, given the structure of IMP, there is no reason to put unstable > and stable classes in different directories. We just mark which > restraints and score states are stable and which are unstable, with, for > example, a doxygen group. Moving something from one module to another is > a pain for anyone who uses it. > > >> As for the kernel, clearly there is a need for guidelines for what lives >> in there. >> > I propose that the kernel is the IMP/*.h only. After all, that is the > kernel... Everything else is optional. > > >> For now, however, we could start by moving the experimental >> stuff that is not relied upon by other projects into the 'unstable' >> module. (Seems that this would be some or all of CoverBondsScoreState, >> ParticleRefiner and subclasses, BrownianDynamics, RefineOncePairScore, >> TunnelSingletonScore, WormLikeChain.) >> >> > For me, the important thing is to have NonbondedList, HierarchyDecorator > (which the pdb reader depends on) and things like that somewhere were I > can actually modify them and can commit patches in a reasonable fashion. > Weird things like most of the ones in the list have no good reason to go > in the IMP svn at all. > > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev >
Daniel Russel wrote: > Anyway, we should get something done soon, this is taking a bit long > given that few people are inclined to comment. > > It seems to me the reasonable options are: > > 1) an IMP kernel which is just the kernel (so the classes in IMP/*.h) > and then various modules with other classes including a core module > which has most of the current restraints and is open to everyone to > commit things. > > 2) use a more normal revision control system with release and unstable > branches. Anyone can commit to the unstable branch and Ben is in charge > of moving things from unstable to release. > > As always, I prefer the second, especially since IMP is fairly modular > and so one can pick and choose which unstable things one uses use. > > So, one or two? > As a follow up. Ultimately, they will look pretty much the same as there will be two separate copies of more or less everything with the added complication that in the second case both versions can be mixed and matched. However, mixing and matching is unlikely to work reliably, as it won't really be tested.
Daniel Russel wrote: >> I proposed creating the 'unstable' module to deal with (1). > I really don't see the point of that. Two things: > - first, the traditional approach is to have stable and unstable > branches in SVN. The advantage of this is that we don't have to change > all the calls (by changing or removing the module) if we want to move > stuff from one place to another.
An SVN branch is just a directory anyway. And I'd rather not branch all of IMP.
> - second, given the structure of IMP, there is no reason to put unstable > and stable classes in different directories.
Perhaps we should not call this module 'unstable', because you seem to be getting the wrong end of the stick here; perhaps 'misc' is a better name. The intention is for it to be used for kernel additions that don't happily sit anywhere else. Because the access is more open, it is likely that people will put in unstable, untested code, but I certainly don't think we should encourage that.
>> As for the kernel, clearly there is a need for guidelines for what lives >> in there. > I propose that the kernel is the IMP/*.h only. After all, that is the > kernel... Everything else is optional.
Well, that's the obvious definition, but it could be argued that that's too inclusive. For example, ParticleRefiner.h probably doesn't need to be in there.
Ben
Ben Webb wrote: > Daniel Russel wrote: > >>> I proposed creating the 'unstable' module to deal with (1). >>> >> I really don't see the point of that. Two things: >> - first, the traditional approach is to have stable and unstable >> branches in SVN. The advantage of this is that we don't have to change >> all the calls (by changing or removing the module) if we want to move >> stuff from one place to another. >> > > An SVN branch is just a directory anyway. And I'd rather not branch all > of IMP.
>> - second, given the structure of IMP, there is no reason to put unstable >> and stable classes in different directories. >> > > Perhaps we should not call this module 'unstable', because you seem to > be getting the wrong end of the stick here; perhaps 'misc' is a better > name. The intention is for it to be used for kernel additions that don't > happily sit anywhere else. Because the access is more open, it is likely > that people will put in unstable, untested code, but I certainly don't > think we should encourage that. > So to make the next iteration of the proposal: 1) the IMP kernel module is simplified to contain only the IMP kernel proper which is basically some subset of IMP/*.h, perhaps without decoratorbase and ParticleRefiner.
2) We add a module IMP_misc or IMP_core which contains all the existing restraints, states and decorators and stuff and has a more open access policy.
3) We adopt a convention of labelling things in IMP_misc as to their maturity. Perhaps "stable" for objects whose interfaces are unlikely to change and which are well tested and "unstable" for newer and less mature things and "unsupported" for things which are there because they may be of interest, but the level of interest is unclear and unless people say something they may not be actively developed.
4) IMP_misc will have an open commit policy, but convention that changes to the interface of things marked stable should be proceeded by a warning on imp-dev and that tests on them must not be broken.
I have a mostly marked version of the non-kernel imp code so I can easily produce the code for an IMP_misc module given a place to put it in SVN. Once it is in place, then we can remove the stuff from kernel.
On Aug 27, 2008, at 10:18 AM, Daniel Russel wrote:
> Ben Webb wrote: >> Daniel Russel wrote: >> >>>> I proposed creating the 'unstable' module to deal with (1). >>>> >>> I really don't see the point of that. Two things: >>> - first, the traditional approach is to have stable and unstable >>> branches in SVN. The advantage of this is that we don't have to >>> change >>> all the calls (by changing or removing the module) if we want to >>> move >>> stuff from one place to another. >>> >> >> An SVN branch is just a directory anyway. And I'd rather not branch >> all >> of IMP. > >>> - second, given the structure of IMP, there is no reason to put >>> unstable >>> and stable classes in different directories. >>> >> >> Perhaps we should not call this module 'unstable', because you seem >> to >> be getting the wrong end of the stick here; perhaps 'misc' is a >> better >> name. The intention is for it to be used for kernel additions that >> don't >> happily sit anywhere else. Because the access is more open, it is >> likely >> that people will put in unstable, untested code, but I certainly >> don't >> think we should encourage that. >> > So to make the next iteration of the proposal: > 1) the IMP kernel module is simplified to contain only the IMP kernel > proper which is basically some subset of IMP/*.h, perhaps without > decoratorbase and ParticleRefiner. > > 2) We add a module IMP_misc or IMP_core which contains all the > existing > restraints, states and decorators and stuff and has a more open access > policy. > > 3) We adopt a convention of labelling things in IMP_misc as to their > maturity. Perhaps "stable" for objects whose interfaces are unlikely > to > change and which are well tested and "unstable" for newer and less > mature things and "unsupported" for things which are there because > they > may be of interest, but the level of interest is unclear and unless > people say something they may not be actively developed. > > 4) IMP_misc will have an open commit policy, but convention that > changes > to the interface of things marked stable should be proceeded by a > warning on imp-dev and that tests on them must not be broken. > > I have a mostly marked version of the non-kernel imp code so I can > easily produce the code for an IMP_misc module given a place to put it > in SVN. Once it is in place, then we can remove the stuff from kernel.
So the suggestion is that reviewed and tested code will only be a few *.h files? and the IMP_misc module (labeled stable, unstable or unsupported) is all of IMP ?? Assembler ( Frido's and my 26S stuff) , for example, relays on code that according to the last proposal would be part of IMP_misc (restraints, optimizers , the hierarchy) , which means that we can expect the interfaces to change on a daily basis without any sufficient tests on the python level, cluster etc. It does not make sense to encourage people to rely on the IMP kernel when it is going to be mostly unstable. I think that there should be some level between kernel and IMP_misc, which will include the core functionalities ( basic optimizers, restraints, hierarchy and atomic stuff). If we are going to continue with the suggested plan, we have to have many more test-cases and as Ben suggested assembler stuff should probably be part of the test-cases. I would really really appreciate holding the reorg until we'll have assembler test-cases ready for IMP (probably in the next two weeks).
> _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev
Keren Lasker wrote: > On Aug 27, 2008, at 10:18 AM, Daniel Russel wrote: > > >> Ben Webb wrote: >> >>> Daniel Russel wrote: >>> >>> >>>>> I proposed creating the 'unstable' module to deal with (1). >>>>> >>>>> >>>> I really don't see the point of that. Two things: >>>> - first, the traditional approach is to have stable and unstable >>>> branches in SVN. The advantage of this is that we don't have to >>>> change >>>> all the calls (by changing or removing the module) if we want to >>>> move >>>> stuff from one place to another. >>>> >>>> >>> An SVN branch is just a directory anyway. And I'd rather not branch >>> all >>> of IMP. >>> >>>> - second, given the structure of IMP, there is no reason to put >>>> unstable >>>> and stable classes in different directories. >>>> >>>> >>> Perhaps we should not call this module 'unstable', because you seem >>> to >>> be getting the wrong end of the stick here; perhaps 'misc' is a >>> better >>> name. The intention is for it to be used for kernel additions that >>> don't >>> happily sit anywhere else. Because the access is more open, it is >>> likely >>> that people will put in unstable, untested code, but I certainly >>> don't >>> think we should encourage that. >>> >>> >> So to make the next iteration of the proposal: >> 1) the IMP kernel module is simplified to contain only the IMP kernel >> proper which is basically some subset of IMP/*.h, perhaps without >> decoratorbase and ParticleRefiner. >> >> 2) We add a module IMP_misc or IMP_core which contains all the >> existing >> restraints, states and decorators and stuff and has a more open access >> policy. >> >> 3) We adopt a convention of labelling things in IMP_misc as to their >> maturity. Perhaps "stable" for objects whose interfaces are unlikely >> to >> change and which are well tested and "unstable" for newer and less >> mature things and "unsupported" for things which are there because >> they >> may be of interest, but the level of interest is unclear and unless >> people say something they may not be actively developed. >> >> 4) IMP_misc will have an open commit policy, but convention that >> changes >> to the interface of things marked stable should be proceeded by a >> warning on imp-dev and that tests on them must not be broken. >> >> I have a mostly marked version of the non-kernel imp code so I can >> easily produce the code for an IMP_misc module given a place to put it >> in SVN. Once it is in place, then we can remove the stuff from kernel. >> > > So the suggestion is that reviewed and tested code will only be a few > *.h files? and the IMP_misc module (labeled stable, unstable or > unsupported) is all of IMP ?? > The reviewed and tested code will be exactly the same as today. No real review and sporadic testing :-) More seriously, the practical differences will be negligible for most people other then we would have a convention for marking what is stable and what is not, something we currently lack. We will have some things that are stable and some things which are not. Stable things will be well tested and stable, things which are .
> Assembler ( Frido's and my 26S stuff) , for example, relays on code > that according to the last proposal would be part of IMP_misc > (restraints, optimizers , the hierarchy) , which means that we can > expect the interfaces to change on a daily basis without any > sufficient tests on the python level, cluster etc. > Of course not. You are being silly.
People are conflating three things: - stability of the interface of certain files - location of the files - who can commit the files to svn
The three are entirely separate and we can make decisions about all three independently. Tying them together is is not necessarily a good thing.
> It does not make sense to encourage people to rely on the IMP kernel > when it is going to be mostly unstable. > I think that there should be some level between kernel and IMP_misc, > which will include the core functionalities ( basic optimizers, > restraints, hierarchy and atomic stuff). > yes, all that should be in IMP_whatever_we call it and marked as stable and so infrequently changing.
> If we are going to continue with the suggested plan, we have to have > many more test-cases and as Ben suggested assembler stuff should > probably be part of the test-cases. I would really really appreciate > holding the reorg until we'll have assembler test-cases ready for IMP > (probably in the next two weeks). > That will just result in more code that has to be modified when we reorganize things. Wouldn't you rather write tests against something that is not going to change immediately?
Daniel Russel wrote: > So to make the next iteration of the proposal: > 1) the IMP kernel module is simplified to contain only the IMP kernel > proper which is basically some subset of IMP/*.h, perhaps without > decoratorbase and ParticleRefiner.
That makes sense, but not until dependent projects are in the build system, because obviously they would need to be updated.
> 2) We add a module IMP_misc or IMP_core which contains all the existing > restraints, states and decorators and stuff and has a more open access > policy.
There are two separate modules here: misc and core. misc is for stuff that doesn't fit anywhere else (e.g. ParticleRefiner) and can be very open, even today. Other stuff that is relied upon by others will move from the kernel into core (e.g. nonbonded lists). But that won't happen today - it needs to wait for the dependent projects to be updated. By its very nature, it should also be less open than misc.
> 3) We adopt a convention of labelling things in IMP_misc as to their > maturity.
That makes sense for misc. But everything in core should be stable.
Ben
Ben Webb wrote: > Daniel Russel wrote: > >> So to make the next iteration of the proposal: >> 1) the IMP kernel module is simplified to contain only the IMP kernel >> proper which is basically some subset of IMP/*.h, perhaps without >> decoratorbase and ParticleRefiner. >> > > That makes sense, but not until dependent projects are in the build > system, because obviously they would need to be updated. > Well, people can update their own or not update svn for a while. I would rather not wait too long as we will have another diverged mess to clean up. > There are two separate modules here: misc and core. misc is for stuff > that doesn't fit anywhere else (e.g. ParticleRefiner) and can be very > open, even today. Other stuff that is relied upon by others will move > from the kernel into core (e.g. nonbonded lists). But that won't happen > today - it needs to wait for the dependent projects to be updated. By > its very nature, it should also be less open than misc. > Again, I don't see any point to doing that. You don't buy anything and you add a hassle when you decide that something should move from misc to core.
Further, that doesn't really buy us anything over the current mess other than slightly faster compilation times and creates a great deal of work.
-
Ben Webb -
Daniel Russel -
Keren Lasker