> > I don't agree. There is no way that we can do sufficient testing of the > > code itself, packaging, documentation etc. etc. to warrant making this > > our primary distribution channel. > It currently is our primary distribution channel currently

Right, and that isn't a long term solution. An untested, always sort-of-broken nightly build cheapens the whole product. People expect to see well tested stable builds from time to time.

> I suspect that if we are going to make releases work we need to have > them every 6 months or so

Yes, that's what we already committed ourselves to.

Ben

Let me throw something in the discussion so other people say what they think also. After all, this is a collaborative project, and now's the time to hear what people want to do.

I agree with Daniel that we should try to be cleaner and not commit things that break the whole compile process. But simultaneously, I think it's more interesting for us all to have a stable release for the following reasons: - it has more impact, looks more serious, and does not run the risk of not working. - it is cool for us: There are 29 modules in the $IMP/src/modules folder. Does anyone know what all of these models do? Does anyone even know what *any* of these modules do (except for their own modules, and maybe core and atom)? I don't! I'm sure rmf and parallel are super useful for me but never dared opening the folder. I believe like Ben that a new release is a good thing because it forces us to advertise what we've been doing so far, and by advertising I just mean telling our labmates and maybe give them hints and new ideas on how to do things. Now as Daniel pointed out, we don't have much time to spend on that, so I'm suggesting the following. Why don't we set a date in a more or less short term, and decide that by then, every module should be working without bugs, and have a decent documentation and some examples. A release for which it's *not* the right time to code new neat features, but that's just like a snapshot of what can be done at the moment. I mean the shortest path to what looks reasonably like a new release, not perfect but working great. We could even do this on a per-module basis, saying that for the ISD module (for example), revision number 124567 is the good one that should be used for the new release. Once everyone has it's revision number ready, Daniel or Ben (just a suggestion) try to fix the last bugs where modules don't work together, and then we're good. If we did so, I believe this would represent less than a week's work, mainly to write a documentation, so having a release in a few months should be no big deal. We could even celebrate that by having a beer party and some talks of each module writers so they run us through the examples of their modules to understand how they work.

what do you think?

Y

Le 15/12/11 05:35, Daniel Russel a écrit : > In my understanding the things provided by periodic software releases are: > 1) a chance to fix outstanding bugs > 2) a way of solving packaging and distribution problems > 3) a way to avoid having people use code during major changes > 4) an opportunity to apply more manual testing > 5) a target time for implementation of new features > > As a result of various properties of IMP discussed below, I'm not > seeing that we would much benefit from them compared to providing > given slightly better access to nightly builds. As a result, it seems > to me that effort would be better spent on keeping the day to day > state of things better (writing more nightly tests, providing > infrastructure for weekly tests of more involved things, better > automation of things etc). > > > The above problems addressed by released don't really apply to IMP since > 1) we don't tend to have outstanding bugs sitting around as most bugs > encountered are blocking for projects and so fixed in relatively short > order > 2) these days, packaging for downloadable software is pretty automatic > 3) given the way IMP is structured as various loosely coupled classes > and modules, significant changes can be implemented as a parallel > module or class and then swapped for the old one when ready, so we > don't really have period where previously working code is broken for a > long time > 4) we are add automatic tests as we think of things and through using > IMP constantly are doing more manual testing of it every day than we > would do for a release > 5) people's schedules tend to be dictated by their other projects. > Features are implemented as needed for research/publications and it is > hard to get anyone to devote significant time to an external imposed > deadline. > > > The IMP 1.0 release was very much delayed, and by about a month out > was something that someone should probably think hard about using > instead of a carefully chosen svn build (based on bugs I found and > fixed after the release and which were never backported). We have then > had an extremely long time pass since and pushed back the next release > by months while waiting for a couple of weeks work. We can say that we > will, in the future, devote more resources to release-related > functionality, but people will always want to spend their time on > other things and I think they are, in general, justified given the > role that IMP plays. > > As an alternative, we could have been providing transparent access to > builds (so you can easily figure out if the build yesterday passed all > the tests you care about and check it out or download it with a copy > and paste or a click), then we make the cost of having people upgrade > when bug fixes or new features are added. > > > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev

On Dec 16, 2011, at 8:03 AM, Ben Webb wrote:

> On 12/16/11 5:08 AM, Yannick Spill wrote: >> I agree with Daniel that we should try to be cleaner and not commit >> things that break the whole compile process. > > Of course - I don't think anybody is arguing that we should break things > all the time. ;) This is one reason why tests are useful - you can see > pretty quickly if you (or somebody else) has broken something. (On the > other hand, if you have a bunch of tests that don't work, it's that much > harder to see, because the difference between "0 failures" and "1 > failure" is obvious, but that between "500 failures" and "501 failures" > is not. The IMP.isd tests are a good example here ;) Yeah, it would be really nice to have better feedback than this as the current setup is quite painful to figure out what is broken. At the very least, running each module independently (eg lots of "scons modulename-test" calls) would allow us to get more accurate reporting of success and failure (and having separate files would make looking through them at errors much easier).

And I think it would be not too hard to return a more numerical result (eg percent that faiiled) and display that. If python unit testing doesn't support returning statistics, then we can just make sure that each test string starts with "Testing" and then do, eg, "scons kernel-test" and grep for the "Testing" lines and either "OK" or "ERROR". It is probably easier though to insert some python code to return the success for failure and write it to a file.

Having previous nights results nicely available would be very nice too as one could then get an idea if a change broke something new.

On Dec 16, 2011, at 5:08 AM, Yannick Spill wrote:

> Let me throw something in the discussion so other people say what they think also. After all, this is a collaborative project, and now's the time to hear what people want to do. > > I agree with Daniel that we should try to be cleaner and not commit things that break the whole compile process. But simultaneously, I think it's more interesting for us all to have a stable release for the following reasons: > - it has more impact, looks more serious, and does not run the risk of not working. > - it is cool for us: There are 29 modules in the $IMP/src/modules folder. Does anyone know what all of these models do? Does anyone even know what *any* of these modules do (except for their own modules, and maybe core and atom)? I don't! I'm sure rmf and parallel are super useful for me but never dared opening the folder. > I believe like Ben that a new release is a good thing because it forces us to advertise what we've been doing so far, and by advertising I just mean telling our labmates and maybe give them hints and new ideas on how to do things. While I think this is something we need to do more of, I'm not sure that releases automatically help (I don't recall too much of this on the least release). IMP-dev meetings and adding things to the change history might be a better way to do this. To start on this, there is a file, doc/history.dox where people should announce any interesting changes they make (and should look whenever they update IMP to find what changes have been made).

> Now as Daniel pointed out, we don't have much time to spend on that, so I'm suggesting the following. Why don't we set a date in a more or less short term, and decide that by then, every module should be working without bugs, and have a decent documentation and some examples. A release for which it's *not* the right time to code new neat features, but that's just like a snapshot of what can be done at the moment. I mean the shortest path to what looks reasonably like a new release, not perfect but working great. We could even do this on a per-module basis, saying that for the ISD module (for example), revision number 124567 is the good one that should be used for the new release. Once everyone has it's revision number ready, Daniel or Ben (just a suggestion) try to fix the last bugs where modules don't work together, and then we're good. Yes, I think this is the right direction. But if you push this a little further, I think you end up with what I'm suggesting. Just provide better tools to keep track of and access how well each module/class is working and then just "release" that module or class whenever it is working well. For a module, this can be done by adding a label to SVN or the docs whenever the module writer feels like it. And if the nightly results are more usefully presented, people can more easily make up their own minds about when to update.

The tricky case is when there are dependencies between modules (eg a change that changes the API used by another module). This requires update of several things at once, but, I think, could be handled as a mini-release just coordinating among the involved people.

> If we did so, I believe this would represent less than a week's work, mainly to write a documentation, so having a release in a few months should be no big deal. We could even celebrate that by having a beer party and some talks of each module writers so they run us through the examples of their modules to understand how they work. My main concern is that finding a week were more than a couple of people can work on cleaning up IMP is very difficult. and as a result is perpetually sliding. And the result is that releases tend to slide and slide and hence become rather less useful than the would otherwise be. The more frequent they are, the less the cost of missing them, and so the easier it is to not wait for someone.