On Dec 13, 2010, at 11:08 AM, Javier Velazquez wrote:

> > There is one thing that I didn't understand. If you develop on that > directory, can you commit from it to the proper svn directory? What I > mean is: > > If I am developing in ~/mydir_dir_where_i_work/src/my_file.cpp , when I > commit, is it going to be placed in imp/modules/mymodule/src/my_file.cpp > on the repository? If not, I see a potential problem of duplicated files. Files get committed wherever svn says they go. As a couple examples: - one could have a module that is not in the imp svn but in another svn repository (eg my npc module). I then have a directory (say npcdir), containing a checkout of my npc module (and links to the SConstuct and scons_tools and an appropriate config file pointing to a shared IMP build). I can then build my npc module ("scons -j 4") using the IMP nightly build for the rest of IMP and test it ("scons -j 4 npc-test"). Any changes I commit go to my npc svn repository.

- alternatively, only could be interested only in, say saxs from the IMP svn repository. You could then have a directory containing a checkout of the saxs module from the IMP repository ("svn co https://svn.salilab.org/imp/trunk/modules/saxs") and the sconstruct and scons_tools. Saxs can be built in isolation against the nightly build using "scons -j 4" and tested using "scons -j4 saxs-test" (the nightly build of saxs will be ignored). Changes made to saxs here would, when committed, go into the IMP repository.

Does that make sense and make things clearer?

> > Another thing that I don't get is the tests. Assuming a nightly build > for everyone, could I assume that other modules are tested, and > therefore only test my module? Yes. The idea would be that anything in the IMP svn would be tested as is currently the case (and the builds are exposed). Modules and things which are not in the IMP svn could then be tested on their own (with scons "modulename-test").

> > On 12/13/10 8:54 AM, Daniel Russel wrote: >> IMP now supports building of modules/applications/biological_systems without a full checkout of IMP svn. This means that you can create a directory containing only your module/application/biological_system (henceforth mabs) in a subdir, link the IMP SConstruct and scons_tools entries, make your config's includepath, libpath, pythonpath, and swigpath point to valid IMP headers, libraries, python modules and swig files, you can then happily build and test your module without >> - dealing with scons randomly rebuilding parts of IMP >> - having to compiling IMP yourself at all >> - sit through scons setting up other modules >> >> This makes sense to do if >> - you are not working on IMP core code (just adding stuff to your own mabs or working on a module like saxs or multifit that is not used by other modules) >> - what you are doing is not so bleeding edge that you need IMP updates more than once a day or so >> - we provide some sort of usable nightly build... >> >> My suggestion for how we could best take advantage of this would be to have useful nightly builds (release and fast with all dependencies against the desktop linux version) installed into a shared location each night in a directory labeled by the svn version or the date (eg /diva1/imp/release/12_11_2010). You could then point your local config.py at a given date's build rebuild only your code. Updating to a newer version of IMP would simply requiring changing where your config.py is pointing and rebuild only your code (so very fast). Having a repository of old builds of IMP would be very nice (I think Dina would agree) in tracking down changes in behavior and performance. >> >> The same thing could be done on the cluster. I'm not sure if there is much one could do with laptops though. >> >> Does this sound useful? >> _______________________________________________ >> IMP-dev mailing list >> IMP-dev@salilab.org >> https://salilab.org/mailman/listinfo/imp-dev > > -- > Javier Velazquez > Postdoc at Salilab, UCSF > 1700 4th st. Byers Hall, office 503. > 94158 San Francsico > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev

>> That wouldn't be that hard to do if people wanted it. It would obviously > be difficult to do that for every combination of architecture, build > flags and dependencies, however Aren't all lab desktops running the same distribution? Given that, there is no dependency/variation issue (as there is no reason to not have all the required rpms installed), just one fast and one release are needed. Likewise, presumably for 64 bit machines on the cluster.

> (and old versions will become unusable > if any dependency's API changes, when there is a Fedora update, etc.). Yes, it is a problem when packages are updated during the day. But it breaks individual builds of IMP too. And as it doesn't seem to happen too often, just rebuilding during the day when it does probably should handle it painlessly. And it could, presumably, be solved by timing desktop updates with builds.

> You also can't patch a nightly build to fix a problem in it, or tweak > something in it to see how it affects your code. Of course. It only makes sense for people who are not modifying core parts of IMP frequently. But, at least judging from commits, most people using it don't. And one can often check out a single module and fiddle with it (except for memory layout in algebra and the kernel), while using the build of the rest of IMP.

> What do you gain over > simply building your own copy of IMP and point your lone module's > config.py at that? Obviously you can check out whichever version from > SVN you like and update it as frequently (or infrequently) as you choose. Currently the cost for moving between IMP versions is quite high due to the compilation time. With such a setup, one could update to the new version quite quickly (and revert quickly when there are problems), which I suspect would make people do it more often. And that means that changes that cause trouble will raise flags sooner.

It also makes it easier to track down bugs and performance hiccups as one can easily try old versions (at least back to the last boost upgrade), something which I find prohibitively painful currently.

On Dec 13, 2010, at 1:52 PM, Ben Webb wrote:

> On 12/13/10 1:38 PM, Daniel Russel wrote: >> Aren't all lab desktops running the same distribution? > > Not today they're not. Your machine, for example, runs Fedora 13, and > mine, Fedora 14. I typically don't upgrade all of the desktops > simultaneously because 1) this would be massively disruptive and 2) > early adopters play with the new Fedora releases for a little while (and > compare them with the old) so we have a chance to report any bugs back > to the Fedora folks. Supporting two distributions isn't that much extra (and having builds on both seems like a useful thing to do during a transition anyway). And if it is only a few early adopters then not supporting a few machines isn't really much of a drawback. Likewise, have occasionally short periods where things are awkward (eg during the upgrade to a new fedora), are much of a drawback. One can just move the builds over the new fedora at the start of a general upgrade and people using them who haven't been upgraded yet have to use the pre-upgrade version.

> >>> (and old versions will become unusable if any dependency's API >>> changes, when there is a Fedora update, etc.). >> Yes, it is a problem when packages are updated during the day. > > No, your suggestion was that we have builds dating back months. There > will obviously be many packages updated between the time such a build is > made and today. Yes, a build from some months ago is less likely to work than one from more recently. The frequency of such breaking changes is reasonably small though (based on running old IMP builds around my machine), so it doesn't decrease the utility too much (and provides a natural flushing out time for old builds).

Make sense?