I think it very important that people run their code with checks turned on at least once to make sure there aren't obvious usage errors and most people seem to want speed and hence also want to run production code against a fast IMP build. Has this been has the current setup discouraged anyone from doing multiple builds?
Currently, having two separate builds requires doing something like: - svn co https://svn.salilab.org/imp/trunk imp - mkdir release - ./imp/tools/setup-out-of-source release - cd release - scons -j 4 config.py build=release path=/opt/local/bin libpath=/opt/local/lib includepath=/opt/local/include - scons -j 4 - ./tools/imppy.sh python myscript.py - mkdir fast - ./imp/tools/setup-out-of-source fast - cd fast - scons -j 4 config.py build=fast path=/opt/local/bin libpath=/opt/local/lib includepath=/opt/local/include - scons -j 4 - ./tools/imppy.sh python myscript.py
It could be made easier eg:
Options: 1) provide release and fast subdirectories in the svn checkout. This would be very easy to implement. Usage would then be something like, - svn co https://svn.salilab.org/imp/trunk imp - cd imp - ./configure path=/opt/local/bin libpath=/opt/local/lib includepath=/opt/local/include - cd release - scons -j 4 - ./imppy.sh python myscript.py - cd ../fast - scons -j 4 - ./imppy.sh python myscript.py
Unfortunately with both this and the current setup, installing release and fast versions of IMP requires installing them in separate locations and then switching around your LD_LIBRARY_PATH and PYTHON_PATH to switch between them. But a model where by you run with release on your machine and fast on the cluster would work fine. We could provide imppy.sh equivalents for once IMP is installed.
2) use an environment or python variable to switch on load time (warning, I don't actually know how to implement this, but suspect it can be done). - svn co https://svn.salilab.org/imp/trunk imp - cd imp - scons config.py path=/opt/local/bin libpath=/opt/local/lib includepath=/opt/local/include or ./configure path=/opt/local/bin libpath=/opt/local/lib includepath=/opt/local/include - scons -j 4 #build both fast and release - ./tools/imppy.release.sh python myscript.sh - ./tools/imppy.fast.sh python myscript.sh One could then install IMP normally (there would be libs like lib_imp_fast and lib_imp_release) and the right one could be chosen at runtime with an environment variable (or perhaps something more clever entirely within python)
I think that it is simple enough today to have two separate builds. The major obstacle is that those builds take forever to build, and you really don't want to wait for them, when you want to commit 3 lines of code.
On Tue, Nov 23, 2010 at 9:19 AM, Daniel Russel drussel@gmail.com wrote: > I think it very important that people run their code with checks turned on at least once to make sure there aren't obvious usage errors and most people seem to want speed and hence also want to run production code against a fast IMP build. Has this been has the current setup discouraged anyone from doing multiple builds? > > Currently, having two separate builds requires doing something like: > - svn co https://svn.salilab.org/imp/trunk imp > - mkdir release > - ./imp/tools/setup-out-of-source release > - cd release > - scons -j 4 config.py build=release path=/opt/local/bin libpath=/opt/local/lib includepath=/opt/local/include > - scons -j 4 > - ./tools/imppy.sh python myscript.py > - mkdir fast > - ./imp/tools/setup-out-of-source fast > - cd fast > - scons -j 4 config.py build=fast path=/opt/local/bin libpath=/opt/local/lib includepath=/opt/local/include > - scons -j 4 > - ./tools/imppy.sh python myscript.py > > It could be made easier eg: > > Options: > 1) provide release and fast subdirectories in the svn checkout. This would be very easy to implement. Usage would then be something like, > - svn co https://svn.salilab.org/imp/trunk imp > - cd imp > - ./configure path=/opt/local/bin libpath=/opt/local/lib includepath=/opt/local/include > - cd release > - scons -j 4 > - ./imppy.sh python myscript.py > - cd ../fast > - scons -j 4 > - ./imppy.sh python myscript.py > > Unfortunately with both this and the current setup, installing release and fast versions of IMP requires installing them in separate locations and then switching around your LD_LIBRARY_PATH and PYTHON_PATH to switch between them. But a model where by you run with release on your machine and fast on the cluster would work fine. We could provide imppy.sh equivalents for once IMP is installed. > > 2) use an environment or python variable to switch on load time (warning, I don't actually know how to implement this, but suspect it can be done). > - svn co https://svn.salilab.org/imp/trunk imp > - cd imp > - scons config.py path=/opt/local/bin libpath=/opt/local/lib includepath=/opt/local/include or ./configure path=/opt/local/bin libpath=/opt/local/lib includepath=/opt/local/include > - scons -j 4 #build both fast and release > - ./tools/imppy.release.sh python myscript.sh > - ./tools/imppy.fast.sh python myscript.sh > One could then install IMP normally (there would be libs like lib_imp_fast and lib_imp_release) and the right one could be chosen at runtime with an environment variable (or perhaps something more clever entirely within python) > > > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev >
On Nov 23, 2010, at 11:10 AM, Dina Schneidman wrote:
> I think that it is simple enough today to have two separate builds. > The major obstacle is that those builds take forever to build, and you > really don't want to wait for them, when you want to commit 3 lines of > code. Yeah, but that is a bit harder to solve (at least part is due to deep flaws in scons which I haven't yet managed to work around). On that note, once ccache 3 makes it to our linux distribution, it could be really nice to have a lab ccache repository as we are basically all doing the same compinaltions and ccache would simply fetch the results someone else built earlier ones rather than rebuild them.
> > On Tue, Nov 23, 2010 at 9:19 AM, Daniel Russel drussel@gmail.com wrote: >> I think it very important that people run their code with checks turned on at least once to make sure there aren't obvious usage errors and most people seem to want speed and hence also want to run production code against a fast IMP build. Has this been has the current setup discouraged anyone from doing multiple builds? >> >> Currently, having two separate builds requires doing something like: >> - svn co https://svn.salilab.org/imp/trunk imp >> - mkdir release >> - ./imp/tools/setup-out-of-source release >> - cd release >> - scons -j 4 config.py build=release path=/opt/local/bin libpath=/opt/local/lib includepath=/opt/local/include >> - scons -j 4 >> - ./tools/imppy.sh python myscript.py >> - mkdir fast >> - ./imp/tools/setup-out-of-source fast >> - cd fast >> - scons -j 4 config.py build=fast path=/opt/local/bin libpath=/opt/local/lib includepath=/opt/local/include >> - scons -j 4 >> - ./tools/imppy.sh python myscript.py >> >> It could be made easier eg: >> >> Options: >> 1) provide release and fast subdirectories in the svn checkout. This would be very easy to implement. Usage would then be something like, >> - svn co https://svn.salilab.org/imp/trunk imp >> - cd imp >> - ./configure path=/opt/local/bin libpath=/opt/local/lib includepath=/opt/local/include >> - cd release >> - scons -j 4 >> - ./imppy.sh python myscript.py >> - cd ../fast >> - scons -j 4 >> - ./imppy.sh python myscript.py >> >> Unfortunately with both this and the current setup, installing release and fast versions of IMP requires installing them in separate locations and then switching around your LD_LIBRARY_PATH and PYTHON_PATH to switch between them. But a model where by you run with release on your machine and fast on the cluster would work fine. We could provide imppy.sh equivalents for once IMP is installed. >> >> 2) use an environment or python variable to switch on load time (warning, I don't actually know how to implement this, but suspect it can be done). >> - svn co https://svn.salilab.org/imp/trunk imp >> - cd imp >> - scons config.py path=/opt/local/bin libpath=/opt/local/lib includepath=/opt/local/include or ./configure path=/opt/local/bin libpath=/opt/local/lib includepath=/opt/local/include >> - scons -j 4 #build both fast and release >> - ./tools/imppy.release.sh python myscript.sh >> - ./tools/imppy.fast.sh python myscript.sh >> One could then install IMP normally (there would be libs like lib_imp_fast and lib_imp_release) and the right one could be chosen at runtime with an environment variable (or perhaps something more clever entirely within python) >> >> >> _______________________________________________ >> IMP-dev mailing list >> IMP-dev@salilab.org >> https://salilab.org/mailman/listinfo/imp-dev >> > > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev
On 11/23/10 11:10 AM, Dina Schneidman wrote: > The major obstacle is that those builds take forever to build, and you > really don't want to wait for them, when you want to commit 3 lines of > code.
There are several things you can do as a developer to avoid this:
1. Use the -j flag to scons (just like for make) to parallelize the build. e.g. -j4 or -j8 (depending on how many processors your machine has).
2. Work most of the time with debug builds. Compiling without optimization is a lot faster (although the resulting code is slower, of course).
3. If all you're doing is making small changes to existing files (as opposed to adding whole new modules and the like) you can skip the configure stage of each build by running SCons in interactive mode (--interactive flag).
4. scons is conservative and always errs on the side of a more complete build if it's not sure, so you may get some unnecessary rebuilds (the most obvious is that it relinks any dynamic library if any of its dependencies are rebuilt). I'm assuming this and the rebuild of SWIG modules are the "deep flaws" that Daniel alludes darkly to, since I'm not aware of any others. So for example, if you modify a single file in the SAXS module, libimp_saxs.so will be rebuilt but also the Python IMP.saxs module (since it links against libimp_saxs.so). If you're sure you don't need the Python module to be rebuilt, just ask scons to build libimp_saxs.so itself, i.e. "b build/lib/libimp_saxs.so" from the scons --interactive prompt. (In principle "b saxs-lib" should do the same thing if the alias is set up properly, but also work on Macs and Windows systems.)
Otherwise, sorry, compiling C++ code is slow. Ask for a faster computer. ;)
Ben
> 4. scons is conservative and always errs on the side of a more complete > build if it's not sure, so you may get some unnecessary rebuilds (the > most obvious is that it relinks any dynamic library if any of its > dependencies are rebuilt). I'm assuming this and the rebuild of SWIG > modules are the "deep flaws" that Daniel alludes darkly to, since I'm > not aware of any others. Yes. And unfortunately the swig modules are the biggest files in IMP...
Thanks Ben. I am already familiar with the way compilers and builders work, so these suggestions are used by default. And I am happy with the speed of my computer, 8 cores and 32G memory are not that bad :) Related to Daniel's question, I want to distinguish between working on the code and committing the code. When we work on the code, the re-build is indeed very local, so it's not a big deal. But when we commit code, we have to update our IMP version (in order to have the latest version), everything has to be re-built, since IMP main modules (kernel, core, etc..) change almost everyday. And this is where most of the compilation time is spent. Also as Daniel pointed out, swig does takes lots of time to build, and while I can skip these builds when I work on C++ code, I can't do it when I commit the code and have to run python tests.
On Tue, Nov 23, 2010 at 4:19 PM, Ben Webb ben@salilab.org wrote: > On 11/23/10 11:10 AM, Dina Schneidman wrote: >> The major obstacle is that those builds take forever to build, and you >> really don't want to wait for them, when you want to commit 3 lines of >> code. > > There are several things you can do as a developer to avoid this: > > 1. Use the -j flag to scons (just like for make) to parallelize the > build. e.g. -j4 or -j8 (depending on how many processors your machine has). > > 2. Work most of the time with debug builds. Compiling without > optimization is a lot faster (although the resulting code is slower, of > course). > > 3. If all you're doing is making small changes to existing files (as > opposed to adding whole new modules and the like) you can skip the > configure stage of each build by running SCons in interactive mode > (--interactive flag). > > 4. scons is conservative and always errs on the side of a more complete > build if it's not sure, so you may get some unnecessary rebuilds (the > most obvious is that it relinks any dynamic library if any of its > dependencies are rebuilt). I'm assuming this and the rebuild of SWIG > modules are the "deep flaws" that Daniel alludes darkly to, since I'm > not aware of any others. So for example, if you modify a single file in > the SAXS module, libimp_saxs.so will be rebuilt but also the Python > IMP.saxs module (since it links against libimp_saxs.so). If you're sure > you don't need the Python module to be rebuilt, just ask scons to build > libimp_saxs.so itself, i.e. "b build/lib/libimp_saxs.so" from the scons > --interactive prompt. (In principle "b saxs-lib" should do the same > thing if the alias is set up properly, but also work on Macs and Windows > systems.) > > Otherwise, sorry, compiling C++ code is slow. Ask for a faster computer. ;) > > Ben > -- > ben@salilab.org http://salilab.org/~ben/ > "It is a capital mistake to theorize before one has data." > - Sir Arthur Conan Doyle > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev >
I find it best to preemptively build IMP (and keep it up to date). eg on my desktop IMP is updated and built each night by cron, primarily for benchmarking, but it has nice side effects when it comes to building small changes during the day. And likewise, I just build on my laptop when I don't need it, so that I don't have to wait when I do. Not as nice as something like ccache and works best when you are the primary person breaking svn :-), but it is relatively easy and means I don't often wait (other than for the swig wrappers to be rebuilt).
As for the swig wrapper, I have another idea to try for bypassing the dependency checking on pre-updated files, but haven't gotten a chance to try it yet.
On Nov 23, 2010, at 6:06 PM, Dina Schneidman wrote:
> Thanks Ben. I am already familiar with the way compilers and builders > work, so these suggestions are used by default. And I am happy with > the speed of my computer, 8 cores and 32G memory are not that bad :) > Related to Daniel's question, I want to distinguish between working on > the code and committing the code. > When we work on the code, the re-build is indeed very local, so it's > not a big deal. > But when we commit code, we have to update our IMP version (in order > to have the latest version), everything has to be re-built, since IMP > main modules (kernel, core, etc..) change almost everyday. And this > is where most of the compilation time is spent. > Also as Daniel pointed out, swig does takes lots of time to build, and > while I can skip these builds when I work on C++ code, I can't do it > when I commit the code and have to run python tests. > > On Tue, Nov 23, 2010 at 4:19 PM, Ben Webb ben@salilab.org wrote: >> On 11/23/10 11:10 AM, Dina Schneidman wrote: >>> The major obstacle is that those builds take forever to build, and you >>> really don't want to wait for them, when you want to commit 3 lines of >>> code. >> >> There are several things you can do as a developer to avoid this: >> >> 1. Use the -j flag to scons (just like for make) to parallelize the >> build. e.g. -j4 or -j8 (depending on how many processors your machine has). >> >> 2. Work most of the time with debug builds. Compiling without >> optimization is a lot faster (although the resulting code is slower, of >> course). >> >> 3. If all you're doing is making small changes to existing files (as >> opposed to adding whole new modules and the like) you can skip the >> configure stage of each build by running SCons in interactive mode >> (--interactive flag). >> >> 4. scons is conservative and always errs on the side of a more complete >> build if it's not sure, so you may get some unnecessary rebuilds (the >> most obvious is that it relinks any dynamic library if any of its >> dependencies are rebuilt). I'm assuming this and the rebuild of SWIG >> modules are the "deep flaws" that Daniel alludes darkly to, since I'm >> not aware of any others. So for example, if you modify a single file in >> the SAXS module, libimp_saxs.so will be rebuilt but also the Python >> IMP.saxs module (since it links against libimp_saxs.so). If you're sure >> you don't need the Python module to be rebuilt, just ask scons to build >> libimp_saxs.so itself, i.e. "b build/lib/libimp_saxs.so" from the scons >> --interactive prompt. (In principle "b saxs-lib" should do the same >> thing if the alias is set up properly, but also work on Macs and Windows >> systems.) >> >> Otherwise, sorry, compiling C++ code is slow. Ask for a faster computer. ;) >> >> Ben >> -- >> ben@salilab.org http://salilab.org/~ben/ >> "It is a capital mistake to theorize before one has data." >> - Sir Arthur Conan Doyle >> _______________________________________________ >> IMP-dev mailing list >> IMP-dev@salilab.org >> https://salilab.org/mailman/listinfo/imp-dev >> > > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev
Given that most of the IMP changes are to quite peripheral code, I think it is OK to only update IMP occasionally (say weekly) when only making changes to a particular module. SVN will notify you if the file was changed elsewhere and otherwise it is cheap to clean up the rare issues.
On Nov 23, 2010, at 6:06 PM, Dina Schneidman wrote:
> Thanks Ben. I am already familiar with the way compilers and builders > work, so these suggestions are used by default. And I am happy with > the speed of my computer, 8 cores and 32G memory are not that bad :) > Related to Daniel's question, I want to distinguish between working on > the code and committing the code. > When we work on the code, the re-build is indeed very local, so it's > not a big deal. > But when we commit code, we have to update our IMP version (in order > to have the latest version), everything has to be re-built, since IMP > main modules (kernel, core, etc..) change almost everyday. And this > is where most of the compilation time is spent. > Also as Daniel pointed out, swig does takes lots of time to build, and > while I can skip these builds when I work on C++ code, I can't do it > when I commit the code and have to run python tests. > > On Tue, Nov 23, 2010 at 4:19 PM, Ben Webb ben@salilab.org wrote: >> On 11/23/10 11:10 AM, Dina Schneidman wrote: >>> The major obstacle is that those builds take forever to build, and you >>> really don't want to wait for them, when you want to commit 3 lines of >>> code. >> >> There are several things you can do as a developer to avoid this: >> >> 1. Use the -j flag to scons (just like for make) to parallelize the >> build. e.g. -j4 or -j8 (depending on how many processors your machine has). >> >> 2. Work most of the time with debug builds. Compiling without >> optimization is a lot faster (although the resulting code is slower, of >> course). >> >> 3. If all you're doing is making small changes to existing files (as >> opposed to adding whole new modules and the like) you can skip the >> configure stage of each build by running SCons in interactive mode >> (--interactive flag). >> >> 4. scons is conservative and always errs on the side of a more complete >> build if it's not sure, so you may get some unnecessary rebuilds (the >> most obvious is that it relinks any dynamic library if any of its >> dependencies are rebuilt). I'm assuming this and the rebuild of SWIG >> modules are the "deep flaws" that Daniel alludes darkly to, since I'm >> not aware of any others. So for example, if you modify a single file in >> the SAXS module, libimp_saxs.so will be rebuilt but also the Python >> IMP.saxs module (since it links against libimp_saxs.so). If you're sure >> you don't need the Python module to be rebuilt, just ask scons to build >> libimp_saxs.so itself, i.e. "b build/lib/libimp_saxs.so" from the scons >> --interactive prompt. (In principle "b saxs-lib" should do the same >> thing if the alias is set up properly, but also work on Macs and Windows >> systems.) >> >> Otherwise, sorry, compiling C++ code is slow. Ask for a faster computer. ;) >> >> Ben >> -- >> ben@salilab.org http://salilab.org/~ben/ >> "It is a capital mistake to theorize before one has data." >> - Sir Arthur Conan Doyle >> _______________________________________________ >> IMP-dev mailing list >> IMP-dev@salilab.org >> https://salilab.org/mailman/listinfo/imp-dev >> > > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev
participants (3)
-
Ben Webb -
Daniel Russel -
Dina Schneidman