On May 29, 2008, at 9:50 AM, Ben Webb wrote: > But the nightly builds do > not complete if any of the tests fail, regardless of whether they're > run > in separate processes, and then people queue up at my desk the next > day > to complain about New Feature X not being usable yet in the nightlies. Ahhh.
That said, it is annoying to have to turn off a test when we discover a bug which is not going to be fixed immediately. It should just stay as a failed test as a reminder of non-working functionality (after all, it is still a bug). If it weren't for the memory corruption issue, we could presumably just tell Python.unittest how many tests we expect to have fail and have it modify the return state accordingly. At the very least, such bugs should be stuck somewhere on the wiki (or the doxygen todo list) rather than just working around the failure, but I would much rather have a test failure.
> The Python unittest tests are not for testing C++ code, but the Python > interface. If you want to write tests for something which is not > exposed > to Python, it makes more sense to write C++ tests rather than exposing > it just so you can test it. I seem to recall that Boost has a unittest > framework, which would probably be the best thing to use here. So where do I put C++ tests when I have written them? I currently just delete them when I am done. As a note, I mostly test in C++ until I get rid of the obvious bugs and then change it to python since it is hard to debug across swig.
I don't think we really want to have two separate test frameworks (or rather, I don't think you really want to have two separate test frameworks :-)
> Of course, if the functionality > *is* exposed to Python, you'll need a Python testcase, at a minimum to > make sure the SWIG wrapping is working correctly, so you may as well > use > this Python testcase to also test the underlying C++ code unless you > want to have a lot of overlap in the testcases.
Is it really worth writing code just to make sure that swig works properly? Anything where we have code in IMP.i which modifies return types or memory management needs to be tested in python, but I don't see the point of just checking that each, more or less identical, C++ function call is handled properly by swig.