On Dec 16, 2011, at 8:03 AM, Ben Webb wrote:
> On 12/16/11 5:08 AM, Yannick Spill wrote: >> I agree with Daniel that we should try to be cleaner and not commit >> things that break the whole compile process. > > Of course - I don't think anybody is arguing that we should break things > all the time. ;) This is one reason why tests are useful - you can see > pretty quickly if you (or somebody else) has broken something. (On the > other hand, if you have a bunch of tests that don't work, it's that much > harder to see, because the difference between "0 failures" and "1 > failure" is obvious, but that between "500 failures" and "501 failures" > is not. The IMP.isd tests are a good example here ;) Yeah, it would be really nice to have better feedback than this as the current setup is quite painful to figure out what is broken. At the very least, running each module independently (eg lots of "scons modulename-test" calls) would allow us to get more accurate reporting of success and failure (and having separate files would make looking through them at errors much easier).
And I think it would be not too hard to return a more numerical result (eg percent that faiiled) and display that. If python unit testing doesn't support returning statistics, then we can just make sure that each test string starts with "Testing" and then do, eg, "scons kernel-test" and grep for the "Testing" lines and either "OK" or "ERROR". It is probably easier though to insert some python code to return the success for failure and write it to a file.
Having previous nights results nicely available would be very nice too as one could then get an idea if a change broke something new.