[TIP] Column questions.

Tim Head betatim at gmail.com
Thu Jul 24 15:06:24 PDT 2008


2008/7/24 Andrew Dalke <dalke at dalkescientific.com>:
> Tim Head wrote:
>> How to test code that involves random numbers. So far I have not found
>> a good "howto" for this yet and curious to hear peoples thoughts on
>> it.
>
> Make sure you can specify the random number seed.  This helps
> reduce or eliminate variability across tests.  All this tests
> is reproducibility, not correctness.
>

True. Still it seems awfully shaky, a change in the number of rand()
calls between you setting the seed and checking the answer of your
function will make your test fail. I would like to be able to rely on
tests to tell me whether or not I broke something with my changes, so
this isnt very helpful or at least takes away some of their usefulness
;)

>
> It's possible sometimes to do numerical analysis to figure out
> the range of error.  I've not been in this circumstance.
>
> For correctness you can look at macro properties, like checking
> that energy/momentum are conserved over time.
>

Indeed, but I always thought it was a bit dirty to only be able to say
"well on average our MC generated W and Z bosons have the right mass".
Ideally I'd like to be able to say that it works always, not just on
average, there might be some subtle canceling out effect going on,
plus these statistical tests tend to be a bit tedious to do so maybe
you dont run them after every late night "tuning up" of the
algorithms.

>
> Greg Wilson gave a presentation recently titled
>
>    "High-Performance Computing Considered Harmful"
>
> The slides are at
>   http://www.cs.toronto.edu/~gvwilson/articles/hpc-considered-
> harmful-2008.pdf
>
> The relevant one for this discussion is:
>
>  o We don't know how to test floating-point code
>      * In a way that a grad student in civil engineering can
>        understand and would willingly do
>      * [Einarsson 2005] describes the problem several
>        times, but doesn't offer solutions
>      * |(actual-expected)/expected)| < 10-6 is superstition,
>        not science
>   o The real grand challenge in scientific computing
>      * At least, the one we ought to solve first
>

As an eager follower of his blog I have already looked at these ;). It
makes me unhappy to see that so many people get away with not being
able to prove that their code has no (obvious) bugs. This certainly is
my impression at undergraduate level.

tim


-- 
http://tim.jottit.com/



More information about the testing-in-python mailing list