[TIP] why you should distribute tests with your application / module

[C. Titus Brown]

On Thu, Sep 18, 2008 at 12:42:05PM +1000, Michael Gratton wrote:
-> On Wed, 2008-09-17 at 19:17 -0700, C. Titus Brown wrote:
-> > Win 95, Win 98, Win NT, Win XP, Win 2000, Vista.  There are binaries
-> > that will work on all of them, and binaries that will work on only one
-> > of them.
-> > 
-> > Mac OS X 10.0, 10.1, 10.2, 10.3, 10.4, 10.5, ditto.
-> 
-> That's besides the point. If you ship binaries on those platforms, you
-> should have already run the tests on them (or, given they are
-> forwards-compatible, on the lowest common denominator). If not, then a
-> porter will download the source and can run the tests from there.

This is a naive "perfect world" argument.  I don't have the money, time,
or inclination to keep all of those versions of OS X around, and I
certainly don't trust Apple when they tell me things are "forwards
compatible", much less Microsoft.  There are also patch sets and Service
Packs to consider, too.  I can't test on all of them but I see no harm
in making it possible for my users to do so, quickly and easily.

It is possible to build binaries that, in theory, should work across all
those platforms and service packs.  The value of testing lies between
theory and practice.

-> > -> Mozilla has a large suite of test cases. Do you wish they shipped
-> > those
-> > -> so you could run them every time you download Firefox? If they did,
-> > -> would you?
-> > 
-> > If I were a Firefox developer, I would want to be able to tell users
-> > getting weird errors to run the tests, yes.
-> 
-> So I guess your answer to the question is "no - you wouldn't run the
-> tests unless asked".

I might, but I am not a typical user.  I don't see why it would be to my
disadvantage as a developer to block my users from running the tests.

-> Still no point in shipping the tests with the binaries then... :)

Why not make it easy for the users to run the tests when you ask them
to?  Is there an actual drawback in the face of the potential positives?

-t
-- 
C. Titus Brown, ctb at msu.edu