[TIP] why you should distribute tests with your application / module

[C. Titus Brown]

On Wed, Sep 17, 2008 at 10:06:59AM -0500, Kumar McMillan wrote:
-> On Tue, Sep 16, 2008 at 3:07 PM, Jesse Noller <jnoller at gmail.com> wrote:
-> > On Tue, Sep 16, 2008 at 3:49 PM, Pete <pfein at pobox.com> wrote:
-> >> What about fixture data though?  That can easily get larger than the
-> >> size of the rest of your distribution...
-> >>
-> > Why not generate the fixture data on the fly though?
-> 
-> This is a good point about distributing fixture data.  I can see for
-> an audio based package you might have wav files in your test suite
-> that you probably wouldn't want to distribute.  As for generating
-> fixtures on the fly, there could be a speed penalty for large datasets
-> that would otherwise do just fine in a static text file.  However, I
-> think in most cases fixture data is pretty small and zip does a good
-> job at text compression.  Distributing fixtures is probably ok most of
-> the time.

Hmm, wait, let's distinguish between regression tests and functional
tests.

Regression tests are tests that you run against some static data.  You
then compare the results of that analysis with prior results.  Whether
or not the prior results were *correct* -- the usual assumption is that
they were, of course! -- you can then figure out if something has
*changed*.  Usually unwanted change is bad.

Functional tests are really more about testing specific functionality in
your code.  So there, the difference betweeen creating static fixture
data or generating it on the fly only matters insofar as you can do one
or the other easily.

If you need to include large datasets with functional tests, I'd guess
that pseudorandom generation might be the way I would go, if only to
maximize your testing flexibility -- look, ma, I can generate whole new
test data sets just by changing the seed!

But obviously YMMV.  Asking for "best practices" is less important than
asking about "practices that work for me" IMO.

--titus
-- 
C. Titus Brown, ctb at msu.edu