[TIP] why you should distribute tests with your application / module

[C. Titus Brown]

On Wed, Sep 17, 2008 at 12:30:01PM -0500, Bob Clancy wrote:
-> On Wed, Sep 17, 2008 at  1:19 PM, C. Titus Brown wrote:
-> >Hmm, wait, let's distinguish between regression tests and functional
-> >tests.
-> 
-> Perhaps my "conceptual terminology" is a little different.  I'd tend to 
-> categorize functional (acceptance) tests from prior interations as 
-> regression tests in iterations past the one they were 
-> developed/implemented in.  Would you be willing to clarify the 
-> distinction you make a bit more?

OK, let's see :)

First, I see unit, functional, and regression tests as
programmer-facing, that is, about the *code*, while acceptance tests are
more customer-facing.  If you don't have a customer defining separate
tests that your software must pass in order to be "accepted", then
they're not acceptance tests -- note that this means you can't have
acceptance tests if you're your own customer :).

The breakdown between unit and functional is less important to me (I
would say that unit tests test small units of code, functional tests
test entire code paths).  I regard them both as "white box" tests,
however: I am using intimate knowledge of code and functions to build
setup/teardown/test code to make sure that things are behaving *as I
expect them to* from knowledge of the code and my design goals.

Regression tests are more *black box*.  I run the code as a separate
script or executable against lots of input conditions, and try to
manually verify that I get correct output.  Whether or not I can verify
that, I then record that pairing of input and output and keep on
re-running those tests whenever I can.  That tells me if my code is
behaving *differently* now than it was before.

It's really all about expectations, IMO:

Unit and functional tests tell me when the code fails my expectations.

Regression tests tell me when something unexpected happens.

Acceptance tests tell me when the code fails my customer's expectations.

--

Note that Wikipedia has a broader definition of "regression tests":

	http://en.wikipedia.org/wiki/Regression_testing

cheers,
--titus
-- 
C. Titus Brown, ctb at msu.edu