[TIP] Testing Hierarchy

[Robert Collins]

On Mon, 2009-07-20 at 11:24 -0700, Noah Gift wrote:
> 
> It is very easy to say all code should have 100% unit test coverage,
> have integration tests, functional tests, etc.  What I haven't seen
> someone talk about yet, is when that is appropriate in the real world
> and when it isn't, and an honest assessment of the tradeoff.  I am
> very sold on testing code, but how much depends on the situation I am
> in.

In previous debates I've had with people about this, I've had people
argue that because their time is limited, they should do less or not
automated testing because testing is pure waste.

Now, I don't think that argument is being made here :). However, the
rebuttal to that point can provide some clues for choosing how much
testing is enough.

And my rebuttal of choice is: Testing, and automated testing
specifically is part of development - not a completely separate thing.
So the time for testing is coming from the same time allowance that lets
you sit in front of a function thinking 'does that do what I want'.

And from there it should be obvious: whenever spending time on testing
will help you be more confident that the system does what you want, and
you'd be willing to stare/think/write-code to increase your confidence,
you should be willing to increase test coverage too. I know this is
facile, but I find it really does work well as a rule of thumb. After
all one doesn't stop coding a function until you're confident it does
what you want, but you may stop while you still have some doubts, if the
risk is felt to be low/not worth reducing more.

For a given project, the amount of testing needed to get to this point
where you feel more testing isn't worth it will vary - and this is the
hard bit, I think. One can't simply say '100% coverage' or other similar
rules: they can at most be possible goals to aim for. As tests have
costs (time to maintain, time to execute) exactly the same as any new
functions do, you can often get to a situation where the cost of a new
test isn't worth the increased confidence it will add. 

-Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url : http://lists.idyll.org/pipermail/testing-in-python/attachments/20090721/af542d97/attachment.pgp