[TIP] Ideology

[Ned Batchelder]

I'm coming late to this, but it seems like a classic dilemma.  I've run 
into the same problem: as my testing efforts intensify, my tests become 
more complex.  The natural impulse is to factor common complexity out of 
the tests into something else.  That other thing is the test framework.  
Ideally, I'd "refactor" it all the way into someone else's code (for 
example, nose).  But depending on the project and its audience and 
distribution, I may not want to require another dependency, so I have to 
make the tradeoff between maintaining my own framework (however slight), 
or requiring a third-party package with all of those headaches (tracking 
changes, helping my customers install someone else's package, etc).

> > But don't you then have to debug your test framework?

> Your test framework should have unit tests - and yes, you'll need to
> treat it like any other piece of software, tests, docs, and debugging.

I personally haven't ever gotten to the point of needing separate tests 
just for the test scaffolding.  Generally, the product tests themselves 
will alert me to failures in the test runner.  I can see a large enough 
project getting to that point, though: there is the risk that your test 
scaffold will break in a way that means all tests pass without actually 
checking results.

> I'd rather it be simple. No complexity allowed. Didn't you read my blog post??

I'm assuming this is hyperbole for dramatic effect.  Of course we all 
accept (and even create) complexity.  The trick is to find the 
Goldilocks point: not too little, not too much, just the right amount.  
And then to wrap it in abstractions that don't leak too much.

> A lot of this sounds like something I'd use nose for, but we're trying to keep things stdlib-ish.

Maybe this is the point to examine more closely.  The best way to get 
rid of complexity is to outsource it to someone else.  Then you don't 
have to maintain it, but you still get to benefit from its power.  What 
is it that is keeping you from requiring nose? Is it sacrilege to say 
that it's ok to require 3rd-party stuff to run tests, since not all 
users of the product will need to run the tests?

--Ned.

-- 
Ned Batchelder, http://nedbatchelder.com