[TIP] Ideology

[Jesse Noller]

On Tue, Apr 21, 2009 at 11:34 AM, C. Titus Brown <ctb at msu.edu> wrote:

> -> Maybe if you gave more detail about the nature of the complexity
> -> you're thinking about adding we could give more direct answers.
>
> It's across the board -- cleanup of intermediate files vs no cleanup;
> pdb integration; order randomization; hooks to run test functions
> before/after each test; test tagging; verification of what tests have
> been run; test skipping; etc.

Cleanup of intermediate files should be handled in the teardown() of
the individuals tests :)
pdb: use nose
order randomization: nose
hooks to run test functions: nose
test tagging: nose
verification of what tests have been run: nose
test skipping: nose

> A lot of this sounds like something I'd use nose for, but we're trying
> to keep things stdlib-ish.  I've been arguing for keeping the test
> runner as simple as possible and using nose with plugins for some of the
> specialized stuff, but the boundary between customizing the test runner
> and making use of nose plugins is still under discussion.  (Hence this
> plaintive e-mail.)

You can see my bias above - I'd rather have it done by the runner (so
I can alter behavior there, in a plugin) than anywhere else in my code
base. In my mind, these are things the runner should manage.

Something *like* nose supports plugins to alter behavior for this
express reason, and you can compartmentalize each "thing" in a plugin
of it's own so that you can cleanly separate the concerns and not have
a monolith.

Well, that and you can optionally enable/disable where needed. I'm
obviously biased; having been burned by writing a giant framework
monolith, and not doing things this way, I may have run to the other
extreme (a little bit).

> One specific example -- we have our own 'get_suite' functions for each
> test file, so that we can skip test suites when certain programs aren't
> installed.

Hmm. A class level Skiptest would be nice. I have needs for this exact
thing as well - right now I just have a hook in the setup() to raise
skiptest if a particular resource is missing.

>  I proposed to add unittest TestLoader test selection code
> into our test runner (TestLoader.loadTestsFromName) so that we could run
> individual tests easily, but that breaks the test suite skipping built
> into get_suite().  So now I can backpatch SkipTest from python trunk
> into a customized unittest version, OR I can develop a specialized test
> loader, OR ... probably a lot of other things.

Ew. I wouldn't keep your own version of unittest around. Why not
enhance the nose test runner?

Yeah, I know, obvious bias, but your problems definitely seem to be
the responsibility of the runner - adding plugins to override behavior
makes sense to me in this case.

jesse