[TIP] The test suite that was too dangerous to live

[Olemis Lang]

2009/9/27 Michał Kwiatkowski <constant.beta at gmail.com>:
> Hi!
>
> First, please check out this quote from Gerard Meszaros' excellent
> "xUnit Test Patterns" (also available on
> <http://xunitpatterns.com/Philosophy%20Of%20Test%20Automation.html#State
> or Behavior Verification?>)
>

Could be useful to read TAM [1]_ (the same author). You'll find
general guidelines
about testing automation frameworks and test artifacts.

> Needing to make a couple of changes to the API and underlying behaviour,
> I started in on the test suite, making changes in the expectations. Over
> the past several months they've grown into rather a forest, with many
> layers of indirection, dwarfing the size of the code they're testing;

Something like that happened to me when I started to write tests for
Trac plug-ins using the std style. Test cases were more complex and
than the actual implementation (I swear ... I just did it once ...)

> everything I did to my tests had unexpected consequences elsewhere in
> the test suite, leaving me scratching my head at the lack of insulation
> between test cases.

Oh ! yes ! Once again Meszaros wrote about that in the book (should we
build a monument or something ? :) mentioned by Michal K. (and I've
also experienced his too :-/ ). This is some kind of anti-pattern or
problem that occurs frequently, even for simple test suites,
especially when using «global APIs» .

For instance, few weeks ago there was a thread about testing and
warnings @ AFPy.fr . In this case the warnings mechanism is
centralized, stateful and may be manipulated by multiple packages,
which may setup global filters, ... and effects are propagated far
beyond. This kind of APIs may lead to many problems because they are a
potential source of non-determinism, and thus fragile tests.

There are more complex scenarios as well ...

.. [1] LNCS 2753 - The Test Automation Manifesto
        (http://www.springerlink.com/index/xnjn8fn1lwmae4hc.pdf)
        (http://xunitpatterns.com/~gerard/xpau2003-test-automation-manifesto-paper.pdf)

.. [2] Fragile Test at XUnitPatterns.com
        (http://xunitpatterns.com/Fragile%20Test.html)

-- 
Regards,

Olemis.

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:
Sobrepasa las 300 descargas el módulo dutest  -
http://feedproxy.google.com/~r/simelo-es/~3/U5rff5iTQPI/sobrepasa-las-300-descargas-el-modulo.html