[TIP] testmon - ideas to speed up execution of test suites

[Tibor Arpas]

The more I think about it and experiment around, the more I'm convinced the
trace based (coverage.py) approach is the best. On the couple of projects I
tested the computational overhead was ~20%. That is a small price to pay,
given that you get orders of magnitude speed up for incremental changes.

It's a pity the implementations are all over the place and even everybody
calls it some other name. The best terminology I bumped into was
"incremental" test-runnner.

There was a thread on twitter about this recently
> https://twitter.com/ncoghlan_dev/status/566373773830397952


Thanks! So apparently Ruby has no finished product like that and some
papers call the concept "Regression Test Selection"
The sentiment expressed in the article is exactly what led us to work on
this:

Running tests is the worst. Seriously. It takes forever, and by the time
they’re all done running, I forgot what I was doing. Some apps take 20 to
30 min to run all the tests, and I just can’t wait that long. What bothers
me even more than waiting so long, is that after I make a change to the
code, 99% of the tests aren’t even running the code that I changed! Why am
I spending time running code that is unrelated to the change I made?



>
> The idea goes back years :).
>
> The basic idea is to have an oracle that tells you working tree state
> -> tests to run
>
> Some oracles:
>  - naming conventions; name tests such that you can tell the modules
> they are relevant to.
>    pros: easy to maintain, helps mental association to the importance of
> layers.
>    cons: very easy to fail to run tests where unexpected layer
> associations have formed
>  - trace based: using a debug/profiling hook build a mapping of test X
> ran lines Y. Whenever you run the test again
>    update the database, and map backwards from diffs to the db. You
> can in principle also use this to decide what
>    tests need to run when changing a dependency, though I'm not aware
> of anyone doing that yet.
>    pros: much more reliable at determining what tests to run
>    cons: have to build the db first, and maintain it as e.g. lines
> move around, which makes first-time use expension
>  - stochastically: run some subset of tests randomly, perhaps biased
> by naming conventions or other data like most-recently changed. Aegis
> uses this to select tests to run.
>    pros: very simple
>    cons: reproducability, and lack of coverage
>
> There was an implementation of trace based selection put together for
> LP about 8 years ago, and I did a thing for bzr shortly after that -
> there are implementations all over the place ;). None in the Python
> world that are generally reusable and consumable until recently AFAIK
> though.
>
> -Rob
>
> --
> Robert Collins <rbtcollins at hp.com>
> Distinguished Technologist
> HP Converged Cloud
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20150316/b0b53a0a/attachment-0001.htm>