[TIP] testmon - ideas to speed up execution of test suites
Robert Collins
robertc at robertcollins.net
Mon Mar 16 12:46:22 PDT 2015
On 17 March 2015 at 07:43, Thomi Richards <thomi.richards at canonical.com> wrote:
> Hi,
>
> On Sat, Mar 14, 2015 at 3:57 AM, Tibor Arpas <tibor.arpas at infinit.sk> wrote:
>>
>> E.g. it automatically selects only tests dependent on recent changes and
>> deselects all the others. The concept is quite general, technically feasible
>> and it's possible to finish this and also implement the same for other test
>> runners if there is interest.
>
>
>
> This seems.. too good to be true. Would you mind explaining a bit about how
> it works? Off the top of my head, I can't think of any reliable way to
> determine a list of test cases that need to be run, given a diff of source
> code. Am I missing something?
There was a thread on twitter about this recently
https://twitter.com/ncoghlan_dev/status/566373773830397952
The idea goes back years :).
The basic idea is to have an oracle that tells you working tree state
-> tests to run
Some oracles:
- naming conventions; name tests such that you can tell the modules
they are relevant to.
pros: easy to maintain, helps mental association to the importance of layers.
cons: very easy to fail to run tests where unexpected layer
associations have formed
- trace based: using a debug/profiling hook build a mapping of test X
ran lines Y. Whenever you run the test again
update the database, and map backwards from diffs to the db. You
can in principle also use this to decide what
tests need to run when changing a dependency, though I'm not aware
of anyone doing that yet.
pros: much more reliable at determining what tests to run
cons: have to build the db first, and maintain it as e.g. lines
move around, which makes first-time use expension
- stochastically: run some subset of tests randomly, perhaps biased
by naming conventions or other data like most-recently changed. Aegis
uses this to select tests to run.
pros: very simple
cons: reproducability, and lack of coverage
There was an implementation of trace based selection put together for
LP about 8 years ago, and I did a thing for bzr shortly after that -
there are implementations all over the place ;). None in the Python
world that are generally reusable and consumable until recently AFAIK
though.
-Rob
--
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud
More information about the testing-in-python
mailing list