[TIP] testmon - ideas to speed up execution of test suites

[Robert Collins]

On 17 March 2015 at 07:43, Thomi Richards <thomi.richards at canonical.com> wrote:
> Hi,
>
> On Sat, Mar 14, 2015 at 3:57 AM, Tibor Arpas <tibor.arpas at infinit.sk> wrote:
>>
>> E.g. it automatically selects only tests dependent on recent changes and
>> deselects all the others. The concept is quite general, technically feasible
>> and it's possible to finish this and also implement the same for other test
>> runners if there is interest.
>
>
>
> This seems.. too good to be true. Would you mind explaining a bit about how
> it works? Off the top of my head, I can't think of any reliable way to
> determine a list of test cases that need to be run, given a diff of source
> code. Am I missing something?

There was a thread on twitter about this recently
https://twitter.com/ncoghlan_dev/status/566373773830397952

The idea goes back years :).

The basic idea is to have an oracle that tells you working tree state
-> tests to run

Some oracles:
 - naming conventions; name tests such that you can tell the modules
they are relevant to.
   pros: easy to maintain, helps mental association to the importance of layers.
   cons: very easy to fail to run tests where unexpected layer
associations have formed
 - trace based: using a debug/profiling hook build a mapping of test X
ran lines Y. Whenever you run the test again
   update the database, and map backwards from diffs to the db. You
can in principle also use this to decide what
   tests need to run when changing a dependency, though I'm not aware
of anyone doing that yet.
   pros: much more reliable at determining what tests to run
   cons: have to build the db first, and maintain it as e.g. lines
move around, which makes first-time use expension
 - stochastically: run some subset of tests randomly, perhaps biased
by naming conventions or other data like most-recently changed. Aegis
uses this to select tests to run.
   pros: very simple
   cons: reproducability, and lack of coverage

There was an implementation of trace based selection put together for
LP about 8 years ago, and I did a thing for bzr shortly after that -
there are implementations all over the place ;). None in the Python
world that are generally reusable and consumable until recently AFAIK
though.

-Rob

-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud