[TIP] testmon - ideas to speed up execution of test suites

[Tibor Arpas]

Hi,

so after my email to TIP the *testmon* crowdfunding was successul, thanks!
And now there is a pretty stable version of testmon on PYPI
https://pypi.python.org/pypi/testmon/0.2.8 . Please try it and let us know
your feedback. Having the test-suite re-execute on every file save has a
great value and *testmon *is a big step towards that vision.

One of the things which turned out pretty well and I haven't seen
implemented anywhere else is our source code processing/versioning. It
easily and quickly tracks changes between test runs and not commits, so it
has better granularity and works with any VCS. I did a little write-up
about it : https://github.com/tarpas/testmon/wiki/Determining-affected-tests

Tibor



On Mon, Mar 16, 2015 at 8:46 PM, Robert Collins <robertc at robertcollins.net>
wrote:

> On 17 March 2015 at 07:43, Thomi Richards <thomi.richards at canonical.com>
> wrote:
> > Hi,
> >
> > On Sat, Mar 14, 2015 at 3:57 AM, Tibor Arpas <tibor.arpas at infinit.sk>
> wrote:
> >>
> >> E.g. it automatically selects only tests dependent on recent changes and
> >> deselects all the others. The concept is quite general, technically
> feasible
> >> and it's possible to finish this and also implement the same for other
> test
> >> runners if there is interest.
> >
> >
> >
> > This seems.. too good to be true. Would you mind explaining a bit about
> how
> > it works? Off the top of my head, I can't think of any reliable way to
> > determine a list of test cases that need to be run, given a diff of
> source
> > code. Am I missing something?
>
> There was a thread on twitter about this recently
> https://twitter.com/ncoghlan_dev/status/566373773830397952
>
> The idea goes back years :).
>
> The basic idea is to have an oracle that tells you working tree state
> -> tests to run
>
> Some oracles:
>  - naming conventions; name tests such that you can tell the modules
> they are relevant to.
>    pros: easy to maintain, helps mental association to the importance of
> layers.
>    cons: very easy to fail to run tests where unexpected layer
> associations have formed
>  - trace based: using a debug/profiling hook build a mapping of test X
> ran lines Y. Whenever you run the test again
>    update the database, and map backwards from diffs to the db. You
> can in principle also use this to decide what
>    tests need to run when changing a dependency, though I'm not aware
> of anyone doing that yet.
>    pros: much more reliable at determining what tests to run
>    cons: have to build the db first, and maintain it as e.g. lines
> move around, which makes first-time use expension
>  - stochastically: run some subset of tests randomly, perhaps biased
> by naming conventions or other data like most-recently changed. Aegis
> uses this to select tests to run.
>    pros: very simple
>    cons: reproducability, and lack of coverage
>
> There was an implementation of trace based selection put together for
> LP about 8 years ago, and I did a thing for bzr shortly after that -
> there are implementations all over the place ;). None in the Python
> world that are generally reusable and consumable until recently AFAIK
> though.
>
> -Rob
>
> --
> Robert Collins <rbtcollins at hp.com>
> Distinguished Technologist
> HP Converged Cloud
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20150420/a0e28335/attachment.html>