<div dir="ltr">The more I think about it and experiment around, the more I'm convinced the trace based (coverage.<span class="" id=":3fv.2" tabindex="-1"><span class="" id=":3fv.13" tabindex="-1">py</span></span>) approach <span style="background-color:rgb(255,255,0)">is</span> the best. On the couple of projects I tested the computational overhead was ~20%. That is a small price to pay, given that you get orders of magnitude speed up for incremental changes.<div><div><br></div><div>It's a pity the implementations are all over the place and even everybody calls it some other name. The best terminology I bumped into was "incremental" test-<span class="" id=":3fv.5" tabindex="-1"><span class="" id=":3fv.14" tabindex="-1">runnner</span></span>.</div></div><div><br></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">There was a thread on twitter about this recently<br>
<a href="https://twitter.com/ncoghlan_dev/status/566373773830397952" target="_blank">https://twitter.com/ncoghlan_dev/status/566373773830397952</a></blockquote><div><br></div><div>Thanks! So apparently Ruby has no finished product like that and some papers call the concept "Regression Test Selection"</div><div>The sentiment expressed in the article is exactly what led us to work on this:</div><div><span style="color:rgb(51,51,51);font-family:'Helvetica Neue',Helvetica,Arial,sans-serif;font-size:14px;line-height:20px"><br></span></div><div><span style="color:rgb(51,51,51);font-family:'Helvetica Neue',Helvetica,Arial,sans-serif;font-size:14px;line-height:20px">Running tests is the worst. Seriously. It takes forever, and by the time they’re all done running, I forgot what I was doing. Some apps take 20 to 30 min to run all the tests, and I just can’t wait that long. What bothers me even more than waiting so long, is that after I make a change to the code, 99% of the tests aren’t even running the code that I changed! Why am I spending time running code that is unrelated to the change I made?</span><br></div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>
<br>
The idea goes back years :).<br>
<br>
The basic idea is to have an oracle that tells you working tree state<br>
-> tests to run<br>
<br>
Some oracles:<br>
- naming conventions; name tests such that you can tell the modules<br>
they are relevant to.<br>
pros: easy to maintain, helps mental association to the importance of layers.<br>
cons: very easy to fail to run tests where unexpected layer<br>
associations have formed<br>
- trace based: using a debug/profiling hook build a mapping of test X<br>
ran lines Y. Whenever you run the test again<br>
update the database, and map backwards from diffs to the db. You<br>
can in principle also use this to decide what<br>
tests need to run when changing a dependency, though I'm not aware<br>
of anyone doing that yet.<br>
pros: much more reliable at determining what tests to run<br>
cons: have to build the db first, and maintain it as e.g. lines<br>
move around, which makes first-time use expension<br>
- stochastically: run some subset of tests randomly, perhaps biased<br>
by naming conventions or other data like most-recently changed. Aegis<br>
uses this to select tests to run.<br>
pros: very simple<br>
cons: reproducability, and lack of coverage<br>
<br>
There was an implementation of trace based selection put together for<br>
LP about 8 years ago, and I did a thing for bzr shortly after that -<br>
there are implementations all over the place ;). None in the Python<br>
world that are generally reusable and consumable until recently AFAIK<br>
though.<br>
<br>
-Rob<br>
<span class=""><font color="#888888"><br>
--<br>
Robert Collins <<a href="mailto:rbtcollins@hp.com">rbtcollins@hp.com</a>><br>
Distinguished Technologist<br>
HP Converged Cloud<br>
</font></span></blockquote></div><br></div></div>