<div dir="ltr">On Sun, Feb 17, 2013 at 11:55 PM, Robert Collins <span dir="ltr"><<a href="mailto:robertc@robertcollins.net" target="_blank">robertc@robertcollins.net</a>></span> wrote:<br><div class="gmail_extra"><div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">As I mentioned recently I'm drafting a new TestResult, intended to<br>
overhaul the current one (while providing a backward compat path).<br>
<br>
One thing the current TestResult does that doesn't work well in<br>
practice is countTests.<br>
<br>
countTests is usually used as a progress indicator : "14/132 tests run".<br>
<br>
This has two issues:<br>
- it interacts poorly with subtests and other post-collection test<br>
generators/multipliers. This includes 'tests' that actually run an<br>
external script, or source results from some other test system. We may<br>
have a system for resolving this ambiguity soon with the subtests<br>
patch, but...<br>
- It is a poor estimator for test run progress. Test runs that are<br>
not near-instant generally have huge variation in test performance.<br>
2000/4000 tests executed in 3 minutes could mean 3 minutes to go, or<br>
20. Folk running tests know their tests suites, and thus can see<br>
2000/4000 and know it means '4 minutes thereabouts'.<br>
<br>
The really valuable signal that is provided by the 14/132 style UI is<br>
detecting hung tests.<br>
<br>
Relatedly, for running cross-language and distributing its very handy<br>
to be able to enumerate and run separately all the individual tests.<br>
This provides both more data than countTests, but generally requires<br>
poking underneath the TestSuite structure today.<br>
<br>
I drafted an estimator for the new TestResult that doesn't require<br>
enumerating up front, but I wonder - perhaps we can just delete the<br>
concept entirely as an overly optimistic idea, and instead say:<br>
- we'll require enumeration support of TestCase/TestSuite (but not of subtests)<br>
- runners that want a 14/132 style UI can enumerate up front.<br>
- runners that want to be clever can remember the previous set of<br>
tests and record timing data and present as sophisticated a UI as they<br>
wish<br>
- runners that want to be fast can start running during discovery and<br>
not enumerate at all.<br>
<br>
Thoughts?<br><br></blockquote><div><br></div><div style>I agree with your logic and like your proposal. Makes me wonder whether TestSuite really has an value at all. </div><div style><br></div><div style>jml</div></div>
<br></div></div>