[TIP] countTests and progress bars.

[Robert Collins]

As I mentioned recently I'm drafting a new TestResult, intended to
overhaul the current one (while providing a backward compat path).

One thing the current TestResult does that doesn't work well in
practice is countTests.

countTests is usually used as a progress indicator : "14/132 tests run".

This has two issues:
 - it interacts poorly with subtests and other post-collection test
generators/multipliers. This includes 'tests' that actually run an
external script, or source results from some other test system. We may
have a system for resolving this ambiguity soon with the subtests
patch, but...
 - It is a poor estimator for test run progress. Test runs that are
not near-instant generally have huge variation in test performance.
2000/4000 tests executed in 3 minutes could mean 3 minutes to go, or
20. Folk running tests know their tests suites, and thus can see
2000/4000 and know it means '4 minutes thereabouts'.

The really valuable signal that is provided by the 14/132 style UI is
detecting hung tests.

Relatedly, for running cross-language and distributing its very handy
to be able to enumerate and run separately all the individual tests.
This provides both more data than countTests, but generally requires
poking underneath the TestSuite structure today.

I drafted an estimator for the new TestResult that doesn't require
enumerating up front, but I wonder - perhaps we can just delete the
concept entirely as an overly optimistic idea, and instead say:
 - we'll require enumeration support of TestCase/TestSuite (but not of subtests)
 - runners that want a 14/132 style UI can enumerate up front.
 - runners that want to be clever can remember the previous set of
tests and record timing data and present as sophisticated a UI as they
wish
 - runners that want to be fast can start running during discovery and
not enumerate at all.

Thoughts?
-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Cloud Services