[TIP] countTests and progress bars.

Jonathan Lange jml at mumak.net
Wed Feb 27 01:43:57 PST 2013


On Sun, Feb 17, 2013 at 11:55 PM, Robert Collins
<robertc at robertcollins.net>wrote:

> As I mentioned recently I'm drafting a new TestResult, intended to
> overhaul the current one (while providing a backward compat path).
>
> One thing the current TestResult does that doesn't work well in
> practice is countTests.
>
> countTests is usually used as a progress indicator : "14/132 tests run".
>
> This has two issues:
>  - it interacts poorly with subtests and other post-collection test
> generators/multipliers. This includes 'tests' that actually run an
> external script, or source results from some other test system. We may
> have a system for resolving this ambiguity soon with the subtests
> patch, but...
>  - It is a poor estimator for test run progress. Test runs that are
> not near-instant generally have huge variation in test performance.
> 2000/4000 tests executed in 3 minutes could mean 3 minutes to go, or
> 20. Folk running tests know their tests suites, and thus can see
> 2000/4000 and know it means '4 minutes thereabouts'.
>
> The really valuable signal that is provided by the 14/132 style UI is
> detecting hung tests.
>
> Relatedly, for running cross-language and distributing its very handy
> to be able to enumerate and run separately all the individual tests.
> This provides both more data than countTests, but generally requires
> poking underneath the TestSuite structure today.
>
> I drafted an estimator for the new TestResult that doesn't require
> enumerating up front, but I wonder - perhaps we can just delete the
> concept entirely as an overly optimistic idea, and instead say:
>  - we'll require enumeration support of TestCase/TestSuite (but not of
> subtests)
>  - runners that want a 14/132 style UI can enumerate up front.
>  - runners that want to be clever can remember the previous set of
> tests and record timing data and present as sophisticated a UI as they
> wish
>  - runners that want to be fast can start running during discovery and
> not enumerate at all.
>
> Thoughts?
>
>
I agree with your logic and like your proposal. Makes me wonder whether
TestSuite really has an value at all.

jml
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20130227/91dd8296/attachment.html>


More information about the testing-in-python mailing list