[TIP] proto-pep: plugin proposal (for unittest)

[Michael Foord]

For those who don't want to read the proto-pep, which outlines the *whole*
system and is a little bit long, I wrote up a *much* shorter intro to the
plugin system (including how to get the prototype) on my blog:

    http://www.voidspace.org.uk/python/weblog/arch_d7_2010_07_24.shtml#e1186

Michael

- Show quoted text -


On 29 July 2010 23:55, Michael Foord <fuzzyman at voidspace.org.uk> wrote:

> Hello all,
>
> My apologies in advance if email mangles whitespace in the code examples. I
> can reformulate as a PEP if that is deemed useful and this document can be
> found online at:
>
>    http://hg.python.org/unittest2/file/tip/description.txt
>
> (Please excuse errors and omissions - but do feel free to point them out.)
>
> This is a description, and request for feedback, of the unittest plugin
> system that I am currently prototyping in the plugins branch of unittest2_.
> My goal is to merge the plugin system back into unittest itself in Python
> 3.2.
>
> .. _unittest2: http://hg.python.org/unittest2
>
> As part of the prototype I have been implementing some example plugins (in
> unittest2/plugins/) so I can develop the mechanism against real rather than
> imagined use cases. Jason Pellerin, creator of nose, has been providing me
> with feedback and has been trying it out by porting some of the nose plugins
> to unittest [#]_. He comments on the system "it looks very flexible and
> clean". ;-)
>
> Example plugins available and included:
>
>    * a pep8 and pyflakes checker
>    * a debugger plugin that drops you into pdb on test fail / error
>    * a doctest loader (looks for doctests in all text files in the project)
>    * use a regex for matching files in test discovery instead of a glob
>    * growl notifications on test run start and stop
>    * filter individual *test methods* using a regex
>    * load test functions from modules as well as TestCases
>    * integration with the coverage module for coverage reporting
>
> In addition I intend to create a plugin that outputs junit compatible xml
> from a test run (for integration with tools like the hudson continuous
> integration server) and a test runner that runs tests in parallel using
> multiprocessing.
>
> Not all of these will be included in the merge to unittest. Which ones will
> is still an open question.
>
> I'd like feedback on the proposal, and hopefully approval to port it into
> unittest after discussion / amendment / completion. In particular I'd like
> feedback on the basic system, plus which events should be available and what
> information should be available in them. Note that the system is *not*
> complete in the prototype. Enough is implemented to get "the general idea"
> and to formalise the full system. It still needs extensive tests and the
> extra work in TestProgram makes it abundantly clear that refactoring there
> is well overdue...
>
> In the details below open questions and todos are noted. I *really* value
> feedback (but will ignore bikeshedding ;-)
>
> .. note::
>
>    Throughout this document I refer to the prototype implementation using
> names like ``unittest2.events.hooks``. Should this proposal be accepted then
> the names will live in the unittest package instead of unittest2.
>
>    The core classes for the event system live in the current implementation
> in the ``unittest2.events`` namespace.
>
>
> Abstract
> ========
>
> unittest lacks a standard way of extending it to provide commonly requested
> functionality, other than subclassing and overriding (and reimplementing)
> parts of its behaviour. This document describes a plugin system already
> partially prototyped in unittest2.
>
> Aspects of the plugin system include:
>
> * an events mechanism where handlers can be registered and called during a
> test run
> * a Plugin class built over the top of this for easy creation of plugins
> * a configuration system for specifying which plugins should be loaded and
> for configuring individual plugins
> * command line integration
> * the specific set of events and the information / actions available to
> them
>
> As the plugin system essentially just adds event calls to key places it has
> few backwards compatibility issues. Unfortunately existing extensions that
> override the parts of unittest that call these events will not be compatible
> with plugins that use them. Framework authors who re-implement parts of
> unittest, for example custom test runners, may want to add calling these
> events in appropriate places.
>
>
> Rationale
> =========
>
> Why a plugin system for unittest?
>
> unittest is the standard library test framework for Python but in recent
> years has been eclipsed in functionality by frameworks like nose and
> py.test. Among the reasons for this is that these frameworks are easier to
> extend with plugins than unittest. unittest makes itself particularly
> difficult to extend by using subclassing as its basic extension mechanism.
> You subclass and override behaviour in its core classes like the loader,
> runner and result classes.
>
> This means that where you have more than one "extension" working in the
> same area it is very hard for them to work together. Whilst various
> extensions to unittest do exist (e.g. testtools, zope.testrunner etc) they
> don't tend to work well together. In contrast the plugin system makes
> creating extensions to unittest much simpler and less likely that extensions
> will clash with each other.
>
> nose itself exists as a large system built over the top of unittest.
> Extending unittest in this way was very painful for the creators of nose,
> and every release of Python breaks nose in some way due to changes in
> unittest. One of the goals of the extension mechanism is to allow nose2 to
> be a much thinner set of plugins over unittest(2) that is much simpler to
> maintain [#]_. The early indications are that the proposed system is a good
> fit for this goal.
> ...
>
-- 
http://www.voidspace.org.uk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20100730/66dd6771/attachment.html>