[TIP] unittest2, distribute, and doctests

[Michael Foord]

On 11/06/2010 16:34, Barry Warsaw wrote:
> On Jun 07, 2010, at 11:34 PM, Michael Foord wrote:
>
>    
>> On 07/06/2010 22:43, Barry Warsaw wrote:
>>      
>>> Hey guys, I have a few quick questions about unittest2.  Apologies if these
>>> have been discussed before.
>>>
>>> Distribute adds the ability to run 'python setup.py test' by including the
>>> `test_suite` keyword to your setup().  What's the best way to integrate
>>> unittest2 and distribute to enable the former to do test discovery when run
>>> via the setup.py?
>>>        
>> I'm afraid that I don't know how to do that *currently*, given that I'm
>> not a user of setuptools or distribute.
>>
>> The *plan* is for distutils2 to include a test command that will
>> *default* to test discovery if unittest2 is available or Python 2.7+ is
>> being used.
>>      
> The unittest2.collector function you have in the hg trunk seems to work well,
> with a couple of caveats.  I've pushed a branch that uses unittest2 with my
> flufl.enum package:
>
> % bzr branch lp:~barry/flufl.enum/unit2
>
> First problem, unlike the "standard" distribute 'test' command, you have to
> build the package first.  Otherwise you get a traceback.  Full traceback is
> here:
>
> http://pastebin.ubuntu.com/448281/
>
> excerpted:
>
> % python setup.py test
> Traceback (most recent call last):
> ...
> ImportError: 'test_docs' module incorrectly imported from '/home/barry/projects/flufl/enum.ut2/build/lib.linux-x86_64-2.6/flufl/enum/tests'. Expected '/home/barry/projects/flufl/enum.ut2/flufl/enum/tests'. Is this module globally installed?
>
> However, if you 'python setup.py build' then 'python setup.py test' it works
> just fine.  I haven't investigated further.
>    

Ok, I can look into this. distribute / setuptools must be doing 
something odd with the path - when the collector imports the enum tests 
it is picking them up from the wrong location.

> Second caveat: It would be nice if distribute and unit2 could agree on an API
> for locating "extra" tests.  For example, in flufl.enum, I have a function
> that turns doctests into DocFileSuites with appropriate flags, setups, etc.
>
> def additional_tests():
>      "Run the doc tests (README.txt and docs/*, if any exist)"
>      doctest_files = [
>          os.path.abspath(resource_filename('flufl.enum', 'README.txt'))]
>      if resource_exists('flufl.enum', 'docs'):
>          for name in resource_listdir('flufl.enum', 'docs'):
>              if name.endswith('.txt'):
>                  doctest_files.append(
>                      os.path.abspath(
>                          resource_filename('flufl.enum', 'docs/%s' % name)))
>      kwargs = dict(module_relative=False,
>                    optionflags=DOCTEST_FLAGS,
>                    setUp=setup,
>                    )
>      atexit.register(cleanup_resources)
>      return unittest.TestSuite((
>          doctest.DocFileSuite(*doctest_files, **kwargs)))
>
> This adheres to what I think the Distribute API requires.  unit2's API is
> different so I had to add this little bit of glue:
>
> def load_tests(loader, tests, pattern):
>      """unittest2 API."""
>      return additional_tests()
>
> I'm not sure how to reconcile these.  It would be nice if at least Distribute
> and unit2 agreed on the function name, and then I could set it up to take
> default arguments.  I could kind of do this now:
>
> def additional_tests(loader=None, tests=None, pattern=None):
>      ...
>
> # This line shouldn't be necessary.
> load_tests = additional_tests
>
>
>    

Well... unittest2 is a backport of unittest in Python 2.7 which has just 
got to release candidate and isn't about to change dramatically. On top 
of that the load_tests protocol is more flexible than 'additional_tests' 
and so is *better*. setuptools isn't about to change either, so I don't 
know what we can do here?

The good news is that distutils2 will use unittest / unittest2 and so 
load_tests alone will be sufficient. Of course if you're actually 
*using* unittest2 then load_tests is still sufficient, and if you're not 
using unittest2 then you don't need it.


>> http://bugs.python.org/issue8324
>>      
>>> `python setup.py test` supports a -m and -s option to filter (minimally) the
>>> tests that are run.  Can unittest2's richer command line options be supported?
>>> (This may be more of a distutils/setuptools/distribute question.)
>>>
>>>        
>> My understanding is that some of the discovery options will be
>> configurable. I defer to Tarek and his GSOC team who are actually
>> implementing for a definitive answer though.
>>      
> Cool.  Tarek, do you want to move that discussion to distutils-sig?
>
>    
>>> unit2 -p takes shell globs, but I really like zope.testrunner's regexp-based
>>> filters.  Any chance unittest2 will support something like that?
>>>        
>> Hmmm... as an intermediate step I exposed the filename matching as a
>> method on the TestLoader (_match_path) so that subclasses can be
>> overridden. After the release of Python 2.7 my next goal is to make
>> unittest extensible and file matching would be one of the extension
>> points (so regex file matching would be trivially easy to provide as an
>> extension).
>>
>> I'm not sure about building it into unit2 / unittest itself though. Can
>> you give me an example of something you want to do that requires regexes
>> and doesn't work with shell globs.
>>      
> There are a couple of issues.  First, in some cases, regexps are actually
> *easier* to type and more user friendly.  For example if I just want to run
> the tests in my foobar subsystem, I can generally just type:
>
> % bin/test -vv -t foobar
>
> and that will pick up all test_foobar_* tests.  This also picks up all the
> doctests with foobar in their name, e.g. foobar_api.txt, using_foobar.txt and
> so on.
>
> Second, I think regexps map better to the way you (well, at least *I*<wink>)
> think about your tests than file globs.  As in the example above, I care much
> less about the file names that my tests happen to be in than I do about the
> test names.  Zope's test runner's provides some nice ability to actually match
> against either IIRC, but I tend to use -t because I want to run a subset of
> individual unit or doc tests irrespective of the file they happen to live in
> at the moment.
>
> Third, I've found that for a very big system like Launchpad, regexps (along
> with the ability to specify multiple -t options) provides the flexibility to
> hone the tests down to exactly what you care about.  E.g.
>
> % bin/test -vv -t foo_(api|basic)
>
> would run tests like foo_api_version, foo_api_backwards_compatible,
> foo_basic_wsgi, foo_basic_rest, etc.  This is very nice when you're only
> working on a small part of a large system.
>    

Right. So I'm convinced that sometimes regular expressions are sometimes 
better. I don't think that we can change the default and I'm not sure 
that proliferating command line options is ideal either. I would like a 
way of filtering tests from the command line though.

Filtering files like this - including loading text files and turning 
them into doctests - should be straightforward with the plugin mechanism 
I have 'in mind'. It could initially be developed as a plugin, but a 
standard set of useful plugins could ship with unittest / unittest2. 
These could be enabled globally with a user config file or on a per 
project basis. They could even (conceivably) add new command line 
options (both nose and py.test have systems similar to this).


> Of all the test runners out there, I really do like Zope's command line
> interface the best.  Just something to keep in mind I guess.
>
>    
I'm not at all familiar with it, but would certainly appreciate input on 
any improvements to the unittest(2) command line interface. Specific 
feature requests (or even just one explaining which features from the 
zope test runner you particularly like) would be helpful - either on the 
Python or the unittest2 bug tracker. I'll try and have a look at it 
before I make any further changes.


>>> One of the
>>> reasons I like zope's test discovery is because it works much better with
>>> doctests.  E.g. I can do (after a buildout):
>>>
>>>       % bin/test -vv -t using
>>>
>>> and it will only run my using.txt doctest.  Note that I have a test_docs.py
>>> module that does individual .txt doctest discovery, wrapping each one in a
>>> DocFileSuite with appropriate optionflags, and setUp.  IWBNI unittest2
>>> eventually grew better support for doctests.
>>>
>>>        
>> I'd be interested in hearing how it should grow that support. Again, it
>> is something ripe for being implemented as an extension - but I'd like
>> to see better doctest support in the test runner.
>>      
> Cool.  Part of it is the command line interface outlined above, but the other
> critical part is the ability to set doctest flags and a setup function.
> Basically, we need a convention for turning doctests into DocFileSuites, along
> with of course the discovery of the appropriate .txt or .rst files.  I don't
> much mind having to write that little additional_tests() function given above,
> but I tend to cargo cult that into every package I write, so it would be nice
> to refactor that into the infrastructure.  I'll have to think about a nice API
> for generalizing it.
>
>    

Sounds good. Again, the plugin machinery *must* be flexible enough to 
allow this and it could eventually ship as a standard plugin with unittest.

>>> Is it intentional that a module's load_tests() not called when unit2 -p is
>>> used?
>>>
>>>        
>> No - that is definitely not intentional and from the code I can't see
>> how that is possible, nor can I reproduce it. Can you provide a minimal
>> repro? The way load_tests is called from __init__.py is a bit weird
>> though and I'm not happy with it. Perhaps this is tripping you up?
>>      
> I think I understand a little more about it.  With a load_tests() hacked to
> print something...
>
> % unit2 discover -v
> in load_tests()
> /home/barry/projects/flufl/enum.ut2/flufl/enum/README.txt
> Doctest: README.txt ... ok
> /home/barry/projects/flufl/enum.ut2/flufl/enum/docs/using.txt
> Doctest: using.txt ... ok
>
> ----------------------------------------------------------------------
> Ran 2 tests in 0.014s
>
> OK
> % unit2 discover -v -p using\*
>
> ----------------------------------------------------------------------
> Ran 0 tests in 0.000s
>
> OK
> % unit2 discover -v -p *docs*
> in load_tests()
> /home/barry/projects/flufl/enum.ut2/flufl/enum/README.txt
> Doctest: README.txt ... ok
> /home/barry/projects/flufl/enum.ut2/flufl/enum/docs/using.txt
> Doctest: using.txt ... ok
>
> ----------------------------------------------------------------------
> Ran 2 tests in 0.010s
>
> OK
>
>
> So the problem is probably related to not calling load_tests() when the
> pattern doesn't match a discovered file name.

Right - load_tests is only called for modules (files) that are actually 
loaded.

>    The bummer about that is that I
> was going to try to use load_tests()'s pattern argument to filter the doctests
> that got run, since it makes sense that unit2 doesn't (yet<wink>) know about
> doctest file names.
>
>    

The pattern currently filters files, not tests. I'd like to be able to 
filter tests as well. Loading tests from text files will have to be the 
work of an extension.

All the best,

Michael

> Thanks for all the great work,
> -Barry
>    


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.