[TIP] unittest2, distribute, and doctests
Barry Warsaw
barry at python.org
Fri Jun 11 08:34:11 PDT 2010
On Jun 07, 2010, at 11:34 PM, Michael Foord wrote:
>On 07/06/2010 22:43, Barry Warsaw wrote:
>> Hey guys, I have a few quick questions about unittest2. Apologies if these
>> have been discussed before.
>>
>> Distribute adds the ability to run 'python setup.py test' by including the
>> `test_suite` keyword to your setup(). What's the best way to integrate
>> unittest2 and distribute to enable the former to do test discovery when run
>> via the setup.py?
>
>I'm afraid that I don't know how to do that *currently*, given that I'm
>not a user of setuptools or distribute.
>
>The *plan* is for distutils2 to include a test command that will
>*default* to test discovery if unittest2 is available or Python 2.7+ is
>being used.
The unittest2.collector function you have in the hg trunk seems to work well,
with a couple of caveats. I've pushed a branch that uses unittest2 with my
flufl.enum package:
% bzr branch lp:~barry/flufl.enum/unit2
First problem, unlike the "standard" distribute 'test' command, you have to
build the package first. Otherwise you get a traceback. Full traceback is
here:
http://pastebin.ubuntu.com/448281/
excerpted:
% python setup.py test
Traceback (most recent call last):
...
ImportError: 'test_docs' module incorrectly imported from '/home/barry/projects/flufl/enum.ut2/build/lib.linux-x86_64-2.6/flufl/enum/tests'. Expected '/home/barry/projects/flufl/enum.ut2/flufl/enum/tests'. Is this module globally installed?
However, if you 'python setup.py build' then 'python setup.py test' it works
just fine. I haven't investigated further.
Second caveat: It would be nice if distribute and unit2 could agree on an API
for locating "extra" tests. For example, in flufl.enum, I have a function
that turns doctests into DocFileSuites with appropriate flags, setups, etc.
def additional_tests():
"Run the doc tests (README.txt and docs/*, if any exist)"
doctest_files = [
os.path.abspath(resource_filename('flufl.enum', 'README.txt'))]
if resource_exists('flufl.enum', 'docs'):
for name in resource_listdir('flufl.enum', 'docs'):
if name.endswith('.txt'):
doctest_files.append(
os.path.abspath(
resource_filename('flufl.enum', 'docs/%s' % name)))
kwargs = dict(module_relative=False,
optionflags=DOCTEST_FLAGS,
setUp=setup,
)
atexit.register(cleanup_resources)
return unittest.TestSuite((
doctest.DocFileSuite(*doctest_files, **kwargs)))
This adheres to what I think the Distribute API requires. unit2's API is
different so I had to add this little bit of glue:
def load_tests(loader, tests, pattern):
"""unittest2 API."""
return additional_tests()
I'm not sure how to reconcile these. It would be nice if at least Distribute
and unit2 agreed on the function name, and then I could set it up to take
default arguments. I could kind of do this now:
def additional_tests(loader=None, tests=None, pattern=None):
...
# This line shouldn't be necessary.
load_tests = additional_tests
>http://bugs.python.org/issue8324
>> `python setup.py test` supports a -m and -s option to filter (minimally) the
>> tests that are run. Can unittest2's richer command line options be supported?
>> (This may be more of a distutils/setuptools/distribute question.)
>>
>My understanding is that some of the discovery options will be
>configurable. I defer to Tarek and his GSOC team who are actually
>implementing for a definitive answer though.
Cool. Tarek, do you want to move that discussion to distutils-sig?
>> unit2 -p takes shell globs, but I really like zope.testrunner's regexp-based
>> filters. Any chance unittest2 will support something like that?
>
>Hmmm... as an intermediate step I exposed the filename matching as a
>method on the TestLoader (_match_path) so that subclasses can be
>overridden. After the release of Python 2.7 my next goal is to make
>unittest extensible and file matching would be one of the extension
>points (so regex file matching would be trivially easy to provide as an
>extension).
>
>I'm not sure about building it into unit2 / unittest itself though. Can
>you give me an example of something you want to do that requires regexes
>and doesn't work with shell globs.
There are a couple of issues. First, in some cases, regexps are actually
*easier* to type and more user friendly. For example if I just want to run
the tests in my foobar subsystem, I can generally just type:
% bin/test -vv -t foobar
and that will pick up all test_foobar_* tests. This also picks up all the
doctests with foobar in their name, e.g. foobar_api.txt, using_foobar.txt and
so on.
Second, I think regexps map better to the way you (well, at least *I* <wink>)
think about your tests than file globs. As in the example above, I care much
less about the file names that my tests happen to be in than I do about the
test names. Zope's test runner's provides some nice ability to actually match
against either IIRC, but I tend to use -t because I want to run a subset of
individual unit or doc tests irrespective of the file they happen to live in
at the moment.
Third, I've found that for a very big system like Launchpad, regexps (along
with the ability to specify multiple -t options) provides the flexibility to
hone the tests down to exactly what you care about. E.g.
% bin/test -vv -t foo_(api|basic)
would run tests like foo_api_version, foo_api_backwards_compatible,
foo_basic_wsgi, foo_basic_rest, etc. This is very nice when you're only
working on a small part of a large system.
Of all the test runners out there, I really do like Zope's command line
interface the best. Just something to keep in mind I guess.
>> One of the
>> reasons I like zope's test discovery is because it works much better with
>> doctests. E.g. I can do (after a buildout):
>>
>> % bin/test -vv -t using
>>
>> and it will only run my using.txt doctest. Note that I have a test_docs.py
>> module that does individual .txt doctest discovery, wrapping each one in a
>> DocFileSuite with appropriate optionflags, and setUp. IWBNI unittest2
>> eventually grew better support for doctests.
>>
>
>I'd be interested in hearing how it should grow that support. Again, it
>is something ripe for being implemented as an extension - but I'd like
>to see better doctest support in the test runner.
Cool. Part of it is the command line interface outlined above, but the other
critical part is the ability to set doctest flags and a setup function.
Basically, we need a convention for turning doctests into DocFileSuites, along
with of course the discovery of the appropriate .txt or .rst files. I don't
much mind having to write that little additional_tests() function given above,
but I tend to cargo cult that into every package I write, so it would be nice
to refactor that into the infrastructure. I'll have to think about a nice API
for generalizing it.
>> Is it intentional that a module's load_tests() not called when unit2 -p is
>> used?
>>
>No - that is definitely not intentional and from the code I can't see
>how that is possible, nor can I reproduce it. Can you provide a minimal
>repro? The way load_tests is called from __init__.py is a bit weird
>though and I'm not happy with it. Perhaps this is tripping you up?
I think I understand a little more about it. With a load_tests() hacked to
print something...
% unit2 discover -v
in load_tests()
/home/barry/projects/flufl/enum.ut2/flufl/enum/README.txt
Doctest: README.txt ... ok
/home/barry/projects/flufl/enum.ut2/flufl/enum/docs/using.txt
Doctest: using.txt ... ok
----------------------------------------------------------------------
Ran 2 tests in 0.014s
OK
% unit2 discover -v -p using\*
----------------------------------------------------------------------
Ran 0 tests in 0.000s
OK
% unit2 discover -v -p *docs*
in load_tests()
/home/barry/projects/flufl/enum.ut2/flufl/enum/README.txt
Doctest: README.txt ... ok
/home/barry/projects/flufl/enum.ut2/flufl/enum/docs/using.txt
Doctest: using.txt ... ok
----------------------------------------------------------------------
Ran 2 tests in 0.010s
OK
So the problem is probably related to not calling load_tests() when the
pattern doesn't match a discovered file name. The bummer about that is that I
was going to try to use load_tests()'s pattern argument to filter the doctests
that got run, since it makes sense that unit2 doesn't (yet <wink>) know about
doctest file names.
Thanks for all the great work,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20100611/56b5e3e7/attachment.pgp>
More information about the testing-in-python
mailing list