[TIP] Tasting the Cheeseshop
exarkun at twistedmatrix.com
exarkun at twistedmatrix.com
Mon Oct 17 15:36:18 PDT 2011
On 09:21 pm, barry at python.org wrote:
>On Oct 15, 2011, at 08:55 AM, Alfredo Deza wrote:
>>May I add that it would be ideal that however it is that this process
>>executes tests it provides a way that package authors
>>can use the same tools/processes to run them locally?
>Ideally, the tester would just be running the tests that the package
>itself sets up. I still think `python setup.py test` should be the cli
One thing that turns me off of this approach is that various tools have
taken to executing setup.py in all kinds of strange contexts, so it's
not at all clear what expectations authors should have when trying to
make `python setup.py test` work.
What parts of the project under test will be importable? What kind of
environment setup is `python setup.py test` responsible for? Is `python
setup.py test` supposed to work on a vanilla unpacked tarball, or is it
expected to run only after `python setup.py install` has succeeded?
What if the test runner is part of the project under test? What if
there is already an installed version of the project, different from the
one being tested?
What if there are dependencies needed before the tests can be run?
If you propose what you want to work in more detail, then maybe people
will see that it's possible for them to provide the behavior. It seems
like it might avoid the need to rummage through a lot of baggage to just
avoid the setup.py tarpit though.
>Now, that may not be the *only* interface for running tests, or that
>just the subset of tests that can be completely automated. So if you
>with nose, subunit, py.test, unit2, or whatever, you would continue to
>but you'd make sure that `python setup.py test` worked, which I think
>generally shouldn't be difficult with any of the popular test runners.
>>One thing that would be annoying is this tester reporting failing
>>tests and you are unable to replicate, or unable to fix properly
>>because it may be a unique failure on the environment where it is
>>If for example we assume that tox would be in charge of this task then
>>would solve that problem, because it would allow the dev to run the
>>commands locally and provide fixes right away.
>Well, mostly. There's a similar problem with the Python buildbots. If
>commit a change that breaks Windows, I probably will not be able to
>fix it myself. You might not be able to do the same with failures
>by my ARM buildbot. The hibernating snakebite.org aimed to solve that.
>Still, I think the reporting of such failures is useful to package
>since at least they'll find out about it, as long as the reporting of
>failures doesn't become spammy. E.g. no automatic bug filing. Push
>notifications of test results is another interesting aspect to explore.
More information about the testing-in-python