[TIP] Tasting the Cheeseshop

[exarkun at twistedmatrix.com]

On 09:21 pm, barry at python.org wrote:
>On Oct 15, 2011, at 08:55 AM, Alfredo Deza wrote:
>>May I add that it would be ideal that however it is that this process
>>executes tests it provides a way that package authors
>>can use the same tools/processes to run them locally?
>
>Ideally, the tester would just be running the tests that the package 
>author
>itself sets up.  I still think `python setup.py test` should be the cli 
>for
>that.

One thing that turns me off of this approach is that various tools have 
taken to executing setup.py in all kinds of strange contexts, so it's 
not at all clear what expectations authors should have when trying to 
make `python setup.py test` work.

What parts of the project under test will be importable?  What kind of 
environment setup is `python setup.py test` responsible for?  Is `python 
setup.py test` supposed to work on a vanilla unpacked tarball, or is it 
expected to run only after `python setup.py install` has succeeded?

What if the test runner is part of the project under test?  What if 
there is already an installed version of the project, different from the 
one being tested?

What if there are dependencies needed before the tests can be run?

If you propose what you want to work in more detail, then maybe people 
will see that it's possible for them to provide the behavior.  It seems 
like it might avoid the need to rummage through a lot of baggage to just 
avoid the setup.py tarpit though.

Jean-Paul
>
>Now, that may not be the *only* interface for running tests, or that 
>may run
>just the subset of tests that can be completely automated.  So if you 
>develop
>with nose, subunit, py.test, unit2, or whatever, you would continue to 
>do so,
>but you'd make sure that `python setup.py test` worked, which I think
>generally shouldn't be difficult with any of the popular test runners.
>>One thing that would be annoying is this tester reporting failing
>>tests and you are unable to replicate, or unable to fix properly
>>because it may be a unique failure on the environment where it is
>>being executed.
>>
>>If for example we assume that tox would be in charge of this task then 
>>this
>>would solve that problem, because it would allow the dev to run the 
>>same
>>commands locally and provide fixes right away.
>
>Well, mostly.  There's a similar problem with the Python buildbots.  If 
>I
>commit a change that breaks Windows, I probably will not be able to 
>debug or
>fix it myself.  You might not be able to do the same with failures 
>identified
>by my ARM buildbot.  The hibernating snakebite.org aimed to solve that.
>
>Still, I think the reporting of such failures is useful to package 
>developers
>since at least they'll find out about it, as long as the reporting of 
>such
>failures doesn't become spammy.  E.g. no automatic bug filing.  Push
>notifications of test results is another interesting aspect to explore.
>
>-Barry