[TIP] Tasting the Cheeseshop

[Barry Warsaw]

On Oct 15, 2011, at 08:55 AM, Alfredo Deza wrote:

>May I add that it would be ideal that however it is that this process
>executes tests it provides a way that package authors
>can use the same tools/processes to run them locally?

Ideally, the tester would just be running the tests that the package author
itself sets up.  I still think `python setup.py test` should be the cli for
that.

Now, that may not be the *only* interface for running tests, or that may run
just the subset of tests that can be completely automated.  So if you develop
with nose, subunit, py.test, unit2, or whatever, you would continue to do so,
but you'd make sure that `python setup.py test` worked, which I think
generally shouldn't be difficult with any of the popular test runners.

>One thing that would be annoying is this tester reporting failing
>tests and you are unable to replicate, or unable to fix properly
>because it may be a unique failure on the environment where it is
>being executed.
>
>If for example we assume that tox would be in charge of this task then this
>would solve that problem, because it would allow the dev to run the same
>commands locally and provide fixes right away.

Well, mostly.  There's a similar problem with the Python buildbots.  If I
commit a change that breaks Windows, I probably will not be able to debug or
fix it myself.  You might not be able to do the same with failures identified
by my ARM buildbot.  The hibernating snakebite.org aimed to solve that.

Still, I think the reporting of such failures is useful to package developers
since at least they'll find out about it, as long as the reporting of such
failures doesn't become spammy.  E.g. no automatic bug filing.  Push
notifications of test results is another interesting aspect to explore.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20111017/d3553086/attachment.pgp>