[TIP] including (or not) tests within your package

[Andrew Bennetts]

Ben Finney wrote:
> Andrew Bennetts <andrew at bemusement.org> writes:
> 
> > Ben Finney wrote:
> > > [One downside of installing the tests along with the rest of the
> > > package is that] They would thereby become part of the *public*
> > > interface for the package. I don't want that implied promise; the
> > > test suite is subject to arbitrary unannounced change or removal.
> >
> > Surely you don't consider every importable name to be part of the
> > public interface for a package?
> 
> Every importable name that isn't marked with a leading-underscore name,
> yes. That's what that distinction is for, after all.

So, you don't consider every importable name to be part of the public
interface, then :)

My point is that you are already accepting many names into your namespace
that are just implementation details, without an implied promise that they
are part of the public interface.  There's no reason why the tests have to
be any different, so I don't see why you claim they must be.  You
acknowledge there is a distinction that can be made, so make it.

[...]
> > > They're for use while developing (hence part of the source), not in
> > > their deployed location (hence *not* part of the binary package).
> >
> > What about packages that build on your packages?
> 
> The test code is still available in the source distribution. What use
> case are you thinking of?

I gave some in the message you are replying to.  I'll try again:

  - test infrastructure can be useful in tests for other code using that
    library, e.g. someone writing a bzrlib plugin will want to use APIs
    found in bzrlib.tests, and someone writing a Twisted application will
    want to use APIs found in twisted.trial.  (There is an argument that
    test infrastructure ought to be strictly separated from the tests
    themselves, but you can also argue that detangling those things when
    they aren't already is not be worth the effort unless there's some
    practical benefit.)
  - make it possible — and easy — for users to run your test suite in their
    deployed environment, because they may encounter bugs related to their
    specific installed dependencies, filesystem format, system
    configuration, etc.

> Neither. The test cases are available and useable in the source
> distribution, which someone doing development using this package will
> need anyway.

Really?  Most Python packages I can think of don't require you to acquire a
source distribution using that package.  e.g. Twisted, bzr, Zope, Paste,
PyGTK... if I've installed the binary package, I can start doing development
with these.  Maybe there's something different about the projects you
distribute?

> > Also, there is some value to enabling users to run tests in their
> > deployed environment, where they may have a unique combination of
> > dependency versions, system configuration, etc.
> 
> I can see that, and I would like finer-grained control to be available
> to users. But they do have the ‘python ./setup.py test’ command
> available, at least with Setuptools and Distribute and friends. I don't
> see the pollution of the system namespace to be worth it.

There's an assumption here that running ‘python ./setup.py test’ will give
as accurate results as running tests installed on the system with the rest
of the package.  I can think of some^Wseveral^Wmany ways that might not be
true.  Perhaps I'm overly paranoid.

Also, the pollution of having a foolib.tests name in your namespace in
addition to foolib seems pretty negligible to me.  It's in the namespace
under foolib, after all, not polluting the top-level namespace.

I have some intrinsic sympathy for the “not installing tests is cleaner”
position, but so far it seems to me that “practicality beats purity” here.
If someone wants to do extra work to on their package to make it “cleaner”
in this way, then good for them, but the practical benefit seems pretty
minimal.

-Andrew.