[TIP] including (or not) tests within your package

[Ben Finney]

Andrew Bennetts <andrew at bemusement.org> writes:

> My point is that you are already accepting many names into your
> namespace that are just implementation details, without an implied
> promise that they are part of the public interface.

How so? If the names are just implementation details, they're supposed
to be named with a leading underscore to indicate that. Change the
behaviour of published interface — the no-leading-underscore named
attributes — and prepare for the wrath of users of your API.

> There's no reason why the tests have to be any different, so I don't
> see why you claim they must be. You acknowledge there is a distinction
> that can be made, so make it.

Again, I make the distinction by not polluting the namespace with code
that doesn't belong there.

>   - test infrastructure can be useful in tests for other code using that
>     library, e.g. someone writing a bzrlib plugin will want to use APIs
>     found in bzrlib.tests, and someone writing a Twisted application will
>     want to use APIs found in twisted.trial.

The test cases for my tools are in a separate top-level package, ‘test’.
I point my automated test loader at that package and it runs the lot. I
exclude that directory from installation, and the installation is clean.

Anyone who wants to build on my code has all the tests available from
the source distribution, which is what one gets in order to develop
something further. To *execute* the code doesn't need the tests at all,
though, so they're not installed.

>     (There is an argument that test infrastructure ought to be
>     strictly separated from the tests themselves, but you can also
>     argue that detangling those things when they aren't already is not
>     be worth the effort unless there's some practical benefit.)

How did they get entangled in the first place? You speak as though
entanglement were going to happen without any action on the part of the
programmer; that's clearly false.

>   - make it possible — and easy — for users to run your test suite in
>   their deployed environment, because they may encounter bugs related
>   to their specific installed dependencies, filesystem format, system
>   configuration, etc.

That's a use case with value, certainly. As I said in a previous post,
though, I don't think it's sufficiently valuable; especially because the
tests are useless without the testing framework dependencies, which most
users aren't going to have installed anyway.

> > Neither. The test cases are available and useable in the source
> > distribution, which someone doing development using this package
> > will need anyway.
>
> Really? Most Python packages I can think of don't require you to
> acquire a source distribution using that package.

I get most of my Python libraries by installing them from my operating
system. If I want to do development *of* the code, then it's a given
that I'll need to get the source package.

> Also, the pollution of having a foolib.tests name in your namespace in
> addition to foolib seems pretty negligible to me. It's in the
> namespace under foolib, after all, not polluting the top-level
> namespace.

That's an artifact of (IMO unnecessary and undesirable) entanglement of
the tests with the application code. For reasons you allude to above,
the automated test suite on most packages I work on is separated from
the application code. There's no sensible place to install it on the end
user's system.

> If someone wants to do extra work to on their package to make it
> “cleaner” in this way, then good for them, but the practical benefit
> seems pretty minimal.

Like I say, I don't see any extra work. On the contrary, it makes the
development process much simpler to have a separation of concerns from
the beginning. Not installing the tests is then as easy as just omitting
the top-level test suite from the installation.

-- 
 \     “He may look like an idiot and talk like an idiot but don't let |
  `\              that fool you. He really is an idiot.” —Groucho Marx |
_o__)                                                                  |
Ben Finney