[TIP] including (or not) tests within your package

[holger krekel]

On Tue, Jul 27, 2010 at 12:07 -0400, Alfredo Deza wrote:
> On Tue, Jul 27, 2010 at 10:42 AM, holger krekel <holger at merlinux.eu> wrote:
> 
> > hi Alfredo, all,
> >
> > On Mon, Jul 26, 2010 at 20:16 -0400, Alfredo Deza wrote:
> > > A few weeks ago in our local Python meeting someone mentioned what a drag
> > > was to (most of the time) have to download
> > > the source for a package they have just installed to run the tests
> > because
> > > the 'test(s)' directory was not included at the time
> > > of installation.
> > >
> > > Some packages include them (like unittest2) and some (most?) don't.
> > >
> > > What do you guys feel about including the test directory within your
> > package
> > > so it is included at the time of installation?
> > >
> > > I'm gearing towards including them, but wanted to hear your opinion about
> > > it.
> >
> > To me there are pro's and con's.  These days i like to put my tests
> > external to a package, especially if they are higher-level or functional
> > tests.  This also allows to run the same tests against different versions
> > of a package which i sometimes use for testing cross-version compatibility.
> > The latter is not easy with package-inlined tests.
> >
> > Having said that i would like to have a way to generically
> > test any installed package (including C-extensions where
> > test-inlined-into-package is not really possible).   I rather
> > ponder specifying a test-url such that a tool can go to installed
> > packages, download the specified tests and run them, maybe even reporting
> > them back somewhere.
> >
> > I can see how that could be a problem.
> 
> But not having consistency (or a guideline/best practice) for having (or
> not) tests within a package
> it would be difficult to implement the test-url approach.

Yes, hopefully we can work something out starting from distutils2 metadata. 

Btw, my tests sometimes require extra packages (e.g. pexpect for some functional
interactivity tests) which are not installed with the package-under-test 
and end users could thus not easily run tests with "python -m ..."). 
 
> What happens when network access is restricted? This would severely affect
> the way any test-grabbing/reporting could happen.

In this case test archives need to be available locally, indeed. 
 
> What about having 2 sets of tests? the integration and high-level tests can
> still be outside the package while everything else that would
> not interfere could be "inline"

maybe, yes.  However, especially functional and
integration tests have highest chances of failing in end-user 
environments.   

cheers,
holger