[TIP] including (or not) tests within your package

[Olemis Lang]

On Tue, Aug 10, 2010 at 8:23 AM, Alfredo Deza <alfredodeza at gmail.com> wrote:
> On Tue, Aug 10, 2010 at 8:21 AM, Olemis Lang <olemis at gmail.com> wrote:
>> On Mon, Aug 9, 2010 at 10:37 PM, Fernando Perez <fperez.net at gmail.com>
>> wrote:
>> > On Fri, Jul 30, 2010 at 5:01 AM, Olemis Lang <olemis at gmail.com> wrote:
>>
>> >> Probably this is a matter of flavors . The fact is that I've not seen
>> >> yet (or probably skipped ...) a strong reason (besides my current
>> >> laziness ;o) in favor of *always* including tests inside a package
>> >> prepared to run in production systems .
>> >
>> > This bug in the production, official Debian/Ubuntu delivered version of
>> > ATLAS:
>> >
>> > https://bugs.launchpad.net/ubuntu/+source/atlas/+bug/363510
>> >
>> > was detected because numpy ships its test suite along with it, and
>> > someone ran
>> >
>> > import numpy
>> > numpy.test()
>> >
>> > on a production system, and boom.
>>
>> I'm not saying that testing code in production systems is a crazy idea
>> , I'm just saying that , if that's needed , tests can be installed at
>> any time and , once done with testing , then removed once again.
>>
[...]
>> > I can't count the number of bugs we've found in this manner, simply
>> > because they only appear with some combination of compiler, hardware
>> > endianness, library versions, etc, which developers would *never* have
>> > access to, but some user at some big government lab happens to run on.
>>
>> In theory , that should be tackled by setting up a slave or VM with
>> similar characteristics in a CI environment . If you don't have the
>> means to do so (which I understand perfectly ;o) & the user is really
>> interested in confirming that everything is OK with future versions ,
>> then IMO the right thing to do should be making (him | her) part of
>> the (testing | integration | release) process by setting up a testing
>> environment as similar as possible to the production env so as to
>> test'em (e.g. more or less that's what we do with Trac XmlRpcPlugin
>> ;o) .
>
> No way.

AFAICS that's what the ASF suggests ;o)
http://httpd.apache.org/dev/release.html

> You can't recommend to set up a VM to be 'as similar as possible to the
> production environment'.

that's the first part , the part about «making (him | her) part of the
(testing | integration | release) process» doesn't necessarily involve
VMs , users may use real machines or they production servers (which I
think it's not a good idea ... BTW ;o) or whatever ...

> If you have a server A, model 1 with OS B version 1, then a VM *will not*
>  be able to replicate *fully* that environment.

You'll never be able to do so (I mean the HDDs, power supply,
micro-proc, drivers ..., the atoms and molecules are not the same ...)
, but it's like designing & manufacturing a car (or whatever you might
think of) you have a previous model and use it to get some sort of
confidence about the final product . In software you have MDA ,
emulators , ... , and VMs are AFAIK the closest choice to simulate a
fully working environment (please don't tell me that nobody's never
used a VM for testing purposes) ...

> Having an environment for testing that differs from production *is not OK*.

Well ,

  - does this means that everybody ran the test suites of all
    the software products everybody use on each and every PC
    wherever you know a PC is working (I mean OS, drivers,
    services, system software , apps ...) and that they covered
    all possible situations ?

> There is no way to 'have the means to' replicate every single combination of
> Hardware and Operating Systems out there, hence
> the need to include tests.

I don't really think this reasoning is 100% definitive . As I
mentioned in the reply to Fernando's message , you can get the same
results by installing both pkgs (softw + tests) .

> What you can do/have is a testing environment with VM's all over, which is
> fine. But that doesn't mean you will be able to
> cover your bases.
> I fully agree with Fernando, Jesse, Michael and all the others that
> recommended including tests with your package/module
> and making it easy for end users to be able to run your tests.
>

Sorry , but I still don't see the point of this been a *MUST* or even
*RECOMMENDED* . The fact that you need to have at home glasses and
blenders and tomatoes to prepare & drink your tomato juice doesn't
mean that it's a *MUST* for manufacturers to package the blender , the
glasses and the tomatoes (and so on ...) altogether so that shops will
be able to sell it (nor even that they SHOULD NOT distribute ANY parts
of the blender , just in case someone wants to replace'em or simply
not use it if don't need it) .

The fact that you want to run the test suite in the production
environment doesn't mean

  - that everybody wants to do the same (or even that everybody
    will do it at least once in their life)
  - that you *MUST* have'em both in the same package so as to
    deploy them in the production environment .

Just IMO

-- 
Regards,

Olemis.

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article: