<br><br><div class="gmail_quote">On Tue, Aug 10, 2010 at 8:21 AM, Olemis Lang <span dir="ltr"><<a href="mailto:olemis@gmail.com">olemis@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im">On Mon, Aug 9, 2010 at 10:37 PM, Fernando Perez <<a href="http://fperez.net" target="_blank">fperez.net</a>@<a href="http://gmail.com" target="_blank">gmail.com</a>> wrote:<br>
> On Fri, Jul 30, 2010 at 5:01 AM, Olemis Lang <<a href="mailto:olemis@gmail.com">olemis@gmail.com</a>> wrote:<br>
>> Probably this is a matter of flavors . The fact is that I've not seen<br>
>> yet (or probably skipped ...) a strong reason (besides my current<br>
>> laziness ;o) in favor of *always* including tests inside a package<br>
>> prepared to run in production systems .<br>
><br>
> This bug in the production, official Debian/Ubuntu delivered version of ATLAS:<br>
><br>
> <a href="https://bugs.launchpad.net/ubuntu/+source/atlas/+bug/363510" target="_blank">https://bugs.launchpad.net/ubuntu/+source/atlas/+bug/363510</a><br>
><br>
> was detected because numpy ships its test suite along with it, and someone ran<br>
><br>
> import numpy<br>
> numpy.test()<br>
><br>
> on a production system, and boom.<br>
<br>
</div>I'm not saying that testing code in production systems is a crazy idea<br>
, I'm just saying that , if that's needed , tests can be installed at<br>
any time and , once done with testing , then removed once again.<br>
<div class="im"><br>
> Yes, it could have been found<br>
> otherwise, but it was very, very useful for the numpy developers to<br>
> know this, and also to be able to ask users of other similar systems<br>
> to also run numpy.test() until we could understand where it was coming<br>
> from.<br>
><br>
<br>
</div>+1<br>
<div class="im"><br>
> Production systems are the ones where you want to know that things<br>
> work correctly, right? So they should be the ones where you actually<br>
> test your code.<br>
><br>
<br>
</div>Sorry , but I don't agree . I'd rather say «they should be the ones<br>
where you actually test your code.*once something is going wrong & the<br>
problem cannot be identified in a test environment* »<br>
<div class="im"><br>
> We *always* ship the tests with all projects I am involved with, and<br>
> work very hard to provide really easy ways for users to run the full<br>
> test suite and provide us with useful feedback when something goes<br>
> wrong.<br>
><br>
<br>
</div>you see ? «when something goes wrong.»<br>
;o)<br>
<div class="im"><br>
> I can't count the number of bugs we've found in this manner, simply<br>
> because they only appear with some combination of compiler, hardware<br>
> endianness, library versions, etc, which developers would *never* have<br>
> access to, but some user at some big government lab happens to run on.<br>
><br>
<br>
</div>In theory , that should be tackled by setting up a slave or VM with<br>
similar characteristics in a CI environment . If you don't have the<br>
means to do so (which I understand perfectly ;o) & the user is really<br>
interested in confirming that everything is OK with future versions ,<br>
then IMO the right thing to do should be making (him | her) part of<br>
the (testing | integration | release) process by setting up a testing<br>
environment as similar as possible to the production env so as to<br>
test'em (e.g. more or less that's what we do with Trac XmlRpcPlugin<br>
<div class="im">;o) .<br>
<br></div></blockquote><div><br></div><div>No way. </div><div><br></div><div>You can't recommend to set up a VM to be 'as similar as possible to the production environment'.</div><div><br></div><div>If you have a server A, model 1 with OS B version 1, then a VM *will not* be able to replicate *fully* that environment.</div>
<div><br></div><div>Having an environment for testing that differs from production *is not OK*. </div><div><br></div><div>There is no way to 'have the means to' replicate every single combination of Hardware and Operating Systems out there, hence</div>
<div>the need to include tests.</div><div><br></div><div>What you can do/have is a testing environment with VM's all over, which is fine. But that doesn't mean you will be able to </div><div>cover your bases.</div>
<div><br></div><div>I fully agree with Fernando, Jesse, Michael and all the others that recommended including tests with your package/module </div><div>and making it easy for end users to be able to run your tests.</div><div>
<br></div><div><br></div><div><br></div><div>-Alfredo</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="im">
--<br>
Regards,<br>
<br>
Olemis.<br><br></div></blockquote></div>