[TIP] running unittests on multiple OS, including non-Posix

[John Yeuk Hon Wong]

On 2/28/14 3:56 PM, Albert-Jan Roskam wrote:
> I would like to run my unittests (--> nose) on multiple platforms. I 
> also would like to test different python versions and implementations 
> on each platform (--> tox [2]). These platforms also include Windows, 
> so Travis CI or Docker is not an option, AFAIK. I was thinking about 
> using Vagrant [3] to fire up VirtualBox [4] VMs for each 
> platform-to-be-tested, then either (a) fire up nose or tox through SSH 
> or (b) (the easy way) prepare each VM such that nose/tox is fired up 
> right when the OS starts (ie., edit .bashrc, autoexit.bat or whatever 
> it is called for the OS at hand). But this all feels like reinventing 
> the wheel. Can you recommend a package or strategy to use? Beautiful 
> is better than ugly, although practicality beats purity. In other 
> words: it would nice if it is The Right Way, but I am a lazy bastard 
> so I am also open for quick-and-dirty approaches ;-)
I don't think there is a standalone, open standard how to do this kind 
of things.

Testing is very project dependent and I often run into problems testing 
on external servers, hence why I prefer not to build on Travis, or drone.io.

Docker is a container and as far as I (and you) know, Docker does not 
run on Windows (and Docker is based on LXC technology).

A lot of people run Jenkins to manage continuous integration. Basically, 
Jenkins is a job management software. I don't really run Jenkins these 
days. But you can check this out: 
http://stackoverflow.com/questions/5543413/reconfigure-and-reboot-a-hudson-jenkins-slave-as-part-of-a-build

Basically you can boot slaves on demand when a test is triggered.

For larger software, I have seen people run openstack to handle creating 
new vms and putting them into farms.

Bottom line is:
For Linux, create a docker container of everything you need to run your 
job (call this a base docker image). Export the docker. Each time you 
run a test, tell Jenkins / your own custom bash script to create a new 
docker instance based on the base image. Run the job and report the 
result then remove the instance if you want to. I mention to use docker 
here because it's cheaper to launch docker instance. But for 
consistency, you may consider going for full vagrant vm as windows one 
would use vagrant vm. It's up to you.

For windows, just create a windows machine, bootstrap the machine, 
export the machine into a vagrant box/ova file. On each test, boot up 
the new VM and then run the tests.

That's how I handle things right now, but if anyone have any exceptional 
method, do share please.

In my opinion, the best testing tool is the one that works for your 
need. Generic tool often makes things more complicated.
To add complexity, some people go as far as using puppet/chef/ansible 
instead of vanilla bash/shell scripts. That's great if you have a big 
test farms. If I were you, first get multiple platform testing working, 
then start transiting. I don't know how many hours I lost trying to 
invent complication :)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20140228/20118905/attachment.htm>