Just to be clear, ATS is not a testing harness, that does things like setup/teardown. It is a test suite manager, dispatching tests onto available resources, using some intelligence about getting the 'bottleneck' tests done first, filling up any parallel processor nodes, etc. It simply runs the job you tell it, and gets back the report of whether or not it worked. It also knows how to start parallel (if this is a machine it knows about) or distributed jobs.
<br><br>We used it extensively at LLNL so I'm pretty sure the quality is good; what is not good is the packaging to enable nice addition of new machines. Also, it knows how to run the LLNL batch system for tests that need more resources (such as time) than it has -- but it won't know how to run YOUR batch system (do you? (:->))
<br><br>Creating this tool involved pain, things you never think of off hand. For example, if one test depends on another, we knew to run the parent before the child. Most of our file space was shared. But then we discovered some people creating files on private, not shared, file systems, and expect the child to pick up those files as input. Oops. Back we went, to make sure dependent jobs were run in the same place as their parent. Also we adopted some defensive measures to avoid having multiple variants of the same test stomp on the same files at the same time. Yeah, we could blame the test, but we thought we ought to help out.
<br><br>So, if you do decide to work on it, make sure you don't take things out because you don't know why they are there.<br><br>Also, write to me and I'll tell you who to contact about getting any later sources; I retired six months ago and there is always the possibility that they fixed something. It is possible they might be able to participate in a sprint but I'm unsure about that. I am definitely not available..
<br><br><div class="gmail_quote">On Dec 7, 2007 11:05 AM, Kumar McMillan <<a href="mailto:kumar.mcmillan@gmail.com">kumar.mcmillan@gmail.com</a>> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="Ih2E3d">On Dec 5, 2007 10:44 AM, Noah Gift <<a href="mailto:noah.gift@gmail.com">noah.gift@gmail.com</a>> wrote:<br>> ><br>> ><br>> > The good news is it definitely shortens the execution wall time,
<br>> > depending<br>> > on how many resources you let it have.<br>><br>> Paul,<br>><br>> ATS looks like very interesting stuff. I will take a look at it this<br>> week. Thanks to everyone else, for the great suggestions. On a
<br>> related note, I would love to be involved in a testing related Sprint<br>> at PyCon. I think they, the organizers, are still looking for Sprint<br>> leaders, so if someone is interested in doing more work on parallel
<br>> testing in Python, I will show up.<br><br></div>this is a great idea for a sprint. I've never led a sprint but I am<br>interested in doing so. For it to work we would need a good backlog<br>of task tickets relating to the problem. Off the top of my head, some
<br>tasks:<br><br> - benchmark worker-supervisor communication process (that which sends<br>test results over the wire)<br> - benchmark speed of tests runs in parallel then benchmark speed of<br>same tests run in series, for comparison. Do this for local and
<br>remote workers.<br> - add plenty of unit tests for all the various components involved<br>(this task needs to be more specific)<br> - add plenty of functional tests for running tests in parallel on the<br>same machine, on remote machines, etc
<br> - add tests for many combinations of setup/teardown that could be<br>supported (at the package level, at the module level, etc)<br> - perhaps implement XML output in nose core to address<br>worker-supervisor communication (see
<br><a href="http://code.google.com/p/python-nose/issues/detail?id=140" target="_blank">http://code.google.com/p/python-nose/issues/detail?id=140</a> )<br> - create capistrano (<a href="http://www.capify.org/" target="_blank">
http://www.capify.org/</a>) recipes that install<br>nose, plugins, and test suite to run on remote "worker" servers (or<br>implement in python, but this would be harder)<br><br>... most of these tasks assume we will have a proof of concept
<br>implementation to start with.<br><font color="#888888"><br>-Kumar<br></font><div class="Ih2E3d"><br>> I suppose, I like the idea of<br>> having the ability to scale an arbitrary number of virtual machines as<br>
> test nodes, and having the ability to control them to netboot a<br>> "clean" test operating system, build a database, do test, and then<br>> rebuild themselves. That would be one heck of a Sprint. What is
<br>> great, is that if you have a firewire, or USB 2 drive, you can<br>> simulate something like this on your laptop, quite easily, and then<br>> roll it out into production.<br>><br>><br>><br>> Noah
<br>><br><br></div><div><div></div><div class="Wj3C7c">_______________________________________________<br>testing-in-python mailing list<br><a href="mailto:testing-in-python@lists.idyll.org">testing-in-python@lists.idyll.org
</a><br><a href="http://lists.idyll.org/listinfo/testing-in-python" target="_blank">http://lists.idyll.org/listinfo/testing-in-python</a><br></div></div></blockquote></div><br>