<br><br><div class="gmail_quote">On Sun, Mar 22, 2009 at 07:15, C. Titus Brown <span dir="ltr"><<a href="mailto:ctb@msu.edu">ctb@msu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hi all,<br>
<br>
we're running into an interesting problem over on the pygr project: the<br>
number of tests to run varies according to the installation, and we're<br>
having trouble tracking the expected number of tests.<br>
<br>
Briefly, we include or skip entire test suites depending on whether or<br>
not MySQL, sqlite, and/or BLAST (a sequence comparison package) are<br>
installed. How should we track how many tests *should* actually be run?<br>
<br>
(This is a more general problem, incidentally; I've noticed that in<br>
projects with thousands of tests, a few can slip through the cracks and<br>
become "not executed" without anyone noticing, for a variety of reasons<br>
(inappropriate skipping, poor if statement usage, function renaming,<br>
etc.)<br>
<br>
The two ideas I had are --<br>
<br>
- keep track of aggregate number of tests per test file expected under<br>
the various conditions, e.g.<br>
<br>
test_db_stuff: 20 tests if no MySQL, 50 if MySQL<br>
<br>
- keep track of individual tests by name, and tag them with "not run<br>
if MySQL is not installed", and then check to make sure all of the<br>
expected ones are run.<br>
<br>
Both entail an increase in record-keeping which is annoying but<br>
inevitable, I guess...<br>
<br>
Thoughts?</blockquote><div><br>First issue that comes to mind is you will obviously have to clearly
state what potential resources or variable could be lacking to cause a
test to not be executed. That can be a chore in itself.<br>
<br>Second, depending on how your tests are written, you could
potentially automate a decent amount of the discovery through
decorators. If you have decorators flag what tests require what
settings you can then import a module and have the decorators simply
record the details needed to run that test. This obviously has a
potential issue of errors in mis-stating what the test really needs,
but that could happen with writing them down by hand anyway.<br><br>-Brett <br></div></div><br>