[TIP] getting access to the discovered test list

[Titus Brown]

On Wed, Feb 28, 2007 at 07:10:11PM -0600, Kumar McMillan wrote:
-> On 2/28/07, bear <bear42 at gmail.com> wrote:
-> > What we are doing is giving the developer the normal "run all unit tests"
-> > option which runs them all using the same python instance that was used to
-> > start nose.  But a request was made to also be able to gather the list of
-> > tests and then start each test in its own python - that way any errant code
-> > won't pollute the environment for downstream tests.
-> >
-> > So what I have done is run the loader code to build a list of TestCase
-> > instances and then cycle thru that list calling "python run_test <insert
-> > name of test here>"
-> 
-> you could *maybe* write a nose plugin that would do this but if I were
-> you I'd steer clear of this kind of behavior.  You are going to see a
-> serious slow down in your test suite if you use a python interpretter
-> for each test and what you have is really just a symptom of incorrect
-> setup/teardown state.  Here are some alternatives you can try:
-> 
-> - Write unit tests for your testing tools themselves.  Specifically,
-> test that they are tearing down your application data correctly.  I
-> find that doctests are the easiest way to go for that.
-> - Run tests using the --stop option in nose so that you can focus on
-> the root cause of the error and not the chain reaction of breakage
-> that occurs afterwards
-> - Focus more on your teardown code.  Clear caches, delete registries,
-> remove the entire database, get aggressive with it.

Here's a random idea:

have nose randomly shuffle and/or omit test cases, then run your test
suite five hundred times and focus on what occasionally breaks.  I would
love to write such a nose plugin, personally; perhaps I will, if anyone
thinks it would be useful...

;) monte carlo testing of your test suite!

--titus