[TIP] Ideology

[Jesse Noller]

On Tue, Apr 21, 2009 at 3:13 PM, Douglas Philips <dgou at mac.com> wrote:
> On or about 2009 Apr 21, at 11:17 AM, C. Titus Brown indited:
>> -> Your test framework should have unit tests - and yes, you'll need
>> to
>> -> treat it like any other piece of software, tests, docs, and
>> debugging.
>>
>> Aiee, that sounds like work!
>
> Right. If your tests are not, de facto, testing your framework, your
> framework has bloat that should be eliminated, it isn't carrying its
> own weight.
>

Hooey. Your tests are obviously functional tests for your framework,
however as someone who *writes* the framework, I need unit tests to
make sure I didn't regress the damned thing before I foist it on the
unsuspecting world.

Which reminds me, I still have an amazing 0 tests for nosejobs :)

>> -> It's a matter of choice: where do you want the complexity to lie?
>> In
>> -> the framework designer's bucket, or the test designer's? I tend
>> -> towards the latter obviously (consider the abstractions that Django
>> -> provides).
>>
>> I'd rather it be simple. No complexity allowed. Didn't you read my
>> blog
>> post??
>
> This is the inflection point where unit testing diverges from, say,
> device testing (the kind of stuff I'm fortunately paid to do). And so
> it begs the question of whether unit tests and nose and py.test are
> really all the same, not to mention hardware testing, system/
> functional testing... The more complex the "system under test", the
> more involved the testing framework seems to get...

<soapbox>
A good executor doesn't care what the classification of a given test
is (functional, unit, regression, mymom). A good executor only finds
tests and runs them.

I use nose to run everything from unit tests, to massive, multi-week
tests. All I need to do prior to execution is provision a system (the
system under test) and generate a valid config file to pass to the
tests to tell them "what" system to hammer.
</soapbox>

Ultimately, unittest is geared towards unit testing (duh) - but I (and
others) use it as a perfectly good working base for *any* sort of
test. I've seen it used for drive testing, filesystem testing, etc.

Obviously the more complex the SUT is the more complex your provision,
setup and teardown is. I've had to write a nose plugin which
essentially acts as a singleton to install a given system (in this
case a cluster) and *then* run the tests.

jesse