No subject

"The Python unit testing framework, sometimes referred to as =93PyUnit,=94
is a Python language version of JUnit, by Kent Beck and Erich Gamma.
JUnit is, in turn, a Java version of Kent=92s Smalltalk testing
framework. Each is the de facto standard unit testing framework for
its respective language."

That's nice, but I really have no interest in preserving this legacy. We ha=
ve
no constraint that our testing framework be familiar to people coming from
other languages. We also have no constraint that people can transparently
upgrade from unittest to this new version like unittest2 does. Working unde=
r
those constraints sound like a pain in the ass, but I sure am glad someone
has the patience because it IS important for python. So I in no way mean
to undermine the work going on in unittest2. It's solving a different probl=
em.

Btw, the documentation, which I think is fairly good, can be found here:
http://github.com/Yelp/Testify/wiki

So, with that out of the way, I have had a little time to take a closer loo=
k at
unittest2. As I said, we started testify before unittest2 was around, and n=
ot
being very familiar with this list certainly didn't hear about
development. It's
actually reassuring to me that we solved some of the same problems, but
we took a slightly different approach.

1. Suites
We used to have a system where we centrally managed our 'suites'. This was
built on top of unittest and consisted of tools that built TestSuite object=
s or
something. However, this became very awkward to keep maintained as it was
very easy to forget to add your test to a suite and thus never get
run. So we switched
to more of a 'tagging' system wherin you, with decorators, class
attributes or module
globals set a list of 'suites' your tests belong in.

The same system applies for 'skipping' tests because with suites you
can do a sort
of arithmetic where you add suites you want to run, and subtract out
ones you don't.
A typical execution might look like 'testify yelp.tests -i framework
-x buildbot-disabled'
which would execute all our framework tests but skip any marked
'buildbot-disabled'.

2. Parallel Execution
There are, I suppose, very complex solutions to this problem, but we
solve the 80% case.
We just do a hash based on test class name to select tests to run. The
test runner doesn't actually
do the parallel execution because we have buildbot for that. 'buckets'
are just for selecting what a
single instance of 'testify' will run. Of course sometimes we have
situations where multiple slow
tests end up ganging up on one runner and we have to rebalance by
re-seeding the hash and juggling
stuff around.

Not perfect, but good enough for now.

3. Decorator based setup/teardown (for classes too)
We started developing rather deep and complex class hierarchies for
tests that needed common fixtures.
TestCase classes started being a mess of Mixin's and and forgotten
super() calls.

By having decorators signify our setup and teardown it was way easier
to mix and match.
It's also been way easier to keep complex fixture's organized by
having method names that describe
each step in the setup chain rather than having setUp() methods that
call lots of other methods.

4. Naming convention
Last and maybe least. Dealing with camel case test case API is just
sorta annoying. After beating
into new hire's heads that consistent style is important it's rather
demoralizing to add the caveat: except in test cases.
Again, this is probably where 'taste' rears its ugly head, but having
consistent style throughout your code base is really
*nice*.

Thanks for your guys' feedback. I've probably got a few more emails in
this thread I need to respond to as well.

Rhett