[TIP] A rare philosophical thought

[Kumar McMillan]

On Fri, Aug 1, 2008 at 12:28 PM, C. Titus Brown <ctb at msu.edu> wrote:
> """
> The Python gurus recommend unit testing to make sure code is solid.
> That's great. If I wanted to write dozens of lines of boilerplate
> code in order to make sure stuff worked, I'd have stuck with C++. I want
> to write less code and be confident in the belief that that code is
> correct and error free.
> """

Whenever I see this kind of resistance I've noticed there is often a
misinterpretation of what "automated testing" really means.  I have
asked developers who do not write automated tests how they guarantee
that a feature works.  The good developers respond with: "well, I add
the feature then I fire up the app and verify that it works.  Duh!"

This is a great point of entry for helping developers to understand
automated testing.  What they are already doing is *the same exact
thing* as automated testing except it is manual and not automated.
The real problem lies in that it often doesn't make sense to add
automated tests.  That is, the app may be too small and manual testing
could be as simple as loading up a single page then submitting a form
and verifying the success page.  Done.  You've tested your entire app
and you know it works.

Why automated testing then?  The answer (as we all know) is in
scalability.  That's the point to stress when trying to help a
developer understand the usefulness of automated testing.  OK, so what
happens when you have a sequential form that is 8 pages long?  I.E. to
manually test this you have to start at the beginning *every time*.
For me, this was exactly when the concept of automated testing
clicked.  I had been working on a dating / matchmaking site and the
profile signup was indeed about 8 or 10 form pages and there was a
very abstract base component that drove every single page of the form.
 So if I made a bad change to the base it would break every page of
the form in confusing ways.  I developed this app entirely without
automated tests and had to test the app by manually going through the
form one page at a time!  The result of course was many subtle bugs in
the combinations of form elements that I was too lazy to test.

But even after a developer understands this concept there is still a
problem of discipline.  If you start an app that has 2 pages of forms
and that app grows 4 pages of forms then you will probably admit to
yourself "OK, I should start writing some tests now."  Unfortunately,
for the typical software development team, it is too late to start
writing tests at this point.  With typical pressures from the business
(MAKE MONEY NOW!) it is almost always too hard to sell the fact that
you need to HALT new development for a month while you write automated
tests to cover ALL the use cases you have implemented thus far.

In conclusion, it is my belief that any app to grow from a prototype
into some kind of "stable" tool is doomed to fail if no automated
tests exist at that point of maturity.

Then again, the value of manual testing should not be underestimated!

RE: the original boilerplate argument, don't forget that the average
developer writes BAD TESTS.  Especially when you are prototyping an
app and you know you *should* write tests but you don't know how the
app is going to be implemented this is when you often write a
meaningless test.  Instantiating an object is not a very useful test!
This type of testing is only going to provide more fuel for the
anti-automated-testing developers out there, not to mention give you
and your team a sense of false confidence.  To remedy this, I would
advise any developer I see who writes these kind of tests to *stop*
writing tests for a minute and start manually testing his/her code
instead to gain an understanding of what it means to verify a use
case.  Even unit testing is about verifying a use case -- you have an
object or function [the unit] and another piece of code wants to use
it.  Along those lines, doctest provides another great entry point for
developers to understand the concept of automated testing.  So the
second point is you have to become a "user" to write better tests.
Luckily, this is naturally a part of software development.

Kumar

PS. Don't forget to blame Titus for this rant ;)