[TIP] Thoughts on Fuzz Testing

[Robert Collins]

On 21 October 2015 at 11:03, Randy Syring <randy at thesyrings.us> wrote:
> I recently had a chat with my team about fuzz testing.  The thesis as
> proposed is:
>
> Fuzz tests are useful as tools for a developer to run manually which help
> identify corner cases in the code not covered by
> explicit-branch-based-unit-testing (feel free to help me with the term I
> want here).  They should not run by default in an automated testing suite
> (for performance & time considerations).  They should not be used as an
> excuse for lazy developers to not write explicit tests cases that
> sufficiently cover their code.
>
>
> I'm in interested in feedback on the above.  Agree, disagree, and most
> importantly, why.

So in my view testing - all testing including manual testing,
one-offs, and automated testing - is just an engineering tradeoff. We
know, for instance, that not having tests at all is a terrible
tradeoff - the chance of shipping regressions approaches unity very
very quickly.

the answers for fuzz testing then are context specific. Got something
where fuzzing takes a couple weeks to run - probably don't want to run
it in CI. Got another thing where you can fuzz it pretty exhaustively
after just a few minutes - why wouldn't you run it in CI?

If you're running it in CI (and can be sure you'll keep doing so), is
there much / any value in also explicitly trying to cover the *same*
input space with manually written tests? OTOH if you're writing a fix
for something a fuzzer took a week to find, writing a test to narrowly
and quickly trigger it seems like only a sensible thing to do.

Lastly I find the concept of 'lazy developers' as a shortcut for
whatever you did mean to be more than a little offensive. The most
productive and wonderful developers I know are incredibly lazy in the
best way: they automate things that can be automated so that they're
not production line workers, and are instead being creative and
useful.

And the developers I know that aren't as productive and wonderful all
*want* to be so, and so we should be helping them learn how that
works, not dismissing one of the fundamental things that computers are
useful for.

-Rob


-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud