[TIP] Use unittests for profiling?

[jason kirtland]

On Thu, May 13, 2010 at 7:24 AM, Matthew Wilson <matt at tplus1.com> wrote:
> I've been writing lots of snippets of code and then feeding them into
> cProfile and timeit, looking for stuff to optimize.  It dawned on me
> recently that these snippets look a lot like some of my unittest
> classes: each snippet has a few statements and an optional setup
> block.  In both cases, I'm isolating a component from a bigger system
> and then running just that component.
>
> Has anyone done any work to reuse unit tests for profiling and timing?
>
> Is there something theoretically wrong with this idea?
>
> There's a practical hurdle -- the cProfile and timeit modules were
> originally designed to eat strings of code.  They can work with
> callable objects, but I am finding that to be a little difficult when
> I have non-trivial setups.

The SQLAlchemy test suite implements a profiling test decorator that
asserts that a test case uses only a certain number of function calls
to get the job done.  As written, it does not provide a direct measure
of performance, more of a warning flag that goes up if a refactoring
unexpectedly adds or removes overhead to a measured critical path.

The general approach definitely can work, and with clever inspection
of the stats you could do some interesting assertions on what happened
during the run, or at least some tailored reporting if you're not
interested in pass/fail semantics.

I think we did have some timeit-based testing going on, but those were
converted into non-tests because they were a bit slow to include in
the regular full suite runs.  I wasn't able to figure out how to
account for the many system factors that affect real execution time
and convert that into a definite pass or fail anyhow.

-=j