[TIP] Use unittests for profiling?

[jason kirtland]

On Fri, May 14, 2010 at 9:34 AM, Paul Hildebrandt
<Paul.Hildebrandt at disneyanimation.com> wrote:
> jason kirtland wrote:
>>
>> On Thu, May 13, 2010 at 7:24 AM, Matthew Wilson <matt at tplus1.com> wrote:
>>
>>>
>>> I've been writing lots of snippets of code and then feeding them into
>>> cProfile and timeit, looking for stuff to optimize.  It dawned on me
>>> recently that these snippets look a lot like some of my unittest
>>> classes: each snippet has a few statements and an optional setup
>>> block.  In both cases, I'm isolating a component from a bigger system
>>> and then running just that component.
>>>
>>> Has anyone done any work to reuse unit tests for profiling and timing?
>>>
>>> Is there something theoretically wrong with this idea?
>>>
>>> There's a practical hurdle -- the cProfile and timeit modules were
>>> originally designed to eat strings of code.  They can work with
>>> callable objects, but I am finding that to be a little difficult when
>>> I have non-trivial setups.
>>>
>>
>> The SQLAlchemy test suite implements a profiling test decorator that
>> asserts that a test case uses only a certain number of function calls
>> to get the job done.
>
> Jason,  why does it care?  I know that there is some overhead in function
> calls but is it really that large?  I ask not as a criticism, I am thinking
> about things like Matt is and want to know if I should consider function
> calls aside from general timing.

There is some overhead in Python function calls, but as I mentioned
this particular decorator doesn't really know the difference between 1
slow call and 3 fast ones.  It is handy for finding candidates for
in-lining, for example if breaking some code out into a separate
function turns out to add 100 function calls to a critical but
somewhat invisible path like a __getattr__ implementation.  That kind
of tweak can be a pure speed win.

If you have a fixed environment (say, tests always running on the same
machine and at about the same system load), comparing profile times
vs. previous runs seems reasonable.  With a test decorator it's pretty
easy to run the test function N times to measure a solid result, akin
to what timeit does.  With some futzing in the testing framework it
might be possible to run setup + the profiled test + teardown, too.

>> As written, it does not provide a direct measure
>> of performance, more of a warning flag that goes up if a refactoring
>> unexpectedly adds or removes overhead to a measured critical path.
>>
>> The general approach definitely can work, and with clever inspection
>> of the stats you could do some interesting assertions on what happened
>> during the run, or at least some tailored reporting if you're not
>> interested in pass/fail semantics.
>>
>> I think we did have some timeit-based testing going on, but those were
>> converted into non-tests because they were a bit slow to include in
>> the regular full suite runs.  I wasn't able to figure out how to
>> account for the many system factors that affect real execution time
>> and convert that into a definite pass or fail anyhow.
>>
>> -=j
>>
>> _______________________________________________
>> testing-in-python mailing list
>> testing-in-python at lists.idyll.org
>> http://lists.idyll.org/listinfo/testing-in-python
>>
>