[TIP] Use unittests for profiling?

[Paul Hildebrandt]

jason kirtland wrote:
> On Thu, May 13, 2010 at 7:24 AM, Matthew Wilson <matt at tplus1.com> wrote:
>   
>> I've been writing lots of snippets of code and then feeding them into
>> cProfile and timeit, looking for stuff to optimize.  It dawned on me
>> recently that these snippets look a lot like some of my unittest
>> classes: each snippet has a few statements and an optional setup
>> block.  In both cases, I'm isolating a component from a bigger system
>> and then running just that component.
>>
>> Has anyone done any work to reuse unit tests for profiling and timing?
>>
>> Is there something theoretically wrong with this idea?
>>
>> There's a practical hurdle -- the cProfile and timeit modules were
>> originally designed to eat strings of code.  They can work with
>> callable objects, but I am finding that to be a little difficult when
>> I have non-trivial setups.
>>     
>
> The SQLAlchemy test suite implements a profiling test decorator that
> asserts that a test case uses only a certain number of function calls
> to get the job done.  
Jason,  why does it care?  I know that there is some overhead in 
function calls but is it really that large?  I ask not as a criticism, I 
am thinking about things like Matt is and want to know if I should 
consider function calls aside from general timing.

> As written, it does not provide a direct measure
> of performance, more of a warning flag that goes up if a refactoring
> unexpectedly adds or removes overhead to a measured critical path.
>
> The general approach definitely can work, and with clever inspection
> of the stats you could do some interesting assertions on what happened
> during the run, or at least some tailored reporting if you're not
> interested in pass/fail semantics.
>
> I think we did have some timeit-based testing going on, but those were
> converted into non-tests because they were a bit slow to include in
> the regular full suite runs.  I wasn't able to figure out how to
> account for the many system factors that affect real execution time
> and convert that into a definite pass or fail anyhow.
>
> -=j
>
> _______________________________________________
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
> http://lists.idyll.org/listinfo/testing-in-python
>