[TIP] Interface vs Implementation Testing

holger krekel holger at merlinux.de
Wed Apr 25 02:05:02 PDT 2007


Hi Essien, Max, hi all, 

On Tue, Apr 24, 2007 at 17:02 +0100, Essien Ita Essien wrote:
> Max Ischenko wrote:
> > On 4/24/07, *Titus Brown* <titus at caltech.edu <mailto:titus at caltech.edu>> 
> > wrote:
> > 
> >     -> I coded a huge test suite once that primarily unit tested
> >     -> implementation.  Since there were many tests, any time I wanted to
> >     -> change the way the app was implemented (faster algorithm here, don't
> >     -> need this class anymore, more methods now because of feature X) I
> >     -> usually ended up spending a good deal of time updating and deleting
> >     -> tests.  This just seemed unecessary to me and deterred me from
> >     relying
> >     -> on the test suite as an indication of successful functionality.
> > 
> >     So, isn't this kind of an argument for functional and regression tests
> >     only?
> > 
> >     (I'm honestly curious.  I haven't found much of a need for unit
> >     tests in
> >     my own code.)
> I've been mostly silent and stealing ideas and lessons from others
> so i guess i'd better contribute back a thought or two here ;-)
> 
> For me... I don't even do much of what you call functional testing, but 
> what i find invaluable are those unit tests. How I normally develop is 
> in quick spurts of small ideas that i'll eventually put together into a 
> whole. for instance, I'm gonna need to read a files over the network, so 
> i quickly code a function that does just that, i'm going to need to 
> steganograph streaming data from various sources and stuff to files, so 
> i quicly code functions needed to do just that.
> 
> The problem is that without the whole application, you really can't test 
> these little bits and pieces. But with UNIT tests... the whole ball game 
> changes. You are no more writing a bit application, but a small 
> application that does just A for just B, and you can setup the small 
> environment in your unittest, needed to get A or B to just work. Once 
> its workign... you can safely plunk that functionality int a module
> and be rest assured that when you actually call it in your big 
> application... its gonna do just what you intended it to do.
> 
> When its time to do the putting together... if all your parts are 
> working as expected in other small small applications (aka. Unit Tests).
> You just go to sleep :)

nicely put.  In fact, I guess for complex programs involving many possible 
combinations of user input and involved code/component interactions 
unit tests are kind of neccessary to get to a maintainable and consistent 
code base.  For PyPy (being a full Python Interpreter and a Translation framework 
to various backends) there would have been no way to make progress without unit-testing. 
However, we used large functional tests (such as translating the whole Python 
Interpreter to C) to try to identify some problems for which we then added unit-tests
to prove our theory about the problem. Then we fixed it. 

Then again, the lines blurry - our unit-tests are often not strict unit-tests
but rather small functional tests.  To me, there is a grey area and actually 
i don't care too much.  What matters to me regarding a test is: 

debug information/hints: 
    will a failure and the related information/debugging facility 
    allow me to identify the underlying problem quickly? 

failure-to-problem-relation: 
    are the tests fine-grained enough that i get one failure per problem 
    or do i rather get one failure, but 3 independent problems are underlying it? 

speed: 
    can i run a significant fraction of tests speedily from my development machine? 

i guess that strict unit-tests usually score quite well on these 
criteria, but some functional tests can qualify as well IMO. 

holger



More information about the testing-in-python mailing list