[TIP] Testing multi-threaded applications

[Michael Foord]

Bernhard Mulder wrote:
> I recently also dealt with threads, and ran into similar issues. For 
> example, I was surprised to learn that there is no command to kill 
> threads, instead I had to add code in server threads to get them out 
> of the server loop.
>
> I have concluded a while ago that "sleeping" in tests (your function 
> wait_for_some_time) should really be avoided, for the reasons you 
> state: wait not long enough, and your tests become unstable. Wait too 
> long, and you are wasting time.

The 'waitForCondition' that we use has a 'sleep counter', which sleeps 
for a (configurable time) usually 25ms and then checks the condition. 
You specify how many ticks it should sleep for before timing out.

That way the maximum amount of time you waste is limited.


>
> So why do you sleep? Abstractly speaking, you are waiting for some 
> time and hope that after this time some condition is fulfilled. In our 
> case, you want to wait long enough for the consumer to have consumed 
> all data. In order to avoid the sleep, you need to have your code 
> signal this condition directly, I my code I used the Event objects of 
> the threading module. So the suggestion is that the consumer signal an 
> empty queue via an event, and that the testing code waits for this event
>
> The advantages of this approach are obvious: no need to choose a wait 
> time, your tests run with full speed.
>
> The disadvantage of this approach might be that the signaling of the 
> event is maybe only needed for testing, and is not necessary for 
> production code. I don't know the testing literature, so I don't know 
> if it says something about such testing artifacts. I personally value 
> fast and reliable tests very highly, so I would just leave this 
> artifact in the code.
>

We used to use (still do in some cases) a slightly different pattern, 
using an 'EventWaiter'. You wait (with a timeout) for some event to 
occur, registering for that event. In most cases it has just felt more 
natural to use 'waitforCondition' and most of our tests are migrating to 
that pattern.

Our functional tests always run off the GUI thread - so that we can 
interact with the GUI thread whilst it is running and the test can sleep 
without blocking the GUI. (Calls that interact with the GUI have to be 
'invoked on the GUI thread' - which is very straightforward with .NET 
and Windows Forms.)

> Maybe I should mention that my own testing code is not working yet, 
> since I turned my attention away from the threading aspects of the 
> application to deal with other areas first.
>
> One of the things I want to test is some sort of timeout mechanism: 
> the application hands off some task to some separate process, and 
> needs some fall back mechanism in case this process dies or is 
> inaccessible (it might run on another machine). I will use the 
> time.sleep function in the production code, but for testing, I will 
> run my application on "simulated time" rather than real time, so that, 
> again, the tests can run at full speed. One question I have is: how do 
> I model this? I could replace the function sleep in the time module 
> for testing, but I am afraid that changing standard functions might 
> interfere with the testing framework.

As our functional tests (and some of our unit tests) run on a separate 
thread to the GUI, this hasn't been a problem for us. We've never needed 
to substitute a 'fake time' and we rarely (if ever!) use sleep in our 
production code.

All the best,

Michael Foord
http://www.voidspace.org.uk/ironpython/index.shtml

>
> My current (planned, since the tests are not quite functional yet) 
> solution is to introduce an indirection: import time.sleep into some 
> module called SystemInterfaces, and then import sleep from 
> SystemInterfaces.sleep. This way, my testing scaffolding can 
> substitute a custom sleep function for the standard one.
>
> Similar issues arise with the interface to the file system.
>
> Is there some other recommended way to deal with this issue?
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
> http://lists.idyll.org/listinfo/testing-in-python
>