[TIP] Fwd: Testing multi-threaded applications

[Raphael Marvie]

On May 8, 2007, at 4:38 PM, Will Guaraldi wrote:

> Whoops--forgot to copy the list.
>
> ---------- Forwarded message ----------
> From: Will Guaraldi <will.guaraldi at gmail.com>
> Date: May 8, 2007 10:38 AM
> Subject: Re: [TIP] Testing multi-threaded applications
> To: Raphael Marvie <raphael.marvie at lifl.fr>
>
>
> I'm not really sure I understand what it is you're actually trying to
> test.  Are you testing to make sure there's no possibility for dead
> lock?  Are you testing to make sure threads can't starve?  Are you
> testing that the threads are interacting as you've asked them to?

I am only testing that the threads are interacting as I expect them  
to do so. Being sure, to summarize, that when work appear in the  
queue a consumer will take it and process it.

> I think that non-deterministic tests are way bad because you can't
> rely on the result of the test.  If you can't rely on the result of
> the test, then you can't accurately compare it with your expectations
> of the result and that's not particularly useful.

That is the point of my e-mail. How to avoid as much as possible such  
tests when dealing with multi-threaded applications.

> You mentioned doing a thread join on the threads to wait until they've
> completed.  What's the data flow of your application?  Depending on
> the data flow graph, it might be possible to do a thread join on one
> or two specific threads rather than all the threads.

Nope, all the threads are anonymous (you can see them as a pool of  
threads). The point in the join was only to know that assertions  
could be evaluated.

> Is there a way to have your threads maintain their status on a
> scoreboard of some kind?  For example, when they get a job to perform,
> they update status information somewhere accessible.  Then when the
> thread is done with the job, it updates the status information
> again--perhaps this time marking that it's completed another job.  If
> you can keep some status information somewhere, then you can have your
> test check to see if all the threads have completed what you've asked
> them to complete and end the test at that point in time.

That is what I do, my point is *when* to do it? How my test knows it  
can evaluate the assertions.

> A test that takes too long isn't necessarily bad--you don't _have_ to
> run all of your tests all the time.  Perhaps you should take the
> longer running tests and group them into a "systems test suite" which
> gets run less often than unit tests, functional tests or regression
> tests.  You could throw the systems tests into a build-bot and have it
> automatically run them on what's in version control every night.

I am trying to test first my application because I find much easier  
to develop that way. So I have to run the tests until my code works.  
And I do not want to wait for minutes each time I give my tests  
another run to see if my work is finished.

> I don't have a whole lot of experience with TDD and multi-threaded
> applications, so that's all I'm going to think of without building one
> now and experimenting.  I haven't read much regarding testing
> multi-threaded applications.  In general having multiple threads makes
> application development much more complex to design and debug--I
> imagine this holds true for testing as well.

Yes. That is why I have started this thread in case of some readers  
have good tips for testing such applications.

r.

--
Raphael Marvie, PhD                http://www.lifl.fr/~marvie/
Maître de Conférences / Associate Professor  @  LIFL -- IRCICA
Directeur du Master Informatique Professionnel spécialité IAGL
Head of Master's in Software Engineering     +33 3 28 77 85 83