[TIP] Fwd: Testing multi-threaded applications
raphael.marvie at lifl.fr
Tue May 8 08:29:48 PDT 2007
On May 8, 2007, at 4:38 PM, Will Guaraldi wrote:
> Whoops--forgot to copy the list.
> ---------- Forwarded message ----------
> From: Will Guaraldi <will.guaraldi at gmail.com>
> Date: May 8, 2007 10:38 AM
> Subject: Re: [TIP] Testing multi-threaded applications
> To: Raphael Marvie <raphael.marvie at lifl.fr>
> I'm not really sure I understand what it is you're actually trying to
> test. Are you testing to make sure there's no possibility for dead
> lock? Are you testing to make sure threads can't starve? Are you
> testing that the threads are interacting as you've asked them to?
I am only testing that the threads are interacting as I expect them
to do so. Being sure, to summarize, that when work appear in the
queue a consumer will take it and process it.
> I think that non-deterministic tests are way bad because you can't
> rely on the result of the test. If you can't rely on the result of
> the test, then you can't accurately compare it with your expectations
> of the result and that's not particularly useful.
That is the point of my e-mail. How to avoid as much as possible such
tests when dealing with multi-threaded applications.
> You mentioned doing a thread join on the threads to wait until they've
> completed. What's the data flow of your application? Depending on
> the data flow graph, it might be possible to do a thread join on one
> or two specific threads rather than all the threads.
Nope, all the threads are anonymous (you can see them as a pool of
threads). The point in the join was only to know that assertions
could be evaluated.
> Is there a way to have your threads maintain their status on a
> scoreboard of some kind? For example, when they get a job to perform,
> they update status information somewhere accessible. Then when the
> thread is done with the job, it updates the status information
> again--perhaps this time marking that it's completed another job. If
> you can keep some status information somewhere, then you can have your
> test check to see if all the threads have completed what you've asked
> them to complete and end the test at that point in time.
That is what I do, my point is *when* to do it? How my test knows it
can evaluate the assertions.
> A test that takes too long isn't necessarily bad--you don't _have_ to
> run all of your tests all the time. Perhaps you should take the
> longer running tests and group them into a "systems test suite" which
> gets run less often than unit tests, functional tests or regression
> tests. You could throw the systems tests into a build-bot and have it
> automatically run them on what's in version control every night.
I am trying to test first my application because I find much easier
to develop that way. So I have to run the tests until my code works.
And I do not want to wait for minutes each time I give my tests
another run to see if my work is finished.
> I don't have a whole lot of experience with TDD and multi-threaded
> applications, so that's all I'm going to think of without building one
> now and experimenting. I haven't read much regarding testing
> multi-threaded applications. In general having multiple threads makes
> application development much more complex to design and debug--I
> imagine this holds true for testing as well.
Yes. That is why I have started this thread in case of some readers
have good tips for testing such applications.
Raphael Marvie, PhD http://www.lifl.fr/~marvie/
Maître de Conférences / Associate Professor @ LIFL -- IRCICA
Directeur du Master Informatique Professionnel spécialité IAGL
Head of Master's in Software Engineering +33 3 28 77 85 83
More information about the testing-in-python