<html><head><style type="text/css"><!-- DIV {margin:0px;} --></style></head><body><div style="font-family:arial,helvetica,sans-serif;font-size:12pt">Ben Finney <ben+python@benfinney.id.au> writes:<br><br>> holger krekel <holger@merlinux.eu> writes:<br>> <br>> > On Thu, May 07, 2009 at 10:45 +1000, Ben Finney wrote:<br>> > > So, if I make a change to fix one of the failures and inadvertantly<br>> > > break a test that previously succeeded, I can go an indeterminate<br>> > > number of code-test cycles before I find out about it?<br>> ><br>> > well, you will all the time work on a failing test - most people i<br>> > know like to focus on fixing one particular test and then see if that<br>> > broke any other bits.<br>> <br>> If I make a change that breaks an existing test that used to pass, then<br>> I need to know about that as early as possible to find a better<br>>
solution.<br>> <br>> That implies that all the unit tests need to run when I make that<br>> change, not just the one I'm focussing on. The point of an automated<br>> test suite is that I don't need to remember which of them need to run.<br><br>In that context, I don't see the distinction between unit tests and<br>any other form of tests. I want to know the state of all tests on<br>every change. Some tests are long enough that I am not willing to<br>wait for those tests before I move to my next task. That is what I<br>want from continuous integration (run tests while I move forward).<br><br>In a perfect world, all tests would run instantly on every change and<br>I would not have to wait for any of the results. I don't have a<br>perfect world, so I seek ways to partition tests into sets which will<br>give me the more useful information sooner.<br><br>For me, the distinction between "unit tests", "acceptance
tests",<br>"performance tests" and any other form of automated tests has not been<br>as useful as the distinction between "runs fast enough that it does<br>not interrupt my development work" and "runs slowly enough that it<br>would interrupt my development work".<br><br>There have been other partitions of tests that I dreamt of using, but<br>never found a practical way to implement. For example, what if there<br>were a way to map coverage results back to the test selector, so only<br>those tests which executed code in the modified file were chosen to<br>run on a change?<br><br>> > > That seems like the canonical reason why ???run the entire unit test<br>> > > suite every time??? is good practice, and I can't see why the above<br>> > > behaviour would be desirable.<br>> ><br>> > As Laura points out, if tests take longer it is nice to not have to<br>> > wait before seeing if the intended-to-fix test
works. Btw, py.test<br>> > runs unit, functional and integration tests and i think nosetests also<br>> > does runs more than pure unit tests.<br>> <br>> I think you mean it *can* run all those kinds of tests; my point is that<br>> it can *also* be more specific.<br>> <br>> Acceptance, integration, performance, etc. tests are *not* unit tests,<br>> and unit tests need to be executable as an isolated suite precisely<br>> because they're the type of tests one wants to run as often as possible.<br><br>Can you clarify how you've gained value from the distinction between<br>test types in this context? I'd like to understand how your concepts<br>might apply in my environment.<br><br>Thanks,<br>Mark Waite<br>(amazed at the great discussions on this list. Thanks!)</div></body></html>