[TIP] Incorrect return code py.test on Windows/Jenkins ?

[Jorge Vargas]

On Tue, Mar 22, 2011 at 5:31 AM, Pere Martir <pere.martir4 at gmail.com> wrote:
> On Mon, Mar 21, 2011 at 7:27 PM, Jorge Vargas <jorge.vargas at gmail.com> wrote:
>> On Mon, Mar 21, 2011 at 1:44 PM, Pere Martir <pere.martir4 at gmail.com> wrote:
>>> The return code of py.test on Windows makes Jenkins to consider the
>>> build "failed" if there is any failed testcase in the generated JUnit
>>> XML.
>>
>> Why you want the build to pass if the tests failed ?
>
> A "build" in Jenkins/Hudson has 4 status, the return code of the last
> command distinguish between  (1) and (2):
>
> (1) success/blue - the return code indicates "success" - zero usually
> (2) failed/red - the return code indicates a failure - nonzero usually
> (3) unstable/yellow - there is a failure test case record in JUnit XML
> (4) aborted/gray - manually by user
>
> If the executed Python script raises an exception and it's not
> handled, for example, SyntaxError, the Python interpretor returns a
> non-zero code and Jenkins/Hudson considers the "build" to be "failed"
> - (2).
>
> If the "execution" of the script is successful - that is, all
> exceptions are caught and handled, the return code should be success
> (zero), in spite of failed testcases (1). If I recall correctly, this
> is the behavior of pyunit.
>
> Otherwise there is no way to distinguish between:
> (a) Execution failure of the script - syntax error, interpretor not
> found, Jenkin/Hudson connection reset, anything not directly related
> to the test cases themselves.
> (b) Real failure of test cases due.
>

Right. I have to agree that the difference between a and b is valid.
In practice i consider both a and b a failure in the test suite that
needs to be fixed. Because it's either a problem in the configuration
or the new code.
However in the case of python I think it's ok to have a bad return
code on test failure.