[TIP] Incorrect return code py.test on Windows/Jenkins ?

[Pere Martir]

On Mon, Mar 21, 2011 at 7:27 PM, Jorge Vargas <jorge.vargas at gmail.com> wrote:
> On Mon, Mar 21, 2011 at 1:44 PM, Pere Martir <pere.martir4 at gmail.com> wrote:
>> The return code of py.test on Windows makes Jenkins to consider the
>> build "failed" if there is any failed testcase in the generated JUnit
>> XML.
>
> Why you want the build to pass if the tests failed ?

A "build" in Jenkins/Hudson has 4 status, the return code of the last
command distinguish between  (1) and (2):

(1) success/blue - the return code indicates "success" - zero usually
(2) failed/red - the return code indicates a failure - nonzero usually
(3) unstable/yellow - there is a failure test case record in JUnit XML
(4) aborted/gray - manually by user

If the executed Python script raises an exception and it's not
handled, for example, SyntaxError, the Python interpretor returns a
non-zero code and Jenkins/Hudson considers the "build" to be "failed"
- (2).

If the "execution" of the script is successful - that is, all
exceptions are caught and handled, the return code should be success
(zero), in spite of failed testcases (1). If I recall correctly, this
is the behavior of pyunit.

Otherwise there is no way to distinguish between:
(a) Execution failure of the script - syntax error, interpretor not
found, Jenkin/Hudson connection reset, anything not directly related
to the test cases themselves.
(b) Real failure of test cases due.