[TIP] Test added (and executed) isn't increasing apparent coverage

Skip Montanaro skip.montanaro at gmail.com
Sat Apr 4 07:34:16 PDT 2020


>
> Is your run_tests.py executing 'python ... -m test ...' in a subprocess?
>
> Have you done the necessary groundwork to enable multi-process coverage
> tracking?  I think not -- I don't see '-p' passed to 'coverage run', nor
> a 'coverage combine' command in your Makefile.
>
> Do check the documentation:
> - https://coverage.readthedocs.io/en/latest/subprocess.html (which
>   page surprises me by not mentioning 'coverage combine', TBH)
> - https://coverage.readthedocs.io/en/latest/cmd.html#combining-data-files


Thanks for your help, Marius. I'm still stuck, however. I read the docs,
replaced -a with -p and created a sitecustomize.py file in my checkout's
Lib directory. Despite exiting with a zero status, this command:

~/.local/bin/coverage run -p --source=${PWD}/Lib/rattlesnake
./Tools/scripts/run_tests.py test_rattlesnake


produces no .coverage.HOSTNAME.PID.RANDOM file. This however does:

~/.local/bin/coverage run -p --source=${PWD}/Lib/rattlesnake
$(HOME)/tmp/junk.py


The lack of any coverage data from running the unit test would seem to be
at the root of my problem.

Here's my sitecustomize.py file:

import coverage
print(">> coverage.process_startup()")
coverage.process_startup()


and when executed, you can see the print() call produces the expected
output. Alas, while every Python process clearly executed
coverage.process_startup() and the overall test run exited with zero
status, no .coverage.* files were produced.

% /home/skip/.local/bin/coverage run -p
--source=/home/skip/src/python/cpython/Lib/rattlesnake
./Tools/scripts/run_tests.py test_rattlesnake
>> coverage.process_startup()
/home/skip/src/python/cpython/python -u -W default -bb -E -m test -r -w -j
0 -u all,-largefile,-audio,-gui test_rattlesnake
>> coverage.process_startup()
Using random seed 3070170
0:00:00 load avg: 3.15 Run tests in parallel using 10 child processes
0:00:00 load avg: 3.15 [1/1] test_rattlesnake passed
>> coverage.process_startup()

== Tests result: SUCCESS ==

1 test OK.

Total duration: 462 ms
Tests result: SUCCESS
(python38) cpython% echo $?
0
(python38) cpython% find . -name '.coverage*' | wc -l
0

Explicitly executing the Python process which runs the test_rattlesnake
unit test produces no output despite successful exit status:

(python38) cpython% /home/skip/src/python/cpython/python -u -W default -bb
-E -m test -r -w -j 0 -u all,-largefile,-audio,-gui test_rattlesnake
>> coverage.process_startup()
Using random seed 9943159
0:00:00 load avg: 3.92 Run tests in parallel using 10 child processes
0:00:00 load avg: 3.92 [1/1] test_rattlesnake passed
>> coverage.process_startup()

== Tests result: SUCCESS ==

1 test OK.

Total duration: 467 ms
Tests result: SUCCESS
(python38) cpython% echo $?
0
(python38) cpython% find . -name '.coverage*' | wc -l
0


I added --debug=trace to the coverage run command line. When processing the
junk.py source file it lists the Lib/rattlesnake files I want traced, but
not when processing the test_rattlesnake unit test:

(python38) cpython% make -f Makefile.coverage 2>&1 | egrep 'coverage
run|Tracing'
/home/skip/.local/bin/coverage run -p --debug=trace
--source=/home/skip/src/python/cpython/Lib/rattlesnake
/home/skip/tmp/junk.py
Tracing '/home/skip/src/python/cpython/Lib/rattlesnake/__init__.py'
Tracing '/home/skip/src/python/cpython/Lib/rattlesnake/converter.py'
Tracing '/home/skip/src/python/cpython/Lib/rattlesnake/opcodes.py'
Tracing '/home/skip/src/python/cpython/Lib/rattlesnake/blocks.py'
Tracing '/home/skip/src/python/cpython/Lib/rattlesnake/instructions.py'
Tracing '/home/skip/src/python/cpython/Lib/rattlesnake/util.py'
/home/skip/.local/bin/coverage run -p --debug=trace
--source=/home/skip/src/python/cpython/Lib/rattlesnake
./Tools/scripts/run_tests.py test_rattlesnake


It does tell me about a bunch of other files it won't be tracing, but
doesn't mention my Lib/rattlesnake files at all.

If my reading of the documentation is correct, I am now doing what I need
to do, but for whatever reason coverage is ignoring the files I explicitly
want traced, so I get very incomplete coverage results. Any further insight
you have would be appreciated.

Skip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20200404/80efe4e9/attachment.html>


More information about the testing-in-python mailing list