[TIP] [coverage.py] concept question about branch coverage, statement coverage, and percent coverage

[Ned Batchelder]

On 3/31/20 8:19 PM, Enji Cooper wrote:
> Hi,
> 	I have a general question about a few of the tests that come packaged with coverage.py, related to “line coverage”, “statement coverage”, and “branch coverage”.
> 	The code snippet from BasicCoverageTest.test_simple is as follows:
>
> “””\
>              a = 1
>              b = 2
>              
>              c = 4
>              # Nothing here
>              d = 6
> “””
> 	Per the above example, lines 1, 2, 4, and 6 are executed, but nothing else. The line coverage is 4/4 => 100% and the number of statements executed is also 4/4, resulting in 100% line/statement coverage. There aren’t any branches in the above code, so I would put down something like “N/A”.
> 	The asserted report (abbreviated) is as follows:
>
> 	# The fields in `report` translate to "Statements, Miss, Branch, Partial Branches, Cover"
> 	lines=[1,2,4,6], report="4 0 0 0 100%
>
> 	The test asserts that lines {1, 2, 4, 6} are executed, which is a total of 4/4 lines/statements executed, no statements missed, no branches taken, no partial branches, resulting in 100% line/statement coverage. Seems legit (the expected results match my understanding).
> 	If my above understanding is correct, let’s continue on to the next example.
> 	The code snippet from CompoundStatementTest.test_elif is as follows:
>
> """\
>              a = 1; b = 2; c = 3;
>              if a == 1:
>                  x = 3
>              elif b == 2:
>                  y = 5
>              else:
>                  z = 7
>              assert x == 3
> “””
>
> 	In this case, there are 8 total lines, 11 statements (line 1 contains 3 statements) and 4 possible branches (`if a == 1`, `elif b == 2`, `else:` and `assert x == 3`). Since a=1, lines 1-3 and 8 are executed, but none of the other lines are executed. 6 of the 11 statements are executed, and only 2 of the branch statements (the first being `if a == 1`, the second being `assert x == 3`) are executed. If my understand was correct, this should result in 50% line coverage, 54.55% statement coverage, and 50% branch coverage. This however, isn’t what the test expects:
>
> 	# The fields in `report` translate to "Statements, Miss, Branch, Partial Branches, Cover, Missing"
> 	lines=[1,2,3,4,5,7,8], missing="4-7", report="7 3 4 1 45% 2->4, 4-7”
>
> 	I agree on the `lines` and `missing` value, but the rest of the values seem askew:
>
> 	* There are 10 statements, not 7 statements.
> 	* There are 4 missed statements/lines, not 3 statements/lines.
> 	* There are 4 total branches (I agree on that part at least).
> 	* There is 1 partial branch taken (err… this seems odd — which branch is that?).
> 	* The overall statement/line coverage is assumed to be 45% (how is that calculated?).
> 	* The missing lines are 2->4 (a jump statement), and 4-7.
>
> 	Could someone please explain why these expected values are as noted above?

There are only 7 statements, not 10, for two reasons: first, "else:" 
doesn't result in any compiled code. Execution never jumps to "else", it 
jumps directly to the statement after else.  Second, Python won't report 
on multiple statements in a line, so "a = 1; b = 2; c = 3;" is counted 
as one statement.

There are three missed statements because "else" is not a statement.

The partial branch is "if a == 1", because it was only ever true, never 
false.

The coverage.py FAQ 
(https://coverage.readthedocs.io/en/latest/faq.html#q-how-is-the-total-percentage-calculated) 
explains how the percentage is calculated: (statements - 
missed_statements + branch_exits - missed_branch_exits) / (statements + 
branch_exits)

In this case, that works out to (7 - 3 + 4 - 3) / (7 + 4) = 45%

As the FAQ notes, the coverage.py reports don't show the number of 
branch exits and missed branch exits.  They show the number of branches 
and number of total branches, so it's a bit hard to reconstruct the 
calculation from the reports.

--Ned.


> Thank you,
> -Enji
>
> 1. https://www.cs.odu.edu/~cs252/Book/stmtcov.html
> 2. https://www.cs.odu.edu/~cs252/Book/branchcov.html
> 3. https://www.froglogic.com/coco/statement-coverage/
> _______________________________________________
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
> http://lists.idyll.org/listinfo/testing-in-python