PyGrunn: don’t trust your coverage report - Olga Sentemova¶

Tags: pygrunn, python

(One of my summaries of a talk at the 2021 10th Dutch PyGrunn one-day python conference).

How do we know if our code works? Perhaps the requirements you were given were unclear. Perhaps there simply is an error in your code. Perhaps your software is used in the wrong way.

There are many sorts of tests. The one she focuses on is unit tests, the one we have the most influence on as programmers.

You can use a traceability matrix where you put every individual part of the requirements (“division by zero results in an error”) in columns. In the rows you mention the tests that verify the specific requirement. Lots of work, perhaps only needed for medical equipment. You’re also restrained by the (in)completeness of the requirements. And it is a manual process…

You can also use coverage.py, which checks how many lines of your code are covered/executed by your tests.

When you run coverage, you should configure it properly, so that it only reports your python files. You don’t want to include the standard library or an external library in your report. Also exclude your test files from the report, as they normally have 100% test coverage and thus inflate your score.

But you should be a bit careful with the coverage report. In python, you can have statements with “and/or” or “if/else” in them. a = 20 if b > 4 else d/c The lines might show up as “covered” in your report even though some parts of your line weren’t executed. Luckily there’s a --cov-branch flag, which can mark your complex one-line statements as “only partially tested”.

Tip: use --cov-report=html to create nicely colored html output that shows you visually which lines are covered or not.