[TIP] coverage.py vs QGIS

[Sebastian M. Ernst]

Hi Skip,

thanks for your reply.

> (First and foremost, I think you need to explain more about what you've
> tried so far.

it's a really complex topic, so before wasting too much time on
describing details, I was simply trying to ask for any experience (at
all). Anyway, here are a few more "details" :)

QGIS is a C++/Qt5 app at the moment. At some point in the past, Python
scripting capability, a Python console inside the app and a Python
plugin system were added, see (1). The QGIS-API-to-Python bridge is
based on SIP (2). Python itself is linked into the app through
`libpython`, see (3). Therefore there is no Python command that I can
substitute with coverage.

I am primarily interested in stuff that happens in `utils.py` (4), the
`pyplugin_installer` (5) and, last but not least, individual (Python)
plugins.

For starters, I have tried to add `coverage.py` to a test plugin. QGIS
plugins are sort of Python modules which are imported on demand at QGIS
runtime. Their basic structure is documented here (6). A simple template
can be found here (7) for instance.

Plugin loading happens through (a slightly modified version of)
`builtins.__import__` (not `importlib`), see here (8) and here (9).

My assumption was that I could add `coverage.py` to a plugin's
`__init__.py` file so that it would be started when the plugin is
imported. I tried something like this (directly within `__init__.py`'s
namespace):

```python
import os
import atexit
from coverage import Coverage
_cov = Coverage(source = os.path.dirname(__file__))
_cov.start()
atexit.register(_cov.stop)
```

The use of `atexit` did not work at all, so I experimented with
triggering the stop method manually (through QGIS' Python console). This
resulted in assertion errors coming from `coverage.py`, see (10).
Looking for alternatives, I knew that QGIS does not emit a termination
signal or similar to which I could attach the stop function. So I tried
to call the stop function from within the plugin's `unload` method
inside the plugin's class. `unload` is triggered for every plugin when
QGIS wants to unload it (prior to quitting for instance). This resulted
in the same assertion error. I assumed that `coverage.py` being stared
in one file/module and stopped in another (the plugin class does not
reside within the `__init__.py` file) may be the problem, so I moved the
start of `coverage.py` into the plugin class constructor. This resulted
in either QGIS crashing right at the start or no coverage data at all as
if almost no line was hit. As a last line of defense, I moved the
`coverage.py` start code from the plugin class constructor to the plugin
class `initGui` method. `initGui` is the counterpart to `unload`. It is
called by QGIS after it has imported the plugin. Anyway, having moved
the start code to `initGui` sort of helped. QGIS did not crash anymore
and I saw a few more lines covered (still not every line that definitely
run).

In general, stopping `coverage.py` did result in rather low (if any at
all) coverage. As a workaround, calling `_cov.save()` (and NOT stopping
it at all) resulted in reasonable results. Once `coverage.py` is
invoked, there is a rather high chance that QGIS will crash when I want
to shut it down. The window becomes unresponsive and I see 100% load on
one of my CPU cores. If patient, I can observe that `coverage.py` is
then still working and agonizingly slowly producing output on stdout
(telling me that certain modules were not imported ... letter by letter
by tens of seconds each).

This description is supposed to briefly illustrate some of my
experiments. If relevant, I can share more details, code and output.

> https://coverage.readthedocs.io/en/coverage-5.1/subprocess.html
> 
> I advise you to read it carefully. I missed some things in my first
> go-round, in particular this:

I am pretty sure that I am NOT dealing with (sub-) processes here. There
is just one QGIS process (with a Python thread). My educated guess is
that the problem is much closer to threads, but not Python's threads in
general (or any of the supported libraries listed in the documentation
of `coverage.py`). I am *guessing* that it is the interaction between
Qt's event loop and its thread(s) and Python's thread that leads to
those issues. Maybe there are multiple Python threads, which would sort
of explain some of my observations, but I have not figured out how to
get this kind of information yet.

I am planning to investigate the idea of integrating `coverage.py`
directly into QGIS so I can debug both QGIS' Python code and plugins. I
have not (yet) tried to add `coverage.py` to `utils.py` (i.e. at the
heart of QGIS' Python integration itself). Maybe integrating
`coverage.py` somewhere around (3), i.e. on the C++ side where the
Python interpreter thread is started, is also a (potentially better)
option, though I have no idea how.

I hope this helps to understand where I am coming from.

Best regards,
Sebastian


----


1: https://github.com/qgis/QGIS/tree/master/python
2: https://pypi.org/project/sip/
3:
https://github.com/qgis/QGIS/blob/master/src/python/qgspythonutilsimpl.cpp#L154
4: https://github.com/qgis/QGIS/blob/master/python/utils.py
5: https://github.com/qgis/QGIS/tree/master/python/pyplugin_installer
6:
https://docs.qgis.org/3.10/en/docs/pyqgis_developer_cookbook/plugins/plugins.html#plugin-content
7:
https://github.com/planetfederal/qgis-plugin-template/tree/master/pluginname
8: https://github.com/qgis/QGIS/blob/master/python/utils.py#L298
9: https://github.com/qgis/QGIS/blob/master/python/utils.py#L731
10:
https://github.com/nedbat/coveragepy/blob/master/coverage/collector.py#L332