[TIP] How to test plugins?

[Zbigniew Reszela]

Hello,

I work on a project which highly uses plugins. The plugin developers always
complain that it is very hard to understand how the plugins are really used
by the system and testing plugins while developing them always require
running an instance of the whole system, restarting it to apply changes to
the plugins, etc. To make the developer’s life easier I was thinking on
providing a framework for testing the plugins.


Different plugins must behave the same under given circumstances so I think
it will be even possible to have a battery of tests that after a simple
configuration e.g. the plugin class name and a few parameters will be ready
to use. The tests will need to execute the logic in the same way as the
system does so I think I will write unit tests - there will be no need to
integrate them with the classes from the system. Some of the tests will be
able to call and test a single methods of the plugins - nice. But most of
them will need to perform a sequence of calls to different methods before
we get to the method that we want to test - not so nice. And last but not
least, the plugin works with the real hardware (instrumentation equipment).

I would like to cover the following requirements:

   -

   Using the framework allows TDD of the plugins
   -

   Tests help developers to understand how the system uses the plugins.
   -

   Tests are easily parameterizable.
   -

   Given tests can be easily repeated multiple times - stress testing.
   -

   Tests do time and memory profiling (I was thinking about using profile
   and memory_profiler modules)
   -

   The most complex tests, the ones that execute the whole action involving
   calls to many methods of the plugins, support hook mechanism (execute an
   arbitrary code between the calls to the plugin).


I would appreciate any feedback on the conceptual level and any hint on the
technical level e.g. libraries that could help me (I worked only with the
python’s unittest module up to now and in our project we have already
developed a decorator for easy insertion of the parameterizable tests to
the TestCase class).

I’m also looking for other project that have done something similar? Do you
know any?

Many thanks in advance for sharing your experience!
Zibi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20170315/be9d2800/attachment.html>