[TIP] py.test best way to generate a *lot* of tests

[Thijs Engels]

Holger,

Be aware that due to the use of SQLAlchemy the test code itself is not
really using anything else than (mapped) objects and properties. I am
trying to port a current test script which is using the plain assert
statements from Python itself. This works, but merely to indicate there
is an error. It does not allow reporting or other testing tool specific
output and/or options. However not being limited to any of the required
setup for testing I could just iterate through all the data (mapping is
done via SQLAlchemy).

For instance:

for session in session_list:
    assert session.id > 0
    [...]

    for area in session.areas:
        assert area.curves.keys() == range(1, 25)
        [...]

        for curve in area.curves:
             assert curve.sense in ['supply', 'demand']
             [...]

             for point in curve.points:
                 assert point.quantity >= 0
                 [...]

As indicated in my previous email, without looking up the information
one does not know how many Area objects are linked to a Session, or how
many Point objects are available in the curves. In my current
experimenting I am initially going through all these objects before
testing, in order to feed them to the pytest_generate_tests. But this
way I end up with filling lists containing thousands of objects while
initially it 'felt' better by just iterating my way through the data.

Nevertheless the output options as well as the feedback provided for
failing test make me want to try to get this to work :)

Thijs

On Wed, Jun 13, 2012, at 14:38, holger krekel wrote:
> Hey Thijs,
> 
> not sure i exactly understand your problem yet ...
> 
> On Wed, Jun 13, 2012 at 15:22 +0200, Thijs Engels wrote:
> > I am currently experimenting with py.test to validate result which are
> > written to a database. Hence more functional testing than unittesting.
> > The database interface is completely dealt with via SQLAlchemy, hence
> > the tests merely have to iterate over all the objects.
> > 
> > The issue I am currently running into is the fact that the output
> > consists of sessions, which in itself contains specific data for this
> > session. The amount of data (objects) is however different per session.
> >
> > Which if I understand correctly need to be extracted to create a full
> > list (of objects) for the parametrization. This seems fine if were just
> > these sessions and their child, but there are more levels, and worse
> > there are thousands of objects.
> 
> Could you try to show some example (basic/pseudo) code of what you want
> to
> achieve?
>  
> > I do realize I might be using this test framework for something it was
> > not really meant for, but is it indeed only possible to have py.test go
> > through all these objects if the first step is creating (huge) lists
> > with the actual objects to be fed into the parametrize function?
> 
> Did you look into the pytest_generate_tests hook where you can write
> code (interacting with a database, for example) which then triggers
> parametrization?  Here is a basic example:
> 
>     http://pytest.org/latest/example/parametrize.html#generating-parameters-combinations-depending-on-command-line
>     
> FWIW, i consider it within pytest's mission to support database-driven
> parametrized testing so if there are issues i'd like to help work them
> out.
> 
> best,
> holger