[TIP] Tipp: Ordering Test Files

[Gregory P. Smith]

On Tue, Oct 9, 2018 at 10:06 AM Ned Batchelder <ned at nedbatchelder.com>
wrote:

> Can you say more about why you want the tests to run in a particular
> order?  Often once the test suite grows large, pytest-xdist enters the
> picture, and there isn't a single stream of tests to order anyway.
>
> --Ned.
>

Indeed.  I find more value in test runners that effectively randomize the
order.  It makes flaky tests with side effects that leak out apparent,
improving the overall health of your test suite.  test_b was passing only
because test_a did something that made test_b's test not do what was
intended?  you'll find out rather than potentially hiding a bug.

-gps


>
>
> On 10/9/18 11:57 AM, Daniel Knüttel wrote:
> > Hi testers,
> >
> > this is something I came across several times now and it really helped
> > me and my colleagues.
> >
> > Pytest executes the tests is alphanumeric order. This can be used to
> > order the tests in a meaningful order (often it is reasonable to
> > execute the tests that way).
> >
> > I suggest the following naming scheme (adapted from the POSIX
> > configuration file naming scheme):
> >
> >       test_<XXX>_<name>.py
> >
> > where ``<XXX>`` is an integer value ranging from ``000`` to ``999`` and
> > ``<name>`` is the name you would usually give your test. I usually
> > associate the second digit with a module.
> >
> > Examples:
> >
> >       test_000_io_block_buffer.py
> >       test_001_io_block_read.py
> >       test_002_io_block_write.py
> >       test_010_bootsector.py
> >
> > Here is a real life example:
> >
> >       ============================= test session starts
> ==============================
> >       platform linux -- Python 3.5.3, pytest-3.4.0, py-1.5.2,
> pluggy-0.6.0 -- /usr/bin/python3
> >       cachedir: .pytest_cache
> >       rootdir: /home/daniel/informatik/c/bytecode_interpreter/assembler,
> inifile:
> >       collected 12 items
> >
> >       test/test_010_filecontext.py::test_getc_ungetc
> PASSED                    [  8%]
> >       test/test_011_tokenize.py::test_tokenize_1
> PASSED                        [ 16%]
> >       test/test_011_tokenize.py::test_tokenize_2
> PASSED                        [ 25%]
> >       test/test_011_tokenize.py::test_tokenize_3
> PASSED                        [ 33%]
> >       test/test_011_tokenize.py::test_tokenize_4
> PASSED                        [ 41%]
> >       test/test_012_opcodes.py::test_make_opcodes
> PASSED                       [ 50%]
> >       test/test_013_util.py::test_can_be_mark
> PASSED                           [ 58%]
> >       test/test_013_util.py::test_can_convert_to_int
> PASSED                    [ 66%]
> >       test/test_013_util.py::test_autoint
> PASSED                               [ 75%]
> >       test/test_020_basic_parsing.py::test_commands
> PASSED                     [ 83%]
> >       test/test_020_basic_parsing.py::test_mark
> PASSED                         [ 91%]
> >       test/test_020_basic_parsing.py::test_set_directive
> PASSED                [100%]
> >
> >       ========================== 12 passed in 0.02 seconds
> ===========================
> >
> > Let me know your thoughts on this naming scheme.
> >
>
>
> _______________________________________________
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
> http://lists.idyll.org/listinfo/testing-in-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20181010/1f76bce4/attachment.htm>