[TIP] writing a functional test framework using py.test or proboscis

[holger krekel]

Hi Chris,

On Thu, May 10, 2012 at 06:25 -0400, Pella,Chris wrote:
> We are tasked with writing a test framework which integrates tools written in other languages, and are using python as the glue.  Much of this can be done by wrapping the external tools in either pexpect,  or subprocess. Right now there isn't time to write python bindings for our c or java libraries but that is the preferred way to go in the long run since management wants things happening yesterday (we have virtually no automated testing now).  We are testing systems with a combination of embedded hardware which could be on a local bus or network-attached, with clients that communicate with them either through a library api or various management interfaces, which could be cli-based or have a rest api.
> 
> My issue is that this isn't the normal scenario for using unittest-type frameworks, since there are many so external dependancies.  I liked the way I could organize the test dependencies in proboscis and got something very clean-looking up in minutes. e.g. the first test loads a new firmware build if available and subsequent tests would either run or not run based on that pre-requisite. However, I couldn't get any nose plugins to work ( it is built on top of nose) and would like at least have  xunit output  to send to Jenkins at some point, or at least transform into html for reporting.  I would really like to use this tool but I get the feeling I'm going to have to do some poking around to make it generate the output I need and I'm not sure I can succeed since I'm not sure I have the skill-level at this point.

I don't use nose much myself but i am sure this can be solved by someone
who does.

> We are also trying py.test, but I'm having trouble figuring out how to deal with external dependancies in a clean way.  The person who is doing this work is using some global variables to set a condition and then skipping tests based on that, and I'm really not happy with that. I'm wondering if there is somebody who has dealt with more complex scenarios using py.test or some other framework and can offer an example of how they dealt with it.

You may want to look into using "tox", see http://tox.testrun.org which
automates the creation of virtual environments, installation of
dependencies and running the actual test command.  There also are some
specific docs on how to set it up with Jenkins, see 
http://tox.testrun.org/latest/example/jenkins.html 
irrespective if you are using nose or pytest.

With respect to creating whole-testrun fixtures which require controling
and management of other processes/firmware/hardware you might want to
look into pytest's resource management (funcargs) which can be used 
to effectively manage external state, see for example:
http://pytest.org/latest/example/mysetup.html
If you use tox you may avoid having to skip/check existence of
dependencies.

hope this helps,
holger (who maintains both tox and pytest)

> Also, does anybody have any experience using Jenkins to drive distributed functional tests in python? Our builds engineers are reluctant to move to Jenkins because they have this Stygian stable of make files, perl scripts, batch files etc... driven by a php front-end to run builds. I have an idea that by adopting Jenkins to drive test automation it may help convince them to adopt it and we could have some integration between builds and tests , which we don't have now. I doubt we could have Continuous Integration running easily because of the complexity of builds but it would be a step in the right direction.
> 
> Chris
> 
> 
> The information contained in this electronic mail transmission 
> may be privileged and confidential, and therefore, protected 
> from disclosure. If you have received this communication in 
> error, please notify us immediately by replying to this 
> message and deleting it from your computer without copying 
> or disclosing it.
> 
> 
> 
> _______________________________________________
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
> http://lists.idyll.org/listinfo/testing-in-python
>