[TIP] writing a functional test framework using py.test or proboscis

[holger krekel]

On Thu, May 10, 2012 at 12:00 -0400, Pella,Chris wrote:
> Thanks Holger,
> I'll check out if Tox will map to some of our use cases after some re-factoring. It looks promising.
> The funcargs mechanism will take some time for me to get my head around since I've only done relatively unsophisticated python programming until now.

Hum, I'd like to revisit the funcargs documentation to improve and extend it.
Happy to hear any feedback from your perspective and to answer questions
or give examples related to particular use cases.

best,
holger


> 
> Chris 
> 
> -----Original Message-----
> From: holger krekel [mailto:holger at merlinux.eu] 
> Sent: Thursday, May 10, 2012 11:17 AM
> To: Pella,Chris
> Cc: testing-in-python at lists.idyll.org
> Subject: Re: [TIP] writing a functional test framework using py.test or proboscis
> 
> Hi Chris,
> 
> On Thu, May 10, 2012 at 06:25 -0400, Pella,Chris wrote:
> > We are tasked with writing a test framework which integrates tools written in other languages, and are using python as the glue.  Much of this can be done by wrapping the external tools in either pexpect,  or subprocess. Right now there isn't time to write python bindings for our c or java libraries but that is the preferred way to go in the long run since management wants things happening yesterday (we have virtually no automated testing now).  We are testing systems with a combination of embedded hardware which could be on a local bus or network-attached, with clients that communicate with them either through a library api or various management interfaces, which could be cli-based or have a rest api.
> > 
> > My issue is that this isn't the normal scenario for using unittest-type frameworks, since there are many so external dependancies.  I liked the way I could organize the test dependencies in proboscis and got something very clean-looking up in minutes. e.g. the first test loads a new firmware build if available and subsequent tests would either run or not run based on that pre-requisite. However, I couldn't get any nose plugins to work ( it is built on top of nose) and would like at least have  xunit output  to send to Jenkins at some point, or at least transform into html for reporting.  I would really like to use this tool but I get the feeling I'm going to have to do some poking around to make it generate the output I need and I'm not sure I can succeed since I'm not sure I have the skill-level at this point.
> 
> I don't use nose much myself but i am sure this can be solved by someone
> who does.
> 
> > We are also trying py.test, but I'm having trouble figuring out how to deal with external dependancies in a clean way.  The person who is doing this work is using some global variables to set a condition and then skipping tests based on that, and I'm really not happy with that. I'm wondering if there is somebody who has dealt with more complex scenarios using py.test or some other framework and can offer an example of how they dealt with it.
> 
> You may want to look into using "tox", see http://tox.testrun.org which
> automates the creation of virtual environments, installation of
> dependencies and running the actual test command.  There also are some
> specific docs on how to set it up with Jenkins, see 
> http://tox.testrun.org/latest/example/jenkins.html 
> irrespective if you are using nose or pytest.
> 
> With respect to creating whole-testrun fixtures which require controling
> and management of other processes/firmware/hardware you might want to
> look into pytest's resource management (funcargs) which can be used 
> to effectively manage external state, see for example:
> http://pytest.org/latest/example/mysetup.html
> If you use tox you may avoid having to skip/check existence of
> dependencies.
> 
> hope this helps,
> holger (who maintains both tox and pytest)
> 
> > Also, does anybody have any experience using Jenkins to drive distributed functional tests in python? Our builds engineers are reluctant to move to Jenkins because they have this Stygian stable of make files, perl scripts, batch files etc... driven by a php front-end to run builds. I have an idea that by adopting Jenkins to drive test automation it may help convince them to adopt it and we could have some integration between builds and tests , which we don't have now. I doubt we could have Continuous Integration running easily because of the complexity of builds but it would be a step in the right direction.
> > 
> > Chris
> > 
> > 
> > The information contained in this electronic mail transmission 
> > may be privileged and confidential, and therefore, protected 
> > from disclosure. If you have received this communication in 
> > error, please notify us immediately by replying to this 
> > message and deleting it from your computer without copying 
> > or disclosing it.
> > 
> > 
> > 
> > _______________________________________________
> > testing-in-python mailing list
> > testing-in-python at lists.idyll.org
> > http://lists.idyll.org/listinfo/testing-in-python
> > 
> The information contained in this electronic mail transmission 
> may be privileged and confidential, and therefore, protected 
> from disclosure. If you have received this communication in 
> error, please notify us immediately by replying to this 
> message and deleting it from your computer without copying 
> or disclosing it.
> 
> 
>