[TIP] testing-in-python Digest, Vol 64, Issue 3

[John Minne]

I thought tox is for python source projects.  If so I don't see how it
could help Chris with his current "python as glue" project.

As far as using python and jenkins as a test framework, I highly recommend
it.  I'm using nose with it's xunit plugin for output that is consumed by
jenkins.   But if I was starting a new project today I would have a tough
time picking between pytest and nose.  I might even try Robot if I had the
time.

-John Minne


On Thu, May 10, 2012 at 2:00 PM,
<testing-in-python-request at lists.idyll.org>wrote:

> Send testing-in-python mailing list submissions to
>        testing-in-python at lists.idyll.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>        http://lists.idyll.org/listinfo/testing-in-python
> or, via email, send a message with subject or body 'help' to
>        testing-in-python-request at lists.idyll.org
>
> You can reach the person managing the list at
>        testing-in-python-owner at lists.idyll.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of testing-in-python digest..."
>
> Today's Topics:
>
>   1. writing a functional test framework using py.test or
>      proboscis (Pella,Chris)
>   2. Re: writing a functional test framework using py.test or
>      proboscis (holger krekel)
>   3. Re: writing a functional test framework using py.test or
>      proboscis (Pella,Chris)
>   4. Re: writing a functional test framework using py.test or
>      proboscis (holger krekel)
>
>
> ---------- Forwarded message ----------
> From: "Pella,Chris" <Chris.Pella at safenet-inc.com>
> To: "testing-in-python at lists.idyll.org" <testing-in-python at lists.idyll.org
> >
> Cc:
> Date: Thu, 10 May 2012 06:25:22 -0400
> Subject: [TIP] writing a functional test framework using py.test or
> proboscis
> Hi all,
>
> We are tasked with writing a test framework which integrates tools written
> in other languages, and are using python as the glue.  Much of this can be
> done by wrapping the external tools in either pexpect,  or subprocess.
> Right now there isn't time to write python bindings for our c or java
> libraries but that is the preferred way to go in the long run since
> management wants things happening yesterday (we have virtually no automated
> testing now).  We are testing systems with a combination of embedded
> hardware which could be on a local bus or network-attached, with clients
> that communicate with them either through a library api or various
> management interfaces, which could be cli-based or have a rest api.
>
> My issue is that this isn't the normal scenario for using unittest-type
> frameworks, since there are many so external dependancies.  I liked the way
> I could organize the test dependencies in proboscis and got something very
> clean-looking up in minutes. e.g. the first test loads a new firmware build
> if available and subsequent tests would either run or not run based on that
> pre-requisite. However, I couldn't get any nose plugins to work ( it is
> built on top of nose) and would like at least have  xunit output  to send
> to Jenkins at some point, or at least transform into html for reporting.  I
> would really like to use this tool but I get the feeling I'm going to have
> to do some poking around to make it generate the output I need and I'm not
> sure I can succeed since I'm not sure I have the skill-level at this point.
>
> We are also trying py.test, but I'm having trouble figuring out how to
> deal with external dependancies in a clean way.  The person who is doing
> this work is using some global variables to set a condition and then
> skipping tests based on that, and I'm really not happy with that. I'm
> wondering if there is somebody who has dealt with more complex scenarios
> using py.test or some other framework and can offer an example of how they
> dealt with it.
>
> Also, does anybody have any experience using Jenkins to drive distributed
> functional tests in python? Our builds engineers are reluctant to move to
> Jenkins because they have this Stygian stable of make files, perl scripts,
> batch files etc... driven by a php front-end to run builds. I have an idea
> that by adopting Jenkins to drive test automation it may help convince them
> to adopt it and we could have some integration between builds and tests ,
> which we don't have now. I doubt we could have Continuous Integration
> running easily because of the complexity of builds but it would be a step
> in the right direction.
>
> Chris
>
>
> The information contained in this electronic mail transmission
> may be privileged and confidential, and therefore, protected
> from disclosure. If you have received this communication in
> error, please notify us immediately by replying to this
> message and deleting it from your computer without copying
> or disclosing it.
>
>
>
>
>
>
>
> ---------- Forwarded message ----------
> From: holger krekel <holger at merlinux.eu>
> To: "Pella,Chris" <Chris.Pella at safenet-inc.com>
> Cc: "testing-in-python at lists.idyll.org" <testing-in-python at lists.idyll.org
> >
> Date: Thu, 10 May 2012 15:17:02 +0000
> Subject: Re: [TIP] writing a functional test framework using py.test or
> proboscis
> Hi Chris,
>
> On Thu, May 10, 2012 at 06:25 -0400, Pella,Chris wrote:
> > We are tasked with writing a test framework which integrates tools
> written in other languages, and are using python as the glue.  Much of this
> can be done by wrapping the external tools in either pexpect,  or
> subprocess. Right now there isn't time to write python bindings for our c
> or java libraries but that is the preferred way to go in the long run since
> management wants things happening yesterday (we have virtually no automated
> testing now).  We are testing systems with a combination of embedded
> hardware which could be on a local bus or network-attached, with clients
> that communicate with them either through a library api or various
> management interfaces, which could be cli-based or have a rest api.
> >
> > My issue is that this isn't the normal scenario for using unittest-type
> frameworks, since there are many so external dependancies.  I liked the way
> I could organize the test dependencies in proboscis and got something very
> clean-looking up in minutes. e.g. the first test loads a new firmware build
> if available and subsequent tests would either run or not run based on that
> pre-requisite. However, I couldn't get any nose plugins to work ( it is
> built on top of nose) and would like at least have  xunit output  to send
> to Jenkins at some point, or at least transform into html for reporting.  I
> would really like to use this tool but I get the feeling I'm going to have
> to do some poking around to make it generate the output I need and I'm not
> sure I can succeed since I'm not sure I have the skill-level at this point.
>
> I don't use nose much myself but i am sure this can be solved by someone
> who does.
>
> > We are also trying py.test, but I'm having trouble figuring out how to
> deal with external dependancies in a clean way.  The person who is doing
> this work is using some global variables to set a condition and then
> skipping tests based on that, and I'm really not happy with that. I'm
> wondering if there is somebody who has dealt with more complex scenarios
> using py.test or some other framework and can offer an example of how they
> dealt with it.
>
> You may want to look into using "tox", see http://tox.testrun.org which
> automates the creation of virtual environments, installation of
> dependencies and running the actual test command.  There also are some
> specific docs on how to set it up with Jenkins, see
> http://tox.testrun.org/latest/example/jenkins.html
> irrespective if you are using nose or pytest.
>
> With respect to creating whole-testrun fixtures which require controling
> and management of other processes/firmware/hardware you might want to
> look into pytest's resource management (funcargs) which can be used
> to effectively manage external state, see for example:
> http://pytest.org/latest/example/mysetup.html
> If you use tox you may avoid having to skip/check existence of
> dependencies.
>
> hope this helps,
> holger (who maintains both tox and pytest)
>
> > Also, does anybody have any experience using Jenkins to drive
> distributed functional tests in python? Our builds engineers are reluctant
> to move to Jenkins because they have this Stygian stable of make files,
> perl scripts, batch files etc... driven by a php front-end to run builds. I
> have an idea that by adopting Jenkins to drive test automation it may help
> convince them to adopt it and we could have some integration between builds
> and tests , which we don't have now. I doubt we could have Continuous
> Integration running easily because of the complexity of builds but it would
> be a step in the right direction.
> >
> > Chris
> >
> >
> > The information contained in this electronic mail transmission
> > may be privileged and confidential, and therefore, protected
> > from disclosure. If you have received this communication in
> > error, please notify us immediately by replying to this
> > message and deleting it from your computer without copying
> > or disclosing it.
> >
> >
> >
> > _______________________________________________
> > testing-in-python mailing list
> > testing-in-python at lists.idyll.org
> > http://lists.idyll.org/listinfo/testing-in-python
> >
>
>
>
>
> ---------- Forwarded message ----------
> From: "Pella,Chris" <Chris.Pella at safenet-inc.com>
> To: holger krekel <holger at merlinux.eu>
> Cc: "testing-in-python at lists.idyll.org" <testing-in-python at lists.idyll.org
> >
> Date: Thu, 10 May 2012 12:00:50 -0400
> Subject: Re: [TIP] writing a functional test framework using py.test or
> proboscis
> Thanks Holger,
> I'll check out if Tox will map to some of our use cases after some
> re-factoring. It looks promising.
> The funcargs mechanism will take some time for me to get my head around
> since I've only done relatively unsophisticated python programming until
> now.
>
>
> Chris
>
> -----Original Message-----
> From: holger krekel [mailto:holger at merlinux.eu]
> Sent: Thursday, May 10, 2012 11:17 AM
> To: Pella,Chris
> Cc: testing-in-python at lists.idyll.org
> Subject: Re: [TIP] writing a functional test framework using py.test or
> proboscis
>
> Hi Chris,
>
> On Thu, May 10, 2012 at 06:25 -0400, Pella,Chris wrote:
> > We are tasked with writing a test framework which integrates tools
> written in other languages, and are using python as the glue.  Much of this
> can be done by wrapping the external tools in either pexpect,  or
> subprocess. Right now there isn't time to write python bindings for our c
> or java libraries but that is the preferred way to go in the long run since
> management wants things happening yesterday (we have virtually no automated
> testing now).  We are testing systems with a combination of embedded
> hardware which could be on a local bus or network-attached, with clients
> that communicate with them either through a library api or various
> management interfaces, which could be cli-based or have a rest api.
> >
> > My issue is that this isn't the normal scenario for using unittest-type
> frameworks, since there are many so external dependancies.  I liked the way
> I could organize the test dependencies in proboscis and got something very
> clean-looking up in minutes. e.g. the first test loads a new firmware build
> if available and subsequent tests would either run or not run based on that
> pre-requisite. However, I couldn't get any nose plugins to work ( it is
> built on top of nose) and would like at least have  xunit output  to send
> to Jenkins at some point, or at least transform into html for reporting.  I
> would really like to use this tool but I get the feeling I'm going to have
> to do some poking around to make it generate the output I need and I'm not
> sure I can succeed since I'm not sure I have the skill-level at this point.
>
> I don't use nose much myself but i am sure this can be solved by someone
> who does.
>
> > We are also trying py.test, but I'm having trouble figuring out how to
> deal with external dependancies in a clean way.  The person who is doing
> this work is using some global variables to set a condition and then
> skipping tests based on that, and I'm really not happy with that. I'm
> wondering if there is somebody who has dealt with more complex scenarios
> using py.test or some other framework and can offer an example of how they
> dealt with it.
>
> You may want to look into using "tox", see http://tox.testrun.org which
> automates the creation of virtual environments, installation of
> dependencies and running the actual test command.  There also are some
> specific docs on how to set it up with Jenkins, see
> http://tox.testrun.org/latest/example/jenkins.html
> irrespective if you are using nose or pytest.
>
> With respect to creating whole-testrun fixtures which require controling
> and management of other processes/firmware/hardware you might want to
> look into pytest's resource management (funcargs) which can be used
> to effectively manage external state, see for example:
> http://pytest.org/latest/example/mysetup.html
> If you use tox you may avoid having to skip/check existence of
> dependencies.
>
> hope this helps,
> holger (who maintains both tox and pytest)
>
> > Also, does anybody have any experience using Jenkins to drive
> distributed functional tests in python? Our builds engineers are reluctant
> to move to Jenkins because they have this Stygian stable of make files,
> perl scripts, batch files etc... driven by a php front-end to run builds. I
> have an idea that by adopting Jenkins to drive test automation it may help
> convince them to adopt it and we could have some integration between builds
> and tests , which we don't have now. I doubt we could have Continuous
> Integration running easily because of the complexity of builds but it would
> be a step in the right direction.
> >
> > Chris
> >
> >
> > The information contained in this electronic mail transmission
> > may be privileged and confidential, and therefore, protected
> > from disclosure. If you have received this communication in
> > error, please notify us immediately by replying to this
> > message and deleting it from your computer without copying
> > or disclosing it.
> >
> >
> >
> > _______________________________________________
> > testing-in-python mailing list
> > testing-in-python at lists.idyll.org
> > http://lists.idyll.org/listinfo/testing-in-python
> >
> The information contained in this electronic mail transmission
> may be privileged and confidential, and therefore, protected
> from disclosure. If you have received this communication in
> error, please notify us immediately by replying to this
> message and deleting it from your computer without copying
> or disclosing it.
>
>
>
>
>
>
>
> ---------- Forwarded message ----------
> From: holger krekel <holger at merlinux.eu>
> To: "Pella,Chris" <Chris.Pella at safenet-inc.com>
> Cc: "testing-in-python at lists.idyll.org" <testing-in-python at lists.idyll.org
> >
> Date: Thu, 10 May 2012 16:10:15 +0000
> Subject: Re: [TIP] writing a functional test framework using py.test or
> proboscis
> On Thu, May 10, 2012 at 12:00 -0400, Pella,Chris wrote:
> > Thanks Holger,
> > I'll check out if Tox will map to some of our use cases after some
> re-factoring. It looks promising.
> > The funcargs mechanism will take some time for me to get my head around
> since I've only done relatively unsophisticated python programming until
> now.
>
> Hum, I'd like to revisit the funcargs documentation to improve and extend
> it.
> Happy to hear any feedback from your perspective and to answer questions
> or give examples related to particular use cases.
>
> best,
> holger
>
>
> >
> > Chris
> >
> > -----Original Message-----
> > From: holger krekel [mailto:holger at merlinux.eu]
> > Sent: Thursday, May 10, 2012 11:17 AM
> > To: Pella,Chris
> > Cc: testing-in-python at lists.idyll.org
> > Subject: Re: [TIP] writing a functional test framework using py.test or
> proboscis
> >
> > Hi Chris,
> >
> > On Thu, May 10, 2012 at 06:25 -0400, Pella,Chris wrote:
> > > We are tasked with writing a test framework which integrates tools
> written in other languages, and are using python as the glue.  Much of this
> can be done by wrapping the external tools in either pexpect,  or
> subprocess. Right now there isn't time to write python bindings for our c
> or java libraries but that is the preferred way to go in the long run since
> management wants things happening yesterday (we have virtually no automated
> testing now).  We are testing systems with a combination of embedded
> hardware which could be on a local bus or network-attached, with clients
> that communicate with them either through a library api or various
> management interfaces, which could be cli-based or have a rest api.
> > >
> > > My issue is that this isn't the normal scenario for using
> unittest-type frameworks, since there are many so external dependancies.  I
> liked the way I could organize the test dependencies in proboscis and got
> something very clean-looking up in minutes. e.g. the first test loads a new
> firmware build if available and subsequent tests would either run or not
> run based on that pre-requisite. However, I couldn't get any nose plugins
> to work ( it is built on top of nose) and would like at least have  xunit
> output  to send to Jenkins at some point, or at least transform into html
> for reporting.  I would really like to use this tool but I get the feeling
> I'm going to have to do some poking around to make it generate the output I
> need and I'm not sure I can succeed since I'm not sure I have the
> skill-level at this point.
> >
> > I don't use nose much myself but i am sure this can be solved by someone
> > who does.
> >
> > > We are also trying py.test, but I'm having trouble figuring out how to
> deal with external dependancies in a clean way.  The person who is doing
> this work is using some global variables to set a condition and then
> skipping tests based on that, and I'm really not happy with that. I'm
> wondering if there is somebody who has dealt with more complex scenarios
> using py.test or some other framework and can offer an example of how they
> dealt with it.
> >
> > You may want to look into using "tox", see http://tox.testrun.org which
> > automates the creation of virtual environments, installation of
> > dependencies and running the actual test command.  There also are some
> > specific docs on how to set it up with Jenkins, see
> > http://tox.testrun.org/latest/example/jenkins.html
> > irrespective if you are using nose or pytest.
> >
> > With respect to creating whole-testrun fixtures which require controling
> > and management of other processes/firmware/hardware you might want to
> > look into pytest's resource management (funcargs) which can be used
> > to effectively manage external state, see for example:
> > http://pytest.org/latest/example/mysetup.html
> > If you use tox you may avoid having to skip/check existence of
> > dependencies.
> >
> > hope this helps,
> > holger (who maintains both tox and pytest)
> >
> > > Also, does anybody have any experience using Jenkins to drive
> distributed functional tests in python? Our builds engineers are reluctant
> to move to Jenkins because they have this Stygian stable of make files,
> perl scripts, batch files etc... driven by a php front-end to run builds. I
> have an idea that by adopting Jenkins to drive test automation it may help
> convince them to adopt it and we could have some integration between builds
> and tests , which we don't have now. I doubt we could have Continuous
> Integration running easily because of the complexity of builds but it would
> be a step in the right direction.
> > >
> > > Chris
> > >
> > >
> > > The information contained in this electronic mail transmission
> > > may be privileged and confidential, and therefore, protected
> > > from disclosure. If you have received this communication in
> > > error, please notify us immediately by replying to this
> > > message and deleting it from your computer without copying
> > > or disclosing it.
> > >
> > >
> > >
> > > _______________________________________________
> > > testing-in-python mailing list
> > > testing-in-python at lists.idyll.org
> > > http://lists.idyll.org/listinfo/testing-in-python
> > >
> > The information contained in this electronic mail transmission
> > may be privileged and confidential, and therefore, protected
> > from disclosure. If you have received this communication in
> > error, please notify us immediately by replying to this
> > message and deleting it from your computer without copying
> > or disclosing it.
> >
> >
> >
>
>
>
> _______________________________________________
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
> http://lists.idyll.org/listinfo/testing-in-python
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20120510/1117e6ff/attachment.htm>