[TIP] testing-in-python Digest, Vol 64, Issue 3

Thu May 10 13:11:39 PDT 2012

On Thu, May 10, 2012 at 14:15 -0500, John Minne wrote:
> I thought tox is for python source projects.  If so I don't see how it
> could help Chris with his current "python as glue" project.

You have a point - they probably don't want to distribute python source
packages and this is partly what tox is meant to help with.

However, they have some deps (like pexpect i guess) which theyd like to
see managed automatically.  That is where tox could help.

I am open to add an option to tox to skip the packaging and installation
step so that only the virtualenv-management and test running remains.

Of course there are certainly also solutions that don't involve tox
like writing some of what tox does into a simple script or so.

best,
holger

> As far as using python and jenkins as a test framework, I highly recommend
> it.  I'm using nose with it's xunit plugin for output that is consumed by
> jenkins.   But if I was starting a new project today I would have a tough
> time picking between pytest and nose.  I might even try Robot if I had the
> time.
> 
> -John Minne
> 
> 
> On Thu, May 10, 2012 at 2:00 PM,
> <testing-in-python-request at lists.idyll.org>wrote:
> 
> > Send testing-in-python mailing list submissions to
> >        testing-in-python at lists.idyll.org
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >        http://lists.idyll.org/listinfo/testing-in-python
> > or, via email, send a message with subject or body 'help' to
> >        testing-in-python-request at lists.idyll.org
> >
> > You can reach the person managing the list at
> >        testing-in-python-owner at lists.idyll.org
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of testing-in-python digest..."
> >
> > Today's Topics:
> >
> >   1. writing a functional test framework using py.test or
> >      proboscis (Pella,Chris)
> >   2. Re: writing a functional test framework using py.test or
> >      proboscis (holger krekel)
> >   3. Re: writing a functional test framework using py.test or
> >      proboscis (Pella,Chris)
> >   4. Re: writing a functional test framework using py.test or
> >      proboscis (holger krekel)
> >
> >
> > ---------- Forwarded message ----------
> > From: "Pella,Chris" <Chris.Pella at safenet-inc.com>
> > To: "testing-in-python at lists.idyll.org" <testing-in-python at lists.idyll.org
> > >
> > Cc:
> > Date: Thu, 10 May 2012 06:25:22 -0400
> > Subject: [TIP] writing a functional test framework using py.test or
> > proboscis
> > Hi all,
> >
> > We are tasked with writing a test framework which integrates tools written
> > in other languages, and are using python as the glue.  Much of this can be
> > done by wrapping the external tools in either pexpect,  or subprocess.
> > Right now there isn't time to write python bindings for our c or java
> > libraries but that is the preferred way to go in the long run since
> > management wants things happening yesterday (we have virtually no automated
> > testing now).  We are testing systems with a combination of embedded
> > hardware which could be on a local bus or network-attached, with clients
> > that communicate with them either through a library api or various
> > management interfaces, which could be cli-based or have a rest api.
> >
> > My issue is that this isn't the normal scenario for using unittest-type
> > frameworks, since there are many so external dependancies.  I liked the way
> > I could organize the test dependencies in proboscis and got something very
> > clean-looking up in minutes. e.g. the first test loads a new firmware build
> > if available and subsequent tests would either run or not run based on that
> > pre-requisite. However, I couldn't get any nose plugins to work ( it is
> > built on top of nose) and would like at least have  xunit output  to send
> > to Jenkins at some point, or at least transform into html for reporting.  I
> > would really like to use this tool but I get the feeling I'm going to have
> > to do some poking around to make it generate the output I need and I'm not
> > sure I can succeed since I'm not sure I have the skill-level at this point.
> >
> > We are also trying py.test, but I'm having trouble figuring out how to
> > deal with external dependancies in a clean way.  The person who is doing
> > this work is using some global variables to set a condition and then
> > skipping tests based on that, and I'm really not happy with that. I'm
> > wondering if there is somebody who has dealt with more complex scenarios
> > using py.test or some other framework and can offer an example of how they
> > dealt with it.
> >
> > Also, does anybody have any experience using Jenkins to drive distributed
> > functional tests in python? Our builds engineers are reluctant to move to
> > Jenkins because they have this Stygian stable of make files, perl scripts,
> > batch files etc... driven by a php front-end to run builds. I have an idea
> > that by adopting Jenkins to drive test automation it may help convince them
> > to adopt it and we could have some integration between builds and tests ,
> > which we don't have now. I doubt we could have Continuous Integration
> > running easily because of the complexity of builds but it would be a step
> > in the right direction.
> >
> > Chris
> >
> >
> > The information contained in this electronic mail transmission
> > may be privileged and confidential, and therefore, protected
> > from disclosure. If you have received this communication in
> > error, please notify us immediately by replying to this
> > message and deleting it from your computer without copying
> > or disclosing it.
> >
> >
> >
> >
> >
> >
> >
> > ---------- Forwarded message ----------
> > From: holger krekel <holger at merlinux.eu>
> > To: "Pella,Chris" <Chris.Pella at safenet-inc.com>
> > Cc: "testing-in-python at lists.idyll.org" <testing-in-python at lists.idyll.org
> > >
> > Date: Thu, 10 May 2012 15:17:02 +0000
> > Subject: Re: [TIP] writing a functional test framework using py.test or
> > proboscis
> > Hi Chris,
> >
> > On Thu, May 10, 2012 at 06:25 -0400, Pella,Chris wrote:
> > > We are tasked with writing a test framework which integrates tools
> > written in other languages, and are using python as the glue.  Much of this
> > can be done by wrapping the external tools in either pexpect,  or
> > subprocess. Right now there isn't time to write python bindings for our c
> > or java libraries but that is the preferred way to go in the long run since
> > management wants things happening yesterday (we have virtually no automated
> > testing now).  We are testing systems with a combination of embedded
> > hardware which could be on a local bus or network-attached, with clients
> > that communicate with them either through a library api or various
> > management interfaces, which could be cli-based or have a rest api.
> > >
> > > My issue is that this isn't the normal scenario for using unittest-type
> > frameworks, since there are many so external dependancies.  I liked the way
> > I could organize the test dependencies in proboscis and got something very
> > clean-looking up in minutes. e.g. the first test loads a new firmware build
> > if available and subsequent tests would either run or not run based on that
> > pre-requisite. However, I couldn't get any nose plugins to work ( it is
> > built on top of nose) and would like at least have  xunit output  to send
> > to Jenkins at some point, or at least transform into html for reporting.  I
> > would really like to use this tool but I get the feeling I'm going to have
> > to do some poking around to make it generate the output I need and I'm not
> > sure I can succeed since I'm not sure I have the skill-level at this point.
> >
> > I don't use nose much myself but i am sure this can be solved by someone
> > who does.
> >
> > > We are also trying py.test, but I'm having trouble figuring out how to
> > deal with external dependancies in a clean way.  The person who is doing
> > this work is using some global variables to set a condition and then
> > skipping tests based on that, and I'm really not happy with that. I'm
> > wondering if there is somebody who has dealt with more complex scenarios
> > using py.test or some other framework and can offer an example of how they
> > dealt with it.
> >
> > You may want to look into using "tox", see http://tox.testrun.org which
> > automates the creation of virtual environments, installation of
> > dependencies and running the actual test command.  There also are some
> > specific docs on how to set it up with Jenkins, see
> > http://tox.testrun.org/latest/example/jenkins.html
> > irrespective if you are using nose or pytest.
> >
> > With respect to creating whole-testrun fixtures which require controling
> > and management of other processes/firmware/hardware you might want to
> > look into pytest's resource management (funcargs) which can be used
> > to effectively manage external state, see for example:
> > http://pytest.org/latest/example/mysetup.html
> > If you use tox you may avoid having to skip/check existence of
> > dependencies.
> >
> > hope this helps,
> > holger (who maintains both tox and pytest)
> >
> > > Also, does anybody have any experience using Jenkins to drive
> > distributed functional tests in python? Our builds engineers are reluctant
> > to move to Jenkins because they have this Stygian stable of make files,
> > perl scripts, batch files etc... driven by a php front-end to run builds. I
> > have an idea that by adopting Jenkins to drive test automation it may help
> > convince them to adopt it and we could have some integration between builds
> > and tests , which we don't have now. I doubt we could have Continuous
> > Integration running easily because of the complexity of builds but it would
> > be a step in the right direction.
> > >
> > > Chris
> > >
> > >
> > > The information contained in this electronic mail transmission
> > > may be privileged and confidential, and therefore, protected
> > > from disclosure. If you have received this communication in
> > > error, please notify us immediately by replying to this
> > > message and deleting it from your computer without copying
> > > or disclosing it.
> > >
> > >
> > >
> > > _______________________________________________
> > > testing-in-python mailing list
> > > testing-in-python at lists.idyll.org
> > > http://lists.idyll.org/listinfo/testing-in-python
> > >
> >
> >
> >
> >
> > ---------- Forwarded message ----------
> > From: "Pella,Chris" <Chris.Pella at safenet-inc.com>
> > To: holger krekel <holger at merlinux.eu>
> > Cc: "testing-in-python at lists.idyll.org" <testing-in-python at lists.idyll.org
> > >
> > Date: Thu, 10 May 2012 12:00:50 -0400
> > Subject: Re: [TIP] writing a functional test framework using py.test or
> > proboscis
> > Thanks Holger,
> > I'll check out if Tox will map to some of our use cases after some
> > re-factoring. It looks promising.
> > The funcargs mechanism will take some time for me to get my head around
> > since I've only done relatively unsophisticated python programming until
> > now.
> >
> >
> > Chris
> >
> > -----Original Message-----
> > From: holger krekel [mailto:holger at merlinux.eu]
> > Sent: Thursday, May 10, 2012 11:17 AM
> > To: Pella,Chris
> > Cc: testing-in-python at lists.idyll.org
> > Subject: Re: [TIP] writing a functional test framework using py.test or
> > proboscis
> >
> > Hi Chris,
> >
> > On Thu, May 10, 2012 at 06:25 -0400, Pella,Chris wrote:
> > > We are tasked with writing a test framework which integrates tools
> > written in other languages, and are using python as the glue.  Much of this
> > can be done by wrapping the external tools in either pexpect,  or
> > subprocess. Right now there isn't time to write python bindings for our c
> > or java libraries but that is the preferred way to go in the long run since
> > management wants things happening yesterday (we have virtually no automated
> > testing now).  We are testing systems with a combination of embedded
> > hardware which could be on a local bus or network-attached, with clients
> > that communicate with them either through a library api or various
> > management interfaces, which could be cli-based or have a rest api.
> > >
> > > My issue is that this isn't the normal scenario for using unittest-type
> > frameworks, since there are many so external dependancies.  I liked the way
> > I could organize the test dependencies in proboscis and got something very
> > clean-looking up in minutes. e.g. the first test loads a new firmware build
> > if available and subsequent tests would either run or not run based on that
> > pre-requisite. However, I couldn't get any nose plugins to work ( it is
> > built on top of nose) and would like at least have  xunit output  to send
> > to Jenkins at some point, or at least transform into html for reporting.  I
> > would really like to use this tool but I get the feeling I'm going to have
> > to do some poking around to make it generate the output I need and I'm not
> > sure I can succeed since I'm not sure I have the skill-level at this point.
> >
> > I don't use nose much myself but i am sure this can be solved by someone
> > who does.
> >
> > > We are also trying py.test, but I'm having trouble figuring out how to
> > deal with external dependancies in a clean way.  The person who is doing
> > this work is using some global variables to set a condition and then
> > skipping tests based on that, and I'm really not happy with that. I'm
> > wondering if there is somebody who has dealt with more complex scenarios
> > using py.test or some other framework and can offer an example of how they
> > dealt with it.
> >
> > You may want to look into using "tox", see http://tox.testrun.org which
> > automates the creation of virtual environments, installation of
> > dependencies and running the actual test command.  There also are some
> > specific docs on how to set it up with Jenkins, see
> > http://tox.testrun.org/latest/example/jenkins.html
> > irrespective if you are using nose or pytest.
> >
> > With respect to creating whole-testrun fixtures which require controling
> > and management of other processes/firmware/hardware you might want to
> > look into pytest's resource management (funcargs) which can be used
> > to effectively manage external state, see for example:
> > http://pytest.org/latest/example/mysetup.html
> > If you use tox you may avoid having to skip/check existence of
> > dependencies.
> >
> > hope this helps,
> > holger (who maintains both tox and pytest)
> >
> > > Also, does anybody have any experience using Jenkins to drive
> > distributed functional tests in python? Our builds engineers are reluctant
> > to move to Jenkins because they have this Stygian stable of make files,
> > perl scripts, batch files etc... driven by a php front-end to run builds. I
> > have an idea that by adopting Jenkins to drive test automation it may help
> > convince them to adopt it and we could have some integration between builds
> > and tests , which we don't have now. I doubt we could have Continuous
> > Integration running easily because of the complexity of builds but it would
> > be a step in the right direction.
> > >
> > > Chris
> > >
> > >
> > > The information contained in this electronic mail transmission
> > > may be privileged and confidential, and therefore, protected
> > > from disclosure. If you have received this communication in
> > > error, please notify us immediately by replying to this
> > > message and deleting it from your computer without copying
> > > or disclosing it.
> > >
> > >
> > >
> > > _______________________________________________
> > > testing-in-python mailing list
> > > testing-in-python at lists.idyll.org
> > > http://lists.idyll.org/listinfo/testing-in-python
> > >
> > The information contained in this electronic mail transmission
> > may be privileged and confidential, and therefore, protected
> > from disclosure. If you have received this communication in
> > error, please notify us immediately by replying to this
> > message and deleting it from your computer without copying
> > or disclosing it.
> >
> >
> >
> >
> >
> >
> >
> > ---------- Forwarded message ----------
> > From: holger krekel <holger at merlinux.eu>
> > To: "Pella,Chris" <Chris.Pella at safenet-inc.com>
> > Cc: "testing-in-python at lists.idyll.org" <testing-in-python at lists.idyll.org
> > >
> > Date: Thu, 10 May 2012 16:10:15 +0000
> > Subject: Re: [TIP] writing a functional test framework using py.test or
> > proboscis
> > On Thu, May 10, 2012 at 12:00 -0400, Pella,Chris wrote:
> > > Thanks Holger,
> > > I'll check out if Tox will map to some of our use cases after some
> > re-factoring. It looks promising.
> > > The funcargs mechanism will take some time for me to get my head around
> > since I've only done relatively unsophisticated python programming until
> > now.
> >
> > Hum, I'd like to revisit the funcargs documentation to improve and extend
> > it.
> > Happy to hear any feedback from your perspective and to answer questions
> > or give examples related to particular use cases.
> >
> > best,
> > holger
> >
> >
> > >
> > > Chris
> > >
> > > -----Original Message-----
> > > From: holger krekel [mailto:holger at merlinux.eu]
> > > Sent: Thursday, May 10, 2012 11:17 AM
> > > To: Pella,Chris
> > > Cc: testing-in-python at lists.idyll.org
> > > Subject: Re: [TIP] writing a functional test framework using py.test or
> > proboscis
> > >
> > > Hi Chris,
> > >
> > > On Thu, May 10, 2012 at 06:25 -0400, Pella,Chris wrote:
> > > > We are tasked with writing a test framework which integrates tools
> > written in other languages, and are using python as the glue.  Much of this
> > can be done by wrapping the external tools in either pexpect,  or
> > subprocess. Right now there isn't time to write python bindings for our c
> > or java libraries but that is the preferred way to go in the long run since
> > management wants things happening yesterday (we have virtually no automated
> > testing now).  We are testing systems with a combination of embedded
> > hardware which could be on a local bus or network-attached, with clients
> > that communicate with them either through a library api or various
> > management interfaces, which could be cli-based or have a rest api.
> > > >
> > > > My issue is that this isn't the normal scenario for using
> > unittest-type frameworks, since there are many so external dependancies.  I
> > liked the way I could organize the test dependencies in proboscis and got
> > something very clean-looking up in minutes. e.g. the first test loads a new
> > firmware build if available and subsequent tests would either run or not
> > run based on that pre-requisite. However, I couldn't get any nose plugins
> > to work ( it is built on top of nose) and would like at least have  xunit
> > output  to send to Jenkins at some point, or at least transform into html
> > for reporting.  I would really like to use this tool but I get the feeling
> > I'm going to have to do some poking around to make it generate the output I
> > need and I'm not sure I can succeed since I'm not sure I have the
> > skill-level at this point.
> > >
> > > I don't use nose much myself but i am sure this can be solved by someone
> > > who does.
> > >
> > > > We are also trying py.test, but I'm having trouble figuring out how to
> > deal with external dependancies in a clean way.  The person who is doing
> > this work is using some global variables to set a condition and then
> > skipping tests based on that, and I'm really not happy with that. I'm
> > wondering if there is somebody who has dealt with more complex scenarios
> > using py.test or some other framework and can offer an example of how they
> > dealt with it.
> > >
> > > You may want to look into using "tox", see http://tox.testrun.org which
> > > automates the creation of virtual environments, installation of
> > > dependencies and running the actual test command.  There also are some
> > > specific docs on how to set it up with Jenkins, see
> > > http://tox.testrun.org/latest/example/jenkins.html
> > > irrespective if you are using nose or pytest.
> > >
> > > With respect to creating whole-testrun fixtures which require controling
> > > and management of other processes/firmware/hardware you might want to
> > > look into pytest's resource management (funcargs) which can be used
> > > to effectively manage external state, see for example:
> > > http://pytest.org/latest/example/mysetup.html
> > > If you use tox you may avoid having to skip/check existence of
> > > dependencies.
> > >
> > > hope this helps,
> > > holger (who maintains both tox and pytest)
> > >
> > > > Also, does anybody have any experience using Jenkins to drive
> > distributed functional tests in python? Our builds engineers are reluctant
> > to move to Jenkins because they have this Stygian stable of make files,
> > perl scripts, batch files etc... driven by a php front-end to run builds. I
> > have an idea that by adopting Jenkins to drive test automation it may help
> > convince them to adopt it and we could have some integration between builds
> > and tests , which we don't have now. I doubt we could have Continuous
> > Integration running easily because of the complexity of builds but it would
> > be a step in the right direction.
> > > >
> > > > Chris
> > > >
> > > >
> > > > The information contained in this electronic mail transmission
> > > > may be privileged and confidential, and therefore, protected
> > > > from disclosure. If you have received this communication in
> > > > error, please notify us immediately by replying to this
> > > > message and deleting it from your computer without copying
> > > > or disclosing it.
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > testing-in-python mailing list
> > > > testing-in-python at lists.idyll.org
> > > > http://lists.idyll.org/listinfo/testing-in-python
> > > >
> > > The information contained in this electronic mail transmission
> > > may be privileged and confidential, and therefore, protected
> > > from disclosure. If you have received this communication in
> > > error, please notify us immediately by replying to this
> > > message and deleting it from your computer without copying
> > > or disclosing it.
> > >
> > >
> > >
> >
> >
> >
> > _______________________________________________
> > testing-in-python mailing list
> > testing-in-python at lists.idyll.org
> > http://lists.idyll.org/listinfo/testing-in-python
> >
> >

> _______________________________________________
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
> http://lists.idyll.org/listinfo/testing-in-python