[TIP] Performance Tests
Grig Gheorghiu
grig at gheorghiu.net
Thu Aug 23 15:03:04 PDT 2007
Excellent! Thanks a lot for posting this. We don't have a TIP web page.
Titus -- maybe we should start a google code project where we'd post
things like these, and any software snippets that looks interesting?
Grig
--- "Julius B. Lucks" <julius at younglucks.com> wrote:
> (I would be happy to write more up, or post this on a TIP web page?)
>
> Below is a brief description of how I set up funkload to do
> functional tests on a SOAP web service that is described by a WSDL
> file. I have been writing the web service code with the Zolera SOAP
>
> Infrastructure in the ZSI.py module.
>
> ZSI can be gotten from http://pywebsvcs.sourceforge.net/
> funkload can be gotten from http://funkload.nuxeo.org/
>
> I will assume you have a web service up and running. Alternatively
> you can probably plug in whatever wsdl file you want (although you
> want to make sure you don't unintentionally launch a DOS attack by
> load testing someone else's web service.)
>
> Step 1: Install funkload. Following the funkload webpage, I
> installed funkload 1.6.2 with easy_install-2.5 (I am using python 2.5
>
> installed via fink on Mac OS X 10.4)
>
> sudo easy_install -U funkload
>
> This also installed webunit-1.3.8. Once installed, there are several
>
> commands now available:
>
> fl-install-demo
> fl-run-test
> fl-run-bench
> fl-build-report
>
> and several others. I'll be describing these four in these notes.
>
> Step 2: Setup your tests with funkload. Funkload is built on top of
>
> unittest, which is the unittesting framework in the stdlib. One of
> the reasons I chose funkload is because I already had my projects
> unit tests written with unittests. This doesn't mean you can just
> run your existing test with funkload and be done. Funkload has a
> class called FunkLoadTestCase which is derived from
> unittest.TestCase. It has the same flavor of unittest.TestCase, but
>
> adds logging information into the tests. We'll see this more
> concretely below.
>
> To get things up and running the fastest, I took a look at the demo
> code. I did
>
> fl-install-demo
>
> which created a funkload-demo in my current directory. The most
> useful example for my case was in the xmlrpc subdirectory. The most
>
> important files to take a look at now, are the test_Credential.py
> file and Credential.conf file. Funkload tests make use of a config
> file that is named after the test class. To create my tests, I
> copied these two files into another directory and named them
> test_ws.py and ws.conf.
>
> The web service I am testing has one method called 'query', which
> takes named arguments: query_string - a query, id_list - a list of
> strings representing id's to filter by the query.
>
> Below are the contents of these files with explanation:
>
> ws.conf
> __BEGIN__
> # FunkLoad test configuration file
> # Main section
> #
> [main]
> title=Test the my web service
> description=Try to test query method
> # the server url to test - I am running the server locally
> url=http://localhost:8080/
> # this is the location of my wsdl file describing the service
> # (same directory as the tests)
> wsdl=ws.wsdl
>
>
> # Tests description and configuration
> #
> [test_ws]
> description=Check query method
>
> # ------------------------------------------------------------
> # Configuration for unit test mode fl-run-test
> #
> [ftest]
>
> # log_to destination =
> # console - to the screen
> # file - to a file
> log_to = console file
>
> # log_path = path and file name to store log file
> log_path = ws-test.log
>
> # result_path = path to store the xml result file
> result_path = ws-test.xml
>
> # sleeptime_min = minimum amount of time in seconds to sleep between
>
> requests
> # to the host
> sleep_time_min = 0
>
> # sleeptime_max = maximum amount of time in seconds to sleep between
>
> requests
> # to the host
> sleep_time_max = 0
>
>
> # ------------------------------------------------------------
> # Configuration for bench mode fl-run-bench
> #
> [bench]
>
> # cycles = list of cycles with their number of concurrent users
> cycles = 1:20:40
>
> # duration = duration of a cycle in seconds
> duration = 20
>
> # startup_delay = time to wait between starting-up threads in seconds
> startup_delay = 0.05
>
> # sleep_time = time to wait between test in seconds
> sleep_time = 0.5
>
> # cycle_time = time to wait between cycle in seconds
> cycle_time = 1
>
> # same keys than in [ftest] section
> log_to = file
> log_path = ws-bench.log
> result_path = ws-bench.xml
>
> sleep_time_min = .1
> sleep_time_max = .2
>
> __END__
>
> The config file lists setup information and parameters to use when
> testing. See http://funkload.nuxeo.org/#benching for an explanation
>
> of the parameters.
>
> To get my tests to work, I had to dig into the funkload code a bit to
>
> figure out how it does the FunkLoadTestCase.xmlrpc method. I needed
>
> to modify it to be able to test my SOAP service described with my
> wsdl. I wanted to make a class that derived from FunkLoadTestCase,
> and further derive that for my test cases (instead of
> unittest.TestCase). However, I got some errors about wrong arguments
>
> called on __init__ with a cryptic trace. So I opted for the quick
> hack (for now), of just adding my soap_wsdl method to my test case
> class directly. This is good enough for now:
>
> test_ws.py
> __BEGIN__
> """Simple FunkLoad test for WS
>
> run test with
> > fl-run-test -v test_ws.py
> run bench with
> > fl-run-bench test_ws.py WS.test_ws
> """
> import unittest
> from random import random
> import time
>
> # I am using ZSI to set up the binding to the SOAP service
> from ZSI.ServiceProxy import ServiceProxy
> from funkload.FunkLoadTestCase import FunkLoadTestCase
>
> class WS(FunkLoadTestCase):
> """This test use a configuration file ws.conf."""
>
> def setUp(self):
> """Setting up test."""
> self.logd("setUp")
>
> #get configuration details from ws.conf
> self.server_url = self.conf_get('main', 'url')
> self.wsdl = self.conf_get('main','wsdl')
>
>
> def soap_wsdl(self, wsdl, url_in, method_name,
> named_params=None, description=None):
> """Call a SOAP method_name specified in wsdl with
> named_params.
>
> named_params is a dict of params.
> Modifying FunkLoadTestCase.xmlrpc method.
> Copying method and providing SOAP w/ wsdl details.
> """
> self.steps += 1
> self.page_responses = 0
> self.logd('SOAP: %s::%s\n\tCall %i: %s ...' % (wsdl+url_in,
> method_name,
> self.steps,
> description
> or ''))
> response = None
> t_start = time.time()
> if self._authinfo is not None:
> url = url_in.replace('//', '//'+self._authinfo)
> else:
> url = url_in
> try:
> server = ServiceProxy(wsdl,url=url)
> method = getattr(server, method_name)
> if named_params is not None:
> response = method(**named_params)
> else:
> response = method()
> except:
> etype, value, tback = sys.exc_info()
> t_stop = time.time()
> t_delta = t_stop - t_start
> self.total_time += t_delta
> self.step_success = False
> self.test_status = 'Error'
> self.logd(' Failed in %.3fs' % t_delta)
> self._log_xmlrpc_response(wsdl+url_in, method_name,
> description,
> response, t_start, t_stop, -1)
> if etype is SocketError:
> raise SocketError("Can't access %s." % wsdl+url)
> raise
> t_stop = time.time()
> t_delta = t_stop - t_start
> self.total_time += t_delta
> self.total_xmlrpc += 1
> self.logd(' Done in %.3fs' % t_delta)
> self._log_xmlrpc_response(wsdl+url_in, method_name,
> description, response,
> t_start, t_stop, 200)
> self.sleep()
> return response
>
>
> def test_ws(self):
> """ my test """
> server_url = self.server_url
> wsdl = self.wsdl
>
> ret = self.soap_wsdl(wsdl, server_url, 'query',
> named_params = {'search_query': 'test query',
> 'id_list': []},
> description="Check simple query")
>
> self.assert_(isinstance(ret,dict),
> 'Wrong Return Type, expected dict %s' % ret)
>
> self.logd('ret %s' % ret)
>
> def tearDown(self):
> """Setting up test."""
> self.logd("tearDown.\n")
>
>
> if __name__ in ('main', '__main__'):
> unittest.main()
>
>
> __END__
>
> Here I mostly followed the examples I cited above. The only
> complication was adding my soap_wsdl method so I could run the test.
>
> 3.) Run the tests. I fired up my server on http://localhost:8080,
> and executed
>
> fl-run-test -v test_ws.py
>
> This gave me what looked like a typical unittest output to the
> console, as well as created ws-test.log and ws-test.xml describing
> the test.
>
> 4.) Run a bench. The following did load testing by running the tests
>
> over with different numbers of concurrent threads, using the configs
>
> we setup earlier:
>
> fl-run-bench test_ws.py WS.test_ws
>
> Once again, output was to the console, with ws-bench.log and ws-
> bench.xml being created which described the benchmark testing.
>
> 5.) Create a report. You can generate a nice report of the
> benchmarking logs by doing
>
> fl-build-report ws-bench.xml
>
> or
>
> fl-build-report --html ws-bench.xml
>
>
> These notes outlined the basics of getting setup with funkloader.
> There are other rich features. See http://funkload.nuxeo.org/ for
> more details.
>
> Hope this was helpful,
>
> Julius
>
>
------------------------------------------------------------------------
>
> ---------------
> Please Reply to My Permanent Address: julius at younglucks.com
> http://www.openwetware.org/wiki/User:Julius_B._Lucks
>
------------------------------------------------------------------------
>
> ----------------
>
>
>
> On Aug 23, 2007, at 4:12 PM, Grig Gheorghiu wrote:
>
> > Hi Julius,
> >
> > A short write-up on how you set up funkload would be greatly
> > appreciated by the TIP audience I'm sure :-) The last time I looked
> at
> > it, it wasn't trivial to configure.
> >
> > Grig
> >
> > --- "Julius B. Lucks" <julius at younglucks.com> wrote:
> >
> >> Hey Sylvain,
> >>
> >> Many thanks for the tips. It looks like funkload [3] is the best
> for
> >>
> >> my needs. The others look a little heavy for this initial
> testing.
> >>
> >> Funkload has been really easy so far to set up and run, especially
> >> since I have been unittest to do my testing anyway. The only
> thing I
> >>
> >> had to do was to write my own method within the FunkLoadTestCase
> >> class to handle calling SOAP methods described by WSDL files -
> >> essentially a modification of the Funkload xmlrpc method.
> >>
> >> Cheers,
> >>
> >> Julius
> >>
> >>
> >
>
----------------------------------------------------------------------
>
> > --
> >>
> >> ---------------
> >> Please Reply to My Permanent Address: julius at younglucks.com
> >> http://www.openwetware.org/wiki/User:Julius_B._Lucks
> >>
> >
>
----------------------------------------------------------------------
>
> > --
> >>
> >> ----------------
> >>
> >>
> >>
> >> On Aug 23, 2007, at 9:37 AM, Sylvain Hellegouarch wrote:
> >>
> >>> Depending on your requirements, you may use expensive tools like
> >>> LoadRunner from HP (formerly Mercury) but be ready to spit the
> >> cash.
> >>>
> >>> More seriously, there are several tools that do a decent job in
> >>> that field for free, such as WebLoad [1] or Tsung[2]. I
> personally
> >>
> >>> like Funkload [3] as you design your performance tests as
> glorified
> >>
> >>> unit tests and funkload does the job of measuring, monitoring
> and
> >>
> >>> gathering data. Simplistic but great to a quick load/perf. test
> >>> mockup I think.
> >>>
> >>> [1] http://www.webload.org/
> >>> [2] http://www.process-one.net/en/tsung/
> >>> [3] http://funkload.nuxeo.org/
> >>>
> >>> - Sylvain
> >>>
> >>> Julius B. Lucks a écrit :
> >>>> Hi All,
> >>>>
> >>>> I am writing a web service server with SOAP/WSDL in python using
> >>>> the Zolera Soap Infrastructure module (ZSI - http://
> >>>> pywebsvcs.sourceforge.net/). I have a stub implementation of my
> >>>> service up and running with unittests covering the functionality
> >>>> of the service. Now I want to create some performance tests/
> >>>> benchmarks so that I can get an idea of the resources the
> service
> >>
> >>>> is requiring, and compare it to future versions.
> >>>>
> >>>> What is the best way to do these performance tests? Time my
> >>>> unittests? Profile a set of calls on the service, and record the
> >>>> timings? Should I seperately do tests on the backend, and
> backend
> >>>> +SOAP messaging? Are there any tools for this?
> >>>>
> >>>> I appreciate any tips.
> >>>>
> >>>> Cheers,
> >>>>
> >>>> Julius
> >>>>
> >>>>
> >>
> ---------------------------------------------------------------------
> >>
> >>>> ------------------
> >>>> Please Reply to My Permanent Address: julius at younglucks.com
> >>>> <mailto:julius at younglucks.com>
> >>>> http://www.openwetware.org/wiki/User:Julius_B._Lucks
> >>>>
> >>
> ---------------------------------------------------------------------
> >>
> >>>> -------------------
> >>>>
> >>>>
> >>>>
> >>>>
> >>
> ---------------------------------------------------------------------
> >>
> >>>> ---
> >>>>
> >>>> _______________________________________________
> >>>> testing-in-python mailing list
> >>>> testing-in-python at lists.idyll.org
> >>>> http://lists.idyll.org/listinfo/testing-in-python
> >>>>
> >>>
> >>
> >>> _______________________________________________
> >> testing-in-python mailing list
> >> testing-in-python at lists.idyll.org
> >> http://lists.idyll.org/listinfo/testing-in-python
> >>
> >
>
>
More information about the testing-in-python
mailing list