[TIP] Performance Tests

Julius B. Lucks julius at younglucks.com
Thu Aug 23 14:04:07 PDT 2007

(I would be happy to write more up, or post this on a TIP web page?)

Below is a brief description of how I set up funkload to do  
functional tests on a SOAP web service that is described by a WSDL  
file.  I have been writing the web service code with the Zolera SOAP  
Infrastructure in the ZSI.py module.

ZSI can be gotten from http://pywebsvcs.sourceforge.net/
funkload can be gotten from http://funkload.nuxeo.org/

I will assume you have a web service up and running.  Alternatively  
you can probably plug in whatever wsdl file you want (although you  
want to make sure you don't unintentionally launch a DOS attack by  
load testing someone else's web service.)

Step 1: Install funkload.  Following the funkload webpage, I  
installed funkload 1.6.2 with easy_install-2.5 (I am using python 2.5  
installed via fink on Mac OS X 10.4)

     sudo easy_install -U funkload

This also installed webunit-1.3.8.  Once installed, there are several  
commands now available:


and several others.  I'll be describing these four in these notes.

Step 2: Setup your tests with funkload.  Funkload is built on top of  
unittest, which is the unittesting framework in the stdlib.  One of  
the reasons I chose funkload is because I already had my projects  
unit tests written with unittests.  This doesn't mean you can just  
run your existing test with funkload and be done.  Funkload has a  
class called FunkLoadTestCase which is derived from  
unittest.TestCase.  It has the same flavor of unittest.TestCase, but  
adds logging information into the tests.  We'll see this more  
concretely below.

To get things up and running the fastest, I took a look at the demo  
code.  I did


which created a funkload-demo in my current directory.  The most  
useful example for my case was in the xmlrpc subdirectory.  The most  
important files to take a look at now, are the test_Credential.py  
file and Credential.conf file.  Funkload tests make use of a config  
file that is named after the test class.  To create my tests, I  
copied these two files into another directory and named them  
test_ws.py and ws.conf.

The web service I am testing has one method called 'query', which  
takes named arguments: query_string - a query, id_list - a list of  
strings representing id's to filter by the query.

Below are the contents of these files with explanation:

# FunkLoad test configuration file
# Main section
title=Test the my web service
description=Try to test query method
# the server url to test - I am running the server locally
# this is the location of my wsdl file describing the service
#  (same directory as the tests)

# Tests description and configuration
description=Check query method

# ------------------------------------------------------------
# Configuration for unit test mode fl-run-test

# log_to destination =
# console - to the screen
# file - to a file
log_to = console file

# log_path = path and file name to store log file
log_path = ws-test.log

# result_path = path to store the xml result file
result_path = ws-test.xml

# sleeptime_min = minimum amount of time in seconds to sleep between  
#                 to the host
sleep_time_min = 0

# sleeptime_max = maximum amount of time in seconds to sleep between  
#                 to the host
sleep_time_max = 0

# ------------------------------------------------------------
# Configuration for bench mode fl-run-bench

# cycles = list of cycles with their number of concurrent users
cycles = 1:20:40

# duration = duration of a cycle in seconds
duration = 20

# startup_delay = time to wait between starting-up threads in seconds
startup_delay = 0.05

# sleep_time = time to wait between test in seconds
sleep_time = 0.5

# cycle_time = time to wait between cycle in seconds
cycle_time = 1

# same keys than in [ftest] section
log_to = file
log_path = ws-bench.log
result_path = ws-bench.xml

sleep_time_min = .1
sleep_time_max = .2


The config file lists setup information and parameters to use when  
testing.  See http://funkload.nuxeo.org/#benching for an explanation  
of the parameters.

To get my tests to work, I had to dig into the funkload code a bit to  
figure out how it does the FunkLoadTestCase.xmlrpc method.  I needed  
to modify it to be able to test my SOAP service described with my  
wsdl.  I wanted to make a class that derived from FunkLoadTestCase,  
and further derive that for my test cases (instead of  
unittest.TestCase).  However, I got some errors about wrong arguments  
called on __init__ with a cryptic trace.  So I opted for the quick  
hack (for now), of just adding my soap_wsdl method to my test case  
class directly.  This is good enough for now:

"""Simple FunkLoad test for WS

run test with
   > fl-run-test -v test_ws.py
run bench with
   > fl-run-bench test_ws.py WS.test_ws
import unittest
from random import random
import time

# I am using ZSI to set up the binding to the SOAP service
from ZSI.ServiceProxy import ServiceProxy
from funkload.FunkLoadTestCase import FunkLoadTestCase

class WS(FunkLoadTestCase):
     """This test use a configuration file ws.conf."""

     def setUp(self):
         """Setting up test."""

         #get configuration details from ws.conf
         self.server_url = self.conf_get('main', 'url')
         self.wsdl = self.conf_get('main','wsdl')

     def soap_wsdl(self, wsdl, url_in, method_name,
                   named_params=None, description=None):
         """Call a SOAP method_name specified in wsdl with named_params.

         named_params is a dict of params.
         Modifying FunkLoadTestCase.xmlrpc method.
         Copying method and providing SOAP w/ wsdl details.
         self.steps += 1
         self.page_responses = 0
         self.logd('SOAP: %s::%s\n\tCall %i: %s ...' % (wsdl+url_in,
or ''))
         response = None
         t_start = time.time()
         if self._authinfo is not None:
             url = url_in.replace('//', '//'+self._authinfo)
             url = url_in
             server = ServiceProxy(wsdl,url=url)
             method = getattr(server, method_name)
             if named_params is not None:
                 response = method(**named_params)
                 response = method()
             etype, value, tback = sys.exc_info()
             t_stop = time.time()
             t_delta = t_stop - t_start
             self.total_time += t_delta
             self.step_success = False
             self.test_status = 'Error'
             self.logd(' Failed in %.3fs' % t_delta)
             self._log_xmlrpc_response(wsdl+url_in, method_name,  
                                       response, t_start, t_stop, -1)
             if etype is SocketError:
                 raise SocketError("Can't access %s." % wsdl+url)
         t_stop = time.time()
         t_delta = t_stop - t_start
         self.total_time += t_delta
         self.total_xmlrpc += 1
         self.logd(' Done in %.3fs' % t_delta)
         self._log_xmlrpc_response(wsdl+url_in, method_name,
                                   description, response,
                                   t_start, t_stop, 200)
         return response

     def test_ws(self):
         """ my test """
         server_url = self.server_url
         wsdl = self.wsdl

         ret = self.soap_wsdl(wsdl, server_url, 'query',
                 named_params = {'search_query': 'test query',
                                 'id_list': []},
                 description="Check simple query")

                         'Wrong Return Type, expected dict %s' % ret)

         self.logd('ret %s' % ret)

     def tearDown(self):
         """Setting up test."""

if __name__ in ('main', '__main__'):


Here I mostly followed the examples I cited above.  The only  
complication was adding my soap_wsdl method so I could run the test.

3.) Run the tests.  I fired up my server on http://localhost:8080,  
and executed

     fl-run-test -v test_ws.py

This gave me what looked like a typical unittest  output to the  
console, as well as created ws-test.log and ws-test.xml describing  
the test.

4.) Run a bench.  The following did load testing by running the tests  
over with different numbers of concurrent threads, using the configs  
we setup earlier:

     fl-run-bench test_ws.py WS.test_ws

Once again, output was to the console, with ws-bench.log and ws- 
bench.xml being created which described the benchmark testing.

5.) Create a report.  You can generate a nice report of the  
benchmarking logs by doing

     fl-build-report ws-bench.xml


     fl-build-report --html ws-bench.xml

These notes outlined the basics of getting setup with funkloader.   
There are other rich features.  See http://funkload.nuxeo.org/ for  
more details.

Hope this was helpful,


Please Reply to My Permanent Address: julius at younglucks.com

On Aug 23, 2007, at 4:12 PM, Grig Gheorghiu wrote:

> Hi Julius,
> A short write-up on how you set up funkload would be greatly
> appreciated by the TIP audience I'm sure :-) The last time I looked at
> it, it wasn't trivial to configure.
> Grig
> --- "Julius B. Lucks" <julius at younglucks.com> wrote:
>> Hey Sylvain,
>> Many thanks for the tips.  It looks like funkload [3] is the best for
>> my needs.  The others look a little heavy for this initial testing.
>> Funkload has been really easy so far to set up and run, especially
>> since I have been unittest to do my testing anyway.  The only thing I
>> had to do was to write my own method within the FunkLoadTestCase
>> class to handle calling SOAP methods described by WSDL files -
>> essentially a modification of the Funkload xmlrpc method.
>> Cheers,
>> Julius
> ---------------------------------------------------------------------- 
> --
>> ---------------
>> Please Reply to My Permanent Address: julius at younglucks.com
>> http://www.openwetware.org/wiki/User:Julius_B._Lucks
> ---------------------------------------------------------------------- 
> --
>> ----------------
>> On Aug 23, 2007, at 9:37 AM, Sylvain Hellegouarch wrote:
>>> Depending on your requirements, you may use expensive tools like
>>> LoadRunner from HP (formerly Mercury) but be ready to spit the
>> cash.
>>> More seriously, there are several tools that do a decent job in
>>> that field for free, such as WebLoad [1] or Tsung[2]. I personally
>>> like Funkload [3] as you design your performance tests as glorified
>>> unit tests and funkload does the job of  measuring, monitoring and
>>> gathering data. Simplistic but great to a quick load/perf. test
>>> mockup I think.
>>> [1] http://www.webload.org/
>>> [2] http://www.process-one.net/en/tsung/
>>> [3] http://funkload.nuxeo.org/
>>> - Sylvain
>>> Julius B. Lucks a écrit :
>>>> Hi All,
>>>> I am writing a web service server with SOAP/WSDL in python using
>>>> the Zolera Soap Infrastructure module (ZSI - http://
>>>> pywebsvcs.sourceforge.net/).  I have a stub implementation of my
>>>> service up and running with unittests covering the functionality
>>>> of the service.  Now I want to create some performance tests/
>>>> benchmarks so that I can get an idea of the resources the service
>>>> is requiring, and compare it to future versions.
>>>> What is the best way to do these performance tests?  Time my
>>>> unittests? Profile a set of calls on the service, and record the
>>>> timings? Should I seperately do tests on the backend, and backend
>>>> +SOAP messaging?  Are there any tools for this?
>>>> I appreciate any tips.
>>>> Cheers,
>>>> Julius
>> ---------------------------------------------------------------------
>>>> ------------------
>>>> Please Reply to My Permanent Address: julius at younglucks.com
>>>> <mailto:julius at younglucks.com>
>>>> http://www.openwetware.org/wiki/User:Julius_B._Lucks
>> ---------------------------------------------------------------------
>>>> -------------------
>> ---------------------------------------------------------------------
>>>> ---
>>>> _______________________________________________
>>>> testing-in-python mailing list
>>>> testing-in-python at lists.idyll.org
>>>> http://lists.idyll.org/listinfo/testing-in-python
>>> _______________________________________________
>> testing-in-python mailing list
>> testing-in-python at lists.idyll.org
>> http://lists.idyll.org/listinfo/testing-in-python

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.idyll.org/pipermail/testing-in-python/attachments/20070823/5f23f832/attachment-0001.htm 

More information about the testing-in-python mailing list