[TIP] Performance Tests
Julius B. Lucks
julius at younglucks.com
Thu Aug 23 14:04:07 PDT 2007
(I would be happy to write more up, or post this on a TIP web page?)
Below is a brief description of how I set up funkload to do
functional tests on a SOAP web service that is described by a WSDL
file. I have been writing the web service code with the Zolera SOAP
Infrastructure in the ZSI.py module.
ZSI can be gotten from http://pywebsvcs.sourceforge.net/
funkload can be gotten from http://funkload.nuxeo.org/
I will assume you have a web service up and running. Alternatively
you can probably plug in whatever wsdl file you want (although you
want to make sure you don't unintentionally launch a DOS attack by
load testing someone else's web service.)
Step 1: Install funkload. Following the funkload webpage, I
installed funkload 1.6.2 with easy_install-2.5 (I am using python 2.5
installed via fink on Mac OS X 10.4)
sudo easy_install -U funkload
This also installed webunit-1.3.8. Once installed, there are several
commands now available:
fl-install-demo
fl-run-test
fl-run-bench
fl-build-report
and several others. I'll be describing these four in these notes.
Step 2: Setup your tests with funkload. Funkload is built on top of
unittest, which is the unittesting framework in the stdlib. One of
the reasons I chose funkload is because I already had my projects
unit tests written with unittests. This doesn't mean you can just
run your existing test with funkload and be done. Funkload has a
class called FunkLoadTestCase which is derived from
unittest.TestCase. It has the same flavor of unittest.TestCase, but
adds logging information into the tests. We'll see this more
concretely below.
To get things up and running the fastest, I took a look at the demo
code. I did
fl-install-demo
which created a funkload-demo in my current directory. The most
useful example for my case was in the xmlrpc subdirectory. The most
important files to take a look at now, are the test_Credential.py
file and Credential.conf file. Funkload tests make use of a config
file that is named after the test class. To create my tests, I
copied these two files into another directory and named them
test_ws.py and ws.conf.
The web service I am testing has one method called 'query', which
takes named arguments: query_string - a query, id_list - a list of
strings representing id's to filter by the query.
Below are the contents of these files with explanation:
ws.conf
__BEGIN__
# FunkLoad test configuration file
# Main section
#
[main]
title=Test the my web service
description=Try to test query method
# the server url to test - I am running the server locally
url=http://localhost:8080/
# this is the location of my wsdl file describing the service
# (same directory as the tests)
wsdl=ws.wsdl
# Tests description and configuration
#
[test_ws]
description=Check query method
# ------------------------------------------------------------
# Configuration for unit test mode fl-run-test
#
[ftest]
# log_to destination =
# console - to the screen
# file - to a file
log_to = console file
# log_path = path and file name to store log file
log_path = ws-test.log
# result_path = path to store the xml result file
result_path = ws-test.xml
# sleeptime_min = minimum amount of time in seconds to sleep between
requests
# to the host
sleep_time_min = 0
# sleeptime_max = maximum amount of time in seconds to sleep between
requests
# to the host
sleep_time_max = 0
# ------------------------------------------------------------
# Configuration for bench mode fl-run-bench
#
[bench]
# cycles = list of cycles with their number of concurrent users
cycles = 1:20:40
# duration = duration of a cycle in seconds
duration = 20
# startup_delay = time to wait between starting-up threads in seconds
startup_delay = 0.05
# sleep_time = time to wait between test in seconds
sleep_time = 0.5
# cycle_time = time to wait between cycle in seconds
cycle_time = 1
# same keys than in [ftest] section
log_to = file
log_path = ws-bench.log
result_path = ws-bench.xml
sleep_time_min = .1
sleep_time_max = .2
__END__
The config file lists setup information and parameters to use when
testing. See http://funkload.nuxeo.org/#benching for an explanation
of the parameters.
To get my tests to work, I had to dig into the funkload code a bit to
figure out how it does the FunkLoadTestCase.xmlrpc method. I needed
to modify it to be able to test my SOAP service described with my
wsdl. I wanted to make a class that derived from FunkLoadTestCase,
and further derive that for my test cases (instead of
unittest.TestCase). However, I got some errors about wrong arguments
called on __init__ with a cryptic trace. So I opted for the quick
hack (for now), of just adding my soap_wsdl method to my test case
class directly. This is good enough for now:
test_ws.py
__BEGIN__
"""Simple FunkLoad test for WS
run test with
> fl-run-test -v test_ws.py
run bench with
> fl-run-bench test_ws.py WS.test_ws
"""
import unittest
from random import random
import time
# I am using ZSI to set up the binding to the SOAP service
from ZSI.ServiceProxy import ServiceProxy
from funkload.FunkLoadTestCase import FunkLoadTestCase
class WS(FunkLoadTestCase):
"""This test use a configuration file ws.conf."""
def setUp(self):
"""Setting up test."""
self.logd("setUp")
#get configuration details from ws.conf
self.server_url = self.conf_get('main', 'url')
self.wsdl = self.conf_get('main','wsdl')
def soap_wsdl(self, wsdl, url_in, method_name,
named_params=None, description=None):
"""Call a SOAP method_name specified in wsdl with named_params.
named_params is a dict of params.
Modifying FunkLoadTestCase.xmlrpc method.
Copying method and providing SOAP w/ wsdl details.
"""
self.steps += 1
self.page_responses = 0
self.logd('SOAP: %s::%s\n\tCall %i: %s ...' % (wsdl+url_in,
method_name,
self.steps,
description
or ''))
response = None
t_start = time.time()
if self._authinfo is not None:
url = url_in.replace('//', '//'+self._authinfo)
else:
url = url_in
try:
server = ServiceProxy(wsdl,url=url)
method = getattr(server, method_name)
if named_params is not None:
response = method(**named_params)
else:
response = method()
except:
etype, value, tback = sys.exc_info()
t_stop = time.time()
t_delta = t_stop - t_start
self.total_time += t_delta
self.step_success = False
self.test_status = 'Error'
self.logd(' Failed in %.3fs' % t_delta)
self._log_xmlrpc_response(wsdl+url_in, method_name,
description,
response, t_start, t_stop, -1)
if etype is SocketError:
raise SocketError("Can't access %s." % wsdl+url)
raise
t_stop = time.time()
t_delta = t_stop - t_start
self.total_time += t_delta
self.total_xmlrpc += 1
self.logd(' Done in %.3fs' % t_delta)
self._log_xmlrpc_response(wsdl+url_in, method_name,
description, response,
t_start, t_stop, 200)
self.sleep()
return response
def test_ws(self):
""" my test """
server_url = self.server_url
wsdl = self.wsdl
ret = self.soap_wsdl(wsdl, server_url, 'query',
named_params = {'search_query': 'test query',
'id_list': []},
description="Check simple query")
self.assert_(isinstance(ret,dict),
'Wrong Return Type, expected dict %s' % ret)
self.logd('ret %s' % ret)
def tearDown(self):
"""Setting up test."""
self.logd("tearDown.\n")
if __name__ in ('main', '__main__'):
unittest.main()
__END__
Here I mostly followed the examples I cited above. The only
complication was adding my soap_wsdl method so I could run the test.
3.) Run the tests. I fired up my server on http://localhost:8080,
and executed
fl-run-test -v test_ws.py
This gave me what looked like a typical unittest output to the
console, as well as created ws-test.log and ws-test.xml describing
the test.
4.) Run a bench. The following did load testing by running the tests
over with different numbers of concurrent threads, using the configs
we setup earlier:
fl-run-bench test_ws.py WS.test_ws
Once again, output was to the console, with ws-bench.log and ws-
bench.xml being created which described the benchmark testing.
5.) Create a report. You can generate a nice report of the
benchmarking logs by doing
fl-build-report ws-bench.xml
or
fl-build-report --html ws-bench.xml
These notes outlined the basics of getting setup with funkloader.
There are other rich features. See http://funkload.nuxeo.org/ for
more details.
Hope this was helpful,
Julius
------------------------------------------------------------------------
---------------
Please Reply to My Permanent Address: julius at younglucks.com
http://www.openwetware.org/wiki/User:Julius_B._Lucks
------------------------------------------------------------------------
----------------
On Aug 23, 2007, at 4:12 PM, Grig Gheorghiu wrote:
> Hi Julius,
>
> A short write-up on how you set up funkload would be greatly
> appreciated by the TIP audience I'm sure :-) The last time I looked at
> it, it wasn't trivial to configure.
>
> Grig
>
> --- "Julius B. Lucks" <julius at younglucks.com> wrote:
>
>> Hey Sylvain,
>>
>> Many thanks for the tips. It looks like funkload [3] is the best for
>>
>> my needs. The others look a little heavy for this initial testing.
>>
>> Funkload has been really easy so far to set up and run, especially
>> since I have been unittest to do my testing anyway. The only thing I
>>
>> had to do was to write my own method within the FunkLoadTestCase
>> class to handle calling SOAP methods described by WSDL files -
>> essentially a modification of the Funkload xmlrpc method.
>>
>> Cheers,
>>
>> Julius
>>
>>
> ----------------------------------------------------------------------
> --
>>
>> ---------------
>> Please Reply to My Permanent Address: julius at younglucks.com
>> http://www.openwetware.org/wiki/User:Julius_B._Lucks
>>
> ----------------------------------------------------------------------
> --
>>
>> ----------------
>>
>>
>>
>> On Aug 23, 2007, at 9:37 AM, Sylvain Hellegouarch wrote:
>>
>>> Depending on your requirements, you may use expensive tools like
>>> LoadRunner from HP (formerly Mercury) but be ready to spit the
>> cash.
>>>
>>> More seriously, there are several tools that do a decent job in
>>> that field for free, such as WebLoad [1] or Tsung[2]. I personally
>>
>>> like Funkload [3] as you design your performance tests as glorified
>>
>>> unit tests and funkload does the job of measuring, monitoring and
>>
>>> gathering data. Simplistic but great to a quick load/perf. test
>>> mockup I think.
>>>
>>> [1] http://www.webload.org/
>>> [2] http://www.process-one.net/en/tsung/
>>> [3] http://funkload.nuxeo.org/
>>>
>>> - Sylvain
>>>
>>> Julius B. Lucks a écrit :
>>>> Hi All,
>>>>
>>>> I am writing a web service server with SOAP/WSDL in python using
>>>> the Zolera Soap Infrastructure module (ZSI - http://
>>>> pywebsvcs.sourceforge.net/). I have a stub implementation of my
>>>> service up and running with unittests covering the functionality
>>>> of the service. Now I want to create some performance tests/
>>>> benchmarks so that I can get an idea of the resources the service
>>
>>>> is requiring, and compare it to future versions.
>>>>
>>>> What is the best way to do these performance tests? Time my
>>>> unittests? Profile a set of calls on the service, and record the
>>>> timings? Should I seperately do tests on the backend, and backend
>>>> +SOAP messaging? Are there any tools for this?
>>>>
>>>> I appreciate any tips.
>>>>
>>>> Cheers,
>>>>
>>>> Julius
>>>>
>>>>
>> ---------------------------------------------------------------------
>>
>>>> ------------------
>>>> Please Reply to My Permanent Address: julius at younglucks.com
>>>> <mailto:julius at younglucks.com>
>>>> http://www.openwetware.org/wiki/User:Julius_B._Lucks
>>>>
>> ---------------------------------------------------------------------
>>
>>>> -------------------
>>>>
>>>>
>>>>
>>>>
>> ---------------------------------------------------------------------
>>
>>>> ---
>>>>
>>>> _______________________________________________
>>>> testing-in-python mailing list
>>>> testing-in-python at lists.idyll.org
>>>> http://lists.idyll.org/listinfo/testing-in-python
>>>>
>>>
>>
>>> _______________________________________________
>> testing-in-python mailing list
>> testing-in-python at lists.idyll.org
>> http://lists.idyll.org/listinfo/testing-in-python
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.idyll.org/pipermail/testing-in-python/attachments/20070823/5f23f832/attachment-0001.htm
More information about the testing-in-python
mailing list