[twill] Testing a web API with twill?

[Ramon Felciano]

Hi -

I'm trying to find a web unit and load testing tool that will let me
test URL-based APIs. This is basically a parameterized URL that will
return JSON-encoded results. I would like to be able to easily set up
100+ test cases that are all test the same API, but with different URL
parameter values and different expected results. I would to be able to
flexibly configure these test cases, ideally as a text table or other
simple-to-create-and-parse data structure, and generate a report of
pass/fail cases for each test. 

I would really like to use something like twill, but I'm getting stuck
on 2 fronts:

1.	How to configure a single test case. The key here is that to
check for valid results I need to unserialize the json returned result.
Twill obviously doesn't have this built in, so I assume I will have to
call a twill script from python and then record its results for
subsequent processing. Is that right? Should the calling code itself
then be part of a unit testing framework?
2.	How to configure the series of test cases. I would like to avoid
writing a separate test class / function for each case, and instead
somehow iterate over the param/result pairs. I can't figure out how to
do this without having the entire suite abort after the first test
failure. I would like to run all 100 tests, and have it report back
which ones succeeded and which ones failed.

Are there options or configurations available that allow the flexibility
of a simple configuration file / data structure but that will still run
through all the tests as individuals? Or is this a mismatch for twill's
design? 

FWIW, I've looked at funkload for this as well, but it isn't clear how
to handle the configuration issue above either, and unfortunately
funkload's nice reporting functionality doesn't work for functional
testing, only load testing, so I'm not actually gaining that much. I'm
looking at other options right now, with twill and nose on the
shortlist. Any pointers would be greatly appreciated!

Thank you in advance for your time.

Thanks,

Ramon


----+----
This email message (and any attached document) contains information from Ingenuity Systems Inc. which may be considered confidential by Ingenuity, or which may be privileged or otherwise exempt from disclosure under law, and is for the sole use of the individual or entity to whom it is addressed.  Any other dissemination, distribution or copying of this message is strictly prohibited.  If you receive this message in error, please notify me and destroy the attached message (and all attached documents) immediately. 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.idyll.org/pipermail/twill/attachments/20070409/bd305b2b/attachment.htm