[TIP] Writing Separate Functional Tests

[Noah Gift]

On Nov 21, 2007, at 2:49 PM, Titus Brown wrote:

> On Wed, Nov 21, 2007 at 09:35:40AM -0500, Noah Gift wrote:
> -> I have a gigantic threaded beast I need to test, and I want to  
> write
> -> functional tests that supplement the doctests and unittests, but
> -> wanted some advice on the proper way to do this.  I have not seen  
> much
> -> documentation on writing non-web functional tests, my tool is a CLI
> -> tool. The basic thing I want to verify is that my tool creates a  
> XML
> -> file with the information I would expect to be there.  For  
> example, is
> -> there a selenium equivalent for command line tools?
>
> Hey Noah,
>
> something like texttest (or 'diff') seems like the simplest way to go.
> You could also use one of the bajillion XML parsers to make assertions
> about the contents of the XML file.

that is a good idea.  I was thinking to grand, but a simple diff is  
probably going to be fine for now, and when I need more control,  
parsing an XML file on an attribute by attribute basis will allow me  
to test verify what I am getting is what I expect.

>
>
> As for testing multithreaded code... well, as I've said in private,
> that's even more difficult than *writing* multithreaded code.  You  
> might
> consider recording code coverage on a per-thread basis; I've been
> thinking about how to do that and I might have a solution.  At the  
> very
> least it would tell you what your individual threads were doing.

Trying to understand what massive pools of threads that spawn other  
massive spools of threads, that spawn other massive pools of threads  
is liking pushing a cart full of toys down a steep hill and then  
running after the cart, and trying to dig for a specific toy you want  
before the cart crashes or you fall, or both, all while running as  
fast as you can.

One idea I thought of today, was to create an environmental variable  
for running pdb and integrating that into a tracing library.  You wrap  
anything fragile in a try/except/finally and then have the ability to  
optionally break into the pdb if you set the right environmental  
variable.  This is all highly theoretical though, as I am still  
getting to know the pdb, but I think it might be one answer to get  
deep into a threaded beast, and walk through it.

>
>
> cheers,
> --titus