[twill] IOError 24 (Too Many Open Files) in twill's mechanize & subprocess calls

[Robert Marianski]

On Sun, Mar 02, 2008 at 04:17:53PM -0800, Ramon Felciano wrote:
> Hi --
>  
> I'm running into a "Too Many Open Files" IOError 24 when looping through several hundred calls to a website. I'm calling twill via the Python API directly, and it looks like it is getting hung up inside mechanize and subprocess calls. Is there an upper bound on how many times you can call twill from within the same interpreter? Or are there explicit calls to make to close or otherwise clean up after a particular twill session / subprocess call?

I've encountered something like this in the past, and for me it was
twill trying to call tidy, which I didn't have. IIRC, the subprocess
call was leaking file descriptors. You *may* be seeing something
similar. Try turning the config option to use tidy off to see if it
helps.

config use_tidy 0

Robert

>  
> Thanks!
>  
> Ramon
>  
> Traceback (most recent call last):
>   File "upload.py", line 306, in <module>
>   File "upload.py", line 278, in cleanUpPubmedUrlsAndSave
>   File "upload.py", line 210, in saveEntries
>   File "upload.py", line 240, in cleanUpPubmedUrls
>   File "upload.py", line 63, in getPubmedIdsForTitle
>   File "upload.py", line 45, in getPubmedIdsForTitle_twill
>   File "c:\python25\lib\site-packages\twill-0.9b1-py2.5.egg\twill\commands.py", line 108, in go
>   File "c:\python25\lib\site-packages\twill-0.9b1-py2.5.egg\twill\browser.py", line 113, in go
>   File "c:\python25\lib\site-packages\twill-0.9b1-py2.5.egg\twill\browser.py", line 516, in _journey
>   File "c:\python25\lib\site-packages\twill-0.9b1-py2.5.egg\twill\other_packages\mechanize\_mechanize.py", line 156, in open
>   File "c:\python25\lib\site-packages\twill-0.9b1-py2.5.egg\twill\other_packages\mechanize\_mechanize.py", line 201, in _mech_open
>   File "c:\python25\lib\site-packages\twill-0.9b1-py2.5.egg\twill\other_packages\mechanize\_mechanize.py", line 255, in _set_response
>   File "c:\python25\lib\site-packages\twill-0.9b1-py2.5.egg\twill\utils.py", line 336, in set_response
>   File "c:\python25\lib\site-packages\twill-0.9b1-py2.5.egg\twill\utils.py", line 372, in _cleanup_html
>   File "c:\python25\lib\site-packages\twill-0.9b1-py2.5.egg\twill\utils.py", line 288, in run_tidy
>   File "c:\python25\lib\site-packages\twill-0.9b1-py2.5.egg\twill\other_packages\subprocess.py", line 536, in __init__
>   File "c:\python25\lib\site-packages\twill-0.9b1-py2.5.egg\twill\other_packages\subprocess.py", line 588, in _get_handles
> IOError: [Errno 24] Too many open files
> 
>  
> 
> 
> ----+----
> This email message (and any attached document) contains information from Ingenuity Systems Inc. which may be considered confidential by Ingenuity, or which may be privileged or otherwise exempt from disclosure under law, and is for the sole use of the individual or entity to whom it is addressed.  Any other dissemination, distribution or copying of this message is strictly prohibited.  If you receive this message in error, please notify me and destroy the attached message (and all attached documents) immediately. 
> 
> _______________________________________________
> twill mailing list
> twill at lists.idyll.org
> http://lists.idyll.org/listinfo/twill
> 
> !DSPAM:4021,47cb43c8186955219720167!
>