[twill] IOError 24 (Too Many Open Files) in twill's mechanize & subprocess calls

[Ramon Felciano]

Hi --
 
I'm running into a "Too Many Open Files" IOError 24 when looping through several hundred calls to a website. I'm calling twill via the Python API directly, and it looks like it is getting hung up inside mechanize and subprocess calls. Is there an upper bound on how many times you can call twill from within the same interpreter? Or are there explicit calls to make to close or otherwise clean up after a particular twill session / subprocess call?
 
Thanks!
 
Ramon
 
Traceback (most recent call last):
  File "upload.py", line 306, in <module>
  File "upload.py", line 278, in cleanUpPubmedUrlsAndSave
  File "upload.py", line 210, in saveEntries
  File "upload.py", line 240, in cleanUpPubmedUrls
  File "upload.py", line 63, in getPubmedIdsForTitle
  File "upload.py", line 45, in getPubmedIdsForTitle_twill
  File "c:\python25\lib\site-packages\twill-0.9b1-py2.5.egg\twill\commands.py", line 108, in go
  File "c:\python25\lib\site-packages\twill-0.9b1-py2.5.egg\twill\browser.py", line 113, in go
  File "c:\python25\lib\site-packages\twill-0.9b1-py2.5.egg\twill\browser.py", line 516, in _journey
  File "c:\python25\lib\site-packages\twill-0.9b1-py2.5.egg\twill\other_packages\mechanize\_mechanize.py", line 156, in open
  File "c:\python25\lib\site-packages\twill-0.9b1-py2.5.egg\twill\other_packages\mechanize\_mechanize.py", line 201, in _mech_open
  File "c:\python25\lib\site-packages\twill-0.9b1-py2.5.egg\twill\other_packages\mechanize\_mechanize.py", line 255, in _set_response
  File "c:\python25\lib\site-packages\twill-0.9b1-py2.5.egg\twill\utils.py", line 336, in set_response
  File "c:\python25\lib\site-packages\twill-0.9b1-py2.5.egg\twill\utils.py", line 372, in _cleanup_html
  File "c:\python25\lib\site-packages\twill-0.9b1-py2.5.egg\twill\utils.py", line 288, in run_tidy
  File "c:\python25\lib\site-packages\twill-0.9b1-py2.5.egg\twill\other_packages\subprocess.py", line 536, in __init__
  File "c:\python25\lib\site-packages\twill-0.9b1-py2.5.egg\twill\other_packages\subprocess.py", line 588, in _get_handles
IOError: [Errno 24] Too many open files

 


----+----
This email message (and any attached document) contains information from Ingenuity Systems Inc. which may be considered confidential by Ingenuity, or which may be privileged or otherwise exempt from disclosure under law, and is for the sole use of the individual or entity to whom it is addressed.  Any other dissemination, distribution or copying of this message is strictly prohibited.  If you receive this message in error, please notify me and destroy the attached message (and all attached documents) immediately.