[twill] Temporary Failure in Name Resolution

[Chris Abraham]

Ok,we think we've found the problem.  In twill/utils.py line 287 we have:
process = subprocess.Popen(_tidy_cmd, stdin=subprocess.PIPE,
                           stdout=subprocess.PIPE,
                           stderr=subprocess.PIPE, bufsize=0, 		
                           shell=False)

This line gets called when use_tidy is set to True.  If this line fails
(like when _tidy_cmd can't be found), an exception is caught and then
the program continues (unless require_tidy is True).  This also leaves
three new PIPE files open and they never get closed.

Since use_tidy is True and require_tidy is False by default, if
_tidy_cmd does not exist, there will be an escalating number of open
files for each "go" statement in twill.

For now, this can be fixed by setting require_tidy to True or use_tidy
to False at the beginning of a script.  But really, we should find out
why there are PIPEs left open when subprocess.Popen fails.

Anyway, this satisfies us for now and we're able to use twill for our
purposes without any other problems.

Chris



Chris Abraham wrote:
> Titus,
> We've been doing some testing here which doesn't confirm what you've
> been finding.
> 
> Test (1)
> 
> We installed the svn version of python 2.5 and ran the following twill test:
> go http://www.google.com
> ...(repeated 1000 times)
> 
> meanwhile we tracked the open files on the computer with this:
> while true; do /usr/sbin/lsof | grep twill | wc -l; sleep 1s; done
> 
> and found an output of:
> 180
> 226
> 275
> 332
> ...
> 999
> 1055
> 
> at which point the twill script fails with
> Error:...'Temporary failure in name resolution'
> 
> 
> Test (2)
> 
> We emulated this test using just python with this script:
> import urllib2
> for i in xrange(1000):
>     f = urllib2.urlopen('http://www.google.com')
> 
> and tracked it with:
> while true; do /usr/sbin/lsof | grep python | wc -l; sleep 1s; done
> 
> and found an output of:
> 387
> 387
> ...
> 387
> 
> with python 2.4 we get an output of:
> 315
> 341
> 368
> 395
> 308
> 308
> 322
> 344
> (it flutters but otherwise remains stable around 340)
> 
> 
> 
> Conclusions:
> 
> urllib2 doesn't seem to be causing the problem.  Even with python 2.4,
> Test(2) runs fine without an escalating number of open files.
> 
> We think that there is a problem in twill that for some reason creates
> an escalating number of open files which eventually crashes the process.
>  We will look into the twill code to try and find a solution.
> 
> Does this make sense?  What have been your findings?  We're not sure why
> you think that the latest patched Python 2.5 fixes this problem.
> 
> 
> Chris
> 
> 
> 
> Titus Brown wrote:
>>> On Tue, Apr 10, 2007 at 12:10:56PM -0400, Chris Abraham wrote:
>>> -> I've run into a problem with twill that has already been talked
> about on
>>> -> this list here:
>>> -> http://thread.gmane.org/gmane.comp.web.twill/646/
>>> ->
>>> -> I get the same behavior in various different tests, that is, in a
> long
>>> -> test, after a certain length of time, twill consistently fails with:
>>> -> Error message: '<urlopen error (-3, 'Temporary failure in name
>>> -> resolution')>'
>>> ->
>>> -> Has anyone found a resolution to this problem?  I'm using twill
> version
>>> -> 0.9b1, and Python 2.4.3.
>>>
>>> Hey Chris,
>>>
>>> John Lee pointed me to a patch he submitted to Python,
>>>
>>> 	http://python.org/sf/1627441
>>>
>>> It was applied to 2.5 in January.  I realize this is a lot to ask,
>>> but... could you try running using the latest Python svn?
>>>
>>> I can also send you a hacked version of twill that contains the updated
>>> files for Python 2.5, so all you'd have to do is switch to using 2.5.
>>> That's probably easier, eh?
>>>
>>> cheers,
>>> --titus
> 
> 
> _______________________________________________
> twill mailing list
> twill at lists.idyll.org
> http://lists.idyll.org/listinfo/twill