[twill] 0.7.3, one site fails in Python, noreadonly patch

Fri Oct 21 05:10:07 PDT 2005

G'day,

Summary:

1.  one particular site fails only if used from a Python script,
2.  twill worked great for me, thanks,
3.  proposed patch to ignore readonly fields with rationale,

--

1.  site fails only in Python script

Using twill-0.7.3 on Python 2.3.5, loading a particular page with a
twill script works and showforms recognises the form on the page, but
if I try it with a Python script I get an error.  Here's the
interactive execution ...

>>> from twill.commands import *
>>> go("https://private.example.com/");
==> at https://private.example.com/
>>> showforms()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/lib/python2.3/site-packages/twill/commands.py", line 507, in showforms
    state.showforms()
  File "/usr/lib/python2.3/site-packages/twill/commands.py", line 204, in showforms
    for n, f in enumerate(self._browser.forms()):
  File "/usr/lib/python2.3/site-packages/twill/wwwsearch.zip/mechanize/_mechanize.py", line 270, in forms
mechanize._mechanize.BrowserStateError: not viewing HTML
>>> show()
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
[... edited out ...]
</html>
>>>

On the other hand, if I do this from twill-sh, it works fine.

>> go https://private.example.com/
>> showforms
[... expected output edited out ...]
>> 

Using save_html in both environments shows that the HTML that has been
delivered is the same.

I'm afraid I can't provide the HTML or the exact URLs, due to security
classification of the content.  But I'm willing to examine program
state and so forth if that will help.  I've already tried to dig
beneath twill to find out what is going on, but I'm very new to Python
and I'm not sure I understand what I see.

--

2.  works great otherwise

I am confident that twill is working fine for other sites.  My first
attempt is the attached script, for finding service utilisation on my
two-way satellite internet service ... automation of a painfully slow
interaction.

An interesting extra step was that the page provided after login
contained an HTTP-EQUIV refresh directive with zero timeout.  There
didn't seem to be a built-in response to this, so I had to dig in and
find the URL from the HTML.

--

3.  readonly controls

One particular curiosity with the bigpond.com site was that the page
contains JavaScript that will set the value of a hidden field just
before submitting the form.  If it isn't set, then the login fails.

So I patched commands.py to set readonly controls to writable.
I figure if JavaScript is allowed to do it, so should twill.

Next time, I'll dig down to change the control to writable manually.
But it wasn't practical in twill-sh when I first started.

-- 
James Cameron                         http://quozl.netrek.org/
HP Open Source, Volunteer             http://opensource.hp.com/
PPTP Client Project, Release Engineer http://pptpclient.sourceforge.net/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: telstra-bigpond-two-way-usage.py
Type: text/x-python
Size: 1821 bytes
Desc: not available
Url : http://lists.idyll.org/pipermail/twill/attachments/20051021/35d56f1a/telstra-bigpond-two-way-usage.py
-------------- next part --------------

--- twill/commands.py.~1~	2005-10-02 16:57:12.000000000 +1000
+++ twill/commands.py	2005-10-20 15:50:36.000000000 +1000
@@ -528,9 +528,6 @@
     There are some ambiguities in the way formvalue deals with lists:
     'fv' will *add* the given value to a multilist.
 
-    Formvalue ignores read-only fields completely; if they're readonly,
-    nothing is done.
-
     Available as 'fv' as well.
     """
     form = state.get_form(formname)
@@ -539,7 +536,8 @@
     if control:
         state.clicked(form, control)
         if control.readonly:
-            return
+            control.readonly = False
+            print 'Forced field to be writable'
 
         set_form_control_value(control, value)
     else:
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://lists.idyll.org/pipermail/twill/attachments/20051021/35d56f1a/attachment.bin
