[twill] more complex uses of twill (was Re: [wwwsearch-general] mechanize and basic auth)

[Titus Brown]

-> since I have your attention (and since I wrote most of this before
-> realizing it wasn't really suited to the mechanize list... though feel
-> free to forward the discussion elsewhere)

CCed to twill at lists.idyll.org for general interest ;)

-> Now that I've shown what mechanize can do for our testing, twill is
-> the next thing for me to look at - if I understand it correctly, it's
-> not actually suited for the kind of fuzz testing I'm doing right now
-> because it has it's own microlanguage which lacks things like
-> conditionals and looping -- but that microlanguage looks pretty
-> expressive for straight-line regression tests, especially those
-> generated from "user did this in a browser" bugs...  

The twill microlanguage does specifically lack conditionals and looping.
At the Agile Testing tutorial we ran at PyCon, someone pointed out that
for functional and regression testing, this was precisely what you
wanted.  Once you have conditionals you have to start testing your
tests!

OTOH, you can use Python to drive twill in more complex ways.  The twill
microlanguage is primarily intended for testing, but twill itself can
be driven entirely from within Python.

-> Or am I misunderstanding something? The examples are all of that form,
-> and the comments in discard-sf-mailman-msgs show the kinds of
-> limitations that leads to...  maybe a some existing templating
-> mechanism would help; "kid" has this almost-too-clever hack to make
-> .kid files directly importable, and then they look like modules with
-> functions that take arguments... something like that (kid itself is
-> xml-only) would let you write modules of twill scripts, and then have
-> python workflow code that treats them as verbs.  Alternatively, I
-> suppose just stuffing twill scripts into strings and wrapping them
-> in... hmm, I don't actually see what, unless it's the cmd object...  I
-> half suspect that it makes sense to use twill-sh to experiment with a
-> page, and then turn the twill-history into straight twill.commands
-> calls, or even directly mechanize code...

You can now use twill.execute_string (in the latest alpha only, I think)
to run twill scripts from strings.

I'm also contemplating adding the ability to add twill execution in
doctests.

I am extraordinarily reluctant to make twill proper more complex.  My
feedback so far suggests that twill is most useful in the niche
lying between urllib.urlopen stuff & full-blown driving-a-browser stuff
like PAMIE, Watir, and Selenium. Anything more complicated than what we
have now is probably going to be overkill.

I'm happy to add hooks for whatever you need at the Python level, of
course.

cheers,
--titus

And I would definitely suggest using twill or mechanize for these, yes
;).

-> I did two personal projects about a month ago that in 20/20 hindsight
-> should have been mechanize projects, at least:
-> 
->    lj_friend_comments -- hit a livejournal friendslist (kind of a
->      specialized blog-aggregator page) and (1) check for new postings
->      (2) check for new comments in recent postings.
-> 
->    iesl-delete -- command line tool for deleting comment spam on a
->      homebrew (not mine, MIT-local) blog site.
-> 
-> Both of these could be divided into "mechanics" and "workflow", where
-> the former could use mechanize or twill instead of urllib and
-> sre... and the latter would be unchanged regardless of the underlying
-> implementation, stuff like:
-> 
->         posts = get_current_posts(blog)
->         for n, (title, href, count) in enumerate(posts):
->             count = int(count)
->             old_count = old_feedback.get(href, -1)
->             if old_count != count:
->                 print "%s NEW:" % (n + 1), title, "(%s was %s)" % (count, old_count)
->             ...
-> 
-> where old_feedback is a "shelve" - this logic would be the same even
-> if I had a "real API" to work with, but get_current_posts could be
-> entirely different.  (Both tasks do have the problem that they've got
-> some minor javascript login-enhancement going on that really needs to
-> be manually translated, as well.)