From james.cameron at hp.com Tue Apr 7 16:16:33 2009 From: james.cameron at hp.com (James Cameron) Date: Wed, 8 Apr 2009 09:16:33 +1000 Subject: [twill] reraise_unmasked_exceptions() In-Reply-To: References: <20090319043116.GA6248@hp.com> Message-ID: <20090407231633.GB2786@hp.com> On Thu, Mar 19, 2009 at 07:27:07PM +0000, John J Lee wrote: > On Thu, 19 Mar 2009, James Cameron wrote: > [...] > > ValueError: invalid literal for int() with base 10: '"1"' > [...] > > This was fixed in mechanize 0.1.10 (I don't know which version, if any, > of twill that corresponds to). Bypassing twill-sh and using mechanize directly allowed me to create an exception handler that would ignore this condition. I'm happy. Mechanize is still 0.1.7b on Debian. Also, Debian's Twill uses Debian's mechanize, not the mechanize inside Twill. I think. -- James Cameron http://ftp.hp.com.au/sigs/jc/ From ctb at msu.edu Tue Apr 7 18:43:24 2009 From: ctb at msu.edu (C. Titus Brown) Date: Tue, 7 Apr 2009 18:43:24 -0700 Subject: [twill] reraise_unmasked_exceptions() In-Reply-To: <20090407231633.GB2786@hp.com> References: <20090319043116.GA6248@hp.com> <20090407231633.GB2786@hp.com> Message-ID: <20090408014324.GD22601@idyll.org> On Wed, Apr 08, 2009 at 09:16:33AM +1000, James Cameron wrote: -> On Thu, Mar 19, 2009 at 07:27:07PM +0000, John J Lee wrote: -> > On Thu, 19 Mar 2009, James Cameron wrote: -> > [...] -> > > ValueError: invalid literal for int() with base 10: '"1"' -> > [...] -> > -> > This was fixed in mechanize 0.1.10 (I don't know which version, if any, -> > of twill that corresponds to). -> -> Bypassing twill-sh and using mechanize directly allowed me to create an -> exception handler that would ignore this condition. I'm happy. -> Mechanize is still 0.1.7b on Debian. Also, Debian's Twill uses Debian's -> mechanize, not the mechanize inside Twill. I think. mechanize will be updated in twill 0.9.2, coming Any Day Now. cheers, --titus -- C. Titus Brown, ctb at msu.edu From tennis at tripit.com Thu Apr 16 16:46:12 2009 From: tennis at tripit.com (Tennis Smith) Date: Thu, 16 Apr 2009 16:46:12 -0700 Subject: [twill] authentication problem Message-ID: Hi, I'm trying to test a website that requires an id/password. When trying to use twill, I get this response: >> show > > 401 Authorization Required > >

401 Authorization Required

nginx/0.6.35 > > > I have a legit id/pswd on the system, but can't figure out how to pass them via twill. Anyone have any ideas? -Tennis -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.idyll.org/pipermail/twill/attachments/20090416/647e0b6d/attachment.htm From james.cameron at hp.com Thu Apr 16 16:59:24 2009 From: james.cameron at hp.com (James Cameron) Date: Fri, 17 Apr 2009 09:59:24 +1000 Subject: [twill] authentication problem In-Reply-To: References: Message-ID: <20090416235924.GD2836@hp.com> On Thu, Apr 16, 2009 at 04:46:12PM -0700, Tennis Smith wrote: > I have a legit id/pswd on the system, but can't figure out how to pass > them via twill. > Anyone have any ideas? How do you normally present the username and password? If it is via a pop-up authentication dialog box, then the usual methods such as twill-sh's add_auth should work. If if is via a username and password prompt on a web page, then you have to direct twill to fill in the form. You will either receive a cookie, or a session ID in a redirection. Everybody does it differently. -- James Cameron http://ftp.hp.com.au/sigs/jc/ From tennis at tripit.com Thu Apr 16 17:16:00 2009 From: tennis at tripit.com (Tennis Smith) Date: Thu, 16 Apr 2009 17:16:00 -0700 Subject: [twill] authentication problem In-Reply-To: <20090416235924.GD2836@hp.com> References: <20090416235924.GD2836@hp.com> Message-ID: On Thu, Apr 16, 2009 at 4:59 PM, James Cameron wrote: > On Thu, Apr 16, 2009 at 04:46:12PM -0700, Tennis Smith wrote: > > I have a legit id/pswd on the system, but can't figure out how to pass > > them via twill. > > Anyone have any ideas? > > How do you normally present the username and password? > > If it is via a pop-up authentication dialog box, then the usual methods > such as twill-sh's add_auth should work. > > If if is via a username and password prompt on a web page, then you have > to direct twill to fill in the form. Yes, I have to fill in a form. Can you point me to an example of how to do that? THanks. :) > You will either receive a cookie, > or a session ID in a redirection. > > Everybody does it differently. > > -- > James Cameron > http://ftp.hp.com.au/sigs/jc/ > > _______________________________________________ > twill mailing list > twill at lists.idyll.org > http://lists.idyll.org/listinfo/twill > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.idyll.org/pipermail/twill/attachments/20090416/ebd8bc2c/attachment.htm From james.cameron at hp.com Thu Apr 16 17:20:18 2009 From: james.cameron at hp.com (James Cameron) Date: Fri, 17 Apr 2009 10:20:18 +1000 Subject: [twill] authentication problem In-Reply-To: References: <20090416235924.GD2836@hp.com> Message-ID: <20090417002018.GF2836@hp.com> On Thu, Apr 16, 2009 at 05:16:00PM -0700, Tennis Smith wrote: > Yes, I have to fill in a form. Can you point me to an example of how > to do that? THanks. :) Sure. My internet service provider has a web page to tell me the current usage, and it works by prompting in a form, then issuing a cookie that is used to validate further requests. For this I used twill-sh in a shell script as follows: twill-sh </dev/null 2>/dev/null go https://example.com/usage/daily/ fv Login username quozl fv Login password qwertyuiop submit save_html /var/tmp/usage.html EOF The remains of my script processed the usage.html file to get the byte counts. -- James Cameron http://ftp.hp.com.au/sigs/jc/ From tennis at tripit.com Fri Apr 17 08:22:38 2009 From: tennis at tripit.com (Tennis Smith) Date: Fri, 17 Apr 2009 08:22:38 -0700 Subject: [twill] authentication problem In-Reply-To: <20090417002018.GF2836@hp.com> References: <20090416235924.GD2836@hp.com> <20090417002018.GF2836@hp.com> Message-ID: Grrrr... it turns out the auth process is putting up a separate window with userid/password fields in it. Can twill deal w/that? -T On Thu, Apr 16, 2009 at 5:20 PM, James Cameron wrote: > On Thu, Apr 16, 2009 at 05:16:00PM -0700, Tennis Smith wrote: > > Yes, I have to fill in a form. Can you point me to an example of how > > to do that? THanks. :) > > Sure. > > My internet service provider has a web page to tell me the current > usage, and it works by prompting in a form, then issuing a cookie that > is used to validate further requests. > > For this I used twill-sh in a shell script as follows: > > twill-sh </dev/null 2>/dev/null > go https://example.com/usage/daily/ > fv Login username quozl > fv Login password qwertyuiop > submit > save_html /var/tmp/usage.html > EOF > > The remains of my script processed the usage.html file to get the byte > counts. > > -- > James Cameron > http://ftp.hp.com.au/sigs/jc/ > > _______________________________________________ > twill mailing list > twill at lists.idyll.org > http://lists.idyll.org/listinfo/twill > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.idyll.org/pipermail/twill/attachments/20090417/51108702/attachment.htm From ctb at msu.edu Fri Apr 17 12:27:18 2009 From: ctb at msu.edu (C. Titus Brown) Date: Fri, 17 Apr 2009 12:27:18 -0700 Subject: [twill] authentication problem In-Reply-To: References: <20090416235924.GD2836@hp.com> <20090417002018.GF2836@hp.com> Message-ID: <20090417192718.GC8574@idyll.org> On Fri, Apr 17, 2009 at 08:22:38AM -0700, Tennis Smith wrote: -> Grrrr... it turns out the auth process is putting up a separate window with -> userid/password fields in it. -> -> Can twill deal w/that? Hey Tennis, is it doing that with JavaScript or with browser authentication, do you know? cheers, --titus -> On Thu, Apr 16, 2009 at 5:20 PM, James Cameron wrote: -> -> > On Thu, Apr 16, 2009 at 05:16:00PM -0700, Tennis Smith wrote: -> > > Yes, I have to fill in a form. Can you point me to an example of how -> > > to do that? THanks. :) -> > -> > Sure. -> > -> > My internet service provider has a web page to tell me the current -> > usage, and it works by prompting in a form, then issuing a cookie that -> > is used to validate further requests. -> > -> > For this I used twill-sh in a shell script as follows: -> > -> > twill-sh </dev/null 2>/dev/null -> > go https://example.com/usage/daily/ -> > fv Login username quozl -> > fv Login password qwertyuiop -> > submit -> > save_html /var/tmp/usage.html -> > EOF -> > -> > The remains of my script processed the usage.html file to get the byte -> > counts. -> > -> > -- -> > James Cameron -> > http://ftp.hp.com.au/sigs/jc/ -> > -> > _______________________________________________ -> > twill mailing list -> > twill at lists.idyll.org -> > http://lists.idyll.org/listinfo/twill -> > -> _______________________________________________ -> twill mailing list -> twill at lists.idyll.org -> http://lists.idyll.org/listinfo/twill -- C. Titus Brown, ctb at msu.edu From james.cameron at hp.com Sun Apr 19 15:26:31 2009 From: james.cameron at hp.com (James Cameron) Date: Mon, 20 Apr 2009 08:26:31 +1000 Subject: [twill] authentication problem In-Reply-To: References: <20090416235924.GD2836@hp.com> <20090417002018.GF2836@hp.com> Message-ID: <20090419222631.GB3771@hp.com> On Fri, Apr 17, 2009 at 08:22:38AM -0700, Tennis Smith wrote: > Grrrr... it turns out the auth process is putting up a separate window > with userid/password fields in it. > Can twill deal w/that? The way I deal with that using twill is to find the URL of the separate window, and begin with that URL instead of the other. -- James Cameron http://ftp.hp.com.au/sigs/jc/ From tennis at tripit.com Mon Apr 20 11:26:55 2009 From: tennis at tripit.com (Tennis Smith) Date: Mon, 20 Apr 2009 11:26:55 -0700 Subject: [twill] Doing "showlinks" on every page Message-ID: Hi, Is there a way to visit each page in a website and do 'showlinks"? I'm testing a website and want to get a page-by-page listing of links so I can test all the combinations. Thanks, -T -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.idyll.org/pipermail/twill/attachments/20090420/e875f242/attachment.htm From ctb at msu.edu Mon Apr 20 11:43:28 2009 From: ctb at msu.edu (C. Titus Brown) Date: Mon, 20 Apr 2009 11:43:28 -0700 Subject: [twill] Doing "showlinks" on every page In-Reply-To: References: Message-ID: <20090420184328.GB27532@idyll.org> On Mon, Apr 20, 2009 at 11:26:55AM -0700, Tennis Smith wrote: -> Is there a way to visit each page in a website and do 'showlinks"? I'm -> testing a website and want to get a page-by-page listing of links so I can -> test all the combinations. Hi Tennis, it's a fairly straightforward thing to do programmatically -- c.f. the check_links extension in twill/extensions/check_links.py -- but it would require some new programming. --titus -- C. Titus Brown, ctb at msu.edu From tennis at tripit.com Mon Apr 20 12:06:00 2009 From: tennis at tripit.com (Tennis Smith) Date: Mon, 20 Apr 2009 12:06:00 -0700 Subject: [twill] Doing "showlinks" on every page In-Reply-To: <20090420184328.GB27532@idyll.org> References: <20090420184328.GB27532@idyll.org> Message-ID: Good idea. I'll have a look at the code. Thanks, -Tennis On Mon, Apr 20, 2009 at 11:43 AM, C. Titus Brown wrote: > On Mon, Apr 20, 2009 at 11:26:55AM -0700, Tennis Smith wrote: > -> Is there a way to visit each page in a website and do 'showlinks"? I'm > -> testing a website and want to get a page-by-page listing of links so I > can > -> test all the combinations. > > Hi Tennis, > > it's a fairly straightforward thing to do programmatically -- c.f. the > check_links extension in twill/extensions/check_links.py -- but it would > require some new programming. > > --titus > -- > C. Titus Brown, ctb at msu.edu > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.idyll.org/pipermail/twill/attachments/20090420/b0ff0397/attachment.htm From amilli88 at gmail.com Thu Apr 23 14:20:51 2009 From: amilli88 at gmail.com (im me) Date: Thu, 23 Apr 2009 17:20:51 -0400 Subject: [twill] Having Trouble With Submit Message-ID: <5867b6fd0904231420j2b094327kb26557d255203516@mail.gmail.com> Hi All, I am having trouble with a submit function on a web form. When I do it through the browser, the submit hits one page and then redirects to another, but when I use twill, the submit button merely hits the first page and hangs there. I get a 200 code, which suggests that it is done... There is some sketchy javascript on the submit button: any ideas? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.idyll.org/pipermail/twill/attachments/20090423/36a731d2/attachment.htm From amilli88 at gmail.com Thu Apr 23 14:26:54 2009 From: amilli88 at gmail.com (im me) Date: Thu, 23 Apr 2009 17:26:54 -0400 Subject: [twill] Having Trouble With Submit In-Reply-To: <5867b6fd0904231420j2b094327kb26557d255203516@mail.gmail.com> References: <5867b6fd0904231420j2b094327kb26557d255203516@mail.gmail.com> Message-ID: <5867b6fd0904231426x7ebc1472lce94bc791ffda946@mail.gmail.com> Here is the javascript behind the postbackwithoptions call. I have no idea what it does though. Can twill work with this? There is nothing else on the form except a checkbox, whose value I set to 'on'. On Thu, Apr 23, 2009 at 5:20 PM, im me wrote: > Hi All, > > I am having trouble with a submit function on a web form. When I do it > through the browser, the submit hits one page and then redirects to another, > but when I use twill, the submit button merely hits the first page and hangs > there. I get a 200 code, which suggests that it is done... > > There is some sketchy javascript on the submit button: > > id="ctl00_centerContent_NextButtonControl" class="go_button go_grow" style > ="margin:0px; padding:0px;" /> > > any ideas? > > Thanks! > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.idyll.org/pipermail/twill/attachments/20090423/d34bd449/attachment.htm From russolanotte at yahoo.it Wed Apr 29 09:22:38 2009 From: russolanotte at yahoo.it (Russo Lanotte) Date: Wed, 29 Apr 2009 16:22:38 +0000 (GMT) Subject: [twill] (no subject) Message-ID: <455357.3903.qm@web27001.mail.ukl.yahoo.com> Hi there, I am having a bit of trouble with the http-equiv refresh limit: the system i am using at the moment uses refresh redirection as a means of waiting for completing resource-intensive jobs. Is there any way around apart from changing my system? :) Thank you very much for your help. Cheers -R From akafubu.kibombo at gmail.com Thu Apr 30 01:03:49 2009 From: akafubu.kibombo at gmail.com (Akafubu Kibombo) Date: Thu, 30 Apr 2009 03:03:49 -0500 Subject: [twill] showforms not showing forms Message-ID: <39210b4b0904300103s404097daq84a75a29bb45547e@mail.gmail.com> Example: >> add_extra_header User-Agent "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6" current page: *empty page* >> go http://www.digg.com ==> at http://digg.com/ current page: http://digg.com/ >> showforms current page: http://digg.com/ No matter what site, immediately after typing 'showforms', no forms are show, just "current page: ". Can someone please help me troubleshoot the problem? Best Regards, -AK -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.idyll.org/pipermail/twill/attachments/20090430/ca3fba85/attachment.html From russolanotte at yahoo.it Thu Apr 30 02:38:27 2009 From: russolanotte at yahoo.it (Russo Lanotte) Date: Thu, 30 Apr 2009 09:38:27 +0000 (GMT) Subject: [twill] R: (no subject) Message-ID: <569196.20262.qm@web27001.mail.ukl.yahoo.com> Really sorry, for some reason it didnt show the subject.. Wa supposed to be "HTTP-EQUIV REFRESH" Cheers -R --- Mer 29/4/09, Russo Lanotte ha scritto: > Da: Russo Lanotte > Oggetto: [twill] (no subject) > A: twill at lists.idyll.org > Data: Mercoled? 29 Aprile 2009, 18:22 > > Hi there, > > I am having a bit of trouble with the http-equiv refresh > limit: > the system i am using at the moment uses refresh > redirection as a means of waiting for completing > resource-intensive jobs. > > Is there any way around apart from changing my system? :) > > Thank you very much for your help. > > Cheers > -R > > > > ? ? ? > > _______________________________________________ > twill mailing list > twill at lists.idyll.org > http://lists.idyll.org/listinfo/twill > From ctb at msu.edu Thu Apr 30 07:53:50 2009 From: ctb at msu.edu (C. Titus Brown) Date: Thu, 30 Apr 2009 07:53:50 -0700 Subject: [twill] showforms not showing forms In-Reply-To: <39210b4b0904300103s404097daq84a75a29bb45547e@mail.gmail.com> References: <39210b4b0904300103s404097daq84a75a29bb45547e@mail.gmail.com> Message-ID: <20090430145350.GA365@idyll.org> On Thu, Apr 30, 2009 at 03:03:49AM -0500, Akafubu Kibombo wrote: -> Example: -> -> >> add_extra_header User-Agent "Mozilla/5.0 (Windows; U; Windows NT 5.1; -> en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6" -> current page: *empty page* -> >> go http://www.digg.com -> ==> at http://digg.com/ -> current page: http://digg.com/ -> >> showforms -> current page: http://digg.com/ -> -> -> No matter what site, immediately after typing 'showforms', no forms are -> show, just "current page: ". Can someone please help me troubleshoot -> the problem? Hi AK, what version of twill are you using for this? A few suggestions: - use the 'agent' command to set the agent; I believe add_extra_header adds an extra header on top of the one already set by twill; - make sure that's a valid agent string. I'm getting some weird parse errors on Linux when I try it out. cheers, --titus -- C. Titus Brown, ctb at msu.edu