[twill] uploading large files

[Michele Simionato]

I was uploading a file with formfile and I have got the following traceback:

Traceback (most recent call last):
  File "/home/msimionato/md/scripts/megaupload.py", line 14, in <module>
    c.submit()
  File "/home/msimionato/lib/python2.5/site-packages/twill-0.9b1-py2.5.egg/twill/commands.py",
line 354, in submit
    browser.submit(submit_button)
  File "/home/msimionato/lib/python2.5/site-packages/twill-0.9b1-py2.5.egg/twill/browser.py",
line 447, in submit
    "", mechanize.Request)
  File "/home/msimionato/lib/python2.5/site-packages/twill-0.9b1-py2.5.egg/twill/other_packages/ClientForm.py",
line 3293, in _click
    return self._switch_click(return_type, request_class)
  File "/home/msimionato/lib/python2.5/site-packages/twill-0.9b1-py2.5.egg/twill/other_packages/ClientForm.py",
line 3363, in _switch_click
    req_data = self._request_data()
  File "/home/msimionato/lib/python2.5/site-packages/twill-0.9b1-py2.5.egg/twill/other_packages/ClientForm.py",
line 3346, in _request_data
    self.controls[control_index]._write_mime_data(mw, k, v)
  File "/home/msimionato/lib/python2.5/site-packages/twill-0.9b1-py2.5.egg/twill/other_packages/ClientForm.py",
line 1449, in _write_mime_data
    fh.write(file_object.read())
MemoryError: out of memory

I am surprised twill is storing the whole file in memory. Actually I
was submitting many files. Each file is 150M and I have 2G of free
RAM.
The code is something like

    for fname in fnames:
        fullname = os.path.abspath(fname)
        twill.commands.go(URL)
        twill.commands.formfile('1', 'file', fullname)
        twill.commands.submit()

and at each loop the memory increases and it is never released. This
is on a 64bit linux system.
Any advice to fix this?