[TIP] Finding a way to support test data (logs) in unittest [WAS] Determine test status in tearDown method

[Robert Collins]

On Thu, 2009-10-22 at 08:21 -0500, Olemis Lang wrote:
> Separate thread because this discussion goes far beyond the scope of
> the original question ... ;o)
> 
> On Tue, Oct 20, 2009 at 4:18 PM, Robert Collins
> <robertc at robertcollins.net> wrote:
> > On Tue, 2009-10-20 at 09:28 -0700, Jesse Thompson wrote:
> >
> [...]
> >
> > In my blog post I also note that we really should provide access to the
> > result object to 'user code' - code like your tearDown.
> >
> 
> -1 ... you're just writing functional tests so the goal here should be
> to handle just the part of the testing (artifacts | process) that's
> related to the SUT and its expected behavior.

This isn't about function||unit tests, its about the connection between
the code of the test, and the reporting function. At the moment the
TestCase acts as a poor proxy: it has an unextendable interface - one
must reimplement 'run' to change it.

> > This suggests two ways of getting the logging data - buffering/callback
> > on the test, and direct to the result.
> >
> > buffering on the test would look something like:
> > def tearDown(self):
> >    # defaults to text/plain;charset=utf8 for unicode objects
> >    self.attach('log', logfile_string)
> >
> > Then the test would have to accumulate all the attached things and pass
> > then to the result object:
> > def attach(self, name, content):
> >    if isinstance(content, unicode):
> >         orig_content = content
> >         content = Content(
> >             ContentType('text', 'plain', {'charset':'utf8'}),
> >             lambda:[orig_content.encode('utf8')])
> >    self.__current_details[name] = content
> >
> 
> At least for warnings you'll need to accumulate logs (i.e. warning
> messages), not override the previous message
> 
> -1 for `__current_details` IMO `_current_details` or a shorter name is
> better (unless a property i)

The __current_details indicates private: in this example approach the
TestCase would provide all the attached details to the TestResult at
once - no other code would access __current_details *at all*. I
mentioned accumulation later, but conceptually you'd attach a logs
detail that is itself an accumulator.

> > Talking directly to the result object would look something like:
> > def tearDown(self, result):
> >    result.attach_details('log', logfile_string)
> >
> 
> -1 ... because :
> 
>   - data should be bound to the test case instance involved in the test.
>     sometimes this is important.

I don't understand this - this doesn't change the responsibility for
data binding, it changes /how things are reported/. Currently the only
thing reportable against a test case is the stacktrace.

>   - IMHO it's a very bad idea and a very poor design decision to allow
>     access to TestResult objects in methods of TestCase

Why? Details!

> > Perhaps we should instead 'just' add the
> > 'attach_details' API,
> 
> Name's toooooo long why not just `tc.log.xxx` or `tc.xxx` thus
> providing an interface compatible with logging module . 

Because logging is one special case of the needs I've seen. 

> This would
> imply that :
> 
>   - interface is well known and std
>   - low learning curve but ...
>   - should be compatible with TestCase semantics (e.g. tc.log.error
>     should behave just like tc.fail)
>   - level of detail could be used to (specify | filter) the data that
>     will be stored during the test (e.g. if log level is INFO then
>     DEBUG calls will be ignored or at least will have no impact on
>     the state of the test result) ...
>   - but error level *MUST* always be reported even if CRITICAL is set
>   - logging module supports GoF command pattern thus log entries
>     could be stored directly inside instances of TestResult
> 
> BTW , why to reinvent the wheel ?

Because logging isn't sufficient, and I'm trying to ensure the interface
for *extenders* is good - as long as that is in place, we can offer a
plethora of different interfaces on TestCase, compatibly.


-Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20091023/bd66c295/attachment.pgp>