[TIP] A quick guide to testing (for the khmer project)

[John Wong]

To continue with my other thoughts. I'd say I am not really a supporter of
TDD. I am more of a supporter of my own DDD: documentation-driven
development (write my docstring).


On Wed, Jul 23, 2014 at 11:00 AM, Marcin Tustin <
Marcin.Tustin at dealertrack.com> wrote:

> This encodes the code first, write tests afterward approach to
> development. It's bad because it's *hard*. It's hard to come up with good
> test cases post hoc, because there's a lot of analysis to do (which is made
> easier when you write the code, because you're already thinking of those
> things), and because code not written to be tested is more difficult to
> test. TDD solves these problems by merging the development and testing
> activities into one step.
>

I don't think that's always true from maybe a narrow interpretation.. You
can be really sloppy at writing test. For example, you can patch out a
bunch of objects/functions using Mock in your test and yet your code
remains bad and untestable without mock. And what makes a code testable? Do
people think testability should be based on how quickly one can test a code
in a unit test / integration test or should testability also be based on
how usable the code is (e.g. is there too many arguments? the function does
too many things at once inline?).

def add_numbers(*pos) vs def add_numbers(num_list)

There is nothing hard about testing the code above. It's a matter of what
makes one code more usable.

self.assertEqual(add_numbers(1,1,1,1), 4)  vs
self.assertEqual(add_numbers([1,1,1,1])).

Is dependency injection always more testable?


def get_user_data(user_obj)
       return { "username": user_obj.username, "created_on":
user_obj.created_on }

vs
def get_user_data(user_id)
      user_obj = _get_user_data(user_id)
      return ...

Also

def connect(ssh_conn_obj, data):

vs

def connect(data):
      make_ssh_conn(data)

vs

def connect(data)
     # paste make_ssh_conn code inline in this function (maybe subprocess,
maybe fabric, maybe paramiko)

The third connect is obviously much harder to test, but not so if you know
how to use mock (for unit test and integration tests).

I am probably guilty of saying DDD is better, but if I have a writer
blocker while writing docstring for the code, I often take that as a sign
of "I have to rethink about this code." Maybe the code should raise an
exception rather than returning true/false, maybe the code is too small and
don't really need to be standlalone, or maybe the code needs a better
abstraction. I say I am guilty because (1) I will forget to write
docstring, and (2) even if I am happy with the docstring I've written
first, I will change my mind later.

Also, when I talk about TDD to others, I will tell people my version of TDD
is having test in my final patch submission I don't care what kind of test
you provide.

* I prefer at least functional test, You can give me your excuse why you
can't write your unit test and/or integration test. I will make a ticket
for you so we come back if they are tricky to write.

* Sometimes even functional tests are tricky to accomplish. For example, in
one case my shell script to stunnel wasn't working properly in a build
environment but I know it works manually by hand. I have my integration and
unit test, so i am okay checking my code into upstream and make a ticket
about fixing the build environment.

* Code review is more important than arguing about TDD vs DDD.

* I am still a big fan of more functional tests the better. If there is one
thing about functional test that sucks is that we write too little
functional tests. There are too many things we can test on a HTTP API
requesst/response. This is where I devote time or have someone professional
to develop a good testing harness for me. This is also important for
security testing. Unit tests are great but if we will test more functional
requirements, we will soon have a security breach / functional problem.
Because often time each unit function does their own things properly, but
the chain of function calls turns out to be evil (think about multiple
encoding XSS attack).

John

On Thu, Jul 24, 2014 at 1:18 PM, John Wong <gokoproject at gmail.com> wrote:

> On Thu, Jul 24, 2014 at 7:04 AM, C. Titus Brown <ctb at msu.edu> wrote:
>
>> Hey Terry,
>>
>> Do you have suggestions for a good, concise discussion of
>> the difference between unit tests and functional tests?
>>
>> thanks,
>> --titus
>>
>>
> Thought I will give my first 0.01 cent as a recent grad, though most
> people here have more experience than I do.
>
> When I was an undergrad I asked here on TIP mailing list:
> http://comments.gmane.org/gmane.comp.python.testing.general/5333
>
> In the end, when I gave my talk at school, I refer to
> http://googletesting.blogspot.com/2010/12/test-sizes.html
>
> There are many types of testing techniques and I'd say screw most of the
> subtle differences. I went with that table and always remember there are
> only three types of tests: unit test, integration and functional.
>
> unit test: true unit test means mocking out all dependencies while testing
> a specific function or method. This includes built-in functions if
> necessary (e.g. Math.random) The fewer module interaction the better.
>
> integration test: mock out necessary module interaction and external
> systems (filesystem, network, database) whenever possible or whenever I
> feel it is necessary. When it comes to database model testing, I test
> individual model and their methods. I can test them with or without a
> database, depending on how I feel about having a database or not (some ORM
> makes testing easier!!). I can test the individual model instance creation,
> update and deletion of models with sqlite or with a real production
> database like postgresql. The rationel is that (1) I might have PostgreSQL
> datatype (like JSON) which is not supported by sqlite, or (2) I am going to
> use postgres in functional test anyway. The time of setting up a fresh
> database in postgres is not that bad in reality. I can waste 5 seconds for
> the total 100 model tests if I have to.
>
> function test: as the google char suggested, go full.
>
> The rest: smoke, regression, etc I just write them as needed. I can
> careless about their actual definition. When I see a bug, I fix the code,
> write the assertion and done. Different organization has different opinion
> on how they write test.
>
> John
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20140724/5281dea6/attachment-0001.htm>