[TIP] A quick guide to testing (for the khmer project)

[Marcin Tustin]

I'd say I am not really a supporter of TDD.

Why not? Have you given it a real try? If so, what did you find didn’t work for you?

From: testing-in-python-bounces at lists.idyll.org [mailto:testing-in-python-bounces at lists.idyll.org] On Behalf Of John Wong
Sent: Thursday, July 24, 2014 2:56 PM
To: titus at idyll.org
Cc: tip at lists.idyll.org
Subject: Re: [TIP] A quick guide to testing (for the khmer project)

To continue with my other thoughts. I'd say I am not really a supporter of TDD. I am more of a supporter of my own DDD: documentation-driven development (write my docstring).


On Wed, Jul 23, 2014 at 11:00 AM, Marcin Tustin <Marcin.Tustin at dealertrack.com<mailto:Marcin.Tustin at dealertrack.com>> wrote:
This encodes the code first, write tests afterward approach to development. It's bad because it's *hard*. It's hard to come up with good test cases post hoc, because there's a lot of analysis to do (which is made easier when you write the code, because you're already thinking of those things), and because code not written to be tested is more difficult to test. TDD solves these problems by merging the development and testing activities into one step.

I don't think that's always true from maybe a narrow interpretation.. You can be really sloppy at writing test. For example, you can patch out a bunch of objects/functions using Mock in your test and yet your code remains bad and untestable without mock. And what makes a code testable? Do people think testability should be based on how quickly one can test a code in a unit test / integration test or should testability also be based on how usable the code is (e.g. is there too many arguments? the function does too many things at once inline?).
def add_numbers(*pos) vs def add_numbers(num_list)
There is nothing hard about testing the code above. It's a matter of what makes one code more usable.
self.assertEqual(add_numbers(1,1,1,1), 4)  vs self.assertEqual(add_numbers([1,1,1,1])).
Is dependency injection always more testable?

def get_user_data(user_obj)
       return { "username": user_obj.username, "created_on": user_obj.created_on }
vs
def get_user_data(user_id)
      user_obj = _get_user_data(user_id)
      return ...
Also

def connect(ssh_conn_obj, data):
vs

def connect(data):
      make_ssh_conn(data)
vs
def connect(data)
     # paste make_ssh_conn code inline in this function (maybe subprocess, maybe fabric, maybe paramiko)
The third connect is obviously much harder to test, but not so if you know how to use mock (for unit test and integration tests).

I am probably guilty of saying DDD is better, but if I have a writer blocker while writing docstring for the code, I often take that as a sign of "I have to rethink about this code." Maybe the code should raise an exception rather than returning true/false, maybe the code is too small and don't really need to be standlalone, or maybe the code needs a better abstraction. I say I am guilty because (1) I will forget to write docstring, and (2) even if I am happy with the docstring I've written first, I will change my mind later.
Also, when I talk about TDD to others, I will tell people my version of TDD is having test in my final patch submission I don't care what kind of test you provide.
* I prefer at least functional test, You can give me your excuse why you can't write your unit test and/or integration test. I will make a ticket for you so we come back if they are tricky to write.
* Sometimes even functional tests are tricky to accomplish. For example, in one case my shell script to stunnel wasn't working properly in a build environment but I know it works manually by hand. I have my integration and unit test, so i am okay checking my code into upstream and make a ticket about fixing the build environment.
* Code review is more important than arguing about TDD vs DDD.
* I am still a big fan of more functional tests the better. If there is one thing about functional test that sucks is that we write too little functional tests. There are too many things we can test on a HTTP API requesst/response. This is where I devote time or have someone professional to develop a good testing harness for me. This is also important for security testing. Unit tests are great but if we will test more functional requirements, we will soon have a security breach / functional problem. Because often time each unit function does their own things properly, but the chain of function calls turns out to be evil (think about multiple encoding XSS attack).
John

On Thu, Jul 24, 2014 at 1:18 PM, John Wong <gokoproject at gmail.com<mailto:gokoproject at gmail.com>> wrote:
On Thu, Jul 24, 2014 at 7:04 AM, C. Titus Brown <ctb at msu.edu<mailto:ctb at msu.edu>> wrote:
Hey Terry,

Do you have suggestions for a good, concise discussion of
the difference between unit tests and functional tests?

thanks,
--titus

Thought I will give my first 0.01 cent as a recent grad, though most people here have more experience than I do.

When I was an undergrad I asked here on TIP mailing list: http://comments.gmane.org/gmane.comp.python.testing.general/5333

In the end, when I gave my talk at school, I refer to http://googletesting.blogspot.com/2010/12/test-sizes.html

There are many types of testing techniques and I'd say screw most of the subtle differences. I went with that table and always remember there are only three types of tests: unit test, integration and functional.
unit test: true unit test means mocking out all dependencies while testing a specific function or method. This includes built-in functions if necessary (e.g. Math.random) The fewer module interaction the better.

integration test: mock out necessary module interaction and external systems (filesystem, network, database) whenever possible or whenever I feel it is necessary. When it comes to database model testing, I test individual model and their methods. I can test them with or without a database, depending on how I feel about having a database or not (some ORM makes testing easier!!). I can test the individual model instance creation, update and deletion of models with sqlite or with a real production database like postgresql. The rationel is that (1) I might have PostgreSQL datatype (like JSON) which is not supported by sqlite, or (2) I am going to use postgres in functional test anyway. The time of setting up a fresh database in postgres is not that bad in reality. I can waste 5 seconds for the total 100 model tests if I have to.
function test: as the google char suggested, go full.

The rest: smoke, regression, etc I just write them as needed. I can careless about their actual definition. When I see a bug, I fix the code, write the assertion and done. Different organization has different opinion on how they write test.
John


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20140724/18516534/attachment-0001.htm>