[TIP] Running tests of externally-maintained apps when testing Django projects

[Jonathan Hartley]

On 07/05/2012 23:37, John Wong wrote:
> Hi,
>
> I am not a testing expert, but my development falls in this domain as 
> well since my django site is only used as a web interface. I don't 
> even use django model except for registration (which has to authorized 
> through LDAP) and django session (the default stuff).
>
> >Is it worth running the tests of these externally-maintained apps when
> > testing the site as a whole?
> Isn't this a system test rather than unit-test? Yes. You want to find 
> out whether they work or not, right?
>
>
> >it'sarguable that someone else would be keeping an eye on the unit 
> tests, so
> > there's no need for me to run them.
>
> Isn't the whole point of unittest really testing the "correctness" of 
> an individual unit (which usually means a method / a function) that 
> actually DOES something (rather than just initialize some data, or 
> print, etc).
> If you don't run integration test on these external apps as a whole, 
> you can't assure the quality of the system as a whole. The idea would 
> be "scope". Test the scope / objective of the service (referring to 
> the purpose of using this particular app in your main site) within the 
> main site, and test the untested purposes within the app level (which 
> could be itself a django project)....
>
> A situation would be, say My-Site (your main Django project) has an 
> upload feature, which runs on a different Django PROJECT (another 
> django site) called My Upload. You will need to perform integration 
> tests on both projects. If the inetgration is so soft that you 
> literally take in the file object and pass it to your upload django 
> service, you really just need a few simple, quick integration tests on 
> the main site to test upload(relevant to your main site). If the 
> upload service ITSELF does many things, test those "untested" one 
> within the upload service django project.
>
> John
>
>
> On Mon, May 7, 2012 at 4:04 PM, Alexander Dutton 
> <lists at alexdutton.co.uk <mailto:lists at alexdutton.co.uk>> wrote:
>
>     Hi all,
>
>     Best practice question. I maintain a Django-based site that comprises
>     various externally-maintained apps and some apps that are
>     site-specific.
>     Should I be viewing these as solely unit
>     tests, or as integration tests?
>
>     Most of the externally-maintained apps (e.g. object_permissions) come
>     with their own settings files and are invoked like:
>
>     $ django-admin test --settings=object_permissions.tests.settings
>
>     These settings often declare a specific ROOT_URLCONF and other
>     settings
>     (e.g. TESTING = True). Just including the app in INSTALLED_APPS is
>     liable to result in test failures. I imagine I'd have to patch
>     ROOT_URLCONF for each app, which means a custom test runner and
>     general
>     ugliness. It also doesn't suffice for integration testing, and it's
>     arguable that someone else would be keeping an eye on the unit
>     tests, so
>     there's no need for me to run them.
>
>     Any thoughts?
>
>     Yours,
>
>     Alex
>
>     _______________________________________________
>     testing-in-python mailing list
>     testing-in-python at lists.idyll.org
>     <mailto:testing-in-python at lists.idyll.org>
>     http://lists.idyll.org/listinfo/testing-in-python
>
>
>
>
> _______________________________________________
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
> http://lists.idyll.org/listinfo/testing-in-python


Personally, I think no, it is not worth running the tests for 3rd-party 
Django apps. Certainly not for pre-commit tests (which I will go to 
enourmous lengths to make run as fast as possible.) I could be persuaded 
that it's worth running them in continuous integration, because the cost 
of a slightly longer build is less significant there, but only provided 
running them is very easy, fast, and requires no nannying.

You do want to be sure that these tests are actually passing in your 
project, but on the other hand, you aren't changing the code that they 
test, so the chance of them breaking is relatively low. I think it's a 
question of judgement rather than absolute right or wrong, but 
personally I've never had any benefit (i.e. useful failures) from 
running these tests. So I'd rather speed up the build by skipping them.

-- 
Jonathan Hartley    tartley at tartley.com    http://tartley.com
Made of meat.       +44 7737 062 225       twitter/skype: tartley


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20120509/681d1d8b/attachment.htm>