Towards a better testing infrastructure
Hi,
I've been talking to a few people on IRC about this over the past few
days, but I wanted to put some of it down in writing, to see if there's
any interest in people helping to take this forward.
In short, I think ZopeTestCase and PloneTestCase have become too
unwieldly. There are too many ways to set up test suites, too many base
classes, and too much is set up even for relatively simple tests. In
projects where I work with people who're not Plone experts, I've all but
given up on trying to set up test layers and minimal test fixtures.
We're lucky if we do a few "plain unittest.TestCase" tests. 90% of tests
are full PloneTestCase integration tests, and most people in my teams
(and, I'd wager, out there in the community) have only a vague
understanding about what PloneTestCase is doing and how it affects their
test environment. Leakage between tests happens, and when it happens,
it's hard to track down.
I don't think we should try to "fix" ZTC/PTC, because the BBB
implications would be heinous. Instead, I think we can create some new,
simpler, better documented (!) test infrastructure. New packages can opt
into this instead of using ZTC/PTC. Old packages will not need to move.
I envisage us ending up with two packages: plone.testing, and
plone.app.testing. The former would be a good test dependency for
plone.* type packages, that do not depend on "Plone-the-application",
and the latter for plone.app.* type packages, that do.
The key to all this, I think, is layers. We should end up with a couple
of useful base classes for tests, and then a set of well-defined,
well-documented layers that people can use and extend for their own
purposes.
Below, I'll outline a few thoughts on what could go in each of these
fictitious packages.
Use cases
---------
I see the following "valid" types of tests. I think we should have
exactly one documented way to do each one of these.
- A unittest.TestCase-style unit test (I'll call this a Python unit
test from now on, as distinct from a doctest)
- A doctest file
- Doctests in docstrings in a module
Layers
------
For each of these, I see the following layers being available:
- No layer - just a plain unit test
- No layer, but full teardown of the ZCA after each test. This makes
it safe to use e.g. provideAdapter() in a test.
- A minimal ZCA layer that hooks up things like the object event
redispatcher.
- A basic ZCA layer that also registers the core ZCML directives
(those in the zope and browser namespaces)
- A Zope 2 layer that wires up Zope 2's configuration, but not to the
point of setting up users or a ZODB root.
- A basic ZODB layer that makes a ZODB root available.
- A combination of the minimal Zope 2 layer + basic ZODB layer, with a
default user installed. This would be more or less like a vanilla Zope 2
installation.
- A Plone layer that configures the minimum possible to get a full
Plone site. In particular, this should *not* auto-load site.zcml or all
of Products.* or anything included with z3c.autoinclude's plugin
mechanism. It also should not install any content or create any users.
As far as possible, this should be isolated from the possibly-changing
Plone default policies, e.g. it should use a well-defined (or not)
workflow. This would probably also need before-test and after-test hooks
so that we could do things like set the site hook.
- A ZServer layer that starts up a ZServer for things like Selenium or
Windmill testing. This would probably be used in combination with either
the "vanilla" Zope 2 layer or the Plone site layer.
Helpers
-------
In addition, I think we should have collective.testcaselayer-style
infrastructure for defining and adding layers, including some layer
helpers for doing common things like adding users or installing
products. I've made some suggestions below. These will obviously only
make sense in some layers, and sometimes be more useful in tests than in
layers (though I think they should be available in both).
- Load the ZCML for a package
- Install a Zope 2-style product (is that still necessary?) - no
difference between "products" and "packages"!
- Add a user to the root user folder
- Add a user to the user folder in a Plone site.
- Apply an extension profile (maybe easier to just use portal_setup
directly)
- Set roles for a user
- Set a current authenticated user
- Clear the current authenticated user
Setup
-----
Right now, there's a lot of confusion around how to properly create
layers, how to register test suites (think ZopeDocFileSuite vs.
DocFileSuite ), etc. I think we want a few basic patterns. This may be
more about documentation that code, though.
- Create a suite of all TestCase-style tests in a module
(unittest.defaultTestLoader helps here)
- Create a suite of all doctests in another module, with a layer (in
one call)
- Create a doctest with a layer (in one call)
- ZODB sandboxing for functional tests. It needs to be quite clear
where transaction rollback and other forms of sandboxing is happening.
We have to support both the normal case, where transactions are rolled
back for each test, and the more complex scenario where we have
functional tests using things like testbrowser, Selenium, Windmill that
execute several requests that may result in commits.
Okay, I think that's enough for now. I'm quite interested in helping
with this effort (obviously), in particular around quality control -
making sure the approaches are consistent, obvious and well documented
enough to be useful. I'm also willing to help with code, although I
think others (like Stefan and Ross) are better able to interpret the
basic infrastructure.
We're all affected by this, and I'm sure that if we added up the time we
all spend looking up code samples and hunting down poorly constrained
tests, we could easily justify spending a bit of time making a better
toolset for ourselves. :)
Martin
--
--
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
RSS Feed