Feature request: <at> Assumes
Stephen Connolly <stephen.alan.connolly <at> gmail.com>
2011-09-14 09:18:29 GMT
Consider the case where you are testing a List class...
we have
public class ListTest {
<at> Test
public void newListIsEmpty() {
assertThat(new List().isEmpty(), is(true);
}
<at> Test
public void newListHasSizeZero() {
assertThat(new List().size(), is(0));
}
<at> Test
public void addPutsAnElementIntoAnEmptyList() {
List l = new List();
l.add(new Object());
assertThat(l.isEmpty(), is(false));
}
<at> Test
public void addIncreasesSizeOfPopulatedListByOne() {
List l = new List();
l.add(new Object());
int s = l.size();
l.add(new Object());
assertThat(l.size(), is(s + 1));
}
}
We now want to add some tests of the delete functionality... but the
reality is that until/unless some of the preceding tests are passing,
the tests for delete are meaningless. We could have a perfectly
functional List.delete() method but until such time as the above tests
are passing, there is no way to tell that the method does not work.
Now I could code my tests like such
<at> Test
public void deleteIsANoOpOnEmptyList() {
List l = new List();
assumeThat(l.isEmpty(), is(true));
l.delete(new Object());
}
But all that I am doing is repeating code from the preceding tests,
having changed all those tests' assertThat(...)s into assumeThat(...)s
That does not seem agile to me, copy & paste & search & replace... ban
code smell there
I would much rather be able to annotate the tests with an <at> Assumes
annotation that indicates that the test assumes that the specified
tests are passing, e.g.
<at> Test
<at> Assumes("newListIsEmpty")
public void deleteIsANoOpOnEmptyList() {
List l = new List();
l.delete(new Object());
}
<at> Test
<at> Assumes({"newListIsEmpty","addPutsAnElementIntoAnEmptyList")
public void deleteRemovesAnElement() {
List l = new List();
Object o = new Object();
l.add(o);
l.delete(o);
assertThat(l.isEmpty(), is(true));
}
In fact in my initial example of tests, there are some additional
assumptions that I didn't make explicit
<at> Test
<at> Assumes("newListIsEmpty")
public void addPutsAnElementIntoAnEmptyList() {
...
}
and
<at> Test
<at> Assumes({"newListIsEmpty","addPutsAnElementIntoAnEmptyList")
public void addIncreasesSizeOfPopulatedListByOne() {
...
}
Now you could get some of this functionality via a TestRule...
You could watch tests to see if they pass, and skip tests annotated
with the annotation if assumed functionality is failing, but that
would result in sporadic failures of, e.g. deleteRemovesAnElement
because of the failing newListIsEmpty being executed _after_
deleteRemovesAnElement rather than before.
The simple point is that the test result of deleteRemovesAnElement is
meaningless until its assumptions are true, and while I could code the
assumptions with assumeThat(..)s C&P&S&R is even worse than C&P.
Another alternative to <at> Assumes would be to invoke the assumed
method(s) at the start of the test, e.g.
<at> Test
public void deleteRemovesAnElement() {
newListIsEmpty(); // verify assumed functionality
addPutsAnElementIntoAnEmptyList(); // verify assumed functionality
...
}
That gets rid of the C&P&S&R, but there are two issues with that:
1. We have to manually invoke any setup/tearDown methods, including
all those of the rules that the test class has... very messy
2. The test fails when the assumed test fails. In actuality we can
say nothing at all about whether deleteRemovesAnElement if a
newListIsEmpty is not passing... yes we could code the test
differently, but that is just moving our assumptions somewhere else.
I am sure that there are others out there who feel there is a point 3...
3. We already ran those tests why waste time running them again?
Well the answer to 3 is that these are UNIT tests which should be very
fast, so what is the harm...
So, in my view, best practice unit testing needs the ability to mark
tests as assuming that other tests are passing, so that those tests
can be skipped when the assumptions are known to be failing or
skipped. [This is a deliberately loaded criteria... if the
org.junit.runner.Request does not include the assumed test, then that
test is neither known failing or known skipped, so we can run the test
and output a warning that the failure may be because of assumed
functionality... the use case of executing one and only one test
repeatedly until you get that test passing]
The annotation would have implications on test sorting, as any assumed
tests would have to always happen before the assuming tests (as long
as the assumed tests are in the org.junit.runner.Request)
Also might have to be two annotations, e.g.
<at> Assumes(methodNames)
<at> AssumesClasses(classes)
though in my view the <at> AssumesClasses is less critical, as these are
UNIT tests and each test class should be independent to a large
extent. However I am willing to consider that some people may have
many test classes for one class under test, one test class containing
all the tests of the constructors, another testing the Add methods,
etc. in which case an <at> AssumesClasses annotation makes sense.
Where tests contain a circular dependency, fail/error both tests
Ok, let the critique begin!
-Stephen
P.S.
I pinged Kent with an earlier version of this idea... but I think that
he missed the point about eliminating C&P&S&R that this feature would
provide because I didn't frame the idea correctly...
---------- Forwarded message ----------
From: "Kent Beck"
Date: 13 Sep 2011 17:11
Subject: Re: JUnit and test dependencies
To: "Stephen Connolly"
Stephen,
Thank you for articulating your idea so clearly. The short answer is that
no, we don't plan to support dependencies. If I have tests that are slow
enough that I care about dependencies, my most productive option is
generally to work on the design of the software until the tests are fast
enough that I no longer care. That said, my voice is only one of many. The
longer answer is that I encourage you to post your idea on the JUnit mailing
list for community discussion.
Regards,
Kent
On Sep 13, 2011, at 8:32 AM, Stephen Connolly wrote:
> Kent,
>
> Are there any plans for JUnit to support some test dependencies, such as:
>
> public class OnlyRunTestsThatMakeSenseTest {
>
> <at> Test
> public void basicFunctionalityWorks() {
> ...
> }
>
> <at> Test
> <at> AssumesPasses("basicFunctionalityWorks")
> public void advancedFunctionalityWorks() {
> ...
> }
>
> <at> Test
> <at> AssumesPasses("basicFunctionalityWorks")
> public void basicFunctionalityWorksWithBevel() {
> ...
> }
>
> <at> Test
>
<at> AssumesPasses({"basicFunctionalityWorksWithBevel","advancedFunctionalityWorks"})
> public void advancedFunctionalityWorksWithBevel() {
> ...
> }
>
> }
>
> In the above example, no matter what sorting is applied,
> basicFunctionalityWorks will always be run first, and the other three
> tests will only be run if basicFunctionalityWorks passed.
>
> I see the above being completely in the spirit of unit testing, the
> point with the above is that the <at> Before and <at> After's will be run
> around each method, you are just saying that there is no point even
> trying to test the advanced functionality when the basic functionality
> is broken, skip those tests which we know cannot pass. That allows the
> person writing advancedFunctionalityWorks to power through the setup
> that depends on the basic functionality and not have to litter their
> advanced test with asserts that are redundant because of the basic
> functionality. Those people who are relying on side-effects should
> really, for unit tests at least, be invoking the method who's
> side-effects they depend on directly within their test method, rather
> than relying on accidental ordering.
>
> Having said that, a second feature that I think would be good is
> something like a <at> RunAfter and/or <at> RunBefore which would ensure that
> the test method is run in sequence even if the before or after tests
> fail/are skipped. with <at> RunAfter and <at> RunBefore I still think the
> <at> Before and <at> After methods should be invoked in-between, this would be
> moving towards more of a general purpose testing framework as opposed
> to being unit-testing focused, but JUnit is just too good 
>
> Thoughts?
>
> -Stephen
------------------------------------------------------------------------------
BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA
Learn about the latest advances in developing for the
BlackBerry® mobile platform with sessions, labs & more.
See new tools and technologies. Register for BlackBerry® DevCon today!
http://p.sf.net/sfu/rim-devcon-copy1
RSS Feed