headers
Thatcher Ulrich | 1 Sep 2008 06:31

Re: How easy is it to unit test

On Sun, Aug 31, 2008 at 12:51 PM, Noel Llopis <llopis <at> gmail.com> wrote:
> On Sat, Aug 30, 2008 at 8:00 PM, Thatcher Ulrich <tu <at> tulrich.com> wrote:
>>
>> * unit tests are dead easy to write.  It's not a chore at all; it in
>> fact makes my job much easier.  PROVIDED there is a good testing
>> framework in place.  Getting a good testing framework in place, and
>> keeping it running, can indeed be an incredible chore.
>
> Funny, I'd almost say the opposite. In my experience getting a good
> testing framework in place and keeping it running is trivial
> (http://unittest-cpp.sourceforge.net/ or the new Google one)

Definitely start there, something like that is important, but in my
opinion the larger framework needs to actually run the tests and send
informative, targeted emails.  I think that's outside the scope of
unittest-cpp or gunit because it gets into your source control and
build system and whatnot.  But if you don't have it, developers need
to remember to run the tests, that introduces a lot of drag, and
you're not really getting full value.  Like, one syndrome that happens
is devs who aren't on board with unit testing don't run the tests,
they inadvertently break the tests, and then test-aware devs later
discover the breakage and have to either nag the culprits or fix
things themselves.  Gah.  Far better and easier for everyone if you
get a specific automated email soon after submitting a breakage,
showing what test broke and what change is implicated.

-T
_______________________________________________
Sweng-Gamedev mailing list
Sweng-Gamedev <at> lists.midnightryder.com
(Continue reading)

Permalink | Reply | 628 comments |
headers
Jim Tilander | 1 Sep 2008 06:45
Picon

Re: How easy is it to unit test

>> Funny, I'd almost say the opposite. In my experience getting a good
>> testing framework in place and keeping it running is trivial
>> (http://unittest-cpp.sourceforge.net/ or the new Google one)
>
> Definitely start there, something like that is important, but in my
> opinion the larger framework needs to actually run the tests and send
> informative, targeted emails.
[snip]

Yes. There are several packages that do exactly this (and more),
google for "continuous integration server". The one I've used in the
past is called CruiseControl.NET, but there are of course a plethora
of these things out there.

/j

--

-- 
Beware of architect astronauts.
_______________________________________________
Sweng-Gamedev mailing list
Sweng-Gamedev <at> lists.midnightryder.com
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
Permalink | Reply | 0 comments |
headers
Tom Plunket | 1 Sep 2008 20:58

Re: How easy is it to unit test

> Definitely start there, something like that is important, but in my
> opinion the larger framework needs to actually run the tests and
> send informative, targeted emails.

General automated testing frameworks should send targeted emails,
but this is not necessary for unit testing, because unit tests
should be fast enough to run every build.  The best setup is to have
them automatically run as part of the build process, so you get
"Build Failed!" when the unit tests fail to run, but I've had
success just putting RunUnitTests() as the first line in main(), and
having the game abort if any unit tests fail.

This pleasantly works around the affliction of some developers on
your team not running your unit tests, although I have personally
experienced folks commenting out unit tests when they broke them
rather than making the code that they just broke actually work
correctly.  <grumble grumble>

Now for functional tests (i.e. those tests that crosscut
functionality), I suppose it depends on your workflow, but if the
set can be run in only a couple of minutes, requiring a runthrough
before submitting code shouldn't be too much of a burden.  However
you'll still specifically want your build backend running them and
letting folks know who were involved in checkins between "not
broken" and "broken."

-tom!
_______________________________________________
Sweng-Gamedev mailing list
Sweng-Gamedev <at> lists.midnightryder.com
(Continue reading)

Permalink | Reply | 0 comments |
headers
Aras Pranckevicius | 2 Sep 2008 09:14
Favicon
Gravatar

Re: unit testing survey

> I wonder how widespread the adoption of unit testing is in the games industry? So
> without getting into the "is unit testing a good idea" debate (which has been done before
> and I'll happily discuss it on another thread), I thought it might be interesting to do a little
> survey to find out how many people and / or studios are using it for game development
> and whether the adoption was successful or not. So... are you using unit testing for your
> game?

We aren't making games, but we're making the tech to make games. Also,
because we have a web plugin out there, we can't generally change
behavior at large between releases (even if fixing bugs - some content
might be out there that happens to depend on the bug being there). So
our situation is probably quite different...

That said, we don't use unit testing in a sense of "test every small
bit of code". We do have somewhat more high level functional tests
(e.g. does this scripting API call do the right thing; do physics
triggers work; are coroutines invoked in correct order; ...).
Basically, we unit test public API entry points that our users can
access. We don't test our own internal code that much.

Some other tests are mostly related to external tools and the asset
pipeline. E.g. FBX importing tests (because Autodesk loves to break or
change functionality between FBX SDK releases... so upgrading to new
version is never a light task). In this case, we have a collection of
input files, and for each we have a file that represents actual
imported and processed data, plus hashes of the imported data. If
reimporting the files produces different results than the "template
data" we have, it's a change or bug somewhere.

> I use unit testing whenever I can, but I wonder how to do it with things
(Continue reading)

Permalink | Reply | 8 comments |
headers
Lucas Meijer | 2 Sep 2008 09:59
Picon
Gravatar

Re: unit testing survey

Aras Pranckevicius wrote:
>> So... are you using unit testing for your
>> game?
>>     
>
> We aren't making games, but we're making the tech to make games. 

We are making a game using Aras' tech, and we tend to write unit tests 
first, and then
write the implementation. In most cases, we really like the code this 
produces, if only
because it forces us to think. Areas where we diverge from this 
behaviour is:

- if we don't know the problem domain very well, and decide to first 
just "try something just
for trying", to better acquaint ourselves with the problem/tool/language 
at hand.
(when the dust settles down, we then write the test afterwards)

- When we get too close to the 3d engine, it gets hard to write unit 
tests that can execute from
the commandline, due to the nature of the unity engine.  So we try to 
only "bind" to the 3d engine
as late as possible, keeping as much code as possible testable.

- When we can't be bothered. This sounds silly but actually happens from 
time to time. I always
regret it in the end, and have to spend more test writing the test 
anyway, often realizing my design
(Continue reading)

Permalink | Reply | 2 comments |
headers
Benoit Miller | 2 Sep 2008 15:27

Re: unit testing survey

Lucas Meijer wrote:
> - if we don't know the problem domain very well, and decide 
> to first just "try something just for trying", to better 
> acquaint ourselves with the problem/tool/language at hand.
> (when the dust settles down, we then write the test afterwards)

My experience has been the exact opposite. When I don't exactly know
where I'm going, writing the tests first are a great way for me to state
what I know about the problem and safely iterate towards the "final"
solution. As I'm writing a test (or fixing one to match my expected
behavior) I get to understand a little bit more of the problem, and get
ideas for more tests.

For me this technique really shines when I /think/ I understand
something, change the code accordingly, then see a whole bunch of
previously-working tests fail spectacularly. Then I /know/ I messed up
somewhere in my thought chain (or I made a silly typo!). I love seeing
tests fail, for the confidence I get when they pass.

In my current project, we've gone so far as to include UnitTest++ in our
code base, but I don't know of any developer that used it. The code is
fairly resistant to testing, so sadly I don't get to use it often
either. There is a very small amount of "legacy" unit tests with a
home-grown framework, but these tests are rarely run. Yes, some failing
parts are commented out... :/

Ben.
_______________________________________________
Sweng-Gamedev mailing list
Sweng-Gamedev <at> lists.midnightryder.com
(Continue reading)

Permalink | Reply | 1 comment |
headers
christer_ericson | 2 Sep 2008 19:56
Picon

Re: unit testing survey (Denzil Long)


Jon Watte wrote:
> Denzil Long wrote:
> > there own dogfood". Generally I have found that code developed under
> > TDD is more robust and easier to use.
>
> I think we all agree. The question is how we address the perception that
> the benefit of that robuster, easier to use code is less than the cost
> of using TDD. Starting with numbers showing it's just a perception, and
> not actual reality, would be great :-)

Nope, we don't all agree.

Hopefully everyone on this list is already clear on the difference, but
since I often see people confuse the two, I wanted to stress that "using
unit tests" and "using TDD" are two VERY different things.

I would be surprised to hear anyone say that unit testing is bad. However,
writing the tests _before_ you write the code (which is the key tenet
behind TDD) is not "obviously better" than alternatives such as
test-last.

Christer Ericson, Director of Tools and Technology
Sony Computer Entertainment, Santa Monica
http://realtimecollisiondetection.net/blog/

_______________________________________________
Sweng-Gamedev mailing list
Sweng-Gamedev <at> lists.midnightryder.com
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
(Continue reading)

Permalink | Reply | 0 comments |
headers
Jon Watte | 2 Sep 2008 20:47
Picon
Gravatar

Re: unit testing survey

Aras Pranckevicius wrote:
> Basically, we unit test public API entry points that our users can
> access. We don't test our own internal code that much.
>   

That is almost the definition of a functional test (for an API), and 
could also be used as an acceptance test.

Unit tests test particular, low-level behaviors in an implementation -- 
you could, conceptually, write a separate unit test for each if(), 
switch() or virtual function in your implementation code. Complete unit 
test coverage means that you have complete code coverage in your tests 
-- each conditional is tested going both ways. To be able to actually 
test like this, your code often needs to be significantly more broken up 
than when you only do functional testing, which is why it's sometimes 
hard to implement the unit testing mindset in a larger engineering 
organization. Writing unit testable code is a new skill, and it's not 
taught in university, nor most other workplaces.

Sincerely,

jw

_______________________________________________
Sweng-Gamedev mailing list
Sweng-Gamedev <at> lists.midnightryder.com
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
Permalink | Reply | 4 comments |
headers
Tom Plunket | 2 Sep 2008 21:19

Re: unit testing survey

> My experience has been the exact opposite. When I don't exactly know
> where I'm going, writing the tests first are a great way for me to
> state what I know about the problem and safely iterate towards the
> "final" solution.

My decision to start unit testing (via TDD) came when I was faced
with some operations that I didn't know how to do what I was tasked
with, literature on the topics was lacking, and "reference"
implementations were very poor and had requirements that were not
acceptable.

I started with a method to generate test data for an algorithm, then
I generated some test data and then tested that the algorithm
yielded the result I was looking for.  The first test was not
sufficient to build the algorithm but allowed me to define inputs
and outputs, but that first test (for each of these projects) failed
for quite a while as I poked and prodded and tried different things
and eventually settled in on a solution.

As with many TDD-driven projects, the resulting code was nothing
like I would have dreamt up without doing tests up-front, but was
instead far superior in its elegance and clarity.

-tom!
_______________________________________________
Sweng-Gamedev mailing list
Sweng-Gamedev <at> lists.midnightryder.com
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
(Continue reading)

Permalink | Reply | 0 comments |
headers
Aras Pranckevicius | 2 Sep 2008 21:24
Favicon
Gravatar

Re: unit testing survey

>> Basically, we unit test public API entry points that our users can
>> access. We don't test our own internal code that much.
>
> That is almost the definition of a functional test (for an API), and could
> also be used as an acceptance test.
> <...>
> Unit tests test particular, low-level behaviors in an implementation

Yes. But it's a fuzzy line to make hard divisions. For example, a test
for "does rendering into a cubemap work?" could be treated as a
functional test (it tests a single, small, but actual feature), or as
a unit test (it tests whether a small, isolated piece of code that
sets up matrices and invokes other rendering code works correctly).

That said, yes, I'd call our tests more like functional tests.
Sometimes they test small isolated piece of functionality that also
happens to be small isolated piece of code, and gets almost complete
coverage of the code. So in those cases it could be called either.

--

-- 
Aras Pranckevičius
work: http://unity3d.com
home: http://aras-p.info
_______________________________________________
Sweng-Gamedev mailing list
Sweng-Gamedev <at> lists.midnightryder.com
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com
Permalink | Reply | 1 comment |

Gmane