curl unit testing

Running suite(s): Curl_llust
0%: Checks: 1, Failures: 1, Errors: 0
llist_test.c:28:F:Core:test_llist_initiate:0: list initial size should be zero

Running suite(s): Curl_llust
100%: Checks: 1, Failures: 0, Errors: 0

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Re: curl unit testing

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Re: curl unit testing

On Sat, 1 Jan 2011, Amr Shahin wrote:

> Hello folks,
>
> i'm attaching a sample very small unit tests using both check unit testing 
> library and the current curl testing framework integrated with curl, i was 
> hoping we could asses both and see which is more suitable to write unit 
> tests.

Hi and thanks a lot for your work on this!

> the code uses check is in the file llist_test.c. It's a classic unit testing 
> code, with all basic functions there setup, teardown and macros for defining 
> your tests, the output is well organized, you can see sample output for both 
> all success, and one failed test in the attached files. The only downside 
> for using check is that we're adding an extra dependency to curl, my 
> suggesstion is to add the .so version and check.h to the code base.

I have some objections and I will suggest another approach:

check doesn't seem to be written to grok C89 compilers, have you checked (no 
pun intended) if it does?

I would *REALLY* like to have the full test code bundled with curl and not 
depend on a separate download/package to get a decent test suite, and I find
>500K for a unittest for C quite much. Did anyone of you see how much we would 
need to copy to get a working check bundled?

> the other option is to use the testing setup already in libtest directory, 
> it's pretty easy to add more tests, but it's not as well-organized as using 
> check, all the tests use a function "int test(char *URL)" which may not be 
> suitable for unit tests.
>
> personally, i'm into using check. I'd love to hear what you guys think?

If you think the macros and verification methods check offers are good and 
nice to use, then I think we can simply implement them ourselves. 
fail_unless() and more.

The function named test() is just what we have so far with our test suite, 
there's no reason we couldn't change this if we want and need to. Personally I 
don't see why that is a problem. We can just have one C file for each family 
of tests that we do, and we can use fail_unless() style macros within them and 
we can have failures output look like other failures in our test suite.

Using my approach, the same llist unittest series could be written like the 
attached file - consider it an example, we can of course tweak the details as 
we think and move ahead. Isn't that good enough? It then also works with our 
current infrastructure perfectly, with things such as valgrind, gdb and our 
own memory leak tool and more.

--

-- 

  / daniel.haxx.se

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Re: curl unit testing

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Re: curl unit testing

On Sat, 1 Jan 2011, Daniel Stenberg wrote:

> Using my approach, the same llist unittest series could be written like the 
> attached file - consider it an example, we can of course tweak the details 
> as we think and move ahead. Isn't that good enough? It then also works with 
> our current infrastructure perfectly, with things such as valgrind, gdb and 
> our own memory leak tool and more.

To put the code where my mouth is, the attached patch is a working version of 
the same llist set of tests adjusted to my way of thinking.

Isn't this "good enough" as a first shot and then we can polish this from 
here?

--

-- 

  / daniel.haxx.se

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Re: curl unit testing

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Re: curl unit testing

On Sat, Jan 01, 2011 at 05:37:28PM +0100, Daniel Stenberg wrote:
> To put the code where my mouth is, the attached patch is a working
> version of the same llist set of tests adjusted to my way of
> thinking.

Looks fine to me. You'll need to generalize the kind of mechanism already
used in unit tests 558 and 577 to skip the tests when --enable-hidden-symbols
has been used.  Maybe by making on a "feature" as in
<features>unittest</features>, or a <precheck> program.

>>> Dan
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Re: curl unit testing

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Re: curl unit testing

On Sat, 1 Jan 2011, Vincent Torri wrote:

> 500 Kb: you mean the tarball ? Because if you look at the source code
> itself, it's 120 Kb (the other files are mostly autotools one, around 460
> Kb) and if you look at the library, the static one is about 38Kb. Not a lot.
> Also, these days, package managers can install it easily.

Sure, for those systems that have package managers for which Checks is 
packaged for and Check works on. I somehow suspect that curl runs on far more 
platforms than Check does.

> What's the problem with using an external tool for unit testing ? If you use 
> your own unit testing framework, you duplicate code (hence more work, 
> possible bugs in your unit testing code).

Why I want the test tools "bundled":

A - We avoid the risk of version skew and that some Check versions have bugs
     and others don't, or that they change format/API at some point etc.

B - We support lots of platforms and targets that don't easily install Check
     with a package manager. Perhaps especially important for the autobuilds
     concept to work as smoothly and easily as possible.

Why I think our own code is a good idea instead of using Check:

1 - I don't know what Check aims for in terms of portability, but I'm afraid
     that our bar is (much) higher. We risk limiting the unit tests to fewer
     architectures.

     And yes this matters. I want everyone to be able to not only use (lib)curl
     but also to be able to develop curl and you cannot do that easily and
     safely if you can't run the tests on your platform.

     Portability is one of the corner-stones curl stands on. I work hard on
     not restricting it any further than what we already do and I much rather
     focus on *increasing* portability.

     I also have no desire to have to work on yet another supporting library or
     tool in the case we WOULD face problems in this or other areas.

2 - There is not a lot of code for the actual unit testing. Most of what Check
     is and does is things we ALREADY support and do in curl (most likely in
     a different way).

3 - We have an established architecture for tests. We would have to adapt
     Check and its tests quite a bit for it to offer the same features and the
     same integration in the way our own can. I think of valgrind support,
     easy repeat of a test case to run gdb on a failure and doing memory leak
     and error detections with our own memory debugging system.

     I think I am one of the most frequent users of these features and I would
     not like to see these things sacrifized.

     (As a somewhat telling sign of how useful this is: my rewrite of the
     unittest for the patch I posted yesterday immediately showed a flaw and
     memory leak in the test - that is now corrected in my version of it.)

4 - We do not sacrifize anything in terms of how hard it is or not to write
     the unit tests themselves as we can basically use the exact same set of
     macros. And the actual unit-tests should be the really important part of
     this exercise I believe.

Calling this "rewriting stuff that others have already written" shows a huge 
lack of understanding of what we already have and what work it takes to have 
Check incorporated into this. I'm not just a "NIH" guy.

> Btw, as we are speaking of unit testing, what we are doing with our unit 
> testing is code coverage: use of gcov and lcov to have html output of the 
> coverage of the unit tests. If you are interested, i have written m4 macro 
> to integrate sich process quite easily in configure.ac (Makefile.am stuff is 
> also very easy).

That'd be great! But of course it could just as well be done for all testing 
and not "just" the unit testing parts.

I would like to get a code coverage for a test "round" on a regular basis so 
that we can forcus on expanding the test suit more on the areas where we have 
poor test coverage right now.

--

-- 

  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Re: curl unit testing

On Sun, 2 Jan 2011, Amr Shahin wrote:

>> check doesn't seem to be written to grok C89 compilers, have you checked
>> (no pun intended) if it does?
>
>   i contacted check-devel. will let you know when i get a reply. I wonder 
> however if this is important, the unit tests will be used by contributors 
> not client-developers, i guess all of contributors will be using a modern OS 
> and compiler.

First, not all contributors will work with C99 compilers.

Then, even non-contributors should as far as possible have the chance to run 
the tests on their builds for their platform.

>   it's not only about macros. to have a full unit testing framework we'll
> need to implement test suit handlers, test case handlers, output formatter
> etc ...

I don't understand. What are those and why do we need them?

> this sounds pretty much like re-writing check.

See my previous response. What exactly are you missing in my first take?

> And quoting from Vincent's reply below, the quality won't be guaranteed

What? Are you suggesting that we can't write test code that is as good as the 
Check team does?

> (check is being used and it has a dedicated team to support).

... but you seem to forget that Check is not integrated into our build and 
test suite and you have not even tried to do that yet so I've saved you a lot 
of work this way...

> The effor made to expand our current test can be used to actually write the 
> tests, few KBs sound like a small price :)

Didn't I already do most of that work (and it was very easy and quickly done) 
and you can now start writing the tests?

--

-- 

  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html