Amr Shahin | 1 Jan 01:50 2011
Picon

curl unit testing

Hello folks,

i'm attaching a sample very small unit tests using both check unit testing library and the current curl testing framework integrated with curl, i was hoping we could asses both and see which is more suitable to write unit tests.

the code uses check is in the file llist_test.c. It's a classic unit testing code, with all basic functions there setup, teardown and macros for defining your tests, the output is well organized, you can see sample output for both all success, and one failed test in the attached files.
The only downside for using check is that we're adding an extra dependency to curl, my suggesstion is to add the .so version and check.h to the code base.

the other option is to use the testing setup already in libtest directory, it's pretty easy to add more tests, but it's not as well-organized as using check, all the tests use a function "int test(char *URL)" which may not be suitable for unit tests.

personally, i'm into using check. I'd love to hear what you guys think?

Amr
Running suite(s): Curl_llust
0%: Checks: 1, Failures: 1, Errors: 0
llist_test.c:28:F:Core:test_llist_initiate:0: list initial size should be zero
Attachment (lib1000.c): text/x-csrc, 1394 bytes
Attachment (llist_test.c): text/x-csrc, 1714 bytes
Running suite(s): Curl_llust
100%: Checks: 1, Failures: 0, Errors: 0
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Vincent Torri | 1 Jan 02:43 2011
Picon

Re: curl unit testing



On Sat, Jan 1, 2011 at 1:50 AM, Amr Shahin <amrnablus <at> gmail.com> wrote:
Hello folks,

i'm attaching a sample very small unit tests using both check unit testing library and the current curl testing framework integrated with curl, i was hoping we could asses both and see which is more suitable to write unit tests.

the code uses check is in the file llist_test.c. It's a classic unit testing code, with all basic functions there setup, teardown and macros for defining your tests, the output is well organized, you can see sample output for both all success, and one failed test in the attached files.
The only downside for using check is that we're adding an extra dependency to curl, my suggesstion is to add the .so version and check.h to the code base.

the other option is to use the testing setup already in libtest directory, it's pretty easy to add more tests, but it's not as well-organized as using check, all the tests use a function "int test(char *URL)" which may not be suitable for unit tests.

personally, i'm into using check. I'd love to hear what you guys think?

I like the Check library: it's C, it's maintained and it works with Unix and Windows.

happy new year :)

Vincent Torri
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Daniel Stenberg | 1 Jan 15:39 2011
Picon

Re: SSL bug buffer is too small !!!!!

On Sat, 1 Jan 2011, 정중 이 wrote:

(I'm Cc'ing the curl-library mailing list for info.)

> static void pubkey_show(struct SessionHandle *data,
>                         int num,
>                         const char *type,
>                         const char *name,
>                         unsigned char *raw,
>                         int len)
>
> {
> char buffer[1024]; < -- too small 
>
> if len value(raw size) is over 340 , occur segment violation ...
>
> thereforebuffer is 2048. !!!

Size 2048 is not the correct fix though, as I'm sure something will hit that 
limit as well. We need to make sure that it doesn't overflow the buffer no 
matter what the size of the buffer is.

I wrote a fix and pushed to the git repo just now, it would be great if you 
could try it out and see how it works!

The exact commit was:

https://github.com/bagder/curl/commit/ae291421984a266176df34f24d3a5e76d76ec7c8


-- 

  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Daniel Stenberg | 1 Jan 16:13 2011
Picon

Re: curl unit testing

On Sat, 1 Jan 2011, Amr Shahin wrote:

> Hello folks,
>
> i'm attaching a sample very small unit tests using both check unit testing 
> library and the current curl testing framework integrated with curl, i was 
> hoping we could asses both and see which is more suitable to write unit 
> tests.

Hi and thanks a lot for your work on this!

> the code uses check is in the file llist_test.c. It's a classic unit testing 
> code, with all basic functions there setup, teardown and macros for defining 
> your tests, the output is well organized, you can see sample output for both 
> all success, and one failed test in the attached files. The only downside 
> for using check is that we're adding an extra dependency to curl, my 
> suggesstion is to add the .so version and check.h to the code base.

I have some objections and I will suggest another approach:

check doesn't seem to be written to grok C89 compilers, have you checked (no 
pun intended) if it does?

I would *REALLY* like to have the full test code bundled with curl and not 
depend on a separate download/package to get a decent test suite, and I find
>500K for a unittest for C quite much. Did anyone of you see how much we would 
need to copy to get a working check bundled?

> the other option is to use the testing setup already in libtest directory, 
> it's pretty easy to add more tests, but it's not as well-organized as using 
> check, all the tests use a function "int test(char *URL)" which may not be 
> suitable for unit tests.
>
> personally, i'm into using check. I'd love to hear what you guys think?

If you think the macros and verification methods check offers are good and 
nice to use, then I think we can simply implement them ourselves. 
fail_unless() and more.

The function named test() is just what we have so far with our test suite, 
there's no reason we couldn't change this if we want and need to. Personally I 
don't see why that is a problem. We can just have one C file for each family 
of tests that we do, and we can use fail_unless() style macros within them and 
we can have failures output look like other failures in our test suite.

Using my approach, the same llist unittest series could be written like the 
attached file - consider it an example, we can of course tweak the details as 
we think and move ahead. Isn't that good enough? It then also works with our 
current infrastructure perfectly, with things such as valgrind, gdb and our 
own memory leak tool and more.

--

-- 

  / daniel.haxx.se
Attachment (llist_test.c): text/x-csrc, 899 bytes
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Vincent Torri | 1 Jan 17:28 2011
Picon

Re: curl unit testing

Hey,

On Sat, Jan 1, 2011 at 4:13 PM, Daniel Stenberg <daniel <at> haxx.se> wrote:

I would *REALLY* like to have the full test code bundled with curl and not depend on a separate download/package to get a decent test suite, and I find
500K for a unittest for C quite much. Did anyone of you see how much we would
need to copy to get a working check bundled?

500 Kb: you mean the tarball ? Because if you look at the source code itself, it's 120 Kb (the other files are mostly autotools one, around 460 Kb) and if you look at the library, the static one is about 38Kb. Not a lot. Also, these days, package managers can install it easily. What's the problem with using an external tool for unit testing ? If you use your own unit testing framework, you duplicate code (hence more work, possible bugs in your unit testing code).

Btw, as we are speaking of unit testing, what we are doing with our unit testing is code coverage: use of gcov and lcov to have html output of the coverage of the unit tests. If you are interested, i have written m4 macro to integrate sich process quite easily in configure.ac (Makefile.am stuff is also very easy).

regards

Vincent Torri
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Daniel Stenberg | 1 Jan 17:37 2011
Picon

Re: curl unit testing

On Sat, 1 Jan 2011, Daniel Stenberg wrote:

> Using my approach, the same llist unittest series could be written like the 
> attached file - consider it an example, we can of course tweak the details 
> as we think and move ahead. Isn't that good enough? It then also works with 
> our current infrastructure perfectly, with things such as valgrind, gdb and 
> our own memory leak tool and more.

To put the code where my mouth is, the attached patch is a working version of 
the same llist set of tests adjusted to my way of thinking.

Isn't this "good enough" as a first shot and then we can polish this from 
here?

--

-- 

  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Vincent Torri | 1 Jan 20:15 2011
Picon

Re: curl unit testing



On Sat, Jan 1, 2011 at 5:37 PM, Daniel Stenberg <daniel <at> haxx.se> wrote:
On Sat, 1 Jan 2011, Daniel Stenberg wrote:

Using my approach, the same llist unittest series could be written like the attached file - consider it an example, we can of course tweak the details as we think and move ahead. Isn't that good enough? It then also works with our current infrastructure perfectly, with things such as valgrind, gdb and our own memory leak tool and more.

To put the code where my mouth is, the attached patch is a working version of the same llist set of tests adjusted to my way of thinking.

Isn't this "good enough" as a first shot and then we can polish this from here?

if you prefer rewriting stuff that others have already written, then fine.

Vincent Torri
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Dan Fandrich | 1 Jan 20:30 2011

Re: curl unit testing

On Sat, Jan 01, 2011 at 05:37:28PM +0100, Daniel Stenberg wrote:
> To put the code where my mouth is, the attached patch is a working
> version of the same llist set of tests adjusted to my way of
> thinking.

Looks fine to me. You'll need to generalize the kind of mechanism already
used in unit tests 558 and 577 to skip the tests when --enable-hidden-symbols
has been used.  Maybe by making on a "feature" as in
<features>unittest</features>, or a <precheck> program.

>>> Dan
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Amr Shahin | 2 Jan 02:13 2011
Picon

Re: curl unit testing



On Sat, Jan 1, 2011 at 5:13 PM, Daniel Stenberg <daniel <at> haxx.se> wrote:
On Sat, 1 Jan 2011, Amr Shahin wrote:

Hello folks,

i'm attaching a sample very small unit tests using both check unit testing library and the current curl testing framework integrated with curl, i was hoping we could asses both and see which is more suitable to write unit tests.

Hi and thanks a lot for your work on this!


the code uses check is in the file llist_test.c. It's a classic unit testing code, with all basic functions there setup, teardown and macros for defining your tests, the output is well organized, you can see sample output for both all success, and one failed test in the attached files. The only downside for using check is that we're adding an extra dependency to curl, my suggesstion is to add the .so version and check.h to the code base.

I have some objections and I will suggest another approach:

check doesn't seem to be written to grok C89 compilers, have you checked (no pun intended) if it does?
 
   i contacted check-devel. will let you know when i get a reply. I wonder however if this is important, the unit tests will be used by contributors not client-developers, i guess all of contributors will be using a modern OS and compiler.

I would *REALLY* like to have the full test code bundled with curl and not depend on a separate download/package to get a decent test suite, and I find
500K for a unittest for C quite much. Did anyone of you see how much we would
need to copy to get a working check bundled?
 
    do we really need to attach the whole source of check? the size of the .so is 32 KBs and the size of check.h is 16 KB.

the other option is to use the testing setup already in libtest directory, it's pretty easy to add more tests, but it's not as well-organized as using check, all the tests use a function "int test(char *URL)" which may not be suitable for unit tests.

personally, i'm into using check. I'd love to hear what you guys think?

If you think the macros and verification methods check offers are good and nice to use, then I think we can simply implement them ourselves. fail_unless() and more.

   it's not only about macros. to have a full unit testing framework we'll need to implement test suit handlers, test case handlers, output formatter etc ...
   this sounds pretty much like re-writing check. And quoting from Vincent's reply below, the quality won't be guaranteed (check is being used and it has a dedicated team to support). The effor made to expand our current test can be used to actually write the tests, few KBs sound like a small price :)

The function named test() is just what we have so far with our test suite, there's no reason we couldn't change this if we want and need to. Personally I don't see why that is a problem. We can just have one C file for each family of tests that we do, and we can use fail_unless() style macros within them and we can have failures output look like other failures in our test suite.

Using my approach, the same llist unittest series could be written like the attached file - consider it an example, we can of course tweak the details as we think and move ahead. Isn't that good enough? It then also works with our current infrastructure perfectly, with things such as valgrind, gdb and our own memory leak tool and more.

 The code looks good and it's a reasonable alternative, my vote still goes for check, but i'm not married to this opinion, if we decide that using the current test code i'll go for it.

Amr

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Daniel Stenberg | 2 Jan 10:07 2011
Picon

Re: curl unit testing

On Sat, 1 Jan 2011, Vincent Torri wrote:

> 500 Kb: you mean the tarball ? Because if you look at the source code
> itself, it's 120 Kb (the other files are mostly autotools one, around 460
> Kb) and if you look at the library, the static one is about 38Kb. Not a lot.
> Also, these days, package managers can install it easily.

Sure, for those systems that have package managers for which Checks is 
packaged for and Check works on. I somehow suspect that curl runs on far more 
platforms than Check does.

> What's the problem with using an external tool for unit testing ? If you use 
> your own unit testing framework, you duplicate code (hence more work, 
> possible bugs in your unit testing code).

Why I want the test tools "bundled":

A - We avoid the risk of version skew and that some Check versions have bugs
     and others don't, or that they change format/API at some point etc.

B - We support lots of platforms and targets that don't easily install Check
     with a package manager. Perhaps especially important for the autobuilds
     concept to work as smoothly and easily as possible.

Why I think our own code is a good idea instead of using Check:

1 - I don't know what Check aims for in terms of portability, but I'm afraid
     that our bar is (much) higher. We risk limiting the unit tests to fewer
     architectures.

     And yes this matters. I want everyone to be able to not only use (lib)curl
     but also to be able to develop curl and you cannot do that easily and
     safely if you can't run the tests on your platform.

     Portability is one of the corner-stones curl stands on. I work hard on
     not restricting it any further than what we already do and I much rather
     focus on *increasing* portability.

     I also have no desire to have to work on yet another supporting library or
     tool in the case we WOULD face problems in this or other areas.

2 - There is not a lot of code for the actual unit testing. Most of what Check
     is and does is things we ALREADY support and do in curl (most likely in
     a different way).

3 - We have an established architecture for tests. We would have to adapt
     Check and its tests quite a bit for it to offer the same features and the
     same integration in the way our own can. I think of valgrind support,
     easy repeat of a test case to run gdb on a failure and doing memory leak
     and error detections with our own memory debugging system.

     I think I am one of the most frequent users of these features and I would
     not like to see these things sacrifized.

     (As a somewhat telling sign of how useful this is: my rewrite of the
     unittest for the patch I posted yesterday immediately showed a flaw and
     memory leak in the test - that is now corrected in my version of it.)

4 - We do not sacrifize anything in terms of how hard it is or not to write
     the unit tests themselves as we can basically use the exact same set of
     macros. And the actual unit-tests should be the really important part of
     this exercise I believe.

Calling this "rewriting stuff that others have already written" shows a huge 
lack of understanding of what we already have and what work it takes to have 
Check incorporated into this. I'm not just a "NIH" guy.

> Btw, as we are speaking of unit testing, what we are doing with our unit 
> testing is code coverage: use of gcov and lcov to have html output of the 
> coverage of the unit tests. If you are interested, i have written m4 macro 
> to integrate sich process quite easily in configure.ac (Makefile.am stuff is 
> also very easy).

That'd be great! But of course it could just as well be done for all testing 
and not "just" the unit testing parts.

I would like to get a code coverage for a test "round" on a regular basis so 
that we can forcus on expanding the test suit more on the areas where we have 
poor test coverage right now.

--

-- 

  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html


Gmane