Legolas | 1 Dec 02:22 2005
Picon

Re: Info request about the zero copy interface

Daniel Stenberg ha scritto:

> On Tue, 29 Nov 2005, Legolas wrote:
>
>> *Do you think using this object would be useful?* Or else do you want 
>> me to extract the memory streams interface code from the main project?
>
>
> Sorry, but I don't quite understand how all that comes into play when 
> we'd add a zero copy interface.
>
> To make downloads zero copy, libcurl must have a place and size to 
> store data. How does it get that? And how does it ask for subsequent 
> buffers?
>
> To make uploads zero copy, libcurl must get a data buffer with a size 
> (without having to copy it). How does it get that? And how does it ask 
> for subsequent buffers?
>
> Please be more specific on how your ideas would get turned into a 
> libcurl API.
>
OK, I will explain it in a few lines:

My idea is to give libcurl an RWops object (which is an input or output 
stream), and then libcurl will call its assigned methods to make any 
read/write operation. I know this is a radical change, but it may speed 
up a lot every operation.

(Continue reading)

Bryan Henderson | 1 Dec 03:12 2005

Re: configure script adds user's LDFLAGS to curl-config

>> As a result of this, my LDFLAGS are added in curl-config, so that 
>> `curl-config --libs` returns flags which I used to build libcurl. Isn't that 
>> wrong?
>
>Perhaps, but the configures script cannot (currently) tell what of the given 
>options that are required for libcurl to get built and what options that are 
>needed for apps that use libcurl to get built.

>Any suggestions on how to fix?

For starters, I would make LDFLAGS not affect curl-config --libs .  I
see LDFLAGS as a request to sidestep the normal configure intelligence
and throw extra options onto a link command in a libcurl build.
Configure shouldn't do anything else with it.

To really fix it, we need configure options that explicitly supply the
required information.  I suggest --extralibs, meaning "Linker options
necessary to include libraries in the link that resolve symbols to
which libcurl refers, and not covered by any more specific option."

Or wait until we have an actual example of a needed extra library and
make an option for that.

--

-- 
Bryan Henderson                                    Phone 408-621-2000
San Jose, California

Daniel Stenberg | 1 Dec 08:35 2005
Picon

Re: Info request about the zero copy interface

On Thu, 1 Dec 2005, Legolas wrote:

> My idea is to give libcurl an RWops object (which is an input or output 
> stream), and then libcurl will call its assigned methods to make any 
> read/write operation. I know this is a radical change, but it may speed up a 
> lot every operation.

"every" ? What other operations than read or write do you think it needs?

--

-- 
  Commercial curl and libcurl Technical Support: http://haxx.se/curl.html

Legolas | 1 Dec 10:19 2005
Picon

Re: Info request about the zero copy interface

Daniel Stenberg ha scritto:

> On Thu, 1 Dec 2005, Legolas wrote:
>
>> My idea is to give libcurl an RWops object (which is an input or 
>> output stream), and then libcurl will call its assigned methods to 
>> make any read/write operation. I know this is a radical change, but 
>> it may speed up a lot every operation.
>
>
> "every" ? What other operations than read or write do you think it needs?
>
I don't understand your question, however passing libcurl an object with 
assigned read/write methods (and others if needed) will avoid the file 
writing or reading memory copy bottleneck (data would not be cached into 
memory and passed to libcurl, but directly read/written from libcurl 
itself through the read/write method).

1) Example of possible libcurl source snippet (WRITE operation, maybe a 
download):
------------------------------------------
RWops   *rw;
void         *socket_buffer_recv;
int            size, blocks;
...
    /* ZERO COPY: RWops object will write to file, memory, or whatever 
it really is behind the RWops */
    if (rw->write(rw, socket_buffer_recv, size, blocks) != blocks) {
          /* eventual cleanup code */
          return CURLE_WRITE_ERROR;
(Continue reading)

Naresh_Sharma | 1 Dec 12:45 2005

Java FTP sample program using libcurl


Hi,

Can anyone point me to the a sample program written in java for FTP upload and download using libcurl.


Regards
Naresh
Daniel Stenberg | 1 Dec 16:48 2005
Picon

Re: Info request about the zero copy interface

On Thu, 1 Dec 2005, Legolas wrote:

>> "every" ? What other operations than read or write do you think it needs?
>> 
> I don't understand your question, however passing libcurl an object with 
> assigned read/write methods (and others if needed) will avoid the file 
> writing or reading memory copy bottleneck (data would not be cached into 
> memory and passed to libcurl, but directly read/written from libcurl itself 
> through the read/write method).

No, that is _not_ what a zero copy interface does as it does not avoid the 
extra copy. We already have that kind of callbacks, and they are call read and 
write callbacks.

To me, it looks like you are too focused on squeezing your own ideas into 
libcurl than to actually provide a usable zero copy interface.

> 1) Example of possible libcurl source snippet (WRITE operation, maybe a 
> download):

You say we call the write callback. In what way does this write callback 
differ from the write callback libcurl already features?

Your write callback seems to take a pointer to a buffer as one of its 
arguments. What buffer? Wouldn't libcurl need to store data in that when it 
receives data from a peer? How would the write callback get and use the data 
without copying it?

> I think this is really the best way to create an actual zero copy interface, 
> in case (1, WRITE) the eventual subsequent buffers needed would be handled 
> automatically by the RWops object.

How?

--

-- 
  Commercial curl and libcurl Technical Support: http://haxx.se/curl.html

Daniel Stenberg | 1 Dec 18:15 2005
Picon

Re: living without global variables

On Tue, 29 Nov 2005, Bryan Henderson wrote:

>> Also, I'm not fond of the specific functions added for the single specific 
>> items that (currently) have a global state or a field in the client object: 
>> openssl, gnutls, sock, charset. I prefer a more extensible approach that 
>> allows us to add a new subsystem tomorrow and remote one of the existing 
>> the day after that, without breaking the API/ABI backwards compatibility.
>
> I presume you're thinking of a single function with named flags to select 
> what it does.  Note that this enables only binary compatibility.

It enables binary compatibility and backwards source compatibility. So that 
when we add new options they won't break old applications, and removing 
options won't either as long as the defines remain.

> At compile time, the compatibility characteristics look the same to me 
> whether you add a new function name or add a new flag name.

Yes, but adding and removing functions to the API/ABI is a lot more expensive 
operation than adding and removing flags. In code and in docs.

> But binary compatibility is good, and so is reducing symbol table clutter. I 
> will try this:
>
>  CURL_EXTERN void curl_client_use_global(CURLC *curlc, long flags);
>
>  #define CURL_GLOBAL_OPENSSL ...
>  #define CURL_GLOBAL_WINSOCK ...
>
> Ultimately, it will have to get messier because the user will be passing a 
> handle for a private facility instead of just saying, "use the global 
> facility."

I don't understand. Why is

   curl_client_use_global(curlc, CURL_GLOBAL_OPENSSL);

"messier" than

   curl_client_use_global_openssl(curlc);

? They both pass the exact same amount of information, just in different ways.

> That handle may not be representable as a generic type (e.g. void *), and 
> even if it is, we may want to use a real type.

True, but us taking a flag or a specific function doesn't change that.

>> Is there really a need for gnutls global data?
>
> Well, I assume the GNU TLS designers wouldn't have provided such a heinous 
> thing as gnutls_global_init(), and curl_global_init() wouldn't call it, if 
> it weren't necessary.

Indeed. Sorry, I forgot there was such a one. It is kind of funny that they 
couldn't avoid globals either.

>> Further, is there a need for the curl_client_global_xxx_ok() public 
>> functions?
>
> I didn't intend for these to be public.  They should go in clientif.h
> instead of client.h, right?

Yes, and they shoule have a Curl_ prefix (instead of curl_) and not use 
CURL_EXTERN.

> But that reminds me of something I forgot to deal with: If the user is going 
> to be responsible for initializing global stuff, he should have a way to 
> determine what global stuff libcurl would like to use.

Now that can become a fun excersize.

> In particular, if a user wants to do SSL connections, the libcurl he links 
> to might be built for OpenSSL or might be built for GNU TLS, and the user 
> needs to know at compile time which one it is so he can #include the right 
> SSL library interface and call the right initialization functions.

Unfortunately that goes pretty much in the opposite direction of my efforts to 
enable libcurl users to be completely unknowing about what particular SSL 
library libcurl uses.

But I don't see how it can be avoided if the app really wants to do the global 
inits by itself.

> This seems to call for an option on curl-config to tell you what facilities 
> it uses.

Yes, but curl-config does not work on all platforms libcurl works on... We 
could perhaps have a function that can return a bitmask with the global flags 
that this particular libcurl uses.

--

-- 
  Commercial curl and libcurl Technical Support: http://haxx.se/curl.html

Daniel Stenberg | 1 Dec 18:19 2005
Picon

Re: configure script adds user's LDFLAGS to curl-config

On Thu, 1 Dec 2005, Bryan Henderson wrote:

> For starters, I would make LDFLAGS not affect curl-config --libs .  I see 
> LDFLAGS as a request to sidestep the normal configure intelligence and throw 
> extra options onto a link command in a libcurl build. Configure shouldn't do 
> anything else with it.

That would be a hard blow to many people. Very often even I set LDFLAGS and 
family before configure is invoked to point out a proper path on where to find 
related libs such as OpenSSL, zlib etc. Thus, these flags must also be used 
when apps that build with libcurl.

> Or wait until we have an actual example of a needed extra library and make 
> an option for that.

Since this isn't scratching any itch of my own and this has proved to be 
working at least pretty good so far, I'll leave this to be fixed by someone 
who actually suffers from this and feels strongly enough to implement 
something that cures the problem.

--

-- 
  Commercial curl and libcurl Technical Support: http://haxx.se/curl.html

Legolas | 1 Dec 18:16 2005
Picon

Re: Info request about the zero copy interface

Daniel Stenberg ha scritto:

> On Thu, 1 Dec 2005, Legolas wrote:
>
>>> "every" ? What other operations than read or write do you think it 
>>> needs?
>>>
>> I don't understand your question, however passing libcurl an object 
>> with assigned read/write methods (and others if needed) will avoid 
>> the file writing or reading memory copy bottleneck (data would not be 
>> cached into memory and passed to libcurl, but directly read/written 
>> from libcurl itself through the read/write method).
>
>
> No, that is _not_ what a zero copy interface does as it does not avoid 
> the extra copy. We already have that kind of callbacks, and they are 
> call read and write callbacks.

I am saying to overwrite those callbacks, i.e. _not_ to use them but 
using instead a custom made object (the one I was talking before) which 
creates the correspondant callbacks for files or memory.

>
> To me, it looks like you are too focused on squeezing your own ideas 
> into libcurl than to actually provide a usable zero copy interface.

To you. And I am afraid you have misunderstood. I am wondering why are 
you taking a ridiculizing tone with me. If you have no time or if you 
think answering me is not worth, simply don't. You should have asked me 
only to better explain but looks like you do don't care about that.
However my aim was just to give suggestions and to receive, I don't 
think I have been arrogant in my replies.

>
>> 1) Example of possible libcurl source snippet (WRITE operation, maybe 
>> a download):
>
>
> You say we call the write callback. In what way does this write 
> callback differ from the write callback libcurl already features?

It differs because a write callback of a file-type object would contain 
directly 'fwrite' calls. Isn't that clear? A file-type object has 
DIFFERENT read and write callbacks from those owned by a memory-type 
object. The object I was talking about in the previous replies SETS them.

>
> Your write callback seems to take a pointer to a buffer as one of its 
> arguments. What buffer? Wouldn't libcurl need to store data in that 
> when it receives data from a peer? How would the write callback get 
> and use the data without copying it?

libcurl wouldn't care how the write callback uses the data present into 
'socket_buffer_recv', which is in my scheme the *real* socket buffer 
used for input by any recv-like call done by libcurl and it is passed to 
the callback.
The write callback may dump the content to a file or copy it into a 
bigger memory stream (enlarging it if necessary). This may not be safe 
for caching-purposes, I know...however my idea was to integrate the 
RWops object into libcurl to prevent this.
Perhaps the internal working of the object is not clear, give a look to 
the following snippet:
- - - - - - - - - - - - -
typedef struct RWops {
    ...
    void      *buffer;
    int        buffer_size;
    int        (*read)(struct RWops *context, void *ptr, int size, int 
maxnum);
    int        (*write)(struct RWops *context, void *ptr, int size, int 
maxnum);
    ...
    int        (*_assert_buffer_size)(struct RWops *context, int 
desired_buffer_size);
    ...
} RWops;
- - - - - - - - - - - -

>
>> I think this is really the best way to create an actual zero copy 
>> interface, in case (1, WRITE) the eventual subsequent buffers needed 
>> would be handled automatically by the RWops object.
>
>
> How?
>
The write callback reallocates the memory when the next block writing 
operation is going to overflow. So it works a dynamic stream.

Steven Harrison | 1 Dec 19:19 2005
Picon
Picon

Re: libcURL, Windows, and SSL

Daniel Stenberg wrote:

> On Wed, 30 Nov 2005, Steven Harrison wrote:
>
>> I'm using the last version that came with SSL support (7.13.1 
>> according the the dll's file version)
>
>
> You mean the latest version precompiled for your platform that works 
> for you? Yes, that might be the case. Compiling a later version 
> yourself should be easy and quick enough though.
>
I've not attempted to roll my own libcURL as I'm not sure what to do to 
get it to use SSL, even the docs are a little hard for me to understand 
sometimes.

>> and I've got my application working ok, but every so often it refuses 
>> to download anything from the server.
>
>
> What happens then in more exact terms?

I work in a help desk, and I'm sorry but I use that phrase a lot when 
I've not heard or read the problem exactly, but:

In more exact terms the application I've wrote does not download 
anything from the server, and reports no errors.  The log file I'm using 
also does not report anything (even the outputdebug string log is 
showing no XML, yet cURL is returning with an "A OK" response.

>
>> (I downloaded the libcurl-7.15.0-win32-ssl-sspi.zip file after 
>> looking though the archives of this mailing list, but that is 
>> returning errors about unallocated memory).
>
>
> That's news to me. Anyone else seeing this problem with this package? 
> I don't use them myself, I only built and provide them.
>
Next time I'm attemp to use those libs I'll post the exact error, or and 
I'm also getting a compile error about the .dll.a file having too many 
.text sections.

>> So am I stuck with using the old 7.13.1 and not having access to the 
>> cookies until I've run the cleanup function, can I upgrade to 7.15.x 
>> and get SSL to work (even if I have to get the openSSL libs), or is 
>> there another way to access the information in 7.13.1?
>
>
> If you really want to extract the cookie information from libcurl, 
> then upgrading seems to be the only option.
>
Not what I wanted to know, but I'd have to do the upgrade at some point 
I suppose since some of the features in the later versions are starting 
to be needed in the near future in the project I'm working on.

Well I'll hunt though the archives for information on how to compile 
libcURL in MSVC 2003.net (C++) with SSL support, failing that, I'll just 
add some other log commands and see if I can work out what's causing my 
problems with downloads from an https server..


Gmane