headers
Edward S. Peschko | 1 Mar 2004 01:30
Favicon

automatic configuration script ps

(ps, 

this is what happens when I manually take one of the proxies out of the 
function and use it directly.

It returns a function revealing the Location that the browser is supposed
to use, with a number associated. This number is a time-sensitive  
session which the browser then uses.

I can write a perl script around curl that captures this location 
and redirects it to another curl command, but I'm stuck as to how
to send a user and password to this redirected statement so it consistently 
works. I can enter "http://user:password <at> ...", but this only causes the
authentication to go away, and it doesn't actually send the page.

And if I enter authentication at the wrong *time*, it doesn't accept
the authentication, and fails. Since this is going to be used in a production
setting, this restriction is annoying to say the least. The best thing
would be for curl to support it natively.

So - are there any plans on supporting this tangled mess natively? FWIW,
we run websense, so if this was fixed, it would effect far more people than
just me - websense is a fairly common site to be run by large corporations.)

Ed

trace follows - I've blackened out the things that are proprietary
to pg&e.. ie, I don't want to get in trouble.. ;-)
)
(Continue reading)

Permalink | Reply | 0 comments |
headers
Daniel Stenberg | 1 Mar 2004 07:49
Picon
Favicon

Re: 'automatic configuration script' as proxy

On Sun, 29 Feb 2004, Edward S. Peschko wrote:

> I was wondering if curl supported an 'automatic configuration script' as do
> mozilla, IE, and opera...

Nope. Try FAQ entry 3.14: http://curl.haxx.se/docs/faq.html

> In other words, we - at my site - have to go through a bunch of hoops to get
> a proxy server name. I need to be able to automate this.

Then I sugges that you get a javascript parser and make sure you intrepret
that response page using it.

The last time I had a look, the (only free one I know) javascript parser code
was a bigger code chunk than curl itself.

> ps - if I can get this working, could I somehow get cvs read access using
> curl in this way? I'm thinking through the cvsweb method.. )

Unfortunately, no. CVS does not work over proxies.

I hope to have the curl CVS repository converted over to Subversion one day in
the future, and then you'd be able to access the source repository.

--
    Daniel Stenberg -- http://curl.haxx.se/ -- http://daniel.haxx.se/
   [[ Do not send mails to this email address. They won't reach me. ]]
Permalink | Reply | 7 comments |
headers
FUN TOES | 1 Mar 2004 08:20

question on "--request":can I remove the " /" after the request?

curl -v0 --request "GET http://www.google.com/search?hl=en" targetproxy.
I found the http request sent  like this:
> GET http://www.google.com/search?hl=en / HTTP/1.0
Can I remove the " /" before " HTTP/1.0"? 
(some httpproxy will give error for it).

I step into url.c with gdb,but I can't find where the " /"
was append to the urlbuffer.After a do_curl() call,this 
request is sent.I can't find what is the  do_curl() used.
Would you like give me some clues?

Because of the targetproxy I used is usually the second 
proxy.(culr -p -U user:pass -x httpproxy1:port1 -X "GET http://target/search?..." -0 proxy2:port2).I
have no other
choice.

Best regards,
funtest
                                           2004 3 2
--

-- 
_____________________________________________________________
Web-based SMS services available at http://www.operamail.com.
>From your mailbox to local or overseas cell phones.

Powered by Outblaze
Permalink | Reply | 1 comment |
headers
Daniel Stenberg | 1 Mar 2004 10:28
Picon
Favicon

Re: question on "--request":can I remove the " /" after the request?

On Mon, 1 Mar 2004, FUN TOES wrote:

> curl -v0 --request "GET http://www.google.com/search?hl=en" targetproxy.
> I found the http request sent  like this:
> > GET http://www.google.com/search?hl=en / HTTP/1.0
> Can I remove the " /" before " HTTP/1.0"?

No you can't. And no, you're not supposed to use that option like that since
it creates weird HTTP requests, exactly as you found out.

Removing the path from the request just because you abuse the 'request' option
is not an option for libcurl.

> (some httpproxy will give error for it).

So why do you use it like this?

> I step into url.c with gdb,but I can't find where the " /" was append to the
> urlbuffer.After a do_curl() call,this request is sent.I can't find what is
> the do_curl() used. Would you like give me some clues?

The code that creates the whole HTTP request is in Curl_http(). I can't give
you any exact line numbers since my devel version is much changed since the
latest official release, but if you checkout from CVS it is line 1345 in
lib/http.c.

> Because of the targetproxy I used is usually the second proxy.(culr -p -U
> user:pass -x httpproxy1:port1 -X "GET http://target/search?..." -0
> proxy2:port2).I have no other choice.
(Continue reading)

Permalink | Reply | 0 comments |
headers
Lenny Sorey | 1 Mar 2004 13:12

Re: Curl In JSP

Daniel,

Thanks for your reply. 

Sorry for my late response.

You were right, it was a path problem.

Once I referenced the path correctly, everything started to 
work properly.

Regards,

Lenny Sorey

On Thu, 26 Feb 2004 12:27:50 +0100 (CET), Daniel Stenberg wrote:

>On Wed, 25 Feb 2004, Lenny Sorey wrote:
>
>> I can run Curl with success from a command line with the parameters show
>> below and get the desired results.
>>
>> When I try to use the same command line within my jsp as shown below, I get
>> a blank page.
>
>I think it sounds as if your path is wrong when you run the command from
>within your program. When you use -v, the command line almost always outputs
>at least something.
>
>--
(Continue reading)

Permalink | Reply | 0 comments |
headers
FUN TOES | 1 Mar 2004 17:07

Re: question on "--request":can I remove the " /" after the request?

> Date: Mon, 1 Mar 2004 10:28:04 +0100 (CET)
> From: Daniel Stenberg <daniel-curl <at> haxx.se>
> Subject: Re: question on "--request":can I remove the " /" after the
> 	request?
> To: curl tool talk <curl-users <at> cool.haxx.se>
> Message-ID: <Pine.LNX.4.58.0403011022360.11959 <at> linux3.contactor.se>
> Content-Type: TEXT/PLAIN; charset=US-ASCII
> 
> On Mon, 1 Mar 2004, FUN TOES wrote:
> 
> > curl -v0 --request "GET http://www.google.com/search?hl=en" targetproxy.
> > I found the http request sent  like this:
> > > GET http://www.google.com/search?hl=en / HTTP/1.0
> > Can I remove the " /" before " HTTP/1.0"?
> 
> No you can't. And no, you're not supposed to use that option like that since
> it creates weird HTTP requests, exactly as you found out.
> 
> Removing the path from the request just because you abuse the 'request' option
Yeah,but I have no idear how to change the http_request to 
fit for the second proxy.  
> is not an option for libcurl.
> 
> > (some httpproxy will give error for it).
> 
> So why do you use it like this?
I want to visit some site through two http proxy.
> 
> > I step into url.c with gdb,but I can't find where the " /" was append to the
> > urlbuffer.After a do_curl() call,this request is sent.I can't find what is
(Continue reading)

Permalink | Reply | 1 comment |
headers
Daniel Stenberg | 1 Mar 2004 17:11
Picon
Favicon

Re: question on "--request":can I remove the " /" after the request?

On Mon, 1 Mar 2004, FUN TOES wrote:

> Yeah!That is it.The complete comand line is like this:

> curl -vp0 -x myproxy:8080 -U user:pass 61.132.137.170:3128 -X "GET
> http://www.google.com/search?hl=en"

> I want to visit http://www.google.com/search?hl=en via
> two proxy (myproxy,61.132.137.170).

> Because curl's "-x" option only work once,I can only find this method to
> through the second proxy. And it failed to some extra request including path
> as you said. Would you like to give me some more idears?

Thanks for the additional info. I think I understand your request a whole lot
better now.

Hm.

I don't think you can trick curl into doing this without changing the source
code.

I wouldn't mind seeing curl support this (admittedly weird) setup. How would
you prefer this to work from a user's point of view?

For a quick patch to get it working, you can prevent curl to include the path
the HTTP request, like you wanted. But that's not a viable option for the main
sources! ;-)

--

--
(Continue reading)

Permalink | Reply | 0 comments |
headers
Edward S. Peschko | 1 Mar 2004 19:57
Favicon

Re: 'automatic configuration script' as proxy

On Mon, Mar 01, 2004 at 07:49:25AM +0100, Daniel Stenberg wrote:
> On Sun, 29 Feb 2004, Edward S. Peschko wrote:
> 
> > I was wondering if curl supported an 'automatic configuration script' as do
> > mozilla, IE, and opera...
> 
> Nope. Try FAQ entry 3.14: http://curl.haxx.se/docs/faq.html

ok, so there's an entry in the FAQ which seems to set in stone the 
lack of a piece of functionality (which would be very useful btw) and
then goes on to explain that there is a freeware javascript engine available,
and that people have used it successfully in the past..

If people have used it successfully in the past, are there instructions on how to 
use it successfully posted somewhere, and if so, could they be put there?

> The last time I had a look, the (only free one I know) javascript parser code
> was a bigger code chunk than curl itself.

That's fine, I would make it a dependency then, like perl's DBD::DB2 depends on 
having db-4 installed. And then make the dependency a voluntary one (ie: if the 
javascript interpreter is there support a given command line flag, if not, don't).

Don't get me wrong, I think curl is very useful, and I do use it for ftp. 
I just think that me and a hell of a lot of other people *can't* use it 
effectively if this is not supported. 

And since all the main browsers support javascript, more and more websites
(not just company access methods which a lot of poeple use in themselves) will
become inaccessible by curl and therefore curl will diminish in value.
(Continue reading)

Permalink | Reply | 6 comments |
headers
Daniel Stenberg | 1 Mar 2004 21:36
Picon
Favicon

Re: 'automatic configuration script' as proxy

On Mon, 1 Mar 2004, Edward S. Peschko wrote:

> > Nope. Try FAQ entry 3.14: http://curl.haxx.se/docs/faq.html
>
> ok, so there's an entry in the FAQ which seems to set in stone the lack of a
> piece of functionality (which would be very useful btw) and then goes on to
> explain that there is a freeware javascript engine available, and that
> people have used it successfully in the past..

Nothing is ever that set in stone that it can't ever be changed. But not
without a lot of persuation, shown interest and most likely proven code that
works.

At least that's my view.

> If people have used it successfully in the past, are there instructions on
> how to use it successfully posted somewhere, and if so, could they be put
> there?

There was a mailing list post once in the past. I guess google might be able
to help you find it. I'm not sure it has any relevance anymore though since it
was a few years back, IIRC.

The javascript engine was the one in Mozilla. Try their site.

> Don't get me wrong, I think curl is very useful, and I do use it for ftp.
> I just think that me and a hell of a lot of other people *can't* use it
> effectively if this is not supported.

Perhaps.
(Continue reading)

Permalink | Reply | 1 comment |
headers
Edward S. Peschko | 1 Mar 2004 22:29
Favicon

Re: 'automatic configuration script' as proxy

> > If people have used it successfully in the past, are there instructions on
> > how to use it successfully posted somewhere, and if so, could they be put
> > there?
> 
> There was a mailing list post once in the past. I guess google might be able
> to help you find it. I'm not sure it has any relevance anymore though since it
> was a few years back, IIRC.
> 
> The javascript engine was the one in Mozilla. Try their site.
> 
> > Don't get me wrong, I think curl is very useful, and I do use it for ftp.
> > I just think that me and a hell of a lot of other people *can't* use it
> > effectively if this is not supported.
> 
> Perhaps.
> 
> > And since all the main browsers support javascript, more and more websites
> > (not just company access methods which a lot of poeple use in themselves)
> > will become inaccessible by curl and therefore curl will diminish in value.
> 
> I don't see the need for it. I don't feel like coding it. I don't agree with
> you about the importance of doing this. I think you excaggerate the frequency
> of this need.

well, to address that last point, consider two scenarios: 

	1) I'm at one site, with about 10000 internet users. We use websense for the 
       purpose of filtering web traffic.  Places that I've worked for in the
	   past have had the same thing. Its going to get more common in the future.
(Continue reading)

Permalink | Reply | 0 comments |

Gmane