users of WWW::Mechanize for Ruby

headers shig odani | 30 Apr 16:21

_______________________________________________
Mechanize-users mailing list
Mechanize-users <at> rubyforge.org
http://rubyforge.org/mailman/listinfo/mechanize-users
_______________________________________________
Mechanize-users mailing list
Mechanize-users <at> rubyforge.org
http://rubyforge.org/mailman/listinfo/mechanize-users
_______________________________________________
Mechanize-users mailing list
Mechanize-users <at> rubyforge.org
http://rubyforge.org/mailman/listinfo/mechanize-users
On Jan 27, 2012, at 12:41 PM, Robert Poor wrote:

> I'm 99% certain that this is what's being discussed in 
> 
> 	https://github.com/tenderlove/mechanize/issues/168
> 
> But I'm not sure what the remedy is.  My production app is running on Heroku, so I need a fix that I can push
there.  Can someone post instructions that this neophyte can follow?  (I gather I'm supposed to fork and add
the current master version from github to my config, but I need simple little steps for muddy little feet.) 

There's several forks of mechanize that have the fix, you can use one of them or downgrade to a previous
version of mechanize.

>  Thanks...
> 
> I'm running Rails 3.2, Mechanize 2.1.  Mechanize is failing with 
> 
>         Encoding::UndefinedConversionError: "\xA2" from ASCII-8BIT to UTF-8

I should have a release out be the end of next week that fixes this bug.

_______________________________________________
Mechanize-users mailing list
Mechanize-users <at> rubyforge.org
http://rubyforge.org/mailman/listinfo/mechanize-users
_______________________________________________
Mechanize-users mailing list
Mechanize-users <at> rubyforge.org
http://rubyforge.org/mailman/listinfo/mechanize-users
I'm a newbie with Mechanize, and I have what seems like a simple question: I'm trying to save a page that has
been loaded and then reload it later for further processing.  I see that Mechanize has a method parse(uri,
response, body) that (I presume) can load the DOM without physically accessing the site over the
internet.  If I save the URI and body from a previous Mechanize.get(), how do I get the "response" from the
page that is also required in order to reload the page?  Or is there a better way to save and restore the page
without incurring the latency of a second internet access?  I've also tried Mechanize::Page.new() but I
have the same problem in that I need a "response" for that call (nil gives an error).

hi all

i use tdiary( The web diary system http://sourceforge.net/projects/tdiary/ ).

i am trying to update the diary with the mechanize.

i run the code (see below), mechanize returns
Net::HTTPInternalServerError (Mechanize::ResponseCodeError)

Googled Treats, that it's a server-side error.

(e.g. but perl... WWW::Mechanize::FAQ - search.cpan.org My Mech
program doesn't work, but it works in the browser.
http://search.cpan.org/dist/WWW-Mechanize/lib/WWW/Mechanize/FAQ.pod#My_Mech_program_doesn%27t_work,_but_it_works_in_the_browser.
)

However, there are no errors on the server side.

Do people know the cause?

In addition, web browser( Firefox ) can safely update my diary.

[Environment]
% ruby --version
ruby 1.8.7 (2011-02-18 patchlevel 334) [i386-netbsdelf]

% gem list

*** LOCAL GEMS ***

domain_name (0.5.1)
mechanize (2.1)
net-http-digest_auth (1.2)
net-http-persistent (2.3.3)
nokogiri (1.5.0)
ntlm-http (0.1.1)
tmail (1.2.7.1)
unf (0.0.4)
unf_ext (0.0.4)
webrobots (0.0.12)

 * NetBSD 5.99.58
 * tdiary 3.1.0

[code]
#!/usr/pkg/bin/ruby -Ku

require 'rubygems'
require 'mechanize'
require 'logger'
require 'pp'

def main
  uri = "http://localhost/~rin/diary/update.rb"
  user = "test"
  pass = "test"
  referer = "http://www.area51.gr.jp/~rin/diary/update.rb"

  text = "!test\nhello tdiary!\n"

  agent = Mechanize.new
  agent.log = Logger.new($stdout)
#  agent.log.level = Logger::INFO
  agent.auth(user, pass)                 <= i use digest authentication
  agent.get(uri, nil, referer)
  agent.page.form_with(:action => "update.rb") { |form|
    form['body'] = text
    form.submit(form.button_with(:name => "append"))           <=
seems to be an error here.
  }

end

main

[mechanize log]
% ./tdwrite.rb
I, [2012-01-08T00:25:20.594807 #9165]  INFO -- : Net::HTTP::Get:
/~rin/diary/update.rb
D, [2012-01-08T00:25:20.595147 #9165] DEBUG -- : request-header:
accept-language => en-us,en;q=0.5
D, [2012-01-08T00:25:20.595197 #9165] DEBUG -- : request-header: accept => */*
D, [2012-01-08T00:25:20.595264 #9165] DEBUG -- : request-header:
user-agent => Mechanize/2.1 Ruby/1.8.7p334
(http://github.com/tenderlove/mechanize/)
D, [2012-01-08T00:25:20.595308 #9165] DEBUG -- : request-header:
accept-encoding => gzip,deflate,identity
D, [2012-01-08T00:25:20.595352 #9165] DEBUG -- : request-header:
referer => http://www.area51.gr.jp/~rin/diary/update.rb
D, [2012-01-08T00:25:20.595395 #9165] DEBUG -- : request-header: host
=> localhost
D, [2012-01-08T00:25:20.595434 #9165] DEBUG -- : request-header:
accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
I, [2012-01-08T00:25:20.598261 #9165]  INFO -- : status:
Net::HTTPUnauthorized 1.1 401 Authorization Required
D, [2012-01-08T00:25:20.598313 #9165] DEBUG -- : response-header:
content-type => text/html; charset=iso-8859-1
D, [2012-01-08T00:25:20.598359 #9165] DEBUG -- : response-header:
connection => Keep-Alive
D, [2012-01-08T00:25:20.598397 #9165] DEBUG -- : response-header:
www-authenticate => Digest realm="tDiary",
nonce="4NEBxvG1BAA=22e28cac4aac01662c270678d2355fbbfbe46395",
algorithm=MD5, qop="auth"
D, [2012-01-08T00:25:20.598433 #9165] DEBUG -- : response-header:
server => Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.9-dev DAV/2
D, [2012-01-08T00:25:20.598469 #9165] DEBUG -- : response-header: date
=> Sat, 07 Jan 2012 15:25:20 GMT
D, [2012-01-08T00:25:20.598504 #9165] DEBUG -- : response-header:
content-length => 290
D, [2012-01-08T00:25:20.598538 #9165] DEBUG -- : response-header:
content-encoding => gzip
D, [2012-01-08T00:25:20.598573 #9165] DEBUG -- : response-header:
keep-alive => timeout=5, max=100
D, [2012-01-08T00:25:20.598794 #9165] DEBUG -- : response-header: vary
=> Accept-Encoding
D, [2012-01-08T00:25:20.598965 #9165] DEBUG -- : Read 290 bytes (290 total)
D, [2012-01-08T00:25:20.599088 #9165] DEBUG -- : gzip body
I, [2012-01-08T00:25:20.605070 #9165]  INFO -- : Net::HTTP::Get:
/~rin/diary/update.rb
D, [2012-01-08T00:25:20.605298 #9165] DEBUG -- : request-header:
accept-language => en-us,en;q=0.5
D, [2012-01-08T00:25:20.605371 #9165] DEBUG -- : request-header: accept => */*
D, [2012-01-08T00:25:20.605417 #9165] DEBUG -- : request-header:
user-agent => Mechanize/2.1 Ruby/1.8.7p334
(http://github.com/tenderlove/mechanize/)
D, [2012-01-08T00:25:20.605462 #9165] DEBUG -- : request-header:
accept-encoding => gzip,deflate,identity
D, [2012-01-08T00:25:20.605499 #9165] DEBUG -- : request-header:
authorization => Digest username="diary", realm="tDiary", qop=auth,
uri="/~rin/diary/update.rb",
nonce="4NEBxvG1BAA=22e28cac4aac01662c270678d2355fbbfbe46395",
nc=00000000, cnonce="a6a1370113ae4db8792628a3e87927e8",
response="c304545256e7e4d0a03c3e2d6ae4ec80"
D, [2012-01-08T00:25:20.605545 #9165] DEBUG -- : request-header:
referer => http://www.area51.gr.jp/~rin/diary/update.rb
D, [2012-01-08T00:25:20.605581 #9165] DEBUG -- : request-header: host
=> localhost
D, [2012-01-08T00:25:20.605616 #9165] DEBUG -- : request-header:
accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
I, [2012-01-08T00:25:20.755405 #9165]  INFO -- : status: Net::HTTPOK 1.1 200 OK
D, [2012-01-08T00:25:20.755802 #9165] DEBUG -- : response-header:
content-type => text/html; charset=UTF-8
D, [2012-01-08T00:25:20.755878 #9165] DEBUG -- : response-header:
connection => Keep-Alive
D, [2012-01-08T00:25:20.755925 #9165] DEBUG -- : response-header:
authentication-info => rspauth="af4628a938bc6d2b04bf73ba60fb4a59",
cnonce="a6a1370113ae4db8792628a3e87927e8", nc=00000000, qop=auth
D, [2012-01-08T00:25:20.755962 #9165] DEBUG -- : response-header:
server => Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.9-dev DAV/2
D, [2012-01-08T00:25:20.755998 #9165] DEBUG -- : response-header: date
=> Sat, 07 Jan 2012 15:25:20 GMT
D, [2012-01-08T00:25:20.756032 #9165] DEBUG -- : response-header:
content-length => 7160
D, [2012-01-08T00:25:20.756067 #9165] DEBUG -- : response-header:
content-encoding => gzip
D, [2012-01-08T00:25:20.756102 #9165] DEBUG -- : response-header:
keep-alive => timeout=5, max=99
D, [2012-01-08T00:25:20.756137 #9165] DEBUG -- : response-header: vary
=> User-Agent,Accept-Encoding
D, [2012-01-08T00:25:20.756250 #9165] DEBUG -- : Read 7160 bytes (7160 total)
D, [2012-01-08T00:25:20.756403 #9165] DEBUG -- : gzip body
I, [2012-01-08T00:25:20.772181 #9165]  INFO -- : form encoding: ISO-8859-1
D, [2012-01-08T00:25:20.774358 #9165] DEBUG -- : query:
"old=20120108&year=2012&month=1&day=8&title=&body=%21test%0Ahello+tdiary%21%0A&plugin_ping_send=true&api=0&plugin_tags_filter=on&plugin_tb_url=&plugin_tb_section=p01&plugin_tb_excerpt=&append=+%E8%BF%BD%E8%A8%98+"
I, [2012-01-08T00:25:20.775397 #9165]  INFO -- : Net::HTTP::Post:
/~rin/diary/update.rb
D, [2012-01-08T00:25:20.775641 #9165] DEBUG -- : request-header:
accept-language => en-us,en;q=0.5
D, [2012-01-08T00:25:20.775879 #9165] DEBUG -- : request-header: accept => */*
D, [2012-01-08T00:25:20.776086 #9165] DEBUG -- : request-header:
content-type => application/x-www-form-urlencoded
D, [2012-01-08T00:25:20.776280 #9165] DEBUG -- : request-header:
user-agent => Mechanize/2.1 Ruby/1.8.7p334
(http://github.com/tenderlove/mechanize/)
D, [2012-01-08T00:25:20.776507 #9165] DEBUG -- : request-header:
accept-encoding => gzip,deflate,identity
D, [2012-01-08T00:25:20.776697 #9165] DEBUG -- : request-header:
content-length => 211
D, [2012-01-08T00:25:20.776922 #9165] DEBUG -- : request-header:
referer => http://diary:diary <at> localhost/~rin/diary/update.rb
D, [2012-01-08T00:25:20.777128 #9165] DEBUG -- : request-header: host
=> localhost
D, [2012-01-08T00:25:20.777381 #9165] DEBUG -- : request-header:
accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
I, [2012-01-08T00:25:20.778814 #9165]  INFO -- : status:
Net::HTTPUnauthorized 1.1 401 Authorization Required
D, [2012-01-08T00:25:20.779127 #9165] DEBUG -- : response-header:
content-type => text/html; charset=iso-8859-1
D, [2012-01-08T00:25:20.779357 #9165] DEBUG -- : response-header:
connection => Keep-Alive
D, [2012-01-08T00:25:20.779569 #9165] DEBUG -- : response-header:
www-authenticate => Digest realm="tDiary",
nonce="6ZYExvG1BAA=98d27d517f0275958fb5065ff69c8088ea850008",
algorithm=MD5, qop="auth"
D, [2012-01-08T00:25:20.779818 #9165] DEBUG -- : response-header:
server => Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.9-dev DAV/2
D, [2012-01-08T00:25:20.780037 #9165] DEBUG -- : response-header: date
=> Sat, 07 Jan 2012 15:25:20 GMT
D, [2012-01-08T00:25:20.780231 #9165] DEBUG -- : response-header:
content-length => 290
D, [2012-01-08T00:25:20.780439 #9165] DEBUG -- : response-header:
content-encoding => gzip
D, [2012-01-08T00:25:20.780628 #9165] DEBUG -- : response-header:
keep-alive => timeout=5, max=98
D, [2012-01-08T00:25:20.780847 #9165] DEBUG -- : response-header: vary
=> Accept-Encoding
D, [2012-01-08T00:25:20.781100 #9165] DEBUG -- : Read 290 bytes (290 total)
D, [2012-01-08T00:25:20.781460 #9165] DEBUG -- : gzip body
I, [2012-01-08T00:25:20.787175 #9165]  INFO -- : Net::HTTP::Post:
/~rin/diary/update.rb
D, [2012-01-08T00:25:20.787408 #9165] DEBUG -- : request-header:
accept-language => en-us,en;q=0.5
D, [2012-01-08T00:25:20.787652 #9165] DEBUG -- : request-header: accept => */*
D, [2012-01-08T00:25:20.787889 #9165] DEBUG -- : request-header:
content-type => application/x-www-form-urlencoded
D, [2012-01-08T00:25:20.788136 #9165] DEBUG -- : request-header:
user-agent => Mechanize/2.1 Ruby/1.8.7p334
(http://github.com/tenderlove/mechanize/)
D, [2012-01-08T00:25:20.788393 #9165] DEBUG -- : request-header:
accept-encoding => gzip,deflate,identity
D, [2012-01-08T00:25:20.788710 #9165] DEBUG -- : request-header:
authorization => Digest username="diary", realm="tDiary", qop=auth,
uri="/~rin/diary/update.rb",
nonce="6ZYExvG1BAA=98d27d517f0275958fb5065ff69c8088ea850008",
nc=00000001, cnonce="a6a1370113ae4db8792628a3e87927e8",
response="a903c6e42ed5896cd1da05516fc5b93d"
D, [2012-01-08T00:25:20.789087 #9165] DEBUG -- : request-header:
content-length => 211
D, [2012-01-08T00:25:20.789302 #9165] DEBUG -- : request-header:
referer => http://diary:diary <at> localhost/~rin/diary/update.rb
D, [2012-01-08T00:25:20.789520 #9165] DEBUG -- : request-header: host
=> localhost
D, [2012-01-08T00:25:20.789711 #9165] DEBUG -- : request-header:
accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
I, [2012-01-08T00:25:20.842091 #9165]  INFO -- : status:
Net::HTTPInternalServerError 1.1 500 Internal Server Error
D, [2012-01-08T00:25:20.842477 #9165] DEBUG -- : response-header:
content-type => text/html
D, [2012-01-08T00:25:20.842718 #9165] DEBUG -- : response-header:
connection => close
D, [2012-01-08T00:25:20.842961 #9165] DEBUG -- : response-header:
authentication-info => rspauth="f92b2000ce402faad6c351d19abdc07c",
cnonce="a6a1370113ae4db8792628a3e87927e8", nc=00000001, qop=auth
D, [2012-01-08T00:25:20.843216 #9165] DEBUG -- : response-header:
server => Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.9-dev DAV/2
D, [2012-01-08T00:25:20.843419 #9165] DEBUG -- : response-header: date
=> Sat, 07 Jan 2012 15:25:20 GMT
D, [2012-01-08T00:25:20.843644 #9165] DEBUG -- : response-header:
content-length => 416
D, [2012-01-08T00:25:20.843865 #9165] DEBUG -- : response-header:
content-encoding => gzip
D, [2012-01-08T00:25:20.844076 #9165] DEBUG -- : response-header: vary
=> Accept-Encoding
D, [2012-01-08T00:25:20.844377 #9165] DEBUG -- : Read 416 bytes (416 total)
D, [2012-01-08T00:25:20.844808 #9165] DEBUG -- : gzip body
/usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize/http/agent.rb:303:in
`fetch': 500 => Net::HTTPInternalServerError
(Mechanize::ResponseCodeError)
        from /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize/http/agent.rb:655:in
`response_authenticate'
        from /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize/http/agent.rb:300:in
`fetch'
        from /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize.rb:1036:in
`post_form'
        from /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize.rb:426:in
`submit'
        from /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize/form.rb:178:in
`submit'
        from ./tdwrite.rb:24:in `main'
        from (eval):23:in `form_with'
        from ./tdwrite.rb:22:in `main'
        from ./tdwrite.rb:29

[apache web server access_log]

127.0.0.1 - - [08/Jan/2012:00:25:20 +0900] "GET /~rin/diary/update.rb
HTTP/1.1" 401 290
127.0.0.1 - diary [08/Jan/2012:00:25:20 +0900] "GET
/~rin/diary/update.rb HTTP/1.1" 200 7160
127.0.0.1 - - [08/Jan/2012:00:25:20 +0900] "POST /~rin/diary/update.rb
HTTP/1.1" 401 290
127.0.0.1 - diary [08/Jan/2012:00:25:20 +0900] "POST
/~rin/diary/update.rb HTTP/1.1" 500 416

[apache web server error_log]

(In particular there are no errors)

--

-- 
miwa

HI,

I'm trying to download file with no success using below code:

  $mech->get( $search_result );
  $mech->form_number( 4 );

  $mech->current_form()->click("eksport_do_pliku");
  print $mech->res();

This is part of HTML source:
"
<button type="submit" name="eksport_do_pliku" class="buty" 
style="margin-left:20px" 
onclick="document.getElementById('cod').value='2';"><img 
src="gfx/przyciski/akceptuj.png" 
style="vertical-align:middle;margin-right:5px">Eksportuj dane do pliku 
</button>
"

The result of "$mech->res();" is "HTTP::Response=HASH(0x8a49d10)"

Please help :)

--

-- 
best regards
Piotr Polok

HI,

I need to click second button on below form and download a file, could 
you please give a advice how to do that :)

HTML source:

             <form method="POST" target="_self">
                  <p style="margin-bottom: 
5px;margin-left:45px;margin-top:-40px;"><input id="radio_5" 
checked="checked" name="format" value="1" 
type="radio"style="vertical-align:middle;"><label for="radio_5"> liniowy 
</label></p></p>
                  <p style="margin-bottom: 5px;margin-left: 45px"><input 
id="radio_6" name="format" value="2" type="radio" 
style="vertical-align:middle;"><label for="radio_6"> xml</label></p>
                  <p style="margin-bottom: 5px;margin-left: 45px"><input 
id="radio_7"  name="format" value="3" 
type="radio"style="vertical-align:middle;"><label for="radio_7"> 
MT942</label></p>
                  <p style="margin-bottom: 5px;margin-left: 45px"><input 
id="radio_8" name="format" value="4" type="radio" 
style="vertical-align:middle;"><label for="radio_8"> 
MT942/Multicash</label></p>
             </td>
          </tr>
          <tr>
              <td colspan="2"><hr width="98%" color="#acacac" 
noshade="noshade" size="1" style="vertical-align: bottom"/></td></tr>
	  </tr>
           <tr>
               <td class="wyciagi_w_td1">
                 <button type="submit" class="buty" 
onclick="document.getElementById('cod').value='1';"><img 
src="gfx/przyciski/akceptuj.png" 
style="vertical-align:middle;margin-right:5px" >Zapisz 
parametry</button>
               </td>
               <td style="text-align: left;">
                 <input id="cod" name="cod" value="" type="hidden">
                 <button type="submit" class="buty" 
style="margin-left:20px" 
onclick="document.getElementById('cod').value='2';"><img 
src="gfx/przyciski/akceptuj.png" 
style="vertical-align:middle;margin-right:5px" >Eksportuj dane do pliku 
</button>

                 <a class="nw"

href='javascript:openWindow("start.php?page=pdf&pdf=wirtualne&szukaj=wy_oddo&data_new1=2011-12-03&data_new2=2012-01-03&sort_typ=data&rows=100&rach=1&idnrb=5&rach_wir=1&rach_wirtualne=&types=oddo")' 
style="margin-left:30px"><img src="gfx/przyciski/pdf.gif" 
style="vertical-align:middle;margin-right:5px;height:28px">Wydruk 
.pdf</a>&nbsp;
                 <a class="nw"

href='javascript:openWindow("start.php?page=wydruk&typdruk=wirtualne&szukaj=wy_oddo&data_new1=2011-12-03&data_new2=2012-01-03&sort_typ=data&rows=100&rach=1&idnrb=5&rach_wir=1&rach_wirtualne=&types=oddo")' 
style="margin-left:30px"><img src="gfx/przyciski/drukarka.png" 
style="vertical-align:middle;margin-right:5px;">Przygotuj wydruk</a>
              </td>
	  </tr>
	</table>
     </FORM>

--

-- 
br
Piotr Polok

_______________________________________________
Mechanize-users mailing list
Mechanize-users <at> rubyforge.org
http://rubyforge.org/mailman/listinfo/mechanize-users

12	May 2012
4	April 2012
1	March 2012
4	February 2012
16	January 2012
4	December 2011
2	October 2011
6	September 2011
2	August 2011
13	July 2011
3	June 2011
5	May 2011
14	April 2011
4	March 2011
2	February 2011
3	January 2011
7	December 2010
20	October 2010
1	September 2010
7	August 2010
12	July 2010
25	June 2010
12	May 2010
5	April 2010
12	March 2010
13	February 2010
52	January 2010
10	December 2009
5	November 2009
2	October 2009
13	September 2009
16	August 2009
36	July 2009
25	June 2009
24	May 2009
15	April 2009
34	March 2009
10	February 2009
4	December 2008
16	November 2008
5	October 2008
6	September 2008
3	August 2008
5	July 2008
2	June 2008
1	May 2008
7	April 2008
18	March 2008
9	February 2008
11	January 2008
5	December 2007
2	November 2007
17	October 2007
12	September 2007
8	August 2007
46	July 2007
3	June 2007
18	May 2007
9	April 2007
24	March 2007
20	February 2007

way to get response headers only?

Resource time out error handling

Mechanize::ResponseCodeError: 500

Re: dealing with Encoding::UndefinedConversionError: "\xA2" from ASCII-8BIT to UTF-8

Re: dealing with Encoding::UndefinedConversionError: "\xA2" from ASCII-8BIT to UTF-8

dealing with Encoding::UndefinedConversionError: "\xA2" from ASCII-8BIT to UTF-8

Reload previously retrieved page

500 error in tdiary using mechanize

How to download file after button click with no URL

how to click button on the form with no names

deobfuscating javascript?

Mon	Tue	Wed	Thu	Fri	Sat	Sun

	1	2	3	4	5	6
7	8	9	10	11	12	13
14	15	16	17	18	19	20
21	22	23	24	25	26	27
28	29	30	31