barsalou | 2 Jul 23:11 2011
Picon

Doing a post using Mechanize 2.0 - a solution

I'm working with a web page where it is required that you log in, then  
use a POST to submit the information via a form.

Since this isn't part of my everyday life, it was a struggle to figure  
out the pieces that were needed to make this happen.

We needed to:
  - login
  - get the cookie info
  - make a new request that included the returned cookie info
  - post the request with necessary form data

The example also shows how to add custom headers to the posted request.

Firefox's TamperData Add-on was needed to see what the server was  
expecting in order to put this together.

Here is my solution:

https://gist.github.com/1061645

Suggestions for slimming this down or other changes are welcome.

Mike B.

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
Eric Hodel | 27 Jun 22:55 2011
Picon

[ANN] Mechanize 2.0

mechanize version 2.0 has been released!

* <http://mechanize.rubyforge.org>
* <http://github.com/tenderlove/mechanize/tree/master>

The Mechanize library is used for automating interaction with websites.
Mechanize automatically stores and sends cookies, follows redirects,
can follow links, and submit forms.  Form fields can be populated and
submitted.  Mechanize also keeps track of the sites that you have visited as
a history.

Changes:

### 2.0 / 2011-06-27

Mechanize is now under the MIT license

* API changes
  * WWW::Mechanize has been removed.  Use Mechanize.
  * Pre connect hooks are now called with the agent and the request.  See
    Mechanize#pre_connect_hooks.
  * Post connect hooks are now called with the agent and the response.  See
    Mechanize#post_connect_hooks.
  * Mechanize::Chain is gone, as an internal API this should cause no problems.
  * Mechanize#fetch_page no longer accepts an options Hash.
  * Mechanize#put now accepts headers instead of an options Hash as the last
    argument
  * Mechanize#delete now accepts headers instead of an options Hash as the
    last argument
  * Mechanize#request_with_entity now accepts headers instead of an options
(Continue reading)

Sathia S | 26 Jun 08:54 2011
Picon

How to submit form for Select box

HI,

to submit form for textbox 
  form['Email'] = 'sathia2704 <at> gmail.com'

<select>
<option>one</option>
<option>Two</option>
<option>Three</option>
</select>

How to submit form for selectbox ?

Please tell how to submit this select box
 
                                                                            Thanks in advance

_______________________________________________
Mechanize-users mailing list
Mechanize-users <at> rubyforge.org
http://rubyforge.org/mailman/listinfo/mechanize-users
Waley Chen | 26 Jun 07:15 2011
Picon

Help Getting Started with Mechanize

I'm new to Ruby and just installed Ruby for Windows. I want to use the mechanize library (https://github.com/tenderlove/mechanize) and so I'm following the guide at https://github.com/tenderlove/mechanize/blob/master/GUIDE.rdoc.
On the Windows cmd line, I installed mechanize by using the cmd "gem install mechanize".
When I run the following code:
require 'rubygems'
require
'mechanize'

agent
= Mechanize.new
I get the error:
<internal:lib/rubygems/custom_require>:29:in `require': no such file to load -- net/http/digest_auth (LoadError)
from <internal:lib/rubygems/custom_require>:29:in `
require'
from C:/Ruby192/lib/ruby/1.9.1/mechanize.rb:5:in `<top (required)>'

from
<internal:lib/rubygems/custom_require>:29:in `require'
from <internal:lib/rubygems/custom_require>:29:in `
require'
from helloworld.rb:2:in `<main>'

Anybody know what's going on?



Thanks!

_______________________________________________
Mechanize-users mailing list
Mechanize-users <at> rubyforge.org
http://rubyforge.org/mailman/listinfo/mechanize-users
Andrew Ross | 17 May 18:51 2011

Rebuild Browser State Later?

Hi all,

I'm looking for a way to completely save and rebuild the browser state to use for a later form submission. 
Since a Mechanize browser instance can't be serialized, this has proved difficult.  I can see how you can
easily store and re-load the cookie jar, but I also need to have the DOM in the exact state it was in
previously.  

Does anyone know of an easy way to do this with mechanize?

Thanks,
Andy
Rob GB | 12 May 18:55 2011
Picon

retrieve metadata

Hi all, When I request a URL I get a refresh URL in the headers: <meta http-equiv="refresh" content="2;url=/html/Splash.action?splash="/>\ When I ask to mechanize to list the metadata I do this: pp $page.meta I get: [#<Mechanize::Page::Meta "" "http://localhost/html/Splash.action?splash=">] How can I cleanly retrieve the refresh URL? Thanks!
_______________________________________________
Mechanize-users mailing list
Mechanize-users <at> rubyforge.org
http://rubyforge.org/mailman/listinfo/mechanize-users
DeNigris Sean | 24 Apr 07:38 2011

Dynamic Captcha

I'm trying to download my EZ-Pass statements from www.e-zpassny.com using Mechanize. I'm having trouble
logging in because the source of the captcha image is https://www.e-zpassny.com/vector/jcaptcha.do,
which generates a new image every time it's loaded. So when I attempt to look at it by opening it in a window,
it's already changed.

How do I see the original image from my "login_page = agent.get 'https://www.e-zpassny.com/vector/account/home/accountLogin.do'"?

Thanks.
Sean
Robert Poor | 3 May 04:37 2011
Picon

parsing and submitting a form from a page with missing content type?

[cross posted on Nokogiri and Mechanize lists]

I have a page that is missing a valid content type, but it appears to
contain (mostly?) well formed HTML.  My goal is to agent.click() on a
button in the form, but I don't know the best way to get there.

The first lines read:

  <html>
    <head>
      <meta name="generator" content="HTML Tidy, see www.w3.org">
      ...

When I coerce Mechanize to use Mechanize::Page as the parser via:

       <at> agent.pluggable_parser.default = Mechanize::Page

I get a Mechanize::ContentTypeError (okay, so maybe it isn't so well
formed).  But Nokogiri is able to parse it without error.

So: what's the best way (or easiest way) to submit the form from the
loaded page?  I see two options:

* xpath to the appropriate form using Nokogiri and then (somehow) pass
the form to Mechanize to submit.  But I'm not sure of the syntax for
that.

* write my own pluggable parser that somehow knows how to deal with
the page.  I'm even foggier on how that would work.

Suggestions or examples are welcome...
Eric Hodel | 18 Apr 07:24 2011
Picon

[ANN] Mechanize 2.0.pre.2

mechanize version 2.0.pre.2 has been released!  Please test this release and let me know if I have broken
anything important.

* <http://mechanize.rubyforge.org>
* <http://github.com/tenderlove/mechanize/tree/master>

The Mechanize library is used for automating interaction with websites.
Mechanize automatically stores and sends cookies, follows redirects,
can follow links, and submit forms.  Form fields can be populated and
submitted.  Mechanize also keeps track of the sites that you have visited as
a history.

Changes:

### 2.0.pre.2 / 2011-04-17

Mechanize is now under the MIT license

* API changes
  * WWW::Mechanize has been removed.  Use Mechanize.
  * Pre connect hooks are now called with the agent and the request.  See
    Mechanize#pre_connect_hooks.
  * Post connect hooks are now called with the agent and the response.  See
    Mechanize#post_connect_hooks.
  * Mechanize::Chain is gone, as an internal API this should cause no problems.
  * Mechanize#fetch_page no longer accepts an options Hash.
  * Mechanize#put now accepts headers instead of an options Hash as the last
    argument
  * Mechanize#delete now accepts headers instead of an options Hash as the
    last argument
  * Mechanize#request_with_entity now accepts headers instead of an options
    Hash as the last argument
  * Mechanize no longer raises RuntimeError directly, Mechanize::Error or
    ArgumentError are raised instead.
  * The User-Agent header has changed.  It no longer includes the WWW- prefix
    and now includes the ruby version.  The URL has been updated as well.
  * Mechanize now requires ruby 1.8.7 or newer.
  * Hpricot support has been removed as webrobots requires nokogiri.
  * Mechanize#get no longer accepts the referer as the second argument.
  * Mechanize#get no longer allows the HTTP method to be changed (:verb
    option).

* Deprecations
  * Mechanize#get with an options hash is deprecated and will be removed after
    October, 2011.
  * Mechanize::Util::to_native_charset is deprecated as it is no longer used
    by Mechanize.

* New Features

  * Add header reference methods to Mechanize::File so that a reponse
    object gets compatible with Net::HTTPResponse.
  * Mechanize#click accepts a regexp or string to click a button/link in the
    current page. It works as expected when not passed a string or regexp.
  * Provide a way to only follow permanent redirects (301)
    automatically: <tt>agent.redirect_ok = :permanent</tt>  GH #73
  * Documented various Mechanize accessors.  GH #66
  * Mechanize now uses net-http-digest_auth.  GH #31
  * Mechanize now implements session cookies.  GH #78
  * Mechanize now implements deflate decoding.  GH #40
  * Mechanize now allows a certificate and key to be passed directly.  GH #71
  * Mechanize::Form::MultiSelectList now implements #option_with and
    #options_with.  GH #42
  * Add Mechanize::Page::Link#rel and #rel?(kind) to read and test the rel
    attribute.
  * Add Mechanize::Page#canonical_uri to read a </tt><link
    rel="canonical"></tt> tag.
  * Add support for Robots Exclusion Protocol (i.e. robots.txt) and
    nofollow/noindex in meta tags and the rel attribute.  Automatic
    exclusion can be turned on by setting:
      agent.robots = true
  * Manual robots.txt test can be performed with
    Mechanize#robots_allowed? and #robots_disallowed?.
  * Mechanize::Form now supports the accept-charset attribute.  GH #96

* Bug Fixes:

  * Fixed a bug where Referer is not sent when accessing a relative
    URI starting with "http".
  * Fix handling of Meta Refresh with relative paths.  GH #39
  * Mechanize::CookieJar now supports RFC 2109 correctly.  GH #85
  * Fixed typo in EXAMPLES.rdoc.  GH #74
  * The base element is now handled correctly for images.  GH #72
  * Image buttons with no name attribute are now included in the form's button
    list.  GH#56
  * Improved handling of non ASCII-7bit compatible characters in links (only
    an issue on ruby 1.8).  GH #36, GH #75
  * Loading cookies.txt is faster.  GH #38
  * Mechanize no longer sends cookies for a.b.example to axb.example.  GH #41
  * Mechanize no longer sends the button name as a form field for image
    buttons.  GH #45
  * Blank cookie values are now skipped.  GH #80
  * Mechanize now adds a '.' to cookie domains if no '.' was sent.  This is
    not allowed by RFC 2109 but does appear in RFC 2965.  GH #86
  * file URIs are now read in binary mode.  GH #83
  * Content-Encoding: x-gzip is now treated like gzip per RFC 2616.
  * Mechanize now unescapes URIs for meta refresh.  GH #68
  * Mechanize now has more robust HTML charset detection.  GH #43
  * Mechanize::Form::Textarea is now created from a textarea element.  GH #94
Minwoo Lee | 11 Apr 10:06 2011
Picon

html parsing problem with encoding

Hi.
I have a question about parsing and encoding.

I tried this out.

>> agent = Mechanize.new
>> p.forms
=> []

However, it has a form.
I think the reason is encoding of 'enc.html'

'enc.html' is with encoding 'euc-kr' and it conflicts with the parser.

I tried out
>> p.encoding = 'utf-8'
or
>> p.encoding = 'euc-kr'
but it doesn't work.


Thank you for reading this email.
_______________________________________________
Mechanize-users mailing list
Mechanize-users <at> rubyforge.org
http://rubyforge.org/mailman/listinfo/mechanize-users
Eric Hodel | 10 Apr 03:31 2011
Picon

[ANN] mechanize 2.0.pre.1

I have released Mechanize 2.0.pre.1 to rubygems.org.

  gem install --pre mechanize

My goal for mechanize is to move the HTTP layer into its own gem and hopefully into ruby itself.  The main
changes to support this are the removal of Mechanize::Chain and the removal of the options hash that
#fetch_page supported.

Passing an options hash to the mechanize methods #get, #head, #post, #put etc. will be deprecated in 2.0
(this has not been done yet).

Changes so far:

Mechanize is now under the MIT license

* API changes
  * WWW::Mechanize has been removed.
  * Pre connect hooks are now called with the agent and the request.  See
    Mechanize#pre_connect_hooks.
  * Post connect hooks are now called with the agent and the response.  See
    Mechanize#post_connect_hooks.
  * Mechanize::Chain is gone, as an internal API this should cause no problems.
  * Mechanize#fetch_page no longer accepts an options Hash.
  * Mechanize#put now accepts headers instead of an options Hash as the last
    argument
  * Mechanize#delete now accepts headers instead of an options Hash as the
    last argument
  * Mechanize#request_with_entity now accepts headers instead of an options
    Hash as the last argument
  * Mechanize no longer raises RuntimeError directly, Mechanize::Error or
    ArgumentError are raised instead.
  * The User-Agent header has changed.  It no longer includes the WWW- prefix
    and now includes the ruby version.  The URL has been updated as well.
  * Mechanize now requires ruby 1.8.7 or newer.

* New Features

  * Add header reference methods to Mechanize::File so that a reponse
    object gets compatible with Net::HTTPResponse.
  * Mechanize#click accepts a regexp or string to click a button/link in the
    current page. It works as expected when not passed a string or regexp.
  * Provide a way to only follow permanent redirects (301)
    automatically: <tt>agent.redirect_ok = :permanent</tt>  GH #73
  * Documented various Mechanize accessors.  GH #66
  * Mechanize now uses net-http-digest_auth.  GH #31
  * Mechanize now implements session cookies.  GH #78
  * Mechanize now implements deflate decoding.  GH #40
  * Mechanize now allows a certificate and key to be passed directly.  GH #71
  * Mechanize::Form::MultiSelectList now implements #option_with and
    #options_with.  GH #42
  * Add Mechanize::Page::Link#rel and #rel?(kind) to read and test the rel
    attribute.
  * Add Mechanize::Page#canonical_uri to read a </tt><link
    rel="canonical"></tt> tag.
  * Add support for Robots Exclusion Protocol (i.e. robots.txt) and
    nofollow/noindex in meta tags and the rel attribute.  Automatic
    exclusion can be turned on by setting:
      agent.robots = true
  * Manual robots.txt test can be performed with
    Mechanize#robots_allowed? and #robots_disallowed?.

* Bug Fixes:

  * Fixed a bug where Referer is not sent when accessing a relative
    URI starting with "http".
  * Fix handling of Meta Refresh with relative paths.  GH #39
  * Mechanize::CookieJar now supports RFC 2109 correctly.  GH #85
  * Fixed typo in EXAMPLES.rdoc.  GH #74
  * The base element is now handled correctly for images.  GH #72
  * Image buttons with no name attribute are now included in the form's button
    list.  GH#56
  * Improved handling of non ASCII-7bit compatible characters in links (only
    an issue on ruby 1.8).  GH #36, GH #75
  * Loading cookies.txt is faster.  GH #38
  * Mechanize no longer sends cookies for a.b.example to axb.example.  GH #41
  * Mechanize no longer sends the button name as a form field for image
    buttons.  GH #45
  * Blank cookie values are now skipped.  GH #80
  * Mechanize now adds a '.' to cookie domains if no '.' was sent.  This is
    not allowed by RFC 2109 but does appear in RFC 2965.  GH #86
  * file URIs are now read in binary mode.  GH #83
  * Content-Encoding: x-gzip is now treated like gzip per RFC 2616.
  * Mechanize now unescapes URIs for meta refresh.  GH #68
  * Mechanize now has more robust HTML charset detection.  GH #43

Gmane