HTML Parser in Java ()

Derrick Oswald | 1 Feb 2007 03:00

Re: Cookie handling

That should be handled OK.
But it doesn't look like what you want from that URL.

----- Original Message ----
From: Gavin Gilmour <gavin <at> brokentrain.net>
To: htmlparser user list <htmlparser-user <at> lists.sourceforge.net>
Sent: Tuesday, January 30, 2007 8:49:25 AM
Subject: Re: [Htmlparser-user] Cookie handling

Wow, good point. The URL seems to load fine here in my browser but on further
inspection:

sokar:~/junk% curl http://www3.interscience.wiley.com/cgi-bin/abstract/68504762/ABSTRACT\?CRETRY\=1\&SRETRY\=0
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a
href="http://www3.interscience.wiley.com/cookie_setting_error.html">here</a>.</p>
</body></html>

Seems to have already decided it's dud, weird.

After a bit of investigating, the full story is a bit worse than I thought and
is involing multiple redirects. The first link I need comes back from this service:
http://eutils.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi?dbfrom=pubmed&id=10629107&retmode=ref&cmd=prlinks
- which just offers up a 302 or whatever and then issues a redirect.

Fair enough, so:

---
sokar:~/junk% curl 'http://eutils.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi?dbfrom=pubmed&id=10629107&retmode=ref&cmd=prlinks'
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a
href="http://dx.doi.org/10.1002/(SICI)1097-4644(1999)75:32+<84::AID-JCB11>3.0.CO;2-M">here</a>.</p>
</body></html>
---

Is giving 'http://dx...' which is what the parser is trying next I'd imagine. So then:

sokar:~/junk% curl 'http://dx.doi.org/10.1002/(SICI)1097-4644(1999)75:32+<84::AID-JCB11>3.0.CO;2-M'
<HTML><HEAD><TITLE>Handle Redirect</TITLE></HEAD>
<BODY><A
HREF="http://doi.wiley.com/10.1002/%28SICI%291097-4644%281999%2975%3A32%2B%3C84%3A%3AAID-JCB11%3E3.0.CO%3B2-M">http://doi.wiley.com/10.1002/%28SICI%291097-4644%281999%2975%3A32%2B%3C84%3A%3AAID-JCB11%3E3.0.CO%3B2-M</A></BODY></HTML>

Looking at this URL, it seems to be the 'final one' which is leading to the
(desired) destination in a browser. (Does that output even look like something
the parser would handle though?)

What a mess :(

Gavin.

P.S. Sorry about the horribly formatted mail due to the unsightly urls
involved.

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

_______________________________________________
Htmlparser-user mailing list
Htmlparser-user <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/htmlparser-user

headers

Gavin Gilmour | 1 Feb 2007 11:49

Re: Cookie handling

Ah, I see what you mean. Looking at the last url it seems even more redirects
are taking place until it's arriving at the final url - I've counted around 7
so far. Regardless though, I'm still at a loss why the redirects aren't being
processed and the cookie stuff handled in the end.

Gavin.

On Wed, Jan 31, 2007 at 06:00:17PM -0800, Derrick Oswald wrote:
> 
> That should be handled OK.
> But it doesn't look like what you want from that URL.
> 
> ----- Original Message ----
> From: Gavin Gilmour <gavin <at> brokentrain.net>
> To: htmlparser user list <htmlparser-user <at> lists.sourceforge.net>
> Sent: Tuesday, January 30, 2007 8:49:25 AM
> Subject: Re: [Htmlparser-user] Cookie handling
> 
> Wow, good point. The URL seems to load fine here in my browser but on further
> inspection:
> 
> sokar:~/junk% curl http://www3.interscience.wiley.com/cgi-bin/abstract/68504762/ABSTRACT\?CRETRY\=1\&SRETRY\=0
> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
> <html><head>
> <title>302 Found</title>
> </head><body>
> <h1>Found</h1>
> <p>The document has moved <a
> href="http://www3.interscience.wiley.com/cookie_setting_error.html">here</a>.</p>
> </body></html>
> 
> Seems to have already decided it's dud, weird.
> 
> After a bit of investigating, the full story is a bit worse than I thought and
> is involing multiple redirects. The first link I need comes back from this service:
> http://eutils.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi?dbfrom=pubmed&id=10629107&retmode=ref&cmd=prlinks
> - which just offers up a 302 or whatever and then issues a redirect.
> 
> Fair enough, so:
> 
> ---
> sokar:~/junk% curl 'http://eutils.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi?dbfrom=pubmed&id=10629107&retmode=ref&cmd=prlinks'
> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
> <html><head>
> <title>302 Found</title>
> </head><body>
> <h1>Found</h1>
> <p>The document has moved <a
> href="http://dx.doi.org/10.1002/(SICI)1097-4644(1999)75:32+&lt;84::AID-JCB11&gt;3.0.CO;2-M">here</a>.</p>
> </body></html>
> ---
> 
> Is giving 'http://dx...' which is what the parser is trying next I'd imagine. So then:
> 
> sokar:~/junk% curl 'http://dx.doi.org/10.1002/(SICI)1097-4644(1999)75:32+<84::AID-JCB11>3.0.CO;2-M'
> <HTML><HEAD><TITLE>Handle Redirect</TITLE></HEAD>
> <BODY><A
> HREF="http://doi.wiley.com/10.1002/%28SICI%291097-4644%281999%2975%3A32%2B%3C84%3A%3AAID-JCB11%3E3.0.CO%3B2-M">http://doi.wiley.com/10.1002/%28SICI%291097-4644%281999%2975%3A32%2B%3C84%3A%3AAID-JCB11%3E3.0.CO%3B2-M</A></BODY></HTML>
> 
> Looking at this URL, it seems to be the 'final one' which is leading to the
> (desired) destination in a browser. (Does that output even look like something
> the parser would handle though?)
> 
> What a mess :(
> 
> Gavin.
> 
> P.S. Sorry about the horribly formatted mail due to the unsightly urls
> involved.
> 
> 
> 
> 
> 

> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier.
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

> _______________________________________________
> Htmlparser-user mailing list
> Htmlparser-user <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

headers

sebb | 11 Feb 2007 22:15

Formal release of 2.0 ?

I'm hoping to release an updated version of JMeter before too long,
and would like to include the updated htmlparser 2.0 (with the new
license, for which many thanks).

The current build of 2.0 is listed as an "Integration" build - is this
OK to use, or is there going to be a formal release of 2.0?

S///

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

headers

Derrick Oswald | 13 Feb 2007 05:35

Re: Formal release of 2.0 ?

I don't think I have time to do a formal release.
Better go with the integration build ... if it suits.

Derrick

----- Original Message ----
From: sebb <sebbaz <at> gmail.com>
To: htmlparser user list <htmlparser-user <at> lists.sourceforge.net>
Sent: Sunday, February 11, 2007 4:15:04 PM
Subject: [Htmlparser-user] Formal release of 2.0 ?

I'm hoping to release an updated version of JMeter before too long,
and would like to include the updated htmlparser 2.0 (with the new
license, for which many thanks).

The current build of 2.0 is listed as an "Integration" build - is this
OK to use, or is there going to be a formal release of 2.0?

S///

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Htmlparser-user mailing list
Htmlparser-user <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/htmlparser-user

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

_______________________________________________
Htmlparser-user mailing list
Htmlparser-user <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/htmlparser-user

headers

Subramanya Sastry | 14 Feb 2007 09:17

Warning message about charset

Hi there,

I have been using HTML Parser for a while now, and happy with it.  But, 
I have now run into a minor glitch and am hoping someone here might be 
able to help me with it.

On some html pages, I get the following warning / error message:
       "unable to determine cannonical charset name for x-user-defined - 
using ISO-8859-1"
The rest of the parsing proceeds fine without a hitch.

I am not able to figure out where the error message is coming from .. I 
know it is being spit out during parsing.  Can anyone tell me how I 
could catch this error and raise an exception -- because this error 
message corresponds to other bad things happening further down the 
pipeline in my application.

Thanks for any leads.

Subbu.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

headers

Derrick Oswald | 15 Feb 2007 03:14

Re: Warning message about charset

Hi Subbu,

This message is spit out if a character set that is specified in a META tag can't be resolved to a real character set name.
See the doSemanticAction() of MetaTag:
charset = getPage ().getCharset (getAttribute ("CONTENT"));
where the call to Page.getCharset() is calling Page.findCharset(), where the message is generated.

Derrick

----- Original Message ----
From: Subramanya Sastry <sastry <at> cs.wisc.edu>
To: htmlparser-user <at> lists.sourceforge.net
Sent: Wednesday, February 14, 2007 3:17:51 AM
Subject: [Htmlparser-user] Warning message about charset

Hi there,

I have been using HTML Parser for a while now, and happy with it.  But,
I have now run into a minor glitch and am hoping someone here might be
able to help me with it.

On some html pages, I get the following warning / error message:
       "unable to determine cannonical charset name for x-user-defined -
using ISO-8859-1"
The rest of the parsing proceeds fine without a hitch.

I am not able to figure out where the error message is coming from .. I
know it is being spit out during parsing.  Can anyone tell me how I
could catch this error and raise an exception -- because this error
message corresponds to other bad things happening further down the
pipeline in my application.

Thanks for any leads.

Subbu.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Htmlparser-user mailing list
Htmlparser-user <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/htmlparser-user

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

_______________________________________________
Htmlparser-user mailing list
Htmlparser-user <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/htmlparser-user

headers

Dipesh Sharma | 16 Feb 2007 03:20

HTML tree creation

Hi there,

I'm trying to build an HTML tree model using your parser. I have come up with this code so far:

import org.htmlparser.Parser;
import org.htmlparser.util.NodeList;
import org.htmlparser.util.ParserException;
import org.htmlparser.parserapplications.filterbuilder.HtmlTreeModel;

class Test
{
public static void main (String[] args)
{
try
{
Parser parser = new Parser ("http://www.deals2buy.com")
NodeList list = parser.parse (null);
HtmlTreeModel Tree= new HtmlTreeModel(list);
System.out.println (Tree.toString());
}
catch (ParserException pe)
{
pe.printStackTrace ();
}
}
}

However toString method simply prints the objects name. Could you please help me with a solution that prints out the whole HTML tree?

Thanks,
Chikki

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

_______________________________________________
Htmlparser-user mailing list
Htmlparser-user <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/htmlparser-user

headers

Dipesh Sharma | 21 Feb 2007 05:20

urgent

From: Dipesh Sharma <dipesh_mca <at> rediffmail.com>
Subject: urgent

Dear Derrick,

I'm trying to extract information out of an HTML page, depending upon the path (of html tags specified).eg if the input path (of tags) is <head><title>, then the title should be retreived. This is the program that i have come up with. It works fine.

import org.htmlparser.Parser;
import org.htmlparser.util.NodeList;
import org.htmlparser.util.ParserException;
import org.htmlparser.beans.StringBean;
import org.htmlparser.filters.TagNameFilter;
import org.htmlparser.util.*;
import org.htmlparser.*;
import org.htmlparser.filters.AndFilter;
import org.htmlparser.filters.HasParentFilter;

class Test
{
public static void main (String[] args)
{
try
{

Parser parser = new Parser ("http://www.deals2buy.com");

StringBean sb = new StringBean ();
NodeList list = new NodeList ();

NodeFilter filter =
   new AndFilter (
   new TagNameFilter ("title"),
   new HasParentFilter (
   new TagNameFilter ("head")));

for (NodeIterator e = parser.elements (); e.hasMoreNodes (); )
e.nextNode ().collectInto (list, filter);

if (list!=null)
list.visitAllNodesWith (sb);
System.out.println (sb.getStrings ());

}
catch (ParserException pe)
{
pe.printStackTrace ();
}
}
}

However i wish to extend the functinality to more than 2 levels by nesting the node-filter even furhter. eg. if the user specifies (<TABLE class=dti> <TR> <TD class=lf> <TABLE class=dtlp> <TR> <TD class=lp>).

something like this

NodeFilter filter =
   new AndFilter (new TagNameFilter ("TABLE "),new AndFilter
            ( new TagNameFilter ("TD"),
                  new HasParentFilter(
                        new TagNameFilter ("TR"))));

Could you plz help me fix the problem. It is urgent.
Thanks in advance.

Dipesh

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

_______________________________________________
Htmlparser-user mailing list
Htmlparser-user <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/htmlparser-user

headers

sajid khan | 26 Feb 2007 12:41

How can I improve the speed of extracting the information of a html page through HtmlParser.

Hi,
I am using HTMLParser for extracting the content of the Html page. I have noticed that bulk of the time is spent in extracting the information than processing the data.
The code looks like this,

// inputStream is of type InputStream. It carries the page Source of a Html page.
Page page = new Page(inputStream, null);
Lexer lexer = new Lexer(page);
Parser parser = new Parser(lexer);
StringBean sb=new StringBean();
parser.visitAllNodesWith (sb);
String text = sb.getStrings();
//Doing something with text.

Here I want to inform you that i have crawled few pages with the help of a crawler. So html pages are in my Hard Disk.

Can anybody please help me to improve the speed of my program.

regards
Sajid Khan.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

_______________________________________________
Htmlparser-user mailing list
Htmlparser-user <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/htmlparser-user

headers

Ian Macfarlane | 28 Feb 2007 17:35

Re: HTML tree creation

Have a look at the NodeTreeWalker class. It includes a depth-first (how you'd probably want to do it) iteration option, so you can go to the root node and simply call next and then print out the name of the current element.

As it works on a Node, if the root has multiple nodes (common if there is a doctype and some other text spacing), you get a NodeList - you can simply iterate through the Nodes of the root NodeList and perform the above on each one.

Ian

On 16 Feb 2007 02:20:16 -0000, Dipesh Sharma <dipesh_mca <at> rediffmail.com> wrote:

Hi there,

I'm trying to build an HTML tree model using your parser. I have come up with this code so far:

import org.htmlparser.Parser;
import org.htmlparser.util.NodeList;
import org.htmlparser.util.ParserException;
import org.htmlparser.parserapplications.filterbuilder.HtmlTreeModel;

class Test
{
public static void main (String[] args)
{
try
{
Parser parser = new Parser ("http://www.deals2buy.com")
NodeList list = parser.parse (null);
HtmlTreeModel Tree= new HtmlTreeModel(list);
System.out.println (Tree.toString());
}
catch (ParserException pe)
{
pe.printStackTrace ();
}
}
}

However toString method simply prints the objects name. Could you please help me with a solution that prints out the whole HTML tree?

Thanks,
Chikki

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Htmlparser-user mailing list
Htmlparser-user <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/htmlparser-user

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

_______________________________________________
Htmlparser-user mailing list
Htmlparser-user <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/htmlparser-user

4	May 2013
6	April 2013
6	March 2013
2	February 2013
4	January 2013
4	December 2012
2	November 2012
2	October 2012
9	September 2012
5	August 2012
1	July 2012
8	June 2012
4	May 2012
3	April 2012
2	March 2012
5	January 2012
2	November 2011
6	October 2011
8	August 2011
7	July 2011
7	June 2011
6	May 2011
6	April 2011
1	March 2011
12	January 2011
5	December 2010
5	November 2010
3	October 2010
8	September 2010
25	August 2010
35	July 2010
43	June 2010
52	May 2010
31	April 2010
9	March 2010
11	February 2010
3	January 2010
16	December 2009
6	November 2009
22	October 2009
17	September 2009
16	August 2009
5	July 2009
9	June 2009
6	May 2009
4	April 2009
10	March 2009
8	February 2009
4	January 2009
1	December 2008
1	November 2008
3	October 2008
2	September 2008
5	August 2008
9	July 2008
20	June 2008
23	May 2008
11	April 2008
14	March 2008
8	February 2008
7	January 2008
15	December 2007
33	November 2007
12	October 2007
26	September 2007
27	August 2007
21	July 2007
11	June 2007
13	May 2007
24	April 2007
16	March 2007
12	February 2007
39	January 2007
3	December 2006
11	November 2006
11	October 2006
11	September 2006
12	August 2006
22	July 2006
34	June 2006
31	May 2006
41	April 2006
51	March 2006
24	February 2006
28	January 2006
40	December 2005
30	November 2005
49	October 2005
40	September 2005
15	August 2005
39	July 2005
18	June 2005
6	May 2005
23	April 2005
16	March 2005
16	February 2005
10	January 2005
34	December 2004
11	November 2004
17	October 2004
13	September 2004
42	August 2004
24	July 2004
6	June 2004
11	May 2004
26	April 2004
23	March 2004
3	February 2004
2	January 2002

Mon	Tue	Wed	Thu	Fri	Sat	Sun

			1	2	3	4
5	6	7	8	9	10	11
12	13	14	15	16	17	18
19	20	21	22	23	24	25
26	27	28