Matthew LeMieux | 1 Sep 02:19 2010

Log messages galore - urgent help recovering

I've been very happy with HBase, and am very much looking forward to more stable releases in the future.   
Today, I had another one of those unfortunate crashes that seems to occur every few days and need some help
understanding how I can speed up the recovery, which is taking longer than usual.   I'm running on CDH3.  

Right now, I'm getting log messages printed out at a rate of 100's / second in the master log file.  

They start with: "2010-08-31 23:55:15,886 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_PROCESS_OPEN:"

And end with:  "a of b"

Where a counts up to b each second.  I seem to remember that I used to see b count down during a previous recover. 
So, for example, I might get 200 messages one second with lines ending in "1 of 200", "2 of 200", ... "200 of
200".  Then the next second  b might be 199, so the lines would end in "1 of 199", "2 of 199", ....  "199 of 199". 

Unfortunately, right now, b seems to stay constant at 148 for a half hour.   The only work HBase appears to be
doing is printing hundreds of log messages.  

It says all the region servers are online.  DFS is healthy with proper replication.  The machines are under
low load, having no other jobs or services running on them.  Region servers have either 4 or 6 GB allocated to
them. The machines appear to all have CPU utilization of under 15%.  

Not all of the region servers are showing progress... on at least one of them I can see messages of the form: 

"2010-09-01 00:14:35,209 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:"

These are appearing VERY SLOWLY, and other region servers appear to be completely idle while this is going
on.  

I really need some help to get things back up and running.  I have people who are waiting to get work done.  
(Continue reading)

Ryan Rawson | 1 Sep 02:23 2010
Picon

Re: Log messages galore - urgent help recovering

When you have a master that is confused like this, you can try this:

- make sure it isnt undergoing log splitting
- kill -9 the master
- restart the master

the startup code will check the cluster status then take appropriate action.

There is a new master which will make all this junk better very soon.
Thanks for putting up with it :-)

-ryan

On Tue, Aug 31, 2010 at 5:19 PM, Matthew LeMieux <mdl@...> wrote:
> I've been very happy with HBase, and am very much looking forward to more stable releases in the future.  
 Today, I had another one of those unfortunate crashes that seems to occur every few days and need some
help understanding how I can speed up the recovery, which is taking longer than usual.   I'm running on CDH3.
>
> Right now, I'm getting log messages printed out at a rate of 100's / second in the master log file.
>
> They start with: "2010-08-31 23:55:15,886 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_PROCESS_OPEN:"
>
> And end with:  "a of b"
>
> Where a counts up to b each second.  I seem to remember that I used to see b count down during a previous
recover.  So, for example, I might get 200 messages one second with lines ending in "1 of 200", "2 of 200",
... "200 of 200".  Then the next second  b might be 199, so the lines would end in "1 of 199", "2 of 199", ....
 "199 of 199".
>
(Continue reading)

Jean-Daniel Cryans | 1 Sep 02:27 2010
Picon

Re: Log messages galore - urgent help recovering

One thing you can do is to kill -9 the master process, then restart it
with bin/hbase-daemon.sh start master

This will clear the master state and it will inspect the cluster when
restarting to figure where things are.

If that doesn't work you can also restart HBase completely.

Are the region servers even able to open the regions? Any exceptions?
Can you show us some logs perhaps? Do use a service like pastebin or
put them on some web server.

It's verbose in this case because there are a lot of regions to
assign, and for debugging purposes (like right now) we need to be able
to trace the movements of every region.

J-D

On Tue, Aug 31, 2010 at 5:19 PM, Matthew LeMieux <mdl@...> wrote:
> I've been very happy with HBase, and am very much looking forward to more stable releases in the future.  
 Today, I had another one of those unfortunate crashes that seems to occur every few days and need some
help understanding how I can speed up the recovery, which is taking longer than usual.   I'm running on CDH3.
>
> Right now, I'm getting log messages printed out at a rate of 100's / second in the master log file.
>
> They start with: "2010-08-31 23:55:15,886 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_PROCESS_OPEN:"
>
> And end with:  "a of b"
>
(Continue reading)

Shuja Rehman | 1 Sep 02:30 2010
Picon

Re: Getting data from Hbase from client/remote computer

Stack,

I have checked the example on the given link

http://hbase.apache.org/docs/r0.89.20100726/apidocs/org/apache/hadoop/hbase/client/package-summary.html#package_description

<http://hbase.apache.org/docs/r0.89.20100726/apidocs/org/apache/hadoop/hbase/client/package-summary.html#package_description>but
it is same as on the other documentation. Secondly I am running hbase in
Distributed mode. I am confused with the term Zookeeper ensemble. Do you
mean that i set the configuration like this

config.set("hbase.zookeeper.quorum", "myserver.mycompany.com:2181");

If yes, then i have tried in this way also but agian getting exceptions when
I write this line

 HTable table = new HTable(config, "myTable");

The Exception details are

Aug 31, 2010 7:19:24 PM org.apache.hadoop.hbase.HBaseConfiguration <init>
WARNING: instantiating HBaseConfiguration() is deprecated. Please use
HBaseConfiguration#create() to construct a plain Configuration
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/zookeeper/KeeperException
        at
org.apache.hadoop.hbase.client.HConnectionManager.getClientZooKeeperWatcher(HConnectionManager.java:176)
        at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getZooKeeperWrapper(HConnectionManager.java:1048)
        at
(Continue reading)

Shuja Rehman | 1 Sep 03:29 2010
Picon

Re: Getting data from Hbase from client/remote computer

Hi Stack

I have solved this problem
Thanks
On Wed, Sep 1, 2010 at 5:30 AM, Shuja Rehman <shujamughal@...> wrote:

> Stack,
>
> I have checked the example on the given link
>
>
> http://hbase.apache.org/docs/r0.89.20100726/apidocs/org/apache/hadoop/hbase/client/package-summary.html#package_description
>
>
> <http://hbase.apache.org/docs/r0.89.20100726/apidocs/org/apache/hadoop/hbase/client/package-summary.html#package_description>but
> it is same as on the other documentation. Secondly I am running hbase in
> Distributed mode. I am confused with the term Zookeeper ensemble. Do you
> mean that i set the configuration like this
>
> config.set("hbase.zookeeper.quorum", "myserver.mycompany.com:2181");
>
> If yes, then i have tried in this way also but agian getting exceptions
> when I write this line
>
>  HTable table = new HTable(config, "myTable");
>
> The Exception details are
>
> Aug 31, 2010 7:19:24 PM org.apache.hadoop.hbase.HBaseConfiguration <init>
> WARNING: instantiating HBaseConfiguration() is deprecated. Please use
(Continue reading)

Imran M Yousuf | 1 Sep 03:37 2010
Picon

Re: Getting data from Hbase from client/remote computer

Hi Shuja-Ur-Rahman,

It would be nice if you would mention how you solved it in the mailing
list so that if someone else runs into the same problem they know how
to solve it.

/Imran

On Wed, Sep 1, 2010 at 7:29 AM, Shuja Rehman <shujamughal@...> wrote:
> Hi Stack
>
> I have solved this problem
> Thanks
> On Wed, Sep 1, 2010 at 5:30 AM, Shuja Rehman <shujamughal@...> wrote:
>
>> Stack,
>>
>> I have checked the example on the given link
>>
>>
>> http://hbase.apache.org/docs/r0.89.20100726/apidocs/org/apache/hadoop/hbase/client/package-summary.html#package_description
>>
>>
>> <http://hbase.apache.org/docs/r0.89.20100726/apidocs/org/apache/hadoop/hbase/client/package-summary.html#package_description>but
>> it is same as on the other documentation. Secondly I am running hbase in
>> Distributed mode. I am confused with the term Zookeeper ensemble. Do you
>> mean that i set the configuration like this
>>
>> config.set("hbase.zookeeper.quorum", "myserver.mycompany.com:2181");
>>
(Continue reading)

Shuja Rehman | 1 Sep 03:49 2010
Picon

Re: Getting data from Hbase from client/remote computer

Hi Imran

I have used these configuration settings and include the zoo keeper and
log4j jars also in class path.

HBaseConfiguration config = new HBaseConfiguration();
  config.clear();
  config.set("hbase.zookeeper.quorum", "myserver.mycompany.com:2181");
  config.set("hbase.zookeeper.property.clientPort","2181");

It resolve the previous problem but now new problem occurs. The new problem
is its trying to connect to local ip address

INFO zookeeper.ZooKeeper: Initiating client connection, connectString=
myserver.mycompany.com:2181 sessionTimeout=60000
watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher <at> fd54d6
10/08/31 20:27:33 INFO zookeeper.ClientCnxn: Opening socket connection to
server myserver.mycompany.com/20314.166.12:2181
10/08/31 20:27:33 INFO zookeeper.ClientCnxn: Socket connection established
to myserver.mycompany.com/203.14.166.12:2181, initiating session
10/08/31 20:27:34 INFO zookeeper.ClientCnxn: Session establishment complete
on server myserver.mycompany.com/203.14.166.12:2181,  sessionid =
0x12acaa705e30004, negotiated timeout = 40000

10/08/31 20:27:54 INFO ipc.HbaseRPC: Problem connecting to server: /
10.0.3.85:60020
10/08/31 20:28:15 INFO ipc.HbaseRPC: Problem connecting to server: /
10.0.3.85:60020
10/08/31 20:28:36 INFO ipc.HbaseRPC: Problem connecting to server: /
10.0.3.85:60020
(Continue reading)

Stack | 1 Sep 05:27 2010
Picon

Re: Getting data from Hbase from client/remote computer

On Tue, Aug 31, 2010 at 5:30 PM, Shuja Rehman <shujamughal@...> wrote:
> HBaseConfiguration#create() to construct a plain Configuration
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/zookeeper/KeeperException

As it says, NoClassDefFoundError.  Try adding zookeeper jar to your CLASSPATH.
St.Ack

Kelvin Rawls | 1 Sep 05:48 2010

RE: Getting data from Hbase from client/remote computer

Shuja

No real magic code here, Google JMX Tutorial and take any hello world JMX example - gut method and change
signature as needed - add HBase access code from any working example - get JMX client from tutorial mod as
required and it just works. 

We are up against a deadline here - but if there is lots of interest I can put together a complete simple
example later -  can't post current app yet ... 

Kelvin
________________________________________
From: Shuja Rehman [shujamughal@...]
Sent: Tuesday, August 31, 2010 3:55 PM
To: user@...
Subject: Re: Getting data from Hbase from client/remote computer

Kelvin

Can you give me some more details or some reference links from where i can
learn how to use JMXBean stuff to connect to master server and then perform
operations on hbase.

Thanks

On Tue, Aug 31, 2010 at 8:58 PM, Kelvin Rawls <kelvin@...> wrote:

> Shuja:
>
> getting back to your original question, we are using JMXBeans and Locater
> code that return references to the proxies for the JMXBeans to allow remote
(Continue reading)

Bradford Stephens | 1 Sep 06:38 2010
Picon

JSONP and Stargate

Hey homies,

I'm trying to write some JavaScript (which I know little about) to
pull data out of HBase via Stargate via jQuery. To get around the
"Single Origin Policy", I'm trying to do gets by using JSONP, which
embeds/retrieves requests in <script> tags. This seems to be a
relatively common way to get around the issue...I think.

Does Stargate have support for something like that, or should I patch it in?

My JS-fu is weak, so if I'm missing something big, let me know.

--

-- 
Bradford Stephens,
Founder, Drawn to Scale
drawntoscalehq.com
727.697.7528

http://www.drawntoscalehq.com --  The intuitive, cloud-scale data
solution. Process, store, query, search, and serve all your data.

http://www.roadtofailure.com -- The Fringes of Scalability, Social
Media, and Computer Science


Gmane