Arijit Banerjee | 26 Oct 00:21 2014
Picon

Problem accessing HBase from jobs running from Oozie

Hello all,

With reference to https://issues.apache.org/jira/browse/HBASE-11118.

Mapreduce jobs that programmatically interacts with HBase fails to use
right classes and pulls wrong config resulting in connecting to correct
Zookeeper or throw a protobuf exception. Now this can be solved by setting
HADOOP_CLASSPATH env variable pointing them to right classpath. Now that
workaround does not work while spawning the job from oozie.

We are using CDH5.1.0 with HBase 0.98.1. It seems from HBASE-11118
<https://issues.apache.org/jira/browse/HBASE-11118> that this issue is
fixed as of hbase 0.98.4 . Our application works fine when submitting
directly from commandline after setting the following environment variable
but it fails when spawned via Oozie. We have set the env variable in all
data nodes, hadoop-env.sh for all users but without luck(probably being
overridden somewhere). Wondering if there is any workaround for Hbase
0.98.1 with Oozie 4.0?

export
HADOOP_CLASSPATH=/usr/share/cmf/lib/cdh5/hbase-protocol-0.98.1-cdh5.1.0.jar:/etc/hbase/conf

Regards,
Arijit Banerjee
Flavio Pompermaier | 24 Oct 10:53 2014
Picon

Error during HBase import

Hi all,

I'm trying to import in my HBase 0.98 an export of 0.96 with the following
command
sudo -u hdfs hbase org.apache.hadoop.hbase.mapreduce.Driver import X
/somedir

But I get this error:

Error: java.lang.NullPointerException
at org.apache.hadoop.hbase.client.Mutation.setClusterIds(Mutation.java:331)
at
org.apache.hadoop.hbase.mapreduce.Import$Importer.writeResult(Import.java:180)
at org.apache.hadoop.hbase.mapreduce.Import$Importer.map(Import.java:142)
at org.apache.hadoop.hbase.mapreduce.Import$Importer.map(Import.java:125)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

Any help?

Best,
Flavio
(Continue reading)

Sznajder ForMailingList | 23 Oct 10:27 2014
Picon

Similar Column names in creation of HBase table

Hi

Is it allowed to provide two similar family names during the creation of an
HBase table?

I was expecting that it is forbidden, **but** this call does not throw any
exception

        String[] families = {"cf", "cf"};
        HTableDescriptor desc = new HTableDescriptor(name);
        for (String cf : families) {
          HColumnDescriptor coldef = new HColumnDescriptor(cf);
          desc.addFamily(coldef);
        }
        try {
            admin.createTable(desc);
        } catch (TableExistsException e) {
            throw new IOException("table \'" + name + "\' already exists");
        }
Hayden Marchant | 23 Oct 07:57 2014

Orphaned aborted snapshot

Hi all,

I am running HBase 0.94.6 on a 20 node cluster, and am taking daily snapshots of our single table (only
keeping snapshots for the last 3 days. Yesterday, I started seeing the following messages in one of the
region servers that had to be restarted:

2014-10-22 08:29:19,982 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC Server handler 1 on 60020: starting
2014-10-22 08:29:19,982 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC Server handler 2 on 60020: starting
2014-10-22 08:29:19,986 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Serving as
njhdslave40,60020,1413980958234, RPC listening on njhdslave40/172.30.120.180:60020, sessionid=0x2482ca09984b22a
2014-10-22 08:29:19,986 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker:
SplitLogWorker njhdslave40,60020,1413980958234 starting
2014-10-22 08:29:19,988 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Registered
RegionServer MXBean
2014-10-22 08:29:20,024 INFO org.apache.hadoop.hbase.procedure.ProcedureMember: Received abort
on procedure with no local subprocedure upd-2014_10_19, ignoring it.
org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via
timer-java.util.Timer <at> 2cef133c:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:Timeout
caused Foreign Exception Start:1413728408736, End:1413728468758, diff:60022, max:60000 ms
    at org.apache.hadoop.hbase.errorhandling.ForeignException.deserialize(ForeignException.java:171)
    at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.abort(ZKProcedureMemberRpcs.java:320)
    at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.watchForAbortedProcedures(ZKProcedureMemberRpcs.java:143)
    at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.start(ZKProcedureMemberRpcs.java:340)
    at org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager.start(RegionServerSnapshotManager.java:141)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:734)
    at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:Timeout
caused Foreign Exception Start:1413728408736, End:1413728468758, diff:60022, max:60000 ms
(Continue reading)

ch huang | 23 Oct 03:34 2014
Picon

issue about snapshot migration

hi,maillist:

       i have two hbase cluster ,one is version 0.94.6 another is 0.98.1 ,i
try to mv data of table A from 0.94.6 to 0.98.1 ,i make the snapshot of A
,then want to use ExportSnapshot to mv snapshot to hdfs cluster which hbase
0.98.1 on,but failed ,why?

# hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot demo_shot
-copy-to hdfs://192.168.26.231:8020/hbase
Exception in thread "main" java.io.IOException: Failed on local exception:
com.google.protobuf.InvalidProtocolBufferException: Message missing
required fields: callId, status; Host Details : local host is: "ch11/
192.168.11.11"; destination host is: "hzih231":8020;
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:763)
        at org.apache.hadoop.ipc.Client.call(Client.java:1241)
        at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
        at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
        at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
        at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
        at
(Continue reading)

krish_571 | 23 Oct 02:50 2014
Picon

setup method in Mapreduce Program


I understand that setup and cleanup methods of Mapper/Reducer  are called
for each task, so if you have 20 mappers running, the setup / cleanup will
be called for each one.

Is there a setup like method to for the entire mapreduce program.  Basically
i want to open only  1  connectionto a HBase table for the entire Mapreduce
job so that all map tasks need not open a separate connection.

--
View this message in context: http://apache-hbase.679495.n3.nabble.com/setup-method-in-Mapreduce-Program-tp4065292.html
Sent from the HBase User mailing list archive at Nabble.com.

jeevi tesh | 22 Oct 13:14 2014
Picon

java.lang.IllegalArgumentException: Path must start with / character

Hi,
I'm using hbase0.94.3, hadoop-2.2.0 and jdk 1.7.67 version..
In hbase shell if i issue any command i'm getting
ERROR: java.lang.IllegalArgumentException: Path must start with / character
hbase-site.xml file content
<property>
    <name>hbase.rootdir</name>
        <value>file:////home/hadoop/hbase_root_dir/hbase</value>
          </property>
            <property>
                <name>hbase.zookeeper.property.dataDir</name>

<value>/home/hadoop/zoo_keeper_root_dir/zookeeper</value>
                      </property>
                        <property>
                                <name>hbase.zookeeper.quorum</name>
                                        <value>10.200.1.145</value>
                                          </property>
                                          <property>

<name>zookeeper.znode.parent</name>

<value>file:////home/hadoop/hbase_root_dir/hbdata</value>
kindly help me to resolve the issue
Junegunn Choi | 22 Oct 03:17 2014
Picon

Performance impact of io.file.buffer.size parameter of HDFS

Hi,

I've read that on a modern hardware I should increase the value of
io.file.buffer.size parameter of HDFS, up to 128kb or so. [1] Does this
advice still hold true in the context of HBase? We've done a series of
performance benchmarks with the different values of it, but couldn't
observe a noticeable difference in performance. Is the default 4kb good
enough? What is the recommended value, or maybe it doesn't matter?

Thanks in advance.

[1]: Book: Hadoop - The Definitive Guide
Kevin | 21 Oct 19:26 2014
Picon

HBase client machine with multiple interfaces

Hi,

I have connected a client machine with two network interfaces to an
internal, isolated HBase cluster and an external network. The HBase cluster
is on its own private LAN, away from the external network. After installing
and updating the Hadoop and HBase configuration files on the client
machine, I can run the HBase shell and look at tables, but
TableInputFormatBase can't find any servers using its DNS checks. The
machine is on the DNS for the external network so, from my understanding,
the HBase client API will use this DNS to try and find the HBase cluster. I
know there are master and regionserver DNS interface configurations, but
are there anything for client libraries? My use case is running Pig scripts
from the client machine. Oddly enough, sometimes a Pig script works, other
times (most times) I get the NullPointerException from the
org.apache.hadoop.net.DNS.reverseDns call.

Any ideas?

Thanks,
Kevin
Serega Sheypak | 21 Oct 17:34 2014
Picon

Why snapshots are disabled by default?

Hi, I tried to create a snapshot and it's not enabled

: java.io.IOException: java.lang.UnsupportedOperationException: To use
snapshots, You must add to the hbase-site.xml of the HBase Master:
'hbase.snapshot.enabled' property with value 'true'.

Is it by-default? If yes, then why? What is the reason?
Jan Lukavský | 21 Oct 17:23 2014
Picon

Delete.deleteColumns not working with HFileOutputFormat?

Hi all,

we are using HBase version 0.94.6-cdh4.3.1 and I have a suspicion that a 
Delete written to hbase through HFileOutputFormat might be ignored (and 
not delete any data) in the following scenario:
  * a Delete object is used to delete the data at the client side
  * call to "deleteColumn" instead of "deleteColumns" is used, which 
means that the underlaying KeyValue will not have an associated 
timestamp (will have HConstants.LATEST_TIMESTAMP)
  * the Delete object is then converted to KeyValues and these are 
written into the output format's record writer

I think (our systems seems to behave this way) the problem is in the way 
the KeyValue is processed in the RegionServer, even though I was not 
able to track the problem in the source code. Can anyone else confirm 
this? When using Delete#deleteColumns everything seems to be working 
fine (the KeyValues have different type). Is this expected or should it 
be considered a bug? And if so, where it should be fixed? I think it 
could be on the side of the record writer (maybe by throwing an 
exception), or in the region server (if possible, this might be 
non-trivial, because of the Delete#deleteColumn semantics).

Any opinions?

Thanks,
  Jan


Gmane