Kanagha | 25 Jul 21:37 2016
Picon

Using hbase for transactional applications

Hi,

I am investigating into designing a simple ecommerce application using
Hbase as the database.

I'm planning to use Apache Phoenix and Hbase (backend) and use a Spring
application.

What are the other recommended approaches?

Thanks
Kanagha
Ma, Sheng-Chen (Aven | 22 Jul 10:12 2016
Picon

how to Dynamic load of Coprocessors

Hi all:
I want to dynamic add coprocessor in order to not restart hbase.

Following is my code:
        Path path = new Path("/coprocessor_jars");
        FileSystem fs = FileSystem.get(conf);
        FileStatus[] status = fs.listStatus(path);
        Path[] listedPaths = FileUtil.stat2Paths(status);
        for (Path p : listedPaths) {
            if(p.getName().contains("test.jar")){
                hdfsPath = p;
            }
        }
        HBaseAdmin hadmin = new HBaseAdmin(conf);
        HTableDescriptor tableDesc = hadmin.getTableDescriptor("testTbl".getBytes());

tableDesc.addCoprocessor("org.apache.hadoop.hbase.coprocessor.transactional.TestRegionEndpoint", hdfsPath,
                Coprocessor.PRIORITY_USER, null);
        // tableDesc.removeCoprocessor("org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint");
        for (Entry<ImmutableBytesWritable, ImmutableBytesWritable> entry :
tableDesc.getValues().entrySet()) {
            System.out.println(Bytes.toString(entry.getKey().get()) + " = " + Bytes.toString(entry.getValue().get()));
        }
        hadmin.disableTable("testTbl".getBytes());
        hadmin.modifyTable("testTbl", tableDesc);
        hadmin.enableTable("testTbl");

the syso print : coprocessor$1 = hdfs://192.168.0.17:17400/coprocessor_jars/test.jar|org.apache.hadoop.hbase.coprocessor.transactional.TestRegionEndpoint|1073741823|

and the remote side return Exception:
(Continue reading)

Daniel Vimont | 22 Jul 03:56 2016

[DISCUSSION/ANNOUNCEMENT] availability of ColumnManager for HBase - beta-02 - via GitHub & Maven

The beta-02 release of ColumnManager for HBase is now available on GitHub
and via the Maven Central Repository.

From the top of the README:
==================
ColumnManagerAPI for HBase™ is an extended METADATA REPOSITORY SYSTEM for
HBase with options for:

COLUMN AUDITING -- captures Column metadata (qualifier and max-length*) as
Tables are updated (or via a discovery facility for previously-existing
Tables).
COLUMN-DEFINITION FACILITIES -- optionally enforces
administratively-managed Column definitions (stipulating valid name,
length, and/or value) as Tables are updated, optionally bringing HBase's
"on-the-fly" column-qualifier creation under centralized control.
SCHEMA CHANGE MONITORING -- tracks and provides an audit trail for
structural modifications made to Namespaces, Tables, and Column Families.
SCHEMA EXPORT/IMPORT -- provides schema (metadata) export and import
facilities for HBase Namespace, Table, and all table-component structures.
----
A basic COMMAND-LINE INTERFACE is also provided for direct invocation of a
number of the above-listed functions without any need for Java coding.

* coming soon in beta-03: discovery of number-of-occurrences for each
unique column-qualifier in a table.
==================

We've already had some useful discussions about ColumnManager over on the
HBase Slack channel, but it would be great to get broader
feedback/commentary on this from the hbase-user collective. And it would be
(Continue reading)

Picon

Re: HBASE Install/Configure

Hi Team,
This has been resolved now. I had to add the classpath for hbase/lib to
hadoop-env.sh
Regards
Anil Khiani

--
View this message in context: http://apache-hbase.679495.n3.nabble.com/HBASE-Install-Configure-tp4081233p4081346.html
Sent from the HBase User mailing list archive at Nabble.com.

Sachin Jain | 21 Jul 08:28 2016
Picon
Gravatar

How to get size of Hbase Table

*Context*
I am using Spark (1.5.1) with HBase (1.1.2) to dump the output of Spark
Jobs into HBase which will be further available as lookups from HBase
Table. BaseRelation extends HadoopFSRelation and is used to read and write
to HBase. Spark Default Source API is used.

*Use Case*
Now, whenever I perform join operation, Spark creates a logical plan and
decides which type of join it should execute and as per Spark Strategies
[0] it checks the size of HBase Table. If it is less than some threshold
(10 MB) it selects Broadcast Hash join otherwise Sort Merge join.

*Problem Statement*
I want to know if there is an API or some approach to calculate the size of
an HBase table.

[0]:
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala#L118

Thanks
-Sachin
Jone Zhang | 20 Jul 05:59 2016
Picon

How can i speculate hbase storage space?

There is a  100G date of one billion records.
If i save it to hbase.
What the size of "hadoop fs -ls /user/hbase/data/default/table_name"?

Best wishes.
Saurabh Malviya (samalviy | 20 Jul 00:50 2016
Picon

Hbase Es -Quava lib conflict

Hi,

I am addressing one issue to make Hbase and ES work together in same spark project

https://community.cloudera.com/t5/Storage-Random-Access-HDFS/Apache-HBase-Shaded-Client-1-1-x-with-Cloudera-Hbase-1-0-0-CDH-5/td-p/35505

If anyone have any pointer let me know, Even Hbase-shaded-client does not help.  If I removed quava
reference ""com.google.guava" % "guava" % "18.0"" It will works but Elastic search failed in that case.

Exception in thread "main" java.lang.IllegalAccessError: tried to access method
com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
               at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:596)
               at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:580)
               at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:559)
               at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
               at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1185)
               at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1152)
               at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1126)
               at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1331)
               at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1155)
               at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:370)
               at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:321)
               at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:206)
               at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
               at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1425)
               at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1018)
               at common.TaskTracker$.updateStatus(TaskTracker.scala:55)

Sean Busbey | 19 Jul 21:35 2016
Gravatar

[DISCUSSION] Updating the hbase shell to Ruby 2.y

Hi Folks!

Over on dev <at> hbase we'd like to update the version of JRuby used in the
`hbase shell` command[1]. Right now, the shell runs as Ruby 1.8, which
has been EOM for a very long time (depending on which community you
ask). We could maintain running Ruby 1.8 compatibility for a bit
longer by upgrading our JRuby from the EOM 1.6 to 1.7, however that
version is also hitting EOM this year[2].

Instead, we'd like to upgrade to the latest release of JRuby, which
only runs in Ruby 2.y mode. I believe it defaults to Ruby 2.3
compatibility[3]. Currently, our plan is to do this for both HBase
2.0.0 and for some future version of HBase 1.y (likely 1.4).

While the normal downstream promises allow for this kind of change in
a minor release for 'operational compatibility'[4], we recognize that
going across 2 major Ruby versions (1.8 -> 1.9, and 1.9 -> 2.2) might
be a considerable burden.

What do folks who are running HBase now think? Do you have a
significant body of Ruby 1.8 tooling in place? Is planning for the end
of this year (likely the soonest a 1.4 release will happen) enough for
updating it?

[1]: https://issues.apache.org/jira/browse/HBASE-16196
[2]: https://twitter.com/jruby/status/751150368839589888
[3]: http://jruby.org/2016/05/27/jruby-9-1-2-0.html
[4]: http://hbase.apache.org/book.html#hbase.versioning.post10

--

-- 
(Continue reading)

tombin | 19 Jul 20:30 2016
Picon

new base cluster help

I am configuring a hbase cluster for the first time.

The issue I am seeing is, when i setup a single server acting as master and
region server, everything seems to be working fine.  If i start region
servers on 2 other boxes, in the base ui i see duplicate region servers.  I
have read that this can be normal, and base will eventually "sort things
out".  But this is showing the ip (with unknown version) and hostname (with
correct base version),  and showing an error of 2 nodes with inconsistent
version.

I also see an error on the UI that one of the regions is in transition and
state=PENDING_OPEN.

I looked at the logs files and see the following error:

2016-07-18 20:38:10,329 WARN  [RS_OPEN_META-hb02:16020-0]
zookeeper.ZKAssign: regionserver:16020-0x255e5cd7f190042, quorum=
zk01.domain.com:2181,zk02.corp.lumsb.com:2181,zk03.domain.com:2181,
baseZNode=/hbase Attempt to transition the unassigned node for 1588230740
from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING failed, the server that
tried to transition was 10.110.2.128,16020,1468874283476 not the expected
hb02.domain.com,16020,1468874283476

This makes me think the duplicate error and the region transition errors
may be related.  I'm running the same hbase-site.xml on every node.  I'm
very confused why this just works with single server, but when i do extra
servers, it fails.  Any ideas?

Thank you in advance.
(Continue reading)

Yu Wei | 19 Jul 18:29 2016
Picon

Is it good choice to access hbase via DAO?

Hi guys,

I'm a little knowledge about hbase.

Is it good choice to access hbase via DAO similar as working with RDBMS?

Any advice?

Thanks,

Jared
Michal Medvecky | 18 Jul 21:34 2016

Cache of region boundaries are out of date.

Hello,

I have ~1B table and trying to select some columns. I get this exception:

java.lang.RuntimeException:
org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 1108
(XCL08): Cache of region boundaries are out of date.
        at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
        at sqlline.TableOutputFormat.print(TableOutputFormat.java:33)
        at sqlline.SqlLine.print(SqlLine.java:1653)
        at sqlline.Commands.execute(Commands.java:833)
        at sqlline.Commands.sql(Commands.java:732)
        at sqlline.SqlLine.dispatch(SqlLine.java:808)
        at sqlline.SqlLine.begin(SqlLine.java:681)
        at sqlline.SqlLine.start(SqlLine.java:398)
        at sqlline.SqlLine.main(SqlLine.java:292)

I tried DELETE FROM SYSTEM.STATS WHERE PHYSICAL_NAME='MEDIA';

but did not help.

Any suggestions?

Thank you,

Michal

Gmane