Ted Yu | 23 Jul 00:07 2014
Picon

Re: HBase appends

When storing new lists using new columns, similar issue would arise, right ?
In Ishan's words:

bq. read all the columns and combine when reading

The combining process applies to the multi-version approach as well.

Cheers

On Tue, Jul 22, 2014 at 12:32 PM, Arun Allamsetty <arun.allamsetty@...
> wrote:

> Hi,
>
> Isn't versioning used for an entirely different purpose? What if I screw up
> a book name and then have to rewrite it? Then I'll have two versions for
> the same book. Also, AFAIK the default number of versions is 1 on table
> creation without additional parameters.
>
> Thanks,
> Arun
> On Jul 22, 2014 12:11 PM, "yonghu" <yongyong313@...> wrote:
>
> > Hi,
> >
> > If a author does not have hundreds of publications, you can directly
> write
> > in one column. Hence, your column will contain multiple data versions.
> The
> > default data version is 3 but you can send more.
(Continue reading)

Guangle Fan | 22 Jul 22:24 2014
Picon

hfile 9.4 to 9.6

Hi, All

I have hfiles in HBase 9.4 cluster that I want to migrate to HBase 9.6
cluster.

What's the proper process to handle this ?

Regards,

Guangle
Tianying Chang | 22 Jul 17:15 2014
Picon

WALPlayer kills many RS when play large number of WALs

Hi

I was running WALPlayer that output HFile for future bulkload. There are
6200 hlogs, and the total size is about 400G.

The mapreduce job finished. But I saw two bad things:
1. More than half of RS died. I checked the syslog, it seems they are
killed by OOM. They also have very high CPU spike for the whole time during
WALPlayer

cpu user usage of 84.4% matches resource limit [cpu user usage>70.0%]

2. Mapreduce job also has failure of Java heap Space error. My job set the
heap usage as 2G,
*mapred.child.java.opts*-Xmx2048m
Does this mean WALPlayer cannot support this load on this kind of setting?

Thanks
Tian-Ying
Picon

how can I use sqoop transfer data from mysql to hbase


   Hi, guys
           In the condition that I have built the enviroment of Hadoop, how can I use sqoop transfer data from mysql.
     I try some methods such as using some scripts but it doesn't work.Please help me if you used it ago.Thaks a lot.

   Best Regards
   Amanda




++++++++++++++++++++++++++++
相晓敏
华东师范大学信息学院计算机系应用所
lanlingxue09 <at> gmail.com
++++++++++++++++++++++++++++
Павел Мезенцев | 22 Jul 12:59 2014

hbase cluster working bad

Hello all!

We have a trouble with hbase
Our hadoop cluster has 4 nodes (plus 1 client node).
There are CHD 4.6 + CM 4.7 hadoop installed
Hadoop versions are:
 - hadoop-hdfs : 2.0.0+1475
 - hadoop-0.20-mapreduce : 2.0.0+1475
 - hbase" : 0.94.6+132
Hadoop and hBase configs are in attachment

We have several tables in hbase with total volume of 2 Tb.
We run mapReduce ETL jobs and analytics queries over them.

There are a lot of warnings like
- The health test result for REGION_SERVER_READ_LATENCY has become bad: The moving average of HDFS read latency is 162 millisecond(s) over the previous 5 minute(s). Critical threshold: 100. 
- The health test result for REGION_SERVER_SYNC_LATENCY has become bad: The moving average of HDFS sync latency is 8.2 second(s) over the previous 5 minute(s). Critical threshold: 5,000. 
- HBase region health: 442 unhealthy regions 
- HDFS_DATA_NODES_HEALTHY has become bad
- HBase Region Health Canary is running slowly on the cluster

mapReduce jobs over hBase with random queries to hBase working very slowly (job is completed on 20% after 18 hours versus 100% after 12 hours on analogue cluster)

Please help use to solve reasons of this alerts and speed up the cluster. 
Could you give us a good advise, what shall we do?

Cheers,
Mezentsev Pavel

Attachment (hbase1-clientconfig.zip): application/zip, 3306 bytes
Attachment (hdfs1-clientconfig.zip): application/zip, 3562 bytes
ch huang | 22 Jul 11:09 2014
Picon

issue about importtsv tool bulk output function

hi,maillist:
         i view the code of importTsv in hbase 0.96, when it do bulk output
,it still use  HFileOutputFormat not HFileOutputFormat2 ,however ,in hbase
0.96 ,it dose not support HFileV1,
Is it a bug of importTsv ?
ch huang | 22 Jul 10:01 2014
Picon

issue about testing importtsv with other field separator

hi,maillist:

i test hbase 0.96.1.1 importtsv tool ,find it do not work with non tab
field separator

# sudo -u hdfs hbase org.apache.hadoop.hbase.mapreduce.ImportTsv
-Dimporttsv.columns=HBASE_ROW_KEY,myco1,mycol2

"-Dmporttsv.separator=|" alex:mymy2 /tmp/alex_test

2014-07-22 15:55:59,746 INFO  [main] mapreduce.Job: Counters: 31
        File System Counters
                FILE: Number of bytes read=0
                FILE: Number of bytes written=113939
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=106
                HDFS: Number of bytes written=0
                HDFS: Number of read operations=2
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=0
        Job Counters
                Launched map tasks=1
                Data-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=2160
                Total time spent by all reduces in occupied slots (ms)=0
                Total time spent by all map tasks (ms)=2160
                Total vcore-seconds taken by all map tasks=2160
                Total megabyte-seconds taken by all map tasks=4423680
        Map-Reduce Framework
                Map input records=1
                Map output records=0
                Input split bytes=96
                Spilled Records=0
                Failed Shuffles=0
                Merged Map outputs=0
                GC time elapsed (ms)=18
                CPU time spent (ms)=190
                Physical memory (bytes) snapshot=278962176
                Virtual memory (bytes) snapshot=2180153344
                Total committed heap usage (bytes)=502267904
        ImportTsv
                Bad Lines=1                        #  why bad lines?
        File Input Format Counters
                Bytes Read=10
        File Output Format Counters
                Bytes Written=0

hbase(main):015:0> scan 'alex:mymy2'
ROW                                           COLUMN+CELL
0 row(s) in 0.0030 seconds

# hadoop fs -cat /tmp/alex_test
aa|bb|dd
ch huang | 22 Jul 08:35 2014
Picon

issue about import hbase table data to different namespace

hi,maillist:

        i try import data to 0.96 hbase ,but when i use import tool ,it can
import data to default namespace, how can i import it to diffent namespace?
Vimal Jain | 22 Jul 07:51 2014
Picon

Replication in Hbase

Hi,
I have 2 Hbase clusters setup in different data center.
Both are configured in pseudo-distributed mode.
I followed the steps in Hbase Replication
<https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/replication/package-summary.html#requirements>
.
But i am getting following logs in master cluster's region server log.

2014-07-22 11:19:19,186 DEBUG
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Opening
log for replication ip-10-14-24-19%2C60020%2C1405945008796.1406006236991 at
134
2014-07-22 11:19:19,193 DEBUG
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
currentNbOperations:0 and seenEntries:0 and size: 0
2014-07-22 11:19:19,193 DEBUG
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Nothing
to replicate, sleeping 1000 times 10

What i am missing here ?
Also in one of the requirements mentioned in the above post , it says
zookeeper should not be managed by Hbase.But i have Hbase managing
zookeeper in both clusters.

Please help here.
Basically , i want to copy all data from one cluster to another which are
geographically distant.

--

-- 
Thanks and Regards,
Vimal Jain
Ishan Chhabra | 22 Jul 04:12 2014

Re: map reduce become much slower when upgrading from 0.94.11 to 0.96.2-hadoop1

Adding back user <at> hbase.

On Mon, Jul 21, 2014 at 7:11 PM, Ishan Chhabra <ichhabra@...>
wrote:

> Remove the line:
>
> urldbScan.setCaching(5000);
>
> and add:
>
> TableMapReduceUtil.setScannerCaching(job, 5000);
>
>
> On Mon, Jul 21, 2014 at 6:58 PM, Li Li <fancyerii@...> wrote:
>
>> it seems we confront this problem. But after reading this issue, I
>> still don't know how to solve it. could you please give me some sample
>> codes?
>> my codes are as follows, what shoud I do in hbase 0.96?
>> List<Scan> scans = new ArrayList<Scan>();
>> Scan urldbScan=new Scan();
>> urldbScan.setCaching(5000);
>> urldbScan.setCacheBlocks(false);
>> urldbScan.setAttribute(Scan.SCAN_ATTRIBUTES_TABLE_NAME,
>> HbaseTools.TB_URL_DB_BT);
>> urldbScan.addFamily(HbaseTools.CF_BT);
>> scans.add(urldbScan);
>> Scan outLinkScan=new Scan();
>> outLinkScan.setCaching(5000);
>> outLinkScan.setCacheBlocks(false);
>> outLinkScan.setAttribute(Scan.SCAN_ATTRIBUTES_TABLE_NAME,
>> HbaseTools.TB_OUT_LINK_BT);
>> outLinkScan.addFamily(HbaseTools.CF_BT);
>> scans.add(outLinkScan);
>> TableMapReduceUtil.initTableMapperJob(scans, Step1Mapper.class,
>> BytesWritable.class,
>> ScheduleData.class, job);
>>
>> On Tue, Jul 22, 2014 at 8:14 AM, Ishan Chhabra <ichhabra@...>
>> wrote:
>> > You might be affected by this:
>> > https://issues.apache.org/jira/browse/HBASE-11558
>> >
>> >
>> > On Wed, Jun 25, 2014 at 4:18 PM, Ishan Chhabra <ichhabra@...
>> >
>> > wrote:
>> >>
>> >> Li Li,
>> >> Were you able to figure out the cause of this? I am seeing something
>> >> similar.
>> >>
>> >>
>> >> On Wed, May 7, 2014 at 10:50 PM, Li Li <fancyerii@...> wrote:
>> >>>
>> >>> today I upgraded hbase 0.94.11 to 0.96.2-hadoop1. I have not changed
>> >>> any client codes except replace 0.94.11 client jar to 0.96.2 's
>> >>> When with old version. when doing mapreduce task. the requests per
>> >>> seconds is about 10,000. But with new one, the value is 300. What's
>> >>> wrong with it?
>> >>> The hbase put and get is fast and Request Per Second is larger than
>> 5,000
>> >>>
>> >>> my codes:
>> >>> List<Scan> scans = new ArrayList<Scan>();
>> >>> Scan urldbScan=new Scan();
>> >>> urldbScan.setCaching(5000);
>> >>> urldbScan.setCacheBlocks(false);
>> >>> urldbScan.setAttribute(Scan.SCAN_ATTRIBUTES_TABLE_NAME,
>> >>> HbaseTools.TB_URL_DB_BT);
>> >>> urldbScan.addFamily(HbaseTools.CF_BT);
>> >>> scans.add(urldbScan);
>> >>> Scan outLinkScan=new Scan();
>> >>> outLinkScan.setCaching(5000);
>> >>> outLinkScan.setCacheBlocks(false);
>> >>> outLinkScan.setAttribute(Scan.SCAN_ATTRIBUTES_TABLE_NAME,
>> >>> HbaseTools.TB_OUT_LINK_BT);
>> >>> outLinkScan.addFamily(HbaseTools.CF_BT);
>> >>> scans.add(outLinkScan);
>> >>> TableMapReduceUtil.initTableMapperJob(scans, Step1Mapper.class,
>> >>> BytesWritable.class,
>> >>> ScheduleData.class, job);
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Ishan Chhabra | Rocket Scientist | RocketFuel Inc.
>> >
>> >
>> >
>> >
>> > --
>> > Ishan Chhabra | Rocket Scientist | RocketFuel Inc.
>>
>
>
>
> --
> *Ishan Chhabra *| Rocket Scientist | RocketFuel Inc.
>

--

-- 
*Ishan Chhabra *| Rocket Scientist | RocketFuel Inc.
Arun Allamsetty | 22 Jul 02:43 2014
Picon

HBase appends

Hi,

If I have a one-to-many relationship in a SQL database (an author might
have written many books), and I want to denormalize it for writing in
HBase, I'll have a table with the Author as the row key and a *list* of
books as values.

Now my question is how do I create a *list* such that I could just append
to it using the HBase Java API *Append* instead of doing a
read-modify-insert on a Java List object containing all the books.

Thanks,
Arun

Gmane