headers
Jean-Daniel Cryans | 1 Mar 2011 01:05
Picon
Favicon

Re: Question on upgrade chapter

Actually, the upgrade doc is wrong. It's 16KB not 16MB so that value
is right, and it's the one you want for 0.20.6 since it lowers greatly
the chance of dataloss for .META. which is pretty catastrophic.

Only change it when you do upgrade.

J-D

On Mon, Feb 28, 2011 at 3:59 PM, Buttler, David <buttler1@...> wrote:
> Hi Guys,
> Thanks for the clarifications.  I did indeed mistype what I did.  I scanned -ROOT- (if you are curious,
it is appended below)
>
> Since I am still on 0.20.6 until I figure out how to make my code work with 0.90.1 on my test cluster, how
should I go about fixing the MEMSTORE_FLUSHSIZE?
>
> Thanks for the help,
> Dave
>
>
>
>
>
>
>
>
>
>
> ---
> hbase(main):004:0> scan '-ROOT-'
(Continue reading)

Permalink | Reply | 0 comments |
headers
Nanheng Wu | 1 Mar 2011 02:04
Picon

Re: Disabling a table taking very long time

I issued two more disable commands and looked at the .META RS. The
last log entry looks like:

11/03/01 00:29:14 INFO regionserver.HLog: Roll
/hbase/.logs/<RS_HOSTNAME>,60020,1298935751670/hlog
.dat.1298935752660, entries=13527, calcsize=3839046, filesize=2560143.
New hlog /hbase/.logs/<RS_HOSTNAME>,60020,1298935751670/hlog.dat.1298939354072

I issued the command at: 00:34:59, the client debug message has
nothing but "11/03/01 00:34:59 DEBUG zookeeper.ClientCnxn: Got ping
response for sessionid:0x12e6e938e9c000b after 90ms" until 00:47:19.
Then all of a sudden it outputs the following:

0:47:28 DEBUG client.HConnectionManager$TableServers: Cache hit for
row <> in tableName .META.: location server 10.146.14.38:60020,
location region name .META.,,1
11/03/01 00:47:28 DEBUG zookeeper.ZooKeeperWrapper: Read ZNode
/hbase/root-region-server got 10.146.14.38:60020
11/03/01 00:47:28 DEBUG client.HConnectionManager$TableServers: Found
ROOT at 10.146.14.38:60020
11/03/01 00:47:28 DEBUG client.HConnectionManager$TableServers: Cached
location for .META.,,1 is 10.146.14.38:60020
11/03/01 00:47:30 DEBUG client.HConnectionManager$TableServers:
Rowscanned=261, rowsOffline=3
11/03/01 00:47:30 DEBUG client.HBaseAdmin: Sleep. Waiting for all
regions to be disabled from myTable
11/03/01 00:47:31 DEBUG client.HBaseAdmin: Wake. Waiting for all
regions to be disabled from myTable

Here's the full thread dump of the .META RS from 00:34:59 to  00:47:31
(Continue reading)

Permalink | Reply | 10 comments |
headers
Jean-Daniel Cryans | 1 Mar 2011 02:17
Picon
Favicon

Re: Disabling a table taking very long time

I didn't get most of the information I was looking for... but I do
have something to work with. So I asked whether the metaScanner was
making progress or not, and from that log I think I see it does:

> 11/03/01 00:47:30 DEBUG client.HConnectionManager$TableServers:
> Rowscanned=261, rowsOffline=3
> 11/03/01 00:47:30 DEBUG client.HBaseAdmin: Sleep. Waiting for all
> regions to be disabled from myTable

Only 3 rows are offline, whereas we expect all the region to be
offline. I expect you see those lines a lot? If so, one thing I'd like
to see is the result of this command:

scan '.META.', {STARTROW => "myTable,,", LIMIT => 261}

It's going to be big. Then grep in the result for the string SPLIT,
and please post back here the lines that match.

J-D

On Mon, Feb 28, 2011 at 5:04 PM, Nanheng Wu <nanhengwu@...> wrote:
> I issued two more disable commands and looked at the .META RS. The
> last log entry looks like:
>
> 11/03/01 00:29:14 INFO regionserver.HLog: Roll
> /hbase/.logs/<RS_HOSTNAME>,60020,1298935751670/hlog
> .dat.1298935752660, entries=13527, calcsize=3839046, filesize=2560143.
> New hlog /hbase/.logs/<RS_HOSTNAME>,60020,1298935751670/hlog.dat.1298939354072
>
> I issued the command at: 00:34:59, the client debug message has
(Continue reading)

Permalink | Reply | 9 comments |
headers
Edward Choi | 1 Mar 2011 03:19
Picon

Re: How to make a CGI with HBase?

Thanks for the reply. 
Didn't know that Thrift was for such purpose. 
Servlet and JSP is totally new to me. I skimmed through the concept on the internet and they look
fascinating. 
I think I am gonna give servlet and JSP a try. 

On 2011. 3. 1., at 오전 12:51, "Usman Waheed" <usmanw@...> wrote:
> HI,
> 
> I have been using the Thrift Perl API to connect to Hbase for my web app. At the moment i only perform random
reads and scans based on date ranges and some other search criteria.
> It works and I am still testing performance.
> 
> -Usman
> 
>> On Mon, Feb 28, 2011 at 2:37 PM, edward choi <mp2893@...> wrote:
>>> Hi,
>>> 
>>> I am planning to make a search engine for news articles. It will probably
>>> have over several billions of news articles so I thought HBase is the way to
>>> go.
>>> 
>>> However, I am very new to CGI. All I know is that you use php, python or
>>> java script with HTML to make a web site and communicate with the backend
>>> database such as MySQL.
>>> 
>>> But I am going to use HBase, not MySQL, and I can't seem to find a script
>>> language that provides any form of API to communicate with HBase.
>>> 
>>> So what do I do?
(Continue reading)

Permalink | Reply | 4 comments |
headers
Bryan Keller | 1 Mar 2011 03:33
Picon

HBase Java client dependencies

In my application I'm using the Java HBase API. I'm using Maven (well, Gradle actually) to declare a
dependency on HBase. Unfortunately, the dependency on HBase drags every transitive dependency but the
kitchen sink into my app, including Ant, Jasper, Jetty, and others. I am hoping not all of these are needed
just for the Java HBase client.

Is there a list somewhere of the minimal set of jars needed for using the Java HBase client? If so, I could just
declare dependencies on these non-transitively.
Permalink | Reply | 1 comment |
headers
陈加俊 | 1 Mar 2011 04:07
Picon

Re: java.lang.OutOfMemoryError: Java heap space

I reassigned 4G heap to each RS and add one more RS to the cluster. Now the
cluster is as follows.I noticed the this line :requests=7, regions=813,
usedHeap=3823, maxHeap=3991,Is it dangerous signs?

-------------------------------2011-03-01 11:01:33
Hbase Version: 0.20.6
runing : yes
AverageLoad: 797.0

Dead: 0

Live Servers: 13

1-> 192.168.5.157:60020 [requests=44, regions=812, usedHeap=3082,
maxHeap=3991]
2-> 192.168.5.153:60020 [requests=7, regions=813, usedHeap=3823,
maxHeap=3991]
3-> 192.168.5.148:60020 [requests=63, regions=814, usedHeap=3670,
maxHeap=3995]
4-> 192.168.5.150:60020 [requests=157, regions=816, usedHeap=3185,
maxHeap=3995]
5-> 192.168.5.145:60020 [requests=103, regions=582, usedHeap=2683,
maxHeap=3995]
6-> 192.168.5.156:60020 [requests=36, regions=819, usedHeap=2987,
maxHeap=3991]
7-> 192.168.5.155:60020 [requests=14, regions=812, usedHeap=3458,
maxHeap=3991]
8-> 192.168.5.146:60020 [requests=35, regions=822, usedHeap=2483,
maxHeap=3995]
9-> 192.168.5.152:60020 [requests=10, regions=813, usedHeap=3897,
(Continue reading)

Permalink | Reply | 1 comment |
headers
Stack | 1 Mar 2011 04:56

Re: HBase Java client dependencies

There are other answers to this question but here is one: hbase,
hadoop, zookeeper and the hbase conf directory which has your
configuration in it.
St.Ack

On Mon, Feb 28, 2011 at 6:33 PM, Bryan Keller <bryanck@...> wrote:
> In my application I'm using the Java HBase API. I'm using Maven (well, Gradle actually) to declare a
dependency on HBase. Unfortunately, the dependency on HBase drags every transitive dependency but the
kitchen sink into my app, including Ant, Jasper, Jetty, and others. I am hoping not all of these are needed
just for the Java HBase client.
>
> Is there a list somewhere of the minimal set of jars needed for using the Java HBase client? If so, I could
just declare dependencies on these non-transitively.
>
>
Permalink | Reply | 0 comments |
headers
Stack | 1 Mar 2011 04:59

Re: java.lang.OutOfMemoryError: Java heap space

Thats a lot of regions per regionserver.  You should up the size of
your regions if they are default sized.  But look in your regionserver
UI.  That'll tell you a better story regards how much of your heap is
given over to keeping storefile indices in memory.  Can you give hbase
more than this 4G?

St.Ack

On Mon, Feb 28, 2011 at 7:07 PM, 陈加俊 <cjjvictory <at> gmail.com> wrote:
> I reassigned 4G heap to each RS and add one more RS to the cluster. Now the
> cluster is as follows.I noticed the this line :requests=7, regions=813,
> usedHeap=3823, maxHeap=3991,Is it dangerous signs?
> -------------------------------2011-03-01 11:01:33
> Hbase Version: 0.20.6
> runing : yes
> AverageLoad: 797.0
>
> Dead: 0
>
>
> Live Servers: 13
>
> 1-> 192.168.5.157:60020 [requests=44, regions=812, usedHeap=3082,
> maxHeap=3991]
> 2-> 192.168.5.153:60020 [requests=7, regions=813, usedHeap=3823,
> maxHeap=3991]
> 3-> 192.168.5.148:60020 [requests=63, regions=814, usedHeap=3670,
> maxHeap=3995]
> 4-> 192.168.5.150:60020 [requests=157, regions=816, usedHeap=3185,
> maxHeap=3995]
(Continue reading)

Permalink | Reply | 0 comments |
headers
tsuna | 1 Mar 2011 09:18
Picon
Gravatar

Large pages, TLB and HBase

Has anyone tried using large pages with HBase?

I was reading up on x86-64 after reading the interview of Linus
Torvalds that Todd linked in another thread
(http://www.itwire.com/opinion-and-analysis/open-sauce/44975-linus-torvalds-looking-back-looking-forward?tmpl=component&layout=default&start=1)
where he says "there's no way to ever really get the Intel PAE model
to work perfectly" (anyone knows what problem(s) he's referring to
there?  I already heard complaints about PAE but don't really know
what are the problems...) and I realized that in long mode in x86-64,
page table walks seem pretty expensive.  If I understand correctly,
the page table has 4 levels, so whenever you have a TLB miss the CPU
needs to dereference 5 pointers!  (dereference CR3, dereference the
page directory pointer table, dereference the  page directory entry,
dereference the page table entry, dereference the page-map level 4
table entry)

So I was wondering whether large pages (aka huge pages) would allow
applications that actively touch a lot of memory (such as HBase) to
make better use of the TLB, and whether this could translate in a
noticeable performance improvement.  Has anyone got experience working
with large pages or tried them on HBase or other similar apps?

When I was doing some tests on context switching, I noticed that when
the size of the working set doesn't fit in the TLB, the time needed to
write a 4KB page went up by a whopping 4x (180ns -> 750ns) on the
Intel Nehalem architecture (aka i7).  [see:
http://blog.tsunanet.net/2010/11/how-long-does-it-take-to-make-context.html
for more details on how I got these measurements].  2MB large pages
reduce by a factor of 512x the number of TLB entries needed, but you
also have fewer entries available.  In practice, on the Nehalem, using
(Continue reading)

Permalink | Reply | 0 comments |
headers
James Ram | 1 Mar 2011 10:36
Picon

HBase Prompt missing

Hi,

I have few basic question related to HBase shell. Please help me out in
these issues.

1. When I start the HBase Shell I am not getting the HBase prompt.
2. When I enter the command wrongly shell is closing abnormally.
3. When I use backspace key also the shell closes abnormally.
4. Is there any technique to find the performance of a HBase script?

--

-- 
With Regards,
Jr.
Permalink | Reply | 8 comments |

Gmane