Vivek Mishra | 15 Apr 21:52 2014
Picon

Unable to complete request: one or more nodes were unavailable.

Hi,
I am trying Cassandra light weight transaction support with Cassandra 2.0.4

cqlsh:twitter> create table user(user_id text primary key, namef text);
cqlsh:twitter> insert into user(user_id,namef) values('v','ff') if not exists;

Unable to complete request: one or more nodes were unavailable.

Any suggestions?

-Vivek
Tony Anecito | 15 Apr 17:24 2014
Picon

OPSCENTER has table row maintenence?

Hi All,

I asked this over a year ago and it seemed as if OPSCENTER did not have the capability to maintain data like SQL Server Enterprise tool did. I need a visual tool that displays rows and allows me to hand edit column info.

iv>
Does this tool now do that? Before it was to create schemas and manage clusters but not edit data.

Thanks!
-Tony
Aravindan T | 15 Apr 10:13 2014

Datastax OPSCENTER

Hi,

Could anyone please guide me on installation procedure of Cassandra using Datastax OPSCENTER? I couldnt find any documentation in regards to the same.

Your help is much appreciated. 


Thanks,
AT

=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you

Donald Smith | 14 Apr 21:50 2014

RE: Lots of commitlog files

Another thing.   cassandra.yaml says:

 

# Total space to use for commitlogs.  Since commitlog segments are

# mmapped, and hence use up address space, the default size is 32

# on 32-bit JVMs, and 1024 on 64-bit JVMs.

#

# If space gets above this value (it will round up to the next nearest

# segment multiple), Cassandra will flush every dirty CF in the oldest

# segment and remove it.  So a small total commitlog space will tend

# to cause more flush activity on less-active columnfamilies.

# commitlog_total_space_in_mb: 4096

 

We’re using a 64 bit linux with a 64 bit JVM:

Java(TM) SE Runtime Environment (build 1.7.0_40-b43)

Java HotSpot(TM) 64-Bit Server VM (build 24.0-b56, mixed mode)

 

but our commit log files are each 32MB in size. Is this indicative of a bug?  Shouldn’t they be 1024MB in size?

 

  Don

 

From: Donald Smith
Sent: Monday, April 14, 2014 12:04 PM
To: 'user <at> cassandra.apache.org'
Subject: Logs of commitlog files

 

1. With cassandra 2.0.6, we have 547G of files in /var/lib/commitlog/.  I started a “nodetool flush” 65 minutes ago; it’s still running.  The 17536 commitlog files have been created in the last 3 days.  (The node has 2.1T of sstables data in /var/lib/cassandra/data/.  This is in staging, not prod.) Why so many commit logs?  Here are our commitlog-related settings in cassandra.yaml:

commitlog_sync: periodic

commitlog_sync_period_in_ms: 10000

# The size of the individual commitlog file segments.  A commitlog

# archiving commitlog segments (see commitlog_archiving.properties),

commitlog_segment_size_in_mb: 32

# Total space to use for commitlogs.  Since commitlog segments are

# segment and remove it.  So a small total commitlog space will tend

# commitlog_total_space_in_mb: 4096

 

Maybe we should set commitlog_total_space_in_mb to something other than the default. According to OpsCenter, commitlog_total_space_in_mb is “None”.    But it seems odd that there’d be so many commit logs.

 

The node is under heavy write load.   There are about 2900 compactions pending.

 

We are NOT archiving commitlogs, via commitlog_archiving.properties.

 

BTW, the documentation for nodetool says:

Flush

Flushes memtables (in memory) to SSTables (on disk), which also enables CommitLog segments to be deleted.

But even after doing a flush, the /var/lib/commitlog dir still has 1G of files, even after waiting 30  minutes.  Each file is 32M in size, plus or minus a few bytes.  I tried this on other clusters, with much smaller amounts of data.   Even restarting Cassandra doesn’t help.

 

I surmise that the 1GB of commit logs are normal: they probably allocate that space as a workspace.

 

 

Thanks,  Don

 

Donald A. Smith | Senior Software Engineer
P: 425.201.3900 x 3866
C: (206) 819-5965
F: (646) 443-2333
donalds <at> AudienceScience.com


 

Donald Smith | 14 Apr 21:04 2014

Logs of commitlog files

1. With cassandra 2.0.6, we have 547G of files in /var/lib/commitlog/.  I started a “nodetool flush” 65 minutes ago; it’s still running.  The 17536 commitlog files have been created in the last 3 days.  (The node has 2.1T of sstables data in /var/lib/cassandra/data/.  This is in staging, not prod.) Why so many commit logs?  Here are our commitlog-related settings in cassandra.yaml:

commitlog_sync: periodic

commitlog_sync_period_in_ms: 10000

# The size of the individual commitlog file segments.  A commitlog

# archiving commitlog segments (see commitlog_archiving.properties),

commitlog_segment_size_in_mb: 32

# Total space to use for commitlogs.  Since commitlog segments are

# segment and remove it.  So a small total commitlog space will tend

# commitlog_total_space_in_mb: 4096

 

Maybe we should set commitlog_total_space_in_mb to something other than the default. According to OpsCenter, commitlog_total_space_in_mb is “None”.    But it seems odd that there’d be so many commit logs.

 

The node is under heavy write load.   There are about 2900 compactions pending.

 

We are NOT archiving commitlogs, via commitlog_archiving.properties.

 

BTW, the documentation for nodetool says:

Flush

Flushes memtables (in memory) to SSTables (on disk), which also enables CommitLog segments to be deleted.

But even after doing a flush, the /var/lib/commitlog dir still has 1G of files, even after waiting 30  minutes.  Each file is 32M in size, plus or minus a few bytes.  I tried this on other clusters, with much smaller amounts of data.   Even restarting Cassandra doesn’t help.

 

I surmise that the 1GB of commit logs are normal: they probably allocate that space as a workspace.

 

 

Thanks,  Don

 

Donald A. Smith | Senior Software Engineer
P: 425.201.3900 x 3866
C: (206) 819-5965
F: (646) 443-2333
donalds <at> AudienceScience.com


 

Yulian Oifa | 14 Apr 18:22 2014
Picon

Cassandra Strange behaviour

Hello to all
I have cassandra cluster with 3 nodes and RF=3 writing with Quorum.
Application wrote today several millions of records to specific CF.
After that one of servers went wild , he eats up the disk.
As i see from logs hinted handoff from 2 other servers are occuring to disk server.
On this server i see that data is flushed to disk each several seconds :
 WARN [CompactionExecutor:249] 2014-04-14 19:17:38,633 CompactionManager.java (line 509) insufficient space to compact all requested files SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-548-Data.db'), SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-1060-Data.db'), SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-890-Data.db'), SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-978-Data.db')
 WARN [CompactionExecutor:249] 2014-04-14 19:17:58,647 CompactionManager.java (line 509) insufficient space to compact all requested files SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-1060-Data.db'), SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-890-Data.db'), SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-978-Data.db')
 INFO [COMMIT-LOG-WRITER] 2014-04-14 19:18:06,232 CommitLogSegment.java (line 59) Creating new commitlog segment /opt/cassandra/commitlog/CommitLog-1397492286232.log
 WARN [CompactionExecutor:249] 2014-04-14 19:18:18,648 CompactionManager.java (line 509) insufficient space to compact all requested files SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-1060-Data.db'), SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-890-Data.db')
ERROR [CompactionExecutor:249] 2014-04-14 19:18:18,649 CompactionManager.java (line 513) insufficient space to compact even the two smallest files, aborting
 INFO [NonPeriodicTasks:1] 2014-04-14 19:18:25,228 MeteredFlusher.java (line 62) flushing high-traffic column family CFS(Keyspace='USER_DATA', ColumnFamily='freeNumbers')
 INFO [NonPeriodicTasks:1] 2014-04-14 19:18:25,228 ColumnFamilyStore.java (line 1128) Enqueuing flush of Memtable-freeNumbers <at> 1950635535(37693440/309874109 serialized/live bytes, 837632 ops)
 INFO [FlushWriter:22] 2014-04-14 19:18:25,229 Memtable.java (line 237) Writing Memtable-freeNumbers <at> 1950635535(37693440/309874109 serialized/live bytes, 837632 ops)
 INFO [FlushWriter:22] 2014-04-14 19:18:26,871 Memtable.java (line 254) Completed flushing /opt/cassandra/data/USER_DATA/freeNumbers-g-1066-Data.db (38103944 bytes)
 INFO [CompactionExecutor:251] 2014-04-14 19:18:26,872 CompactionManager.java (line 542) Compacting Minor: [SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-1065-Data.db'), SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-1063-Data.db'), SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-1064-Data.db'), SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-1066-Data.db')]
 INFO [CompactionExecutor:251] 2014-04-14 19:18:26,878 CompactionController.java (line 146) Compacting large row USER_DATA/freeNumbers:8bdf9678-6d70-11e3-85ab-80e385abf85d (151810145 bytes) incrementally


However total data of this CF is around 4.5 GB , while disk usage for this CF on this server overcomes 20GB.
I have tried restart of this server , cyclic restart of all servers and no luck it continues to write data .i can not run compact also....
How can i stop that?
Best regards
Yulian Oifa
William Oberman | 14 Apr 15:44 2014

bloom filter + suddenly smaller CF

I had a thread on this forum about clearing junk from a CF.  In my case, it's ~90% of ~1 billion rows.

One side effect I had hoped for was a reduction in the size of the bloom filter.  But, according to nodetool cfstats, it's still fairly large (~1.5GB of RAM).

Do bloom filters ever resize themselves when the CF suddenly gets smaller?  

My next test will be restarting one of the instances, though I'll have to wait on that operation so I thought I'd ask in the meantime.

will
Markus Jais | 14 Apr 11:25 2014
Picon

Replication Factor question

Hello,

currently reading the "Practical Cassandra". In the section about replication factors the book says:

"It is generally not recommended to set a replication factor of 3 if you have fewer than six nodes in a data center".

Why is that? What problems would arise if I had a replication factor of 3 and only 5 nodes?

Does that mean that for a replication of 4 I would need at least 8 nodes and for a factor of 5 at least 10 nodes?

Not saying that I would factor 5 andn 10 nodes, just curious about how this works.

All the best,

Markus
Yulian Oifa | 13 Apr 17:26 2014
Picon

Cassandra disk usage

I have column family with 2 raws.
2 raws have overall 100 million columns.
Each columns have name of 15 chars ( digits ) and same 15 chars in value ( also digits ).
Each column should have 30 bytes.
Therefore all data should contain approximately 3GB.
Cassandra cluster has 3 servers , and data is stored in quorum ( 2 servers ).
Therefore each server should have 3GB*2/3=2GB of data for this column family.
Table is almost never changed , data is only removed from this table , which possibly created tombstones , but it should not increase the usage.
However when i check the data i see that each server has more then 4GB of data ( more then twice of what should be).

server 1:
-rw-r--r-- 1 root root 3506446057 Dec 26 12:02 freeNumbers-g-264-Data.db
-rw-r--r-- 1 root root  814699666 Dec 26 12:24 freeNumbers-g-281-Data.db
-rw-r--r-- 1 root root  198432466 Dec 26 12:27 freeNumbers-g-284-Data.db
-rw-r--r-- 1 root root   35883918 Apr 12 20:07 freeNumbers-g-336-Data.db

server 2:
-rw-r--r-- 1 root root 3448432307 Dec 26 11:57 freeNumbers-g-285-Data.db
-rw-r--r-- 1 root root  762399716 Dec 26 12:22 freeNumbers-g-301-Data.db
-rw-r--r-- 1 root root  220887062 Dec 26 12:23 freeNumbers-g-304-Data.db
-rw-r--r-- 1 root root   54914466 Dec 26 12:26 freeNumbers-g-306-Data.db
-rw-r--r-- 1 root root   53639516 Dec 26 12:26 freeNumbers-g-305-Data.db
-rw-r--r-- 1 root root   53007967 Jan  8 15:45 freeNumbers-g-314-Data.db
-rw-r--r-- 1 root root     413717 Apr 12 18:33 freeNumbers-g-359-Data.db


server 3:
-rw-r--r-- 1 root root 4490657264 Apr 11 18:20 freeNumbers-g-358-Data.db
-rw-r--r-- 1 root root     389171 Apr 12 20:58 freeNumbers-g-360-Data.db
-rw-r--r-- 1 root root       4276 Apr 11 18:20 freeNumbers-g-358-Statistics.db
-rw-r--r-- 1 root root       4276 Apr 11 18:24 freeNumbers-g-359-Statistics.db
-rw-r--r-- 1 root root       4276 Apr 12 20:58 freeNumbers-g-360-Statistics.db
-rw-r--r-- 1 root root        976 Apr 11 18:20 freeNumbers-g-358-Filter.db
-rw-r--r-- 1 root root        208 Apr 11 18:24 freeNumbers-g-359-Data.db
-rw-r--r-- 1 root root         78 Apr 11 18:20 freeNumbers-g-358-Index.db
-rw-r--r-- 1 root root         52 Apr 11 18:24 freeNumbers-g-359-Index.db
-rw-r--r-- 1 root root         52 Apr 12 20:58 freeNumbers-g-360-Index.db
-rw-r--r-- 1 root root         16 Apr 11 18:24 freeNumbers-g-359-Filter.db
-rw-r--r-- 1 root root         16 Apr 12 20:58 freeNumbers-g-360-Filter.db

When i try to compact i get the following notification from first server :
INFO [CompactionExecutor:1604] 2014-04-13 18:23:07,260 CompactionController.java (line 146) Compacting large row USER_DATA/freeNumbers:8bdf9678-6d70-11e3-85ab-80e385abf85d (4555076689 bytes) incrementally

Which confirms that there is around 4.5GB of data on that server only.
Why does cassandra takes so much data???

Best regards
Yulian Oifa

Ravil Bayramgalin | 12 Apr 01:38 2014

Eventual consistency with replication factor 1

I've got classical eventual consistency symptoms (read after write returns empty result) but there is a surprising twist. The keyspace has replication factor 1 (it's used as a cache) so how can I get a stale result?

Cassandra version 1.2.15.

Consistency settings (although I think they should not matter with one-replica case):
Read — CL.ONE
Write — CL.ALL

If you need any additional info I would be happy to provide!
Paulo Ricardo Motta Gomes | 12 Apr 00:20 2014

Blog post with Cassandra upgrade tips

Hey,

Some months ago (last year!!) during our previous major upgrade from 1.1 -> 1.2 I started writing a blog post with some tips for a smooth rolling upgrade, but for some reason I forgot to finish the post. I found it recently and decided it to publish anyway, as some of the info may be helpful for future major upgrades:


Cheers,

--
Paulo Motta


Gmane