Omid Aladini | 29 May 11:55 2015

Stream session's index build waiting to be scheduled


Currently both secondary index build and compactions are run by compaction manager within the same thread pool (C* 2.0.x). I'm currently experiencing a situation where repair's stream sessions get stuck waiting because they don't get scheduled to build secondary indexes (and finalise the stream session) because there are already many compaction tasks pending to be finished (the node is recently bootstrapped.).

Is this the expected behaviour? How is it possible to run repair on a column family with secondary indexes while one of the nodes involved in repair have loads of pending compaction tasks?

曹志富 | 29 May 03:39 2015

what this error mean

I have a 25 noedes C* cluster with C* 2.1.3. These days a node occur split brain many times。

check the log I found this:

    INFO  [MemtableFlushWriter:118] 2015-05-29 08:07:39,176 - Completed flushing /home/ant/apache-cassandra-2.1.3/bin/../data/data/system/sstable_activity-5a1ff2
67ace03f128563cfae6103c65e/system-sstable_activity-ka-4371-Data.db (8187 bytes) for commitlog position ReplayPosition(segmentId=1432775133526, position=16684949)
ERROR [IndexSummaryManager:1] 2015-05-29 08:10:30,209 - Exception in thread Thread[IndexSummaryManager:1,1,main]
java.lang.AssertionError: null
        at ~[apache-cassandra-2.1.3.jar:2.1.3]
        at ~[apache-cassandra-2.1.3.jar:2.1.3]
        at ~[apache-cassandra-2.1.3.jar:2.1.3]
        at ~[apache-cassandra-2.1.3.jar:2.1.3]
        at$1.runMayThrow( ~[apache-cassandra-2.1.3.jar:2.1.3]
        at ~[apache-cassandra-2.1.3.jar:2.1.3]
        at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$ ~[apache-cassandra-2.
        at java.util.concurrent.Executors$ [na:1.7.0_71]
        at java.util.concurrent.FutureTask.runAndReset( [na:1.7.0_71]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301( [na:1.7.0_71]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ [na:1.7.0_71]
        at java.util.concurrent.ThreadPoolExecutor.runWorker( [na:1.7.0_71]
        at java.util.concurrent.ThreadPoolExecutor$ [na:1.7.0_71]
        at [na:1.7.0_71]

I  want to know why this and how to fix this

Thanks all
Ranger Tsao
Sachin PK | 28 May 15:15 2015

cassandra.WriteTimeout: code=1100 [Coordinator node timed out waiting for replica nodes' responses]

Hi I'm running Cassandra 2.1.5 ,(single datacenter ,4 node,16GB vps each node ),I have given my configuration below, I'm using python driver on my clients ,when i tried to insert  1049067 items I got an error.
cassandra.WriteTimeout: code=1100 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 0 responses." info={'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}

also I'm getting an error when I check the count of CF from cqlsh

OperationTimedOut: errors={}, last_host=

I've installed Datastax Ops center for monitoring  nodes , it shows about 1000/sec write request even the cluster is idle .

Is there any problem with my cassandra configuration 

cluster_name: 'Test Cluster'

num_tokens: 256

hinted_handoff_enabled: true
max_hint_window_in_ms: 10800000 # 3 hours

hinted_handoff_throttle_in_kb: 1024

max_hints_delivery_threads: 2

batchlog_replay_throttle_in_kb: 1024

authenticator: AllowAllAuthenticator

authorizer: AllowAllAuthorizer

permissions_validity_in_ms: 2000

partitioner: org.apache.cassandra.dht.Murmur3Partitioner

    - /var/lib/cassandra/data

commitlog_directory: /var/lib/cassandra/commitlog

disk_failure_policy: stop

commit_failure_policy: stop


key_cache_save_period: 14400

row_cache_size_in_mb: 0

row_cache_save_period: 0


counter_cache_save_period: 7200

saved_caches_directory: /var/lib/cassandra/saved_caches

commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000

commitlog_segment_size_in_mb: 32


    - class_name: org.apache.cassandra.locator.SimpleSeedProvider

          - seeds: "10.xx.xx.xx,"

concurrent_reads: 45
concurrent_writes: 64
concurrent_counter_writes: 32

memtable_allocation_type: heap_buffers

memtable_flush_writers: 6


index_summary_resize_interval_in_minutes: 60

trickle_fsync: false
trickle_fsync_interval_in_kb: 10240

storage_port: 7000

ssl_storage_port: 7001


start_native_transport: true

native_transport_port: 9042


rpc_port: 9160


rpc_keepalive: true

rpc_server_type: sync

thrift_framed_transport_size_in_mb: 15

incremental_backups: false

snapshot_before_compaction: false

auto_snapshot: true

tombstone_warn_threshold: 1000
tombstone_failure_threshold: 100000

column_index_size_in_kb: 64

batch_size_warn_threshold_in_kb: 5

compaction_throughput_mb_per_sec: 16

sstable_preemptive_open_interval_in_mb: 50

read_request_timeout_in_ms: 5000

range_request_timeout_in_ms: 10000

write_request_timeout_in_ms: 2000

counter_write_request_timeout_in_ms: 5000

cas_contention_timeout_in_ms: 1000

truncate_request_timeout_in_ms: 60000

request_timeout_in_ms: 10000

cross_node_timeout: false

endpoint_snitch: SimpleSnitch

dynamic_snitch_update_interval_in_ms: 100 

dynamic_snitch_reset_interval_in_ms: 600000

dynamic_snitch_badness_threshold: 0.1

request_scheduler: org.apache.cassandra.scheduler.NoScheduler

    internode_encryption: none
    keystore: conf/.keystore
    keystore_password: cassandra
    truststore: conf/.truststore
    truststore_password: cassandra

# enable or disable client/server encryption.
    enabled: false
    keystore: conf/.keystore
    keystore_password: cassandra

internode_compression: all

inter_dc_tcp_nodelay: false

Jason Unovitch | 28 May 02:10 2015

Cassandra 1.2.x EOL date

Evening list,
Simple and quick question, can anyone point me to where the Cassandra
1.2.x series EOL date was announced?  I see archived mailing list
threads for 1.2.19 mentioning it was going to be the last release and
I see CVE-2015-0225 mention it is EOL.   I didn't see it say when the
official EOL date was.



Kaushal Shriyan | 27 May 18:31 2015

EC2snitch in AWS


Can somebody please share me details about setting up of EC2snitch in AWS single region which has availability zone 1a and 1b? I am using Cassandra version 1.2.19 in the setup.

I would appreciate your help.


Thomas Whiteway | 27 May 14:48 2015

Cassandra seems to replace existing node without specifying replace_address



I’ve been investigating using replace_address to replace a node that hasn’t left the cluster cleanly and after upgrading from 2.1.0 to 2.1.4 it seems that adding a new node will automatically replace an existing node with the same IP address even if replace_address isn’t used.  Does anyone know whether this is an expected change (as far as I can tell it doesn’t seem to be)?


I’ve been hitting this by

·         creating a two node cluster (with the nodes in different datacentres)

·         blocking traffic between the two nodes

·         decommissioning one of them

·         unblocking traffic between the two nodes

·         trying to re-add the decommissioned node back into the cluster.


I reproduced this with some extra logging and it seems this change was introduced by CASSANDRA-7939 in 2.1.1.  When StorageService.checkForEndpointCollision calls Gossiper.isFatClient  TokenMetadata.tokenToEndpointMap is empty so isFatClient returns true.




Arun Chaitanya | 27 May 05:32 2015

10000+ CF support from Cassandra

Good Day Everyone,

I am very happy with the (almost) linear scalability offered by C*. We had a lot of problems with RDBMS.

But, I heard that C* has a limit on number of column families that can be created in a single cluster.
The reason being each CF stores 1-2 MB on the JVM heap.

In our use case, we have about 10000+ CF and we want to support multi-tenancy.
(i.e 10000 * no of tenants)

We are new to C* and being from RDBMS background, I would like to understand how to tackle this scenario from your advice.

Our plan is to use Off-Heap memtable approach.

Each node in the cluster has following configuration
16 GB machine (8GB Cassandra JVM + 2GB System + 6GB Off-Heap)
IMO, this should be able to support 1000 CF with no(very less) impact on performance and startup time.

We tackle multi-tenancy using different keyspaces.(Solution I found on the web)

Using this approach we can have 10 clusters doing the job. (We actually are worried about the cost)

Can you please help us evaluate this strategy? I want to hear communities opinion on this.

My major concerns being, 

1. Is Off-Heap strategy safe and my assumption of 16 GB supporting 1000 CF right?

2. Can we use multiple keyspaces to solve multi-tenancy? IMO, the number of column families increase even when we use multiple keyspace.

3. I understand the complexity using multi-cluster for single application. The code base will get tightly coupled with infrastructure. Is this the right approach?

Any suggestion is appreciated.

Mohammed Guller | 27 May 03:17 2015

Spark SQL JDBC Server + DSE

Hi –

As I understand, the Spark SQL Thrift/JDBC server cannot be used with the open source C*. Only DSE supports  the Spark SQL JDBC server.


We would like to find out whether how many organizations are using this combination. If you do use DSE + Spark SQL JDBC server, it would be great if you could share your experience. For example, what kind of issues you have run into? How is the performance? What reporting tools you are using?


Thank  you!




Eax Melanhovich | 26 May 21:00 2015

A few stupid questions...


I'm reading "Beginning Apache Cassandra Development" and there are a few
things I can't figure out.

First. Lets say I have a table (field1, field2, field3, field4), where
(field1, field2) is a primary key and field1 is partition key. There is
a secondary index for field3 column. Do I right understand that in this
case query like:

select ... from my_table where field1 = 123 and field3 > '...'; 

... would be quite efficient, i.e. request would be send only to one
node, not the whole cluster?

Second. Lets say there is some data that almost never changes but is
read all the time. E.g. information about smiles in social network. Or
current sessions. In this case would Cassandra cache "hot" data in
memtable? Or such data should be stored somewhere else, i.e. Redis or


Best regards,
Eax Melanhovich

Matthew Johnson | 26 May 15:24 2015

Start with single node, move to 3-node cluster

Hi gurus,


We have ordered some hardware for a 3-node cluster, but its ETA is 6 to 8 weeks. In the meantime, I have been lent a single server that I can use. I am wondering what the best way is to set up my single node (SN), so I can then move to the 3-node cluster (3N) when the hardware arrives.


Do I:


1.       Create my keyspaces on SN with RF=1, and when 3N is up and running migrate all the data manually (either through Spark, dump-and-load, or write a small script)?

2.       Create my keyspaces on SN with RF=3, bootstrap the 3N nodes into a 4-node cluster when they’re ready, then remove SN from the cluster?

3.       Use SN as normal, and when 3N hardware arrives, physically move the data folder and commit log folder onto one of the nodes in 3N and start it up as a seed?

4.       Any other recommended solutions?


I’m not even sure what the impact would be of running a single node with RF=3 – would this even work?


Any ideas would be much appreciated.





Mahmood Naderan | 26 May 11:38 2015

connection refused error for cassandra-cli

I have downloaded cassandra-1.2.0 for a test. The output of ./bin/cassandra and ./bin/cassandra-cli seems to be fine. Please see

Next, I created a setup-ycsb.cql and wrote
[mahmood <at> tiger ycsb-0.1.4]$ cat setup-ycsb.cql
with placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy'
and strategy_options = {replication_factor:2};
use usertable;
create column family data with comparator=UTF8Type and default_validation_class=UTF8Type and key_validation_class=UTF8Type;

Problem is, when I execute cassandra-cli commnad, I get connection refused error

[mahmood <at> tiger ycsb-0.1.4]$ cassandra-cli -h -f setup-ycsb.cql
org.apache.thrift.transport.TTransportException: Connection refused

Everything seems to be normal as I disabled the firewall. So I wonder why I receive that error.
Please see this output for more detail

Did I miss anything?