Omid Aladini | 29 May 11:55 2015
Picon

Stream session's index build waiting to be scheduled

Hi,

Currently both secondary index build and compactions are run by compaction manager within the same thread pool (C* 2.0.x). I'm currently experiencing a situation where repair's stream sessions get stuck waiting because they don't get scheduled to build secondary indexes (and finalise the stream session) because there are already many compaction tasks pending to be finished (the node is recently bootstrapped.).

Is this the expected behaviour? How is it possible to run repair on a column family with secondary indexes while one of the nodes involved in repair have loads of pending compaction tasks?

Thanks,
Omid
曹志富 | 29 May 03:39 2015
Picon

what this error mean

I have a 25 noedes C* cluster with C* 2.1.3. These days a node occur split brain many times。

check the log I found this:

    INFO  [MemtableFlushWriter:118] 2015-05-29 08:07:39,176 Memtable.java:378 - Completed flushing /home/ant/apache-cassandra-2.1.3/bin/../data/data/system/sstable_activity-5a1ff2
67ace03f128563cfae6103c65e/system-sstable_activity-ka-4371-Data.db (8187 bytes) for commitlog position ReplayPosition(segmentId=1432775133526, position=16684949)
ERROR [IndexSummaryManager:1] 2015-05-29 08:10:30,209 CassandraDaemon.java:167 - Exception in thread Thread[IndexSummaryManager:1,1,main]
java.lang.AssertionError: null
        at org.apache.cassandra.io.sstable.SSTableReader.cloneWithNewSummarySamplingLevel(SSTableReader.java:921) ~[apache-cassandra-2.1.3.jar:2.1.3]
        at org.apache.cassandra.io.sstable.IndexSummaryManager.adjustSamplingLevels(IndexSummaryManager.java:410) ~[apache-cassandra-2.1.3.jar:2.1.3]
        at org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:288) ~[apache-cassandra-2.1.3.jar:2.1.3]
        at org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238) ~[apache-cassandra-2.1.3.jar:2.1.3]
        at org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139) ~[apache-cassandra-2.1.3.jar:2.1.3]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.3.jar:2.1.3]
        at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:82) ~[apache-cassandra-2.
1.3.jar:2.1.3]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_71]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_71]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_71]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_71]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_71]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_71]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]

I  want to know why this and how to fix this

Thanks all
--------------------------------------
Ranger Tsao
Sachin PK | 28 May 15:15 2015
Picon

cassandra.WriteTimeout: code=1100 [Coordinator node timed out waiting for replica nodes' responses]

Hi I'm running Cassandra 2.1.5 ,(single datacenter ,4 node,16GB vps each node ),I have given my configuration below, I'm using python driver on my clients ,when i tried to insert  1049067 items I got an error.
 
cassandra.WriteTimeout: code=1100 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 0 responses." info={'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}

also I'm getting an error when I check the count of CF from cqlsh

OperationTimedOut: errors={}, last_host=127.0.0.1

I've installed Datastax Ops center for monitoring  nodes , it shows about 1000/sec write request even the cluster is idle .

Is there any problem with my cassandra configuration 


cluster_name: 'Test Cluster'

num_tokens: 256


hinted_handoff_enabled: true
.
max_hint_window_in_ms: 10800000 # 3 hours

hinted_handoff_throttle_in_kb: 1024

max_hints_delivery_threads: 2

batchlog_replay_throttle_in_kb: 1024

authenticator: AllowAllAuthenticator

authorizer: AllowAllAuthorizer

permissions_validity_in_ms: 2000

partitioner: org.apache.cassandra.dht.Murmur3Partitioner

data_file_directories:
    - /var/lib/cassandra/data

commitlog_directory: /var/lib/cassandra/commitlog

disk_failure_policy: stop

commit_failure_policy: stop

key_cache_size_in_mb:

key_cache_save_period: 14400

row_cache_size_in_mb: 0


row_cache_save_period: 0

counter_cache_size_in_mb:


counter_cache_save_period: 7200


saved_caches_directory: /var/lib/cassandra/saved_caches

commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000

commitlog_segment_size_in_mb: 32

seed_provider:

    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:

          - seeds: "10.xx.xx.xx,10.xx.xx.xxx"

concurrent_reads: 45
concurrent_writes: 64
concurrent_counter_writes: 32

memtable_allocation_type: heap_buffers


memtable_flush_writers: 6

index_summary_capacity_in_mb:

index_summary_resize_interval_in_minutes: 60

trickle_fsync: false
trickle_fsync_interval_in_kb: 10240

storage_port: 7000

ssl_storage_port: 7001

listen_address: 10.xx.x.xxx

start_native_transport: true

native_transport_port: 9042

rpc_address: 0.0.0.0

rpc_port: 9160

broadcast_rpc_address: 10.xx.x.xxx

rpc_keepalive: true

rpc_server_type: sync

thrift_framed_transport_size_in_mb: 15

incremental_backups: false

snapshot_before_compaction: false

auto_snapshot: true

tombstone_warn_threshold: 1000
tombstone_failure_threshold: 100000

column_index_size_in_kb: 64

batch_size_warn_threshold_in_kb: 5

compaction_throughput_mb_per_sec: 16

sstable_preemptive_open_interval_in_mb: 50

read_request_timeout_in_ms: 5000

range_request_timeout_in_ms: 10000

write_request_timeout_in_ms: 2000

counter_write_request_timeout_in_ms: 5000

cas_contention_timeout_in_ms: 1000

truncate_request_timeout_in_ms: 60000

request_timeout_in_ms: 10000

cross_node_timeout: false

endpoint_snitch: SimpleSnitch


dynamic_snitch_update_interval_in_ms: 100 

dynamic_snitch_reset_interval_in_ms: 600000

dynamic_snitch_badness_threshold: 0.1

request_scheduler: org.apache.cassandra.scheduler.NoScheduler


server_encryption_options:
    internode_encryption: none
    keystore: conf/.keystore
    keystore_password: cassandra
    truststore: conf/.truststore
    truststore_password: cassandra


# enable or disable client/server encryption.
client_encryption_options:
    enabled: false
    keystore: conf/.keystore
    keystore_password: cassandra

internode_compression: all

inter_dc_tcp_nodelay: false




Jason Unovitch | 28 May 02:10 2015
Picon

Cassandra 1.2.x EOL date

Evening list,
Simple and quick question, can anyone point me to where the Cassandra
1.2.x series EOL date was announced?  I see archived mailing list
threads for 1.2.19 mentioning it was going to be the last release and
I see CVE-2015-0225 mention it is EOL.   I didn't see it say when the
official EOL date was.

Thanks!

Jason

Kaushal Shriyan | 27 May 18:31 2015
Picon

EC2snitch in AWS

Hi,

Can somebody please share me details about setting up of EC2snitch in AWS single region which has availability zone 1a and 1b? I am using Cassandra version 1.2.19 in the setup.

I would appreciate your help.

Regards,

Kaushal
Thomas Whiteway | 27 May 14:48 2015

Cassandra seems to replace existing node without specifying replace_address

Hi,

 

I’ve been investigating using replace_address to replace a node that hasn’t left the cluster cleanly and after upgrading from 2.1.0 to 2.1.4 it seems that adding a new node will automatically replace an existing node with the same IP address even if replace_address isn’t used.  Does anyone know whether this is an expected change (as far as I can tell it doesn’t seem to be)?

 

I’ve been hitting this by

·         creating a two node cluster (with the nodes in different datacentres)

·         blocking traffic between the two nodes

·         decommissioning one of them

·         unblocking traffic between the two nodes

·         trying to re-add the decommissioned node back into the cluster.

 

I reproduced this with some extra logging and it seems this change was introduced by CASSANDRA-7939 in 2.1.1.  When StorageService.checkForEndpointCollision calls Gossiper.isFatClient  TokenMetadata.tokenToEndpointMap is empty so isFatClient returns true.

 

Thanks,

Thomas

Arun Chaitanya | 27 May 05:32 2015
Picon

10000+ CF support from Cassandra

Good Day Everyone,

I am very happy with the (almost) linear scalability offered by C*. We had a lot of problems with RDBMS.

But, I heard that C* has a limit on number of column families that can be created in a single cluster.
The reason being each CF stores 1-2 MB on the JVM heap.

In our use case, we have about 10000+ CF and we want to support multi-tenancy.
(i.e 10000 * no of tenants)

We are new to C* and being from RDBMS background, I would like to understand how to tackle this scenario from your advice.

Our plan is to use Off-Heap memtable approach.

Each node in the cluster has following configuration
16 GB machine (8GB Cassandra JVM + 2GB System + 6GB Off-Heap)
IMO, this should be able to support 1000 CF with no(very less) impact on performance and startup time.

We tackle multi-tenancy using different keyspaces.(Solution I found on the web)

Using this approach we can have 10 clusters doing the job. (We actually are worried about the cost)

Can you please help us evaluate this strategy? I want to hear communities opinion on this.

My major concerns being, 

1. Is Off-Heap strategy safe and my assumption of 16 GB supporting 1000 CF right?

2. Can we use multiple keyspaces to solve multi-tenancy? IMO, the number of column families increase even when we use multiple keyspace.

3. I understand the complexity using multi-cluster for single application. The code base will get tightly coupled with infrastructure. Is this the right approach?

Any suggestion is appreciated.

Thanks,
Arun
Mohammed Guller | 27 May 03:17 2015

Spark SQL JDBC Server + DSE

Hi –

As I understand, the Spark SQL Thrift/JDBC server cannot be used with the open source C*. Only DSE supports  the Spark SQL JDBC server.

 

We would like to find out whether how many organizations are using this combination. If you do use DSE + Spark SQL JDBC server, it would be great if you could share your experience. For example, what kind of issues you have run into? How is the performance? What reporting tools you are using?

 

Thank  you!

 

Mohammed

 

Eax Melanhovich | 26 May 21:00 2015

A few stupid questions...

Hello.

I'm reading "Beginning Apache Cassandra Development" and there are a few
things I can't figure out.

First. Lets say I have a table (field1, field2, field3, field4), where
(field1, field2) is a primary key and field1 is partition key. There is
a secondary index for field3 column. Do I right understand that in this
case query like:

select ... from my_table where field1 = 123 and field3 > '...'; 

... would be quite efficient, i.e. request would be send only to one
node, not the whole cluster?

Second. Lets say there is some data that almost never changes but is
read all the time. E.g. information about smiles in social network. Or
current sessions. In this case would Cassandra cache "hot" data in
memtable? Or such data should be stored somewhere else, i.e. Redis or
Couchbase?

--

-- 
Best regards,
Eax Melanhovich
http://eax.me/

Matthew Johnson | 26 May 15:24 2015

Start with single node, move to 3-node cluster

Hi gurus,

 

We have ordered some hardware for a 3-node cluster, but its ETA is 6 to 8 weeks. In the meantime, I have been lent a single server that I can use. I am wondering what the best way is to set up my single node (SN), so I can then move to the 3-node cluster (3N) when the hardware arrives.

 

Do I:

 

1.       Create my keyspaces on SN with RF=1, and when 3N is up and running migrate all the data manually (either through Spark, dump-and-load, or write a small script)?

2.       Create my keyspaces on SN with RF=3, bootstrap the 3N nodes into a 4-node cluster when they’re ready, then remove SN from the cluster?

3.       Use SN as normal, and when 3N hardware arrives, physically move the data folder and commit log folder onto one of the nodes in 3N and start it up as a seed?

4.       Any other recommended solutions?

 

I’m not even sure what the impact would be of running a single node with RF=3 – would this even work?

 

Any ideas would be much appreciated.

 

Thanks!

Matthew

 

Mahmood Naderan | 26 May 11:38 2015
Picon

connection refused error for cassandra-cli

Hello,
I have downloaded cassandra-1.2.0 for a test. The output of ./bin/cassandra and ./bin/cassandra-cli seems to be fine. Please see http://pastebin.com/Tgs7qyjS

Next, I created a setup-ycsb.cql and wrote
[mahmood <at> tiger ycsb-0.1.4]$ cat setup-ycsb.cql
CREATE KEYSPACE usertable
with placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy'
and strategy_options = {replication_factor:2};
use usertable;
create column family data with comparator=UTF8Type and default_validation_class=UTF8Type and key_validation_class=UTF8Type;
 

Problem is, when I execute cassandra-cli commnad, I get connection refused error

[mahmood <at> tiger ycsb-0.1.4]$ cassandra-cli -h 192.168.1.5 -f setup-ycsb.cql
org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused




Everything seems to be normal as I disabled the firewall. So I wonder why I receive that error.
Please see this output for more detail  http://pastebin.com/Bnd3TNCM
 

Did I miss anything?
 
Regards,
Mahmood

Gmane