Juho Mäkinen | 30 Oct 11:37 2014
Picon

"Did not get positive replies from all endpoints" error on incremental repair

I'm having problems running nodetool repair -inc -par -pr on my 2.1.1 cluster due to "Did not get positive replies from all endpoints" error.

Here's an example output:
root <at> db08-3:~# nodetool repair -par -inc -pr                                                                                  
[2014-10-30 10:33:02,396] Nothing to repair for keyspace 'system'
[2014-10-30 10:33:02,420] Starting repair command #10, repairing 256 ranges for keyspace profiles (seq=false, full=false)
[2014-10-30 10:33:17,240] Repair failed with error Did not get positive replies from all endpoints.
[2014-10-30 10:33:17,263] Starting repair command #11, repairing 256 ranges for keyspace OpsCenter (seq=false, full=false)
[2014-10-30 10:33:32,242] Repair failed with error Did not get positive replies from all endpoints.
[2014-10-30 10:33:32,249] Starting repair command #12, repairing 256 ranges for keyspace system_traces (seq=false, full=false)
[2014-10-30 10:33:44,243] Repair failed with error Did not get positive replies from all endpoints.

The local system log shows that the repair commands got started, but it seems that they immediately get cancelled due to that error, which btw can't be seen in the cassandra log.

I tried monitoring all logs from all machines in case another machine would show up with some useful error, but so far I haven't found nothing.

Any ideas where this error comes from?

 - Garo

koray mehmet | 30 Oct 10:36 2014
Picon

bulk data load

Hi, 

I have to add a column and value to current data. I tried with batch statement but got warning at cassandra as 

WARN [Native-Transport-Requests:1423504] 2014-10-22 16:49:11,426 BatchStatement.java (line 223) Batch of prepared statements for [keyspace.table] is of size 26600, exceeding specified threshold of 5120 by 21480.

After that messages, could not write to cassandra also.

How should i go on?

Thanks
Koray Sariteke
Jimmy Lin | 30 Oct 04:41 2014
Picon

tuning concurrent_reads param

Hi,
looking at the docs, the default value for concurrent_reads is 32, which seems bit small to me (comparing to say http server)? because if my node is receiving slight traffic, any more than 32 concurrent read query will have to wait.(?)

Recommend rule is, 16* number of drives. Would that be different if I have SSDs?

I am attempting to increase it because I have a few tables have wide rows that app will fetch them, the pure size of data may already eating up the thread time, which can cause  other read threads need to wait and essential slow.

thanks



venkat sam | 29 Oct 20:49 2014

Commissioning failure

 I have a 5 node cassandra cluster and i commissioned 1 new node to the cluster. when i added 1 node. it received streams from 3 nodes out of which 2 were completed successfully and one stream got failed. how can i resume the stream which has failed?
James Derieg | 29 Oct 20:05 2014

Upgrade to 2.1.1 causes error

Have a cassandra cluster that has been running under 2.1.0 fine. Rebuilt 
the cluster using the same settings on 2.1.1 and get this error, even 
with only one node present:

"Non-system keyspaces don't have the same replication settings, 
effective ownership information is meaningless"

I only have one non-system keyspace.  To be sure, I even set system_auth 
and system_traces to use the same replication factor as my main 
keyspace, but the error still persists.  Tried again with a 2.1.0 node 
and upgraded it to 2.1.1 and the cluster errors out again.  Any ideas or 
hints?  Here is what my keyspaces are set to for RF:

CREATE KEYSPACE system_auth WITH replication = {'class': 
'NetworkTopologyStrategy', 'us-east': '3'}  AND durable_writes = true;

CREATE KEYSPACE testspace WITH replication = {'class': 
'NetworkTopologyStrategy', 'us-east': '3'}  AND durable_writes = true;

CREATE KEYSPACE system_traces WITH replication = {'class': 
'NetworkTopologyStrategy', 'us-east': '3'}  AND durable_writes = true;

CREATE KEYSPACE system WITH replication = {'class': 'LocalStrategy'}  
AND durable_writes = true;

---
This email is free from viruses and malware because avast! Antivirus protection is active.
http://www.avast.com

Peter Haggerty | 29 Oct 01:08 2014

2.0.10 to 2.0.11 upgrade and immediate ParNew and CMS GC storm

On a 3 node test cluster we recently upgraded one node from 2.0.10 to
2.0.11. This is a cluster that had been happily running 2.0.10 for
weeks and that has very little load and very capable hardware. The
upgrade was just your typical package upgrade:

$ dpkg -s cassandra | egrep '^Ver|^Main'
Maintainer: Eric Evans <eevans <at> apache.org>
Version: 2.0.11

Immediately after started it ran a couple of ParNews and then started
executing CMS runs. In 10 minutes the node had become unreachable and
was marked as down by the two other nodes in the ring, which are still
2.0.10.

We have jstack output and the server logs but nothing seems to be
jumping out. Has anyone else run into this? What should we be looking
for?

Peter

Philip Thompson | 28 Oct 18:01 2014

CCM 2.0 Announcement

Hello,

I am pleased to announce CCM 2.0, a new major release that includes breaking changes to the CCMLib API. If you are using CCM via its command line interface, moving to CCM 2.0 should require no changes to your workflow. Direct consumers of CCM's python modules, such as cassandra-dtest, will need to make changes in order to use CCM 2.0.

Notably, CCM 2.0 will add support for running DataStax Enterprise clusters, and running Cassandra 2.1+ clusters on Windows. I would like to thank Mike Adamson for contributing the pull requests that enabled DSE support.

CCM 2.0 is available immediately on PyPI.  Future updates will also be published to PyPI, allowing easy installation via pip.

In a future release of CCM, ownership of the project will be transferred to DataStax. CCM will remain open source and Apache licensed.

Thank you,
Philip Thompson
Adria Arcarons | 28 Oct 17:02 2014

OldGen saturation

Hi,

 

I work for a company that gathers time series data from different sensors. I’ve been trying to set up C* in a single-node test environment in order to have an idea of what performance will Cassandra give in our use case. To do so I have implemented a test to simulate our real insertion pattern.

 

We have about 50.000 CFs of varying size, grouping sensors that are in the same physical location. Our partition key is made up of the id of the sensor and the type of the value that is being measured. Hence, a single row for each combination of (sensorId,parameterId). Our primary key is made up of the partition key + the timestamp and the measured value. Moreover, we have a clustering key by timestamps in order to make slice reads fast.

 

The writing test consists of a continuous flow of inserts. The inserts are done inside BATCH statements in groups of 1.000 to a single CF at a time to make them faster. The client is executed in a separate machine.

 

The problem I’m experiencing is that, eventually, when the script has been running for almost 40mins, the heap gets saturated. OldGen gets full and then there is an intensive GC activity trying to free OldGen objects, but it can only free very little space in each pass. Then GC saturates the CPU. Here are the graphs obtained with VisualVM that show this behavior:

 

CPU: https://www.dropbox.com/s/oqqqg0ygbd72n0n/CPU%202014-10-28%2014_24_06-VisualVM%201.3.8.jpg?dl=0

HEAP usage: https://www.dropbox.com/s/qp7iyc5o0fpr1xa/Estancament%20MEM%202014-10-28%2014_21_53-VisualVM%201.3.8.jpg?dl=0

OLDGEN full (via VisualGC): https://www.dropbox.com/s/5udvqq95qkjuppq/HEAP%202014-10-28%2014_22_27-VisualVM%201.3.8.jpg?dl=0

 

Moreover, when the heap is saturated, IO activity drops, from avg 90% of utilization of HD to roughly 15%. So I end up in a situation where very few data is flushed, very few data is freed from memory, and insert rate gets very slow. If the insert process is stopped, C* completes all its pending flushes and after a certain time GC activity stops but OldGen occupancy remains almost full.

 

Why the GC is not capable of freeing more memory?

Isn’t cassandra supposed to stop accepting writes until a certain amount of memory is freed?

I’m sceptic about increasing the size of the memtables. If the IO subsystem isn’t able to cope with the flush activity, the problem would only be delayed.

Can this problem be related in any way to our CF indexing settings?

Why, after completing all pending flushes and compactions, OldGen is still almost full, even when mct is set to 0.15?

Is the BATCH statement the appropriate to insert multiple values inside the same CF?

 

Any thoughts on this would be appreciated. I can provide full logs or config files to anyone interested.

 

Regards,

Adrià.

 

P.S. Details on the setup:

I’m working with the default values except for:

- offheap_objects enabled

- on-heap memtable size set to 128mb. I’ve experienced that this problem is reproduced also with greater on-heap memtable sizes.

- off-heap memtable size set to 2.5GB.

- The number of memtable flusher threads is 3.

- memtable_flush_threshold is set to 0.15 to perform regular flushes to disk.

 

My total heap size is 1GB and the the NewGen region of 256MB. The C* node has 4GB RAM. Intel Xeon CPU E5520 <at> 2.27GHz (3 cores). SATA 500GB HD. Debian 7+Cassandra 2.1.0 + Oracle Java JRE  (build 1.7.0_71-b14). Regarding the writing client, it is implemented in PHP with the YACassandraPDO CQL library, which is based on thrift. The client is executed in a separate machine.

Alain RODRIGUEZ | 28 Oct 15:46 2014
Picon

Explode a keyspace into multiple ones online

Hi guys,

We have a keyspace with more or less 40 tables. We have been adding things in there since Cassandra 0.8 to our current C* - 1.2.18 there for simplicity and cost reasons. Now we need to duplicate part of the data to an other DC, to improve end user latency.

What would be the best way of splitting this ks but staying online (or with the smallest downtime) ?
What would be advantages / disadvantages of keeping those ks recently created in the same cluster rather in distinct cluster.

The more general question behind this is, what would be according to your experience, the main reason of choosing grouping tables into one ks, multiple ks or directly multiple clusters and how to migrate from these different architectures ?

Any thoughts, blog addressing this question or even documentation on this ?
Tim Dunphy | 28 Oct 15:42 2014
Picon

opscenter with community cassandra

Hey all,

 I'd like to setup datastax opscenter to monitor my cassandra ring. However I'm using the open source version of 2.1.1. And before I expend any time and effort in setting this up, I'm wondering if it will work with the open source version? Or would I need to be running datastax cassandra in order to get this going?

Thanks
Tim

--
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B

Load balancing in C* Cluster

Hi,
I am learning C* and its usage these days. I have a very simple, possibly naive question about load balancing.
 
I know that C* can automatically balance the load itself by using tokens. But what about connecting my cluster to a system. For exp, if we have a client or a set of clients (e.g. 12 client machines) accessing a 3-node C* cluster. All three nodes are independent and talk with each other through gossip. This means that we have three IP addresses to connect to a cluster.
 
What should be the best strategy for clients to access these IP addresses? Should we connect four clients each to only one node? OR all 12 clients should see and connect all three nodes? Which strategy is better? Is there any resources available on web for this kind of issue?
 
Regards,
Basit
 

Gmane