WriteTimeoutException: Cassandra timeout during write query at consistency

11/26/2019

I am trying to run ycsb benchmark on cassandra statefulset deployed through helm chart on a kubernetes cluster. This is the command I am trying to run -

$bin/ycsb load cassandra2-cql -s  -P workloads/workloada -P db.params -p recordcount=5000000 -p operationcount=10000000 -threads 10

params file has the hosts information. I keep getting this error and it seems to correspond with the number of threads I am running.

Error inserting, not retrying any more. number of attempts: 1Insertion Retry Limit: 0
com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency ONE (1 replica were required but only 0 acknowledged the write)
    at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:73)
    at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:26)
    at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
    at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
    at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
    at site.ycsb.db.CassandraCQLClient.insert(CassandraCQLClient.java:586)
    at site.ycsb.DBWrapper.insert(DBWrapper.java:221)
    at site.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:601)
    at site.ycsb.ClientThread.run(ClientThread.java:135)
    at java.lang.Thread.run(Thread.java:748)
Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency ONE (1 replica were required but only 0 acknowledged the write)
    at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:100)
    at com.datastax.driver.core.Responses$Error.asException(Responses.java:122)
    at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:477)
    at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1005)
    at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:928)
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)

End output - 

[OVERALL], RunTime(ms), 32097
[OVERALL], Throughput(ops/sec), 8605.66408075521
[TOTAL_GCS_PS_Scavenge], Count, 27
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 89
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.27728448141570866
[TOTAL_GCS_PS_MarkSweep], Count, 0
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 0
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.0
[TOTAL_GCs], Count, 27
[TOTAL_GC_TIME], Time(ms), 89
[TOTAL_GC_TIME_%], Time(%), 0.27728448141570866
[CLEANUP], Operations, 10
[CLEANUP], AverageLatency(us), 222311.4
[CLEANUP], MinLatency(us), 0
[CLEANUP], MaxLatency(us), 2224127
[CLEANUP], 95thPercentileLatency(us), 2224127
[CLEANUP], 99thPercentileLatency(us), 2224127
[INSERT], Operations, 276216
[INSERT], AverageLatency(us), 900.1717641266256
[INSERT], MinLatency(us), 137
[INSERT], MaxLatency(us), 2502655
[INSERT], 95thPercentileLatency(us), 792
[INSERT], 99thPercentileLatency(us), 2463
[INSERT], Return=OK, 276216
[INSERT], Return=ERROR, 10
[INSERT-FAILED], Operations, 10
[INSERT-FAILED], AverageLatency(us), 4456857.6
[INSERT-FAILED], MinLatency(us), 4427776
[INSERT-FAILED], MaxLatency(us), 4481023
[INSERT-FAILED], 95thPercentileLatency(us), 4481023
[INSERT-FAILED], 99thPercentileLatency(us), 4481023

I have attached my values.yaml here values.yaml. I have tried giving resources - memory and cpu but no success. I have also tried setting

name: WRITE_REQUEST_TIMEOUT_IN_MS
value: "50000"

in different combinations in values.yaml and templates/statefulset.yaml I am using helm chart from here - helm chart and ycsb cassandra - ycsb

Could someone please help me out here ? I can share more details if needed.

-- Harrish A
cassandra
datastax
kubernetes
ycsb

0 Answers