K
Q

Creating a Materialized view from table with big amount of data in Cassandra

2/19/2018

We have a Cassandra Cluster with 3 pods, in Google Cloud Kubernetes. Our Cassandra version is 3.9 we are using the Google images.

I got a problem when I tried to create a Materialized View from a table.

The schema of the table is like:

CREATE TABLE environmental_data (
    block_id int,
    timestamp timestamp,
    device_id int,
    sensor_id int,
    .
    .
    .
    PRIMARY KEY (block_id, timestamp, device_id, sensor_id)

I want to create a view with the device_id as cluster key, I tried to do this:

CREATE MATERIALIZED VIEW environmental_data_by_device AS
       SELECT block_id, timestamp, device_id, sensor_id,... FROM environmental_data
       WHERE block_id is not null
       and timestamp is not null
       and device_id is not null
       and sensor_id is not null
       PRIMARY KEY ((device_id), timestamp, sensor_id, block_id)
       WITH CLUSTERING ORDER BY (timestamp DESC);

In local with a very small amount of data, everything went well. But in production with 80 million lines, 2 pods crashed, and Cassandra looped on this error:

Unknown exception caught while attempting to update MaterializedView! environmental_data

java.lang.IllegalArgumentException: Mutation of XXXX bytes is too large for the maximum size of XXXX

There was also many java.lang.OutOfMemoryError: Java heap space

What can I do to be sure the next try will be successful? To put down the production Cassandra a second time is not really conceivable.

I already succeed to create a view base on a table but it was not that big.

-- Mathieu
cassandra
cassandra-3.0
google-cloud-platform
google-kubernetes-engine
kubernetes

Similar Questions

Draft and Helm vs Ksonnet?
Does Kubernetes supports downloading of resources like mesosphere
Isolate single container in pod
How to view C++ application Core Dumps generated inside Docker Containers through FluentD for GCE

2 Answers

5/13/2018

According to docs, Cassandra considers updates within the same partition as a single mutation. In your case, I suppose, this means that every new insert with the same device id may result in transferring all the data that was previously written into that partition. To avoid it, you should consider splitting the data in the mat. view into smaller pieces using the partition key. For example you may introduce minute or hour wide timebuckets depending on the measurement frequency.

-- Leammas
Source: StackOverflow
2/21/2018

I can give you some hints from the infrastructure point of view since I do not know Cassandra in deep. If I were in charge of the infrastructure I would check that the you have set up in the proper way the Deployments to make sure since you had many java.lang.OutOfMemoryError that:

  1. The pods are scheduled on nodes that are capable or sustain their workload and the scheduler is informed about the memory needed by the pods. In this case you have to set memory request.

This is often not highlighted but can be an issue: if you have 3 nodes having 3 GB of RAM and 2 pods consuming 2 GB with no memory request it might happen that they are scheduled on the very same node being killed after a while. Setting up the memory request everything goes fine.

  1. The pods do not consume more memory than expected and in case of memory leakage healthy pods are killed by pods that are consuming too much memory. In this case you should set memory limit.

Moreover you can find here and interesting article that touches also Java Heap Memory Based AutoScaling in Kubernetes.

You can check how much memory and CPU the pods are consuming through the following command:

$ kubectl top pods --namespace=xxxx

And if the nodes are suffering

$ kubectl top nodes
-- GalloCedrone
Source: StackOverflow