K
Q

Question

OOMKilled Druid Operator. How to add resource in examples/tiny-cluster.yaml

3/30/2021

I am deploying Druid-operator on my enterprise Kube. I am getting OOM Killed error:

NAME                                READY   STATUS      RESTARTS   AGE
druid-operator-c68cf5bc8-n6pzt      1/1     Running     0          107m
druid-tiny-cluster-brokers-0        0/1     OOMKilled   1          62s
druid-tiny-cluster-coordinators-0   0/1     OOMKilled   2          62s
druid-tiny-cluster-historicals-0    0/1     OOMKilled   1          62s
druid-tiny-cluster-routers-0        0/1     OOMKilled   1          62s
tiny-cluster-zk-0                   1/1     Running     0          16h

I tried using "resources" to add resource but getting below error:

error: error validating "examples/tiny-cluster.yaml": error validating data: ValidationError(Druid.spec): unknown field "resources" in org.apache.druid.v1alpha1.Druid.spec; if you choose to ignore these errors, turn validation off with --validate=false

Please suggest how to add "resources" in tiny-cluster.yaml.

Thanks.

-- Ankit

druid

kubernetes

1 Answer

3/31/2021

You can add resource requests and resource limits as in the example: tiny-cluster-hpa.yaml.

I will add resource requests and limits for a druid-tiny-cluster-brokers-0 to illustrate how it works.

As you can see in the beginning, I don't have any resource requests and limits configured:

$ kubectl get pod druid-tiny-cluster-brokers-0 -n druid-operator
NAME                           READY   STATUS    RESTARTS   AGE
druid-tiny-cluster-brokers-0   1/1     Running   0          66s
$ kubectl describe pod druid-tiny-cluster-brokers-0 -n druid-operator | grep -i "requests\|limits"
$

Let's add example requests and limits for the druid-tiny-cluster-brokers-0:
NOTE: I've only added resources inside the brokers section.

$ cat examples/tiny-cluster.yaml
...
  nodes:
    brokers:
      # Optionally specify for running broker as Deployment
      # kind: Deployment
      nodeType: "broker"
      # Optionally specify for broker nodes
      # imagePullSecrets:
      # - name: tutu
      druid.port: 8088
      nodeConfigMountPath: "/opt/druid/conf/druid/cluster/query/broker"
      replicas: 1
      runtime.properties: |
        druid.service=druid/broker

        # HTTP server threads
        druid.broker.http.numConnections=5
        druid.server.http.numThreads=10

        # Processing threads and buffers
        druid.processing.buffer.sizeBytes=1
        druid.processing.numMergeBuffers=1
        druid.processing.numThreads=1
        druid.sql.enable=true
      extra.jvm.options: |-
        -Xmx512M
        -Xms512M
      resources:
        requests:
          memory: "2G"
          cpu: "0.5"
        limits:
          memory: "2G"
          cpu: "2"
...

Then apply the changes:

$ kubectl apply -f examples/tiny-cluster.yaml -n druid-operator
druid.druid.apache.org/tiny-cluster configured

Finally, we can check if the limits and requests have been set correctly:

$ kubectl describe pod druid-tiny-cluster-brokers-0 -n druid-operator | grep -i -A 2 "requests\|limits"
    Limits:
      cpu:     2
      memory:  2G
    Requests:
      cpu:      500m
      memory:   2G

Additionally, make sure that your Kubernetes nodes have enough resources to run your applications as this can also cause problems.

-- matt_j

Source: StackOverflow

KQ

OOMKilled Druid Operator. How to add resource in examples/tiny-cluster.yaml

Similar Questions

1 Answer

K
Q