Kubernetes MySQL pod keeps crashing and shows Unable to lock ./ibdata1 error: 11

10/31/2019

I have created new docker image and added in kubernetes statefulset yaml for mysql pod. When I scaled mysql pod to 1 , it's keeps crashing and throwing message unable to lock ./ibdata1 error: 11

I have googled lot about this error but none of them gave a solution. Appreciate if any one help me!

enter image description here

Docker file:

FROM mysql/mysql-server
CMD [ "--max_connections=10000" ]

And Created MySQL YAML like below:

{
  "kind": "StatefulSet",
  "apiVersion": "apps/v1beta2",
  "metadata": {
    "name": "mysql-test",
    "namespace": "test",
    "selfLink": "/apis/apps/v1beta2/namespaces/test/statefulsets/mysql-test",
    "uid": "e7768a0b-faf9-11e9-a989-d8c497367e2a",
    "resourceVersion": "315823479",
    "generation": 33,
    "creationTimestamp": "2019-10-30T09:44:44Z",
    "labels": {
      "app": "mysql-test",
      "release": "mysql-test"
    }
  },
  "spec": {
    "replicas": 1,
    "selector": {
      "matchLabels": {
        "app": "mysql-test"
      }
    },
    "template": {
      "metadata": {
        "creationTimestamp": null,
        "labels": {
          "app": "mysql-test"
        }
      },
      "spec": {
        "volumes": [
          {
            "name": "data",
            "persistentVolumeClaim": {
              "claimName": "mysqldata"
            }
          },
          {
            "name": "backup",
            "persistentVolumeClaim": {
              "claimName": "mysqlbackup"
            }
          }
        ],
        "containers": [
          {
            "name": "latest",
            "image": "<Enterprise repository>/careercompass/mysql:latest",
            "ports": [
              {
                "name": "mysql",
                "containerPort": 3306,
                "protocol": "TCP"
              }
            ],
            "env": [
              {
                "name": "MYSQL_PASSWORD",
                "valueFrom": {
                  "secretKeyRef": {
                    "name": "mysql",
                    "key": "mysql-password",
                    "optional": true
                  }
                }
              },
              {
                "name": "MYSQL_ROOT_PASSWORD",
                "valueFrom": {
                  "secretKeyRef": {
                    "name": "mysql",
                    "key": "mysql-root-password",
                    "optional": true
                  }
                }
              },
              {
                "name": "MYSQL_USER",
                "value": "mysql"
              },
              {
                "name": "MYSQL_DATABASE"
              }
            ],
            "resources": {
              "limits": {
                "cpu": "10",
                "memory": "10000Mi"
              },
              "requests": {
                "cpu": "200m",
                "memory": "3000Mi"
              }
            },
            "volumeMounts": [
              {
                "name": "data",
                "mountPath": "/var/lib/mysql"
              },
              {
                "name": "backup",
                "mountPath": "/var/lib/mysqlbackup"
              }
            ],
            "terminationMessagePath": "/var/lib/termination-log",
            "terminationMessagePolicy": "File",
            "imagePullPolicy": "IfNotPresent"
          }
        ],
        "restartPolicy": "Always",
        "terminationGracePeriodSeconds": 30,
        "dnsPolicy": "ClusterFirst",
        "securityContext": {
          "runAsUser": 999,
          "fsGroup": 999
        },
        "schedulerName": "default-scheduler"
      }
    },
    "serviceName": "",
    "podManagementPolicy": "OrderedReady",
    "updateStrategy": {
      "type": "RollingUpdate",
      "rollingUpdate": {
        "partition": 0
      }
    },
    "revisionHistoryLimit": 10
  },
  "status": {
    "observedGeneration": 33,
    "replicas": 1,
    "currentReplicas": 1,
    "updatedReplicas": 1,
    "currentRevision": "mysql-test-68cb64885c",
    "updateRevision": "mysql-test-68cb64885c",
    "collisionCount": 0
  }
}
-- Sanjay
docker
kubernetes
mysql

1 Answer

10/31/2019

In the log you will find some like this:

[ERROR] Do you already have another mysqlId sever running on port: 3306?

Let's find out what's already running on port 3306:

lsof -i:3306

Then kill it (i.e. the process number) using kill -9 PROCESS

kill -9 13498

Then try to restart MySQL again.

-- Kamol Hasan
Source: StackOverflow