K
Q

Can't run a MySQL slave instance on GKE

1/8/2020

I have a MySQL service running on Google Kubernetes Engine configured with Master-Slave replication strategy. Unfortunately up until now only I am only managed to have the master (mysql-0) instance running. Slave instances (mysql-n) always stuck on CrashLoopBackOff.

Current pods:

Here is the screenshot of my current pods

When I check the log for mysql-1 instance, the following lines printed:

2020-01-08 08:11:20+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.18-1debian9 started.
2020-01-08 08:11:20+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2020-01-08 08:11:20+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.18-1debian9 started.
2020-01-08 08:11:20+00:00 [Note] [Entrypoint]: Initializing database files
2020-01-08T08:11:20.787668Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2020-01-08T08:11:20.787780Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.18) initializing of server in progress as process 45
2020-01-08T08:11:20.789731Z 0 [ERROR] [MY-010457] [Server] --initialize specified but the data directory has files in it. Aborting.
2020-01-08T08:11:20.789739Z 0 [ERROR] [MY-013236] [Server] The designated data directory /var/lib/mysql/ is unusable. You can remove all files that the server added to it.
2020-01-08T08:11:20.790259Z 0 [ERROR] [MY-010119] [Server] Aborting
2020-01-08T08:11:20.791368Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.18)  MySQL Community Server - GPL.

On the master MySQL Instance (mysql-0), the following lines printed:

2020-01-08 08:08:02+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.18-1debian9 started.
2020-01-08 08:08:03+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2020-01-08 08:08:03+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.18-1debian9 started.
2020-01-08T08:08:03.605134Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2020-01-08T08:08:03.605280Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.18) starting as process 1
2020-01-08T08:08:05.939722Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2020-01-08T08:08:05.945787Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2020-01-08T08:08:06.069048Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.18'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server - GPL.
2020-01-08T08:08:06.303012Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/var/run/mysqld/mysqlx.sock' bind-address: '::' port: 33060

The key difference between the two logs is, on the slave instance, the line Initializing database files are printed, indicating mysqld attempted to initalize already existing mysql data directory (/var/lib/mysql) whereas it doesn't happen when the master instance initializes.

I was following this kubernetes guide but it doesn't work on MySQL version 8. So I modify the StatefulSet deployment. Here is the yaml snippet

Anybody has idea on how to solve this problem? Much appreciated :)

UPDATE: Solution provided by @BinaryBullet actually works with some changes down the line to enable multiple slaves replication. This yaml file is the end result.

-- Bondan RS
google-kubernetes-engine
kubernetes
mysql

Similar Questions

Replicaset doesnot update pods in when pod image is modified
Error: data helm repository doesn't work as expected
How to delete added attestation in google cloud platform's kubernetes engine image authorization process
how to install a kibana-plugin in kibana hosted on kubernetes?
getting error while creating kubernetes deployment

1 Answer

1/8/2020

I could see this is the issue with your volumemount, as you have single VolumeClaimTemplate for both master and slave mysql-bin files, schemas, ib_buffer_pool, etc all other mysql related files are conflicting with each other and trying to save at the same location. Due to this one of your db instance is failing, i.e. which ever starting later is failing, in your case slave db.

This could be avoided by adding separate PV and PVC for your master and slave.

Addition of something like this to your statefulset

volumeClaimTemplates:
    - metadata:
        name: slave-data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi

And volumemount for slave pod is something like this

   volumeMounts:
        - name: slave-data
          mountPath: /var/lib/mysql
          subPath: mysql
        - name: conf
          mountPath: /etc/mysql/conf.d

But make sure you have a separate Persistent Volume for your slave as well. To which all your slave mysql-bin files, ib_buffer_pool, schemas, etc mysql related files get saved in a different location from master.

Hope this helps.

-- BinaryMonster
Source: StackOverflow