I got an issue with the deployment of a WordPress + MySQL application on a Kubernetes cluster.
When using HorizontalPodAutoscaler to autoscale my wordpress and wordpress-mysql deployments, it works fine for the wordpress one but not the wordpress-mysql one.
Indeed, when multiple MySQL pods are created, some go in the CrashLoopBackOff status:
$ kubectl get all -n wordpress
NAME READY STATUS RESTARTS AGE
po/wordpress-3874566264-7031k 1/1 Running 0 16h
po/wordpress-mysql-898811424-2bdnn 0/1 CrashLoopBackOff 6 16h
po/wordpress-mysql-898811424-dxj92 1/1 Running 146 16h
po/wordpress-mysql-898811424-vs29j 0/1 CrashLoopBackOff 148 16h
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/wordpress 10.254.121.190 10.0.0.13 80:30003/TCP 16h
svc/wordpress-mysql None <none> 3306/TCP 16h
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa/wordpress Deployment/wordpress 28% / 80%, 0% / 80% 1 10 1 16h
hpa/wordpress-mysql Deployment/wordpress-mysql 90% / 80%, 0% / 80% 1 10 3 16h
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deploy/wordpress 1 1 1 1 16h
deploy/wordpress-mysql 3 3 3 1 16h
NAME DESIRED CURRENT READY AGE
rs/wordpress-3874566264 1 1 1 16h
rs/wordpress-mysql-898811424 3 3 1 16hAnd when I take a look at their logs, I get this:
$ kubectl logs -p wordpress-mysql-898811424-2bdnn -n wordpress
2018-09-12 08:04:12 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2018-09-12 08:04:12 0 [Note] mysqld (mysqld 5.6.41) starting as process 436 ...
2018-09-12 08:04:12 436 [Note] Plugin 'FEDERATED' is disabled.
2018-09-12 08:04:12 436 [Note] InnoDB: Using atomics to ref count buffer pool pages
2018-09-12 08:04:12 436 [Note] InnoDB: The InnoDB memory heap is disabled
2018-09-12 08:04:12 436 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-09-12 08:04:12 436 [Note] InnoDB: Memory barrier is not used
2018-09-12 08:04:12 436 [Note] InnoDB: Compressed tables use zlib 1.2.3
2018-09-12 08:04:12 436 [Note] InnoDB: Using Linux native AIO
2018-09-12 08:04:12 436 [Note] InnoDB: Using CPU crc32 instructions
2018-09-12 08:04:12 436 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2018-09-12 08:04:12 436 [Note] InnoDB: Completed initialization of buffer pool
2018-09-12 08:04:12 436 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2018-09-12 08:04:12 436 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2018-09-12 08:04:12 436 [Note] InnoDB: Retrying to lock the first data file
2018-09-12 08:04:13 436 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2018-09-12 08:04:13 436 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2018-09-12 08:04:14 436 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2018-09-12 08:04:14 436 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2018-09-12 08:04:15 436 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
[...]
2018-09-12 08:05:51 436 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2018-09-12 08:05:52 436 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2018-09-12 08:05:52 436 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2018-09-12 08:05:52 436 [Note] InnoDB: Unable to open the first data file
2018-09-12 08:05:52 7f57a329f5c0 InnoDB: Operating system error number 11 in a file operation.
InnoDB: Error number 11 means 'Resource temporarily unavailable'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html
2018-09-12 08:05:52 436 [ERROR] InnoDB: Can't open './ibdata1'
2018-09-12 08:05:52 436 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data!
2018-09-12 08:05:52 436 [ERROR] Plugin 'InnoDB' init function returned error.
2018-09-12 08:05:52 436 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2018-09-12 08:05:52 436 [ERROR] Unknown/unsupported storage engine: InnoDB
2018-09-12 08:05:52 436 [ERROR] Aborting
2018-09-12 08:05:52 436 [Note] Binlog end
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'partition'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_METRICS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMPMEM'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMP'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_LOCKS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_TRX'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'BLACKHOLE'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'ARCHIVE'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'MRG_MYISAM'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'MyISAM'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'MEMORY'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'CSV'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'sha256_password'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'mysql_old_password'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'mysql_native_password'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'binlog'
2018-09-12 08:05:52 436 [Note] mysqld: Shutdown completeSo it might be quite normal because each MySQL pod is trying to access ./ibdata1 at the same time, but then here is my question: is it really possible to have several MySQL pods sharing the same data? If the answer is yes, then how should I proceed to avoid these annoying errors?
If you need some other information, just tell me and I will edit my post.
Thank you in advance for your help!
So it might be quite normal because each MySQL pod is trying to access ./ibdata1 at the same time
Yes, if you try to do that (you didn't supply manifests), then that's the very reason you have CrashLoopBackOff state. First started instace will lock it and all subsequent will fail.
... trying to access ./ibdata1 at the same time ... is it really possible to have several MySQL pods sharing the same data?
If we talk about same data folder (the very same persistent volume, or hostpath or nfs share...) over two independent mysql instances - then no, not really, and not advisable for a number of reasons.
If you need to have multiple mysql instances (processes, containers or pods) sharing same data (not data folder!) you need to use replication (with read replicas or whatever...) where each of instances has own data folder structure but they sync data between them in some manner. Here is one example of a MySQL single-master topology with multiple slaves running asynchronous replication on kubernetes official documentation. Note that this is not a production setup, not HA setup, but just an illustration of simple replication scenario to give you an idea.
Now, some simple questions: are you sure you can't handle the load with single mysql instance that is serving several wordpress instances? Are you trying to make HA setup? Because answer to each of those question requires a bit different approach and architecture decisions than "increase number of pods from 1 up.
The port name is the same as any pod within your cluster.
ex:
containers:
- image: mysql:5.6
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-pass
key: password
ports:
- containerPort: 3306
name: mysql-wiki
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: mysql-pv-claim