SQL Server pods with PersistentVolumeClaim

8/2/2020

This is the scenario: a SQL Server linux kubernetes setup with minikube.

It runs fine with default settings, databases/tables are created no problem.

But the database files should not be stored within the container so a PersistentVolumeClaim was added and the pod config changed to use the claim and mount /var/opt/mssql to /sqldata on the minikube VM.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: sqldata
spec:
  capacity:
    storage: 1Gi
  storageClassName: sqlserver
  accessModes:
    - ReadWriteMany
  hostPath:
    path: "/sqldata"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dbclaim
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: sqlserver
  resources:
    requests:
      storage: 1Gi


apiVersion: v1
kind: Pod
spec:
  initContainers:
  - name: volume-permissions
    image: busybox
    command: ["sh", "-c", "chown -R 10001:0 /var/opt/mssql"]
    volumeMounts:
    - mountPath: "/var/opt/mssql"
      name: sqldata-storage
  volumes:
  - name: sqldata-storage
    persistentVolumeClaim:
      claimName: dbclaim

  containers:
  - image: mcr.microsoft.com/mssql/server
    name: foo
env:
- name: ACCEPT_EULA
  value: "Y"
- name: SA_PASSWORD
  valueFrom:
    secretKeyRef:
      name: sql-password
      key: sa_password
  - name: MSSQL_PID
    value: Developer
    volumeMounts:
    - mountPath: "/var/opt/mssql/data"
      name: sqldata-storage

Also tried image: microsoft/mssql-server-linux

chown -R 10001:0 /var/opt/mssql

is called in initcontainer to give mssql user access to the host VM's directory.

But what happens now is that the sql server pod starts up and after a minute or two it stops with a CrashloopBackoff.

The logfile from the pod says:

2020-08-02 14:33:57.55 Server Registry startup parameters: -d /var/opt/mssql/data/master.mdf -l /var/opt/mssql/data/mastlog.ldf -e /var/opt/mssql/log/errorlog 2020-08-02 14:33:57.78 Server Error 87(The parameter is incorrect.) occurred while opening file '/var/opt/mssql/data/master.mdf' to obtain configuration information at startup. An invalid startup option might have caused the error. Verify your startup options, and correct or remove them if necessary

Logging into the minikube VM, it looks like sql server does have access as the master table etc is created in the actual mounted directory although only owner permissions are set which is 10001:

$ ls -l /sqldata
-rw-r----- 1 10001 root  4194304 Aug  9 06:51 master.mdf

What to check for to get it running like this?

-- Serve Laurijssen
docker
kubernetes
sql-server

3 Answers

8/13/2020

I have managed to run this. The only thing that I changed from your spec is removing the storageclassName from the Persistent Volume and PersistentVolumeClaim. This is because I didn't have a storage class created so not specifying the storage class will use the default one.

Here is yaml I run.

#pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: sqldata
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: "/sqldata"
#pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dbclaim
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
#sqlserver.yaml
apiVersion: v1
kind: Pod
metadata: 
  name: sqlserver
spec:
  initContainers:
  - name: volume-permissions
    image: busybox
    command: ["sh", "-c", "chown -R 10001:0 /var/opt/mssql"]
    volumeMounts:
    - mountPath: "/var/opt/mssql"
      name: sqldata-storage
  volumes:
  - name: sqldata-storage
    persistentVolumeClaim:
      claimName: dbclaim

  containers:
  - image: mcr.microsoft.com/mssql/server
    name: foo
    volumeMounts:
    - mountPath: "/var/opt/mssql/data"
      name: sqldata-storage
    env:
    - name: ACCEPT_EULA
      value: "Y"
    - name: SA_PASSWORD
      valueFrom:
        secretKeyRef:
          name: sql-password
          key: sa_password
    - name: MSSQL_PID
      value: Developer

This is how I created a secret

kubectl create secret generic sql-password --from-literal=sa_password=Passw0rd

Here is the output of describing the pod.

vagrant@kubemaster:~$ kubectl describe pod sqlserver
Name:         sqlserver
Namespace:    default
Priority:     0
Node:         kubenode02/192.168.56.4
Start Time:   Thu, 13 Aug 2020 20:10:06 +0000
Labels:       <none>
Annotations:  <none>
Status:       Running
IP:           10.36.0.2
IPs:
  IP:  10.36.0.2
Init Containers:
  volume-permissions:
    Container ID:  docker://dbc81ddda15aa5af4b56085ee1923b530f1154ba147c589dcc76fb80121c2d0a
    Image:         busybox
    Image ID:      docker-pullable://busybox@sha256:4f47c01fa91355af2865ac10fef5bf6ec9c7f42ad2321377c21e844427972977
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      chown -R 10001:0 /var/opt/mssql
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 13 Aug 2020 20:10:11 +0000
      Finished:     Thu, 13 Aug 2020 20:10:11 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/opt/mssql from sqldata-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-w9t6t (ro)
Containers:
  foo:
    Container ID:   docker://f43e9321d85daa1b5695dc2944f42a4e12db34b97ba0f333d8a8b9afeace0f31
    Image:          mcr.microsoft.com/mssql/server
    Image ID:       docker-pullable://mcr.microsoft.com/mssql/server@sha256:1a69a5e5f7b00feae9edab6bd72e2f6fd5bbb4e74e4ca46e3cc46f1b911e1305
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Thu, 13 Aug 2020 20:10:14 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      ACCEPT_EULA:  Y
      SA_PASSWORD:  <set to the key 'sa_password' in secret 'sql-password'>  Optional: false
      MSSQL_PID:    Developer
    Mounts:
      /var/opt/mssql/data from sqldata-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-w9t6t (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  sqldata-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  dbclaim
    ReadOnly:   false
  default-token-w9t6t:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-w9t6t
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age        From                 Message
  ----    ------     ----       ----                 -------
  Normal  Scheduled  <unknown>  default-scheduler    Successfully assigned default/sqlserver to kubenode02
  Normal  Pulling    84s        kubelet, kubenode02  Pulling image "busybox"
  Normal  Pulled     80s        kubelet, kubenode02  Successfully pulled image "busybox"
  Normal  Created    80s        kubelet, kubenode02  Created container volume-permissions
  Normal  Started    80s        kubelet, kubenode02  Started container volume-permissions
  Normal  Pulling    79s        kubelet, kubenode02  Pulling image "mcr.microsoft.com/mssql/server"
  Normal  Pulled     78s        kubelet, kubenode02  Successfully pulled image "mcr.microsoft.com/mssql/server"
  Normal  Created    78s        kubelet, kubenode02  Created container foo
  Normal  Started    77s        kubelet, kubenode02  Started container foo
vagrant@kubemaster:~$ 

And here is logs from the pod.

vagrant@kubemaster:~$ kubectl logs sqlserver
SQL Server 2019 will run as non-root by default.
This container is running as user mssql.
Your master database file is owned by mssql.
To learn more visit https://go.microsoft.com/fwlink/?linkid=2099216.
2020-08-13 20:10:17.89 Server      Setup step is FORCE copying system data file 'C:\templatedata\model_replicatedmaster.mdf' to '/var/opt/mssql/data/model_replicatedmaster.mdf'.
2020-08-13 20:10:17.96 Server      Setup step is FORCE copying system data file 'C:\templatedata\model_replicatedmaster.ldf' to '/var/opt/mssql/data/model_replicatedmaster.ldf'.
2020-08-13 20:10:17.96 Server      Setup step is FORCE copying system data file 'C:\templatedata\model_msdbdata.mdf' to '/var/opt/mssql/data/model_msdbdata.mdf'.
2020-08-13 20:10:17.97 Server      Setup step is FORCE copying system data file 'C:\templatedata\model_msdblog.ldf' to '/var/opt/mssql/data/model_msdblog.ldf'.
2020-08-13 20:10:18.06 Server      Microsoft SQL Server 2019 (RTM-CU6) (KB4563110) - 15.0.4053.23 (X64) 
	Jul 25 2020 11:26:55 
	Copyright (C) 2019 Microsoft Corporation
	Developer Edition (64-bit) on Linux (Ubuntu 18.04.4 LTS) <X64>
2020-08-13 20:10:18.07 Server      UTC adjustment: 0:00
2020-08-13 20:10:18.07 Server      (c) Microsoft Corporation.
2020-08-13 20:10:18.07 Server      All rights reserved.
2020-08-13 20:10:18.07 Server      Server process ID is 36.
2020-08-13 20:10:18.07 Server      Logging SQL Server messages in file '/var/opt/mssql/log/errorlog'.
2020-08-13 20:10:18.07 Server      Registry startup parameters: 
	 -d /var/opt/mssql/data/master.mdf
	 -l /var/opt/mssql/data/mastlog.ldf
	 -e /var/opt/mssql/log/errorlog
2020-08-13 20:10:18.08 Server      SQL Server detected 1 sockets with 2 cores per socket and 2 logical processors per socket, 2 total logical processors; using 2 logical processors based on SQL Server licensing. This is an informational message; no user action is required.
2020-08-13 20:10:18.09 Server      SQL Server is starting at normal priority base (=7). This is an informational message only. No user action is required.
2020-08-13 20:10:18.09 Server      Detected 1594 MB of RAM. This is an informational message; no user action is required.
2020-08-13 20:10:18.09 Server      Using conventional memory in the memory manager.
2020-08-13 20:10:18.09 Server      Page exclusion bitmap is enabled.
2020-08-13 20:10:18.12 Server      Buffer pool extension is not supported on Linux platform.
2020-08-13 20:10:18.12 Server      Buffer Pool: Allocating 262144 bytes for 180348 hashPages.
2020-08-13 20:10:18.34 Server      Buffer pool extension is already disabled. No action is necessary.
2020-08-13 20:10:18.90 Server      Successfully initialized the TLS configuration. Allowed TLS protocol versions are ['1.0 1.1 1.2']. Allowed TLS ciphers are ['ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:!DHE-RSA-AES256-GCM-SHA384:!DHE-RSA-AES128-GCM-SHA256:!DHE-RSA-AES256-SHA:!DHE-RSA-AES128-SHA'].
2020-08-13 20:10:18.94 Server      Query Store settings initialized with enabled = 1, 
2020-08-13 20:10:18.96 Server      The maximum number of dedicated administrator connections for this instance is '1'
2020-08-13 20:10:18.97 Server      Node configuration: node 0: CPU mask: 0x0000000000000003:0 Active CPU mask: 0x0000000000000003:0. This message provides a description of the NUMA configuration for this computer. This is an informational message only. No user action is required.
2020-08-13 20:10:18.98 Server      Using dynamic lock allocation.  Initial allocation of 2500 Lock blocks and 5000 Lock Owner blocks per node.  This is an informational message only.  No user action is required.
2020-08-13 20:10:19.01 Server      In-Memory OLTP initialized on lowend machine.
2020-08-13 20:10:19.05 Server      [INFO] Created Extended Events session 'hkenginexesession'
2020-08-13 20:10:19.06 Server      Database Instant File Initialization: enabled. For security and performance considerations see the topic 'Database Instant File Initialization' in SQL Server Books Online. This is an informational message only. No user action is required.
ForceFlush is enabled for this instance. 
2020-08-13 20:10:19.09 Server      Total Log Writer threads: 1. This is an informational message; no user action is required.
2020-08-13 20:10:19.12 Server      clflushopt is selected for pmem flush operation.
2020-08-13 20:10:19.14 Server      Software Usage Metrics is disabled.
2020-08-13 20:10:19.16 Server      CLR version v4.0.30319 loaded.
2020-08-13 20:10:19.18 spid8s      [1]. Feature Status: PVS: 0. CTR: 0. ConcurrentPFSUpdate: 1.
2020-08-13 20:10:19.18 spid8s      Starting up database 'master'.
ForceFlush feature is enabled for log durability.
2020-08-13 20:10:19.61 Server      Common language runtime (CLR) functionality initialized.
2020-08-13 20:10:19.76 spid8s      Service Master Key could not be decrypted using one of its encryptions. See sys.key_encryptions for details.
2020-08-13 20:10:19.77 spid8s      An error occurred during Service Master Key initialization. SQLErrorCode=33095, State=8, LastOsError=0.
2020-08-13 20:10:19.91 spid8s      Resource governor reconfiguration succeeded.
2020-08-13 20:10:19.91 spid8s      SQL Server Audit is starting the audits. This is an informational message. No user action is required.
2020-08-13 20:10:19.92 spid8s      SQL Server Audit has started the audits. This is an informational message. No user action is required.
2020-08-13 20:10:20.00 spid8s      SQL Trace ID 1 was started by login "sa".
2020-08-13 20:10:20.03 spid8s      Server name is 'sqlserver'. This is an informational message only. No user action is required.
2020-08-13 20:10:20.07 spid23s     Always On: The availability replica manager is starting. This is an informational message only. No user action is required.
2020-08-13 20:10:20.08 spid23s     Always On: The availability replica manager is waiting for the instance of SQL Server to allow client connections. This is an informational message only. No user action is required.
2020-08-13 20:10:20.08 spid8s      [4]. Feature Status: PVS: 0. CTR: 0. ConcurrentPFSUpdate: 1.
2020-08-13 20:10:20.11 spid10s     [32767]. Feature Status: PVS: 0. CTR: 0. ConcurrentPFSUpdate: 1.
2020-08-13 20:10:20.12 spid8s      Starting up database 'msdb'.
2020-08-13 20:10:20.13 spid10s     Starting up database 'mssqlsystemresource'.
2020-08-13 20:10:20.14 spid10s     The resource database build version is 15.00.4053. This is an informational message only. No user action is required.
2020-08-13 20:10:20.19 spid22s     A self-generated certificate was successfully loaded for encryption.
2020-08-13 20:10:20.21 spid22s     Server is listening on [ 0.0.0.0 <ipv4> 1433].
2020-08-13 20:10:20.22 Server      Server is listening on [ ::1 <ipv6> 1434].
2020-08-13 20:10:20.22 Server      Server is listening on [ 127.0.0.1 <ipv4> 1434].
2020-08-13 20:10:20.23 Server      Dedicated admin connection support was established for listening locally on port 1434.
2020-08-13 20:10:20.25 spid22s     Server is listening on [ ::1 <ipv6> 1431].
2020-08-13 20:10:20.25 spid10s     [3]. Feature Status: PVS: 0. CTR: 0. ConcurrentPFSUpdate: 1.
2020-08-13 20:10:20.26 spid22s     Server is listening on [ 127.0.0.1 <ipv4> 1431].
2020-08-13 20:10:20.26 spid10s     Starting up database 'model'.
2020-08-13 20:10:20.28 spid22s     SQL Server is now ready for client connections. This is an informational message; no user action is required.
2020-08-13 20:10:20.57 spid10s     Clearing tempdb database.
2020-08-13 20:10:20.94 spid10s     [2]. Feature Status: PVS: 0. CTR: 0. ConcurrentPFSUpdate: 1.
2020-08-13 20:10:20.95 spid10s     Starting up database 'tempdb'.
2020-08-13 20:10:21.21 spid10s     The tempdb database has 1 data file(s).
2020-08-13 20:10:21.22 spid23s     The Service Broker endpoint is in disabled or stopped state.
2020-08-13 20:10:21.23 spid23s     The Database Mirroring endpoint is in disabled or stopped state.
2020-08-13 20:10:21.24 spid8s      Recovery is complete. This is an informational message only. No user action is required.
2020-08-13 20:10:21.26 spid23s     Service Broker manager has started.
vagrant@kubemaster:~$ 

Here is how I checked if the persistent volume works by creating the test file "test file" inside the mounted path /var/opt/mysql/data and delete the pod and created it again. You can still find the test file I created in the same path.

vagrant@kubemaster:~$ kubectl exec -ti sqlserver -- /bin/bash
mssql@sqlserver:/$ 
mssql@sqlserver:/$ cd /var/opt/mssql/data/
mssql@sqlserver:/var/opt/mssql/data$ ls -lrt
total 72068
-rw-r----- 1 mssql root      256 Aug 13 19:28 Entropy.bin
-rw-r----- 1 mssql root 14090240 Aug 13 20:06 msdbdata.mdf
-rw-r----- 1 mssql root  4194304 Aug 13 20:10 model_replicatedmaster.mdf
-rw-r----- 1 mssql root   524288 Aug 13 20:10 model_replicatedmaster.ldf
-rw-r----- 1 mssql root 14090240 Aug 13 20:10 model_msdbdata.mdf
-rw-r----- 1 mssql root   524288 Aug 13 20:10 model_msdblog.ldf
-rw-r----- 1 mssql root  4194304 Aug 13 20:10 master.mdf
-rw-r----- 1 mssql root   524288 Aug 13 20:10 msdblog.ldf
-rw-r----- 1 mssql root  8388608 Aug 13 20:10 modellog.ldf
-rw-r----- 1 mssql root  8388608 Aug 13 20:10 model.mdf
-rw-r----- 1 mssql root  8388608 Aug 13 20:10 templog.ldf
-rw-r----- 1 mssql root  8388608 Aug 13 20:10 tempdb.mdf
-rw-r----- 1 mssql root  2097152 Aug 13 20:10 mastlog.ldf
mssql@sqlserver:/var/opt/mssql/data$ 
mssql@sqlserver:/var/opt/mssql/data$ touch testfile
mssql@sqlserver:/var/opt/mssql/data$ exit
exit
vagrant@kubemaster:~$ kubectl delete pod sqlserver
pod "sqlserver" deleted
vagrant@kubemaster:~$ kubectl create -f sqlserver.yaml 
pod/sqlserver created
vagrant@kubemaster:~$ 
vagrant@kubemaster:~$ kubectl exec -ti sqlserver -- /bin/bash
mssql@sqlserver:/$ 
mssql@sqlserver:/$ ls -lrt /var/opt/mssql/data/
total 72068
-rw-r----- 1 mssql root      256 Aug 13 19:28 Entropy.bin
-rw-r--r-- 1 mssql root        0 Aug 13 20:17 testfile
-rw-r----- 1 mssql root 14090240 Aug 13 20:17 msdbdata.mdf
-rw-r----- 1 mssql root  4194304 Aug 13 20:18 model_replicatedmaster.mdf
-rw-r----- 1 mssql root   524288 Aug 13 20:18 model_replicatedmaster.ldf
-rw-r----- 1 mssql root 14090240 Aug 13 20:18 model_msdbdata.mdf
-rw-r----- 1 mssql root   524288 Aug 13 20:18 model_msdblog.ldf
-rw-r----- 1 mssql root  4194304 Aug 13 20:18 master.mdf
-rw-r----- 1 mssql root   524288 Aug 13 20:18 msdblog.ldf
-rw-r----- 1 mssql root  8388608 Aug 13 20:18 modellog.ldf
-rw-r----- 1 mssql root  8388608 Aug 13 20:18 model.mdf
-rw-r----- 1 mssql root  8388608 Aug 13 20:18 templog.ldf
-rw-r----- 1 mssql root  8388608 Aug 13 20:18 tempdb.mdf
-rw-r----- 1 mssql root  2097152 Aug 13 20:18 mastlog.ldf
mssql@sqlserver:/$ 
mssql@sqlserver:/$ exit
exit
vagrant@kubemaster:~$ 
-- Rohit
Source: StackOverflow

8/2/2020

The problem is in your mountPath.
Can you please try and change it to /var/opt/mssql/data?

  containers:
  - image: mcr.microsoft.com/mssql/server
    name: foo
    volumeMounts:
    - mountPath: "/var/opt/mssql/data"
      name: sqldata-storage
-- hilsenrat
Source: StackOverflow

9/29/2020

I could not comment but, creating a PV and PVC without the storageClassName breaks the link between the two constructs, you will note that the PVC will create a dynamic PV that is then bound to the default Storage Class. This is especially true when running docker desktop with kubernetes turned on as the orchestrator. I had the same issue on my local install where I wanted to run everything hosted out of docker.

-- AJB
Source: StackOverflow