I am using GlusterFS volume plugin in my Kubernetes cluster on Ubuntu.Installed GlusterFS using Deamon Set.
My dought is How the data is mounted? Is pvc or pv s were replicated on every kubernetes worker or the disks which are configured in topology.json for every worker node sharable amoung the cluster? I read the documents,but i didn't get clarity.
How the GlusterFS works?
Here is the link to the official GlusteFS for Kubernetes repository. Two main components are used in this approach: GlusterFS itself and Heketi.
GlusterFS is a scalable, distributed file system that aggregates disk storage resources from multiple servers into a single global namespace. A Volume in GlusterFS consists of blocks called Bricks. Bricks are located on a different server according to a type of the Volume, there are several types of volumes and for more information about them, you can visit this link.
Heketi provides an interface which can be used to manage the lifecycle of GlusterFS volumes. Heketi is used in Kubernetes for a dynamically provision GlusterFS volumes.
In the example from the repository, 3 Nodes for GlusterFS are used, each Node has block devices. All that is mentioned in topology.json
file for Heketi:
{
"clusters": [
{
"nodes": [
{
"node": {
"hostnames": {
"manage": [
"node0"
],
"storage": [
"192.168.10.100"
]
},
"zone": 1
},
"devices": [
"/dev/vdb",
"/dev/vdc",
"/dev/vdd"
]
},
{
"node": {
"hostnames": {
"manage": [
"node1"
],
"storage": [
"192.168.10.101"
]
},
"zone": 1
},
"devices": [
"/dev/vdb",
"/dev/vdc",
"/dev/vdd"
]
},
{
"node": {
"hostnames": {
"manage": [
"node2"
],
"storage": [
"192.168.10.102"
]
},
"zone": 1
},
"devices": [
"/dev/vdb",
"/dev/vdc",
"/dev/vdd"
]
}
]
}
]
}
These Nodes are Kubernetes Nodes as well, you can use all your Kubernetes Nodes as servers for GlusterFS, but the minimum number of Nodes is three.
Heketi will format the block devices posted in topology.json
file and include them into GlusterFS cluster.
After that, you need to create a Kuberntes StorageClass for dynamic provision of GlusterFS volumes. Example of a Yaml file for Kubernetes:
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
name: gluster-heketi
provisioner: kubernetes.io/glusterfs
parameters: #Here are the setting for access to Heketi RESTapi
resturl: "http://10.42.0.0:8080"
restuser: "joe"
restuserkey: "My Secret Life"
Now you can create a PersistentVolumeClaim with a request to the StorageClass. This action will trigger three processes:
1. Kubernetes will create a GlusterFS Volume using Heketi
2. Kubernetes will create a PersistentVolume configured to use the GlusterFS Volume
3. Kubernetes will assign the PersistentVolume to the PersistentVolumeClaim
Here is the example:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: gluster1
annotations:
volume.beta.kubernetes.io/storage-class: gluster-heketi
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
After that, you can use the PersistentVolumeClaim in Kubernetes Deployments, Pods and etc. The example of the Pod configuration:
apiVersion: v1
kind: Pod
metadata:
name: nginx-pod1
labels:
name: nginx-pod1
spec:
containers:
- name: nginx-pod1
image: gcr.io/google_containers/nginx-slim:0.8
ports:
- name: web
containerPort: 80
volumeMounts:
- name: gluster-vol1
mountPath: /usr/share/nginx/html
volumes:
- name: gluster-vol1
persistentVolumeClaim:
claimName: gluster1 <1>
For more information, you can look through this example on the link.