How can I define a single definition of DaemonSet to create slightly different pods from it? What I mean by different pods is that pods created from different commands. I currently make two different DaemonSet objects to create two different kinds of pods.
Let me tell you the details.
I've been modifying kube-flannel.yml, especially the definition of DaemonSet. I want to add "--iface=" to flanneld [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr"] and I want to use two different ethernet interfaces based on node label.
I have three different kinds of nodes: 1) a master node with two ethernet interfaces: eth0 for internal network and eth3 for external network 2) slave nodes with one ethernet interfaces: eth0 for internal network 3) a newly added node with one ethernet interfaces: p2p1 for internal network
When I had 1) and 2), I was okay with the following command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr", "--iface=--eth0"] I have to add --iface option because otherwise flanneld on the master node automatically detects eth3 other than eth0, which is the reason I manually add the iface option.
Now that I have 3), I have to make two DaemonSet definitions with different commands: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr", "--iface=--eth0"] for 1) and 2) , [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr", "--iface=--p2p1"] for 3) Label selector is used to tell which DaemonSet is used for a node. The entire definition is attached at the end.
Is there a more elegant way of creating slightly different pods from a single DaemonSet definition? It would be nice to have only a single definition with a placeholder which can be substituted by the value of a specific label.
Otherwise I can do this by setting flannel in a different way? It would be nice if we can give a list of ethernet interfaces to --iface option but my colleague found that only one interface/ip can be given to --iface option.
Thanks.
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: flannel
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"type": "flannel",
"delegate": {
"isDefaultGateway": true
}
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: kube-flannel-ds
labels:
tier: node
app: flannel
iface: eth0
spec:
template:
metadata:
labels:
tier: node
app: flannel
iface: eth0
spec:
hostNetwork: true
nodeSelector:
beta.kubernetes.io/arch: amd64
iface: eth0
serviceAccountName: flannel
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.7.0-amd64
command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr", "--iface=eth0" ]
securityContext:
privileged: true
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run
- name: flannel-cfg
mountPath: /etc/kube-flannel/
- name: install-cni
image: quay.io/coreos/flannel:v0.7.0-amd64
command: [ "/bin/sh", "-c", "set -e -x; cp -f /etc/kube-flannel/cni-conf.json /etc/cni/net.d/10-flannel.conf; while true; do sleep 3600; done" ]
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: kube-gpu-flannel-ds
labels:
tier: node
app: flannel
iface: p2p1
spec:
template:
metadata:
labels:
tier: node
app: flannel
iface: p2p1
spec:
hostNetwork: true
nodeSelector:
beta.kubernetes.io/arch: amd64
iface: p2p1
serviceAccountName: flannel
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.7.0-amd64
command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr", "--iface=p2p1" ]
securityContext:
privileged: true
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run
- name: flannel-cfg
mountPath: /etc/kube-flannel/
- name: install-cni
image: quay.io/coreos/flannel:v0.7.0-amd64
command: [ "/bin/sh", "-c", "set -e -x; cp -f /etc/kube-flannel/cni-conf.json /etc/cni/net.d/10-flannel.conf; while true; do sleep 3600; done" ]
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfgIf you need more advanced logic regarding container commands, I can suggest 2 options.
Build a custom docker image based on quay.io/coreos/flannel:v0.7.0-amd64 and add a script that does the logic (e.g. based on the output of ifconfig | grep p2p1). Use the script as the container command (either specify this in the Dockerfile or in the POD spec).
The same script can also be but into a ConfigMap and mounted into the container, the same way you already did it with the flannel config. You can then change the command of the container to execute this script. Example command: ["/bin/sh", "/etc/kube-flannel/entrypoint.sh"]
Option 2 seems to be easier and more flexible in my opinion.