Specifically, there are several resources (currently defined with YAML
) in existing Kubernetes clusters that I would like to modify as part of the build process, and some others that I would like to create from scratch. In each case, I want to do so in multiple regions to keep all regions in sync.
The resources in question are Agones fleets
, which look like this (actual values removed, but representative):
apiVersion: agones.dev/v1
kind: Fleet
metadata:
annotations:
agones.dev/sdk-version: 1.11.0
name: test
namespace: game-servers
resourceVersion: "12324578"
selfLink: /apis/agones.dev/v1/namespaces/game-servers/fleets/test
spec:
replicas: 1
scheduling: Packed
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
branch: test
git_commit: 1b12371827fdea31231230901876ffe
spec:
health:
disabled: false
failureThreshold: 5
initialDelaySeconds: 10
periodSeconds: 5
ports:
- containerPort: 1234
name: default
portPolicy: Dynamic
protocol: UDP
sdkServer:
logLevel: Info
template:
metadata:
creationTimestamp: null
labels:
role: game-servers
spec:
containers:
- image: registry.example.com/gameserver:1b12371827fdea31231230901876ffe
name: agones
resources:
limits:
cpu: 500m
memory: 512m
requests:
cpu: 100m
memory: 256Mi
nodeSelector:
role: game-servers
Where there is an existing fleet
, I would like to inject the latest git commit
into the labels, as well as the image to use (assume it will be tagged appropriately on the registry).
If there is no existing fleet
, for certain values I would like to to loop through and create new fleets from scratch, with similar characteristics as above. I have tried a couple of different ways and failed - from issues with permissions on clusters to odd errors when attempting to use pretty straight forward for
loops in Jenkins/Groovy.
As with many things Jenkins related, to make this work a couple of plugins will be needed, specifically:
Assumptions:
ENV
variables, depending on the setupfleetconfig.yaml
is available on the local filesystem and is a complete fleet config similar to the one posed in the questionIn order to treat the various fleets differently, some selection criteria will need to be applied based on the name of the fleet itself, then a loop is needed to go through each region etc.
To keep this simple, there will be a basic if
statement to select between types (this could easily be expanded), and then a 2 element loop to go loop through more than one region (again easily expandable to more).
This is written as a complete Stage
in Jenkins terms, but obviously is not completely executable on its own. It is as close as it can be to an existing, tested, working configuration that has been run multiple times daily for quite some time.
Although this sticks to the example resources, there is no reason it could not be used to modify, create other resources in Kubernetes generally.
stage('DeployFleets') {
agent {
node {
label 'k8s-node-linux'
}
}
steps {
script {
// assume we can read the Fleet name in from a file
FLEET_NAME = readFile("/path/to/FLEET_NAME")
// let's assume that test is one of the fleets that is being modified, not created, deal with that first
container('jenkins-worker'){
if (FLEET_NAME != 'test') {
script {
// again, assume we can read the commit from a file
def GIT_COMMIT_TAG = readFile("/path/to/GIT_COMMIT")
// create a map of 2 fictional regions with an account to use and an cluster adddress for that region
def DEPLOY_REGIONS = [
"us-east-1": ["jenkins_service_acct_use1", 'https://useast1-cluster.example.com'],
"us-east-2": ["jenkins_service_acct_use2", 'https://useast2-cluster.example.com'],
]
// this each construction is needed in order to get around https://issues.jenkins-ci.org/browse/JENKINS-49732 which prevents using a for(element in DEPLOY_REGIONS)
DEPLOY_REGIONS.each { element ->
withKubeCredentials([[credentialsId: element.value[0], serverUrl: element.value[1]]]) {
sh """
kubectl patch fleet ${FLEET_NAME} -n game-servers --type=json -p='[{"op": "replace", "path": "/spec/template/spec/template/spec/containers/0/image", "value":"registry.example.com/gameserver/${FLEET_NAME}:${GIT_COMMIT_TAG}"}]'
kubectl patch fleet ${FLEET_NAME} -n game-servers --type=json -p='[{"op": "replace", "path": "/spec/template/metadata/labels/git_commit", "value":"${GIT_COMMIT_TAG}"}]'
"""
}
}
}
} else {
// rather than patching here, create a fleet from scratch using a source YAML file as a template
script {
def GIT_COMMIT_TAG = readFile("/path/to/GIT_COMMIT")
def NUM_REPLICAS = 1
def DEPLOY_REGIONS = [
"us-east-1": ["jenkins_service_acct_use1", 'https://useast1-cluster.example.com'],
"us-east-2": ["jenkins_service_acct_use2", 'https://useast2-cluster.example.com'],
]
// see note above about each construct
DEPLOY_REGIONS.each { element ->
// assume template available on file system
def FLEET_CONFIG = readYaml file: "/path/to/fleetconfig.yaml"
FLEET_CONFIG.metadata.name = env.SOME_SANE_NAME
FLEET_CONFIG.spec.template.metadata.labels.git_commit = GIT_COMMIT_TAG
FLEET_CONFIG.spec.replicas = NUM_REPLICAS
FLEET_CONFIG.spec.template.spec.template.spec.containers[0].image = "registry.example.com/gameserver/${FLEET_NAME}:${GIT_COMMIT_TAG}"
writeYaml file: "${env.SOME_SANE_NAME}_fleet.yaml", data: FLEET_CONFIG, overwrite: true
withKubeCredentials([[credentialsId: element.value[0], serverUrl: element.value[1]]]) {
sh """
kubectl -n game-servers apply -f "${env.SOME_SANE_NAME}_fleet.yaml"
"""
}
}
}
}
}
}
}
}
This is not particularly difficult given the YAML
manipulation utilities given to us in Jenkins via the plugin but finding an approach that works from end to end can be a challenge. The use of the patch command in kubectl
makes the patching less "native" to Jenkins, but the convenience is more than worth it (an alternative would be using the REST API instead for example). The foreach
structure looks odd, but is needed to avoid a long-standing bug in Jenkins.