How to handle environment substitution in gradle when working with kubernetes yml files?

7/29/2019

I'm creating zip artifacts of kubernetes api objects during a gradle build. The api objects contain

  • environment variables to be replaced at build time of the artifact, like a version number
  • environment variables to be replaced just before deployment of the artifact like a kubernetes namespace or a url to another system
  • environment variables to be evaluated during runtime of the application

A stripped down example is this snippet containing parts of the nginx configuration:

kind: "ConfigMap"
apiVersion: "v1"
metadata:
  name: "nginx-config"
  namespace: "$APPLICATION_NAMESPACE"
data:
  nginx.conf: |
    http {
        server {
            location /application/ {
                proxy_set_header X-Forwarded-Host $host;
                proxy_set_header X-Forwarded-Server $host;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Prefix /application;
                proxy_pass $APPLICATION_INSTANCE/;
            }
        }
    }
---
kind: "Deployment"
apiVersion: "apps/v1"
metadata:
  name: "application-nginx"
  namespace: "$APPLICATION_NAMESPACE"
  annotations:
    application.version: "$APPLICATION_VERSION"
  • $APPLICATION_VERSION must be replaced during the gradle build
  • $APPLICATION_NAMESPACE and $APPLICATION_INSTANCE must be replaced just before the deployment
  • $host and $proxy_add_x_forwarded_for are evaluated by nginx at runtime and therefore mustn't be altered by the gradle build.

The application will be deployed to minikube using kubectl. Therefore there shall be another gradle task to replace the $APPLICATION_NAMESPACE and $APPLICATION_INSTANCE variables. This gradle task requires that variables for runtime be escaped to \$host for example.

The application will be deployed to kubernetes using jenkins Kubernetes Continuous Deploy Plugin. This requires that variables for runtime are not escaped.

But now this is a conflicting situation. The minikube deployment requires to escape variables two times for runtime variables resulting in \\\$host for example. But the kubernetes deployment with jenkins requires that they only be escaped once.

The gradle file currently looks like this:

plugins {
    id 'base'
}

task processResources(type: Copy) {
    from 'src/main/resources'
    into "$buildDir/resources/main/"
    expand(APPLICATION_VERSION: version)
}

task packageDistributionKubernetesDev(type: Zip) {
    dependsOn processResources
    from fileTree("$buildDir/resources/main/kubernetes-dev")
    classifier 'kubernetes-dev'
    expand()
}

task packageDistributionMinikube(type: Zip) {
    dependsOn processResources
    from fileTree("$buildDir/resources/main/minikube")
    classifier 'minikube'
    expand()
}

task prepareMinikube(type: Copy) {
    dependsOn processResources
    from "$buildDir/resources/main/minikube"
    into "$buildDir/minikube"
    expand(APPLICATION_NAMESPACE: project.kubernetesNamespace, APPLICATION_INSTANCE: project.applicationInstance)
}

assemble {
    dependsOn packageDistributionKubernetesDev, packageDistributionMinikube
}

publishing {
    publications {
        mavenJava(MavenPublication) {
            artifact source: packageDistributionKubernetesDev
            artifact source: packageDistributionMinikube
        }
    }
}

How can I properly process the kuberntes api objects during resource filtering in the gradle build and support deployment with jenkins and to minikube at the same time?

-- SpaceTrucker
environment-variables
gradle
jenkins
kubernetes
minikube

1 Answer

7/30/2019

In the end I have removed all escaping from variables that are to be replaced during build time or before deployment. During build time and while preparing the minikube deployment I'm replacing these variables with the variable itself so that they will be picked up during the next expansion. So the expansion practically uses `APPLICATION_NAMESPACE: '\${APPLICATION_NAMESPACE}'.

Variables that are evaluated at runtime are escaped twice, for example \\\$host.

The preparation of the minikube deployment using gradle uses the 2nd expansion with the actual values for APPLICATION_NAMESPACE and APPLICATION_INSTANCE to create the api object descriptions without any escaping.

The packaging of the artifacts uses the 2nd expansion to produce a file that is suitable for normal environment expansion so that APPLICATION_NAMESPACE and APPLICATION_INSTANCE and the runtime variables like host are not not escaped.

The build file then looks like:

plugins {
    id 'base'
}

task processResources(type: Copy) {
    from 'src/main/resources'
    into "$buildDir/resources/main/"
    expand(APPLICATION_VERSION: version, APPLICATION_NAMESPACE: '\${APPLICATION_NAMESPACE}', APPLICATION_INSTANCE: '\${APPLICATION_INSTANCE}')
}

task packageDistributionKubernetesDev(type: Zip) {
    dependsOn processResources
    from fileTree("$buildDir/resources/main/kubernetes-dev")
    classifier 'kubernetes-dev'
    expand(APPLICATION_NAMESPACE: '\${APPLICATION_NAMESPACE}', APPLICATION_INSTANCE: '\${APPLICATION_INSTANCE}')
}

task packageDistributionMinikube(type: Zip) {
    dependsOn processResources
    from fileTree("$buildDir/resources/main/minikube")
    classifier 'minikube'
    expand(APPLICATION_NAMESPACE: '\${APPLICATION_NAMESPACE}', APPLICATION_INSTANCE: '\${APPLICATION_INSTANCE}')
}

task prepareMinikube(type: Copy) {
    dependsOn processResources
    from "$buildDir/resources/main/minikube"
    into "$buildDir/minikube"
    expand(APPLICATION_NAMESPACE: project.kubernetesNamespace, APPLICATION_INSTANCE: project.applicationInstance)
}

assemble {
    dependsOn packageDistributionKubernetesDev, packageDistributionMinikube
}

publishing {
    publications {
        mavenJava(MavenPublication) {
            artifact source: packageDistributionKubernetesDev
            artifact source: packageDistributionMinikube
        }
    }
}
-- SpaceTrucker
Source: StackOverflow