Goal: To install a kubernetes cluster on AWS using kops without any internet access.

Approach: The kops tool generally has a lot of dependencies from the internet. The first and the most important one is nodeup which gets downloaded from s3. This nodeup binary needs some parameters to be passed in file kube_env.yaml.

Assets:
- 125993c220d1a9b5b60ad20a867a0e7cda63e64c@https://storage.googleapis.com/kubernetes-release/release/v1.8.4/bin/linux/amd64/kubelet
- 8e2314db816b9b4465c5f713c1152cb0603db15e@https://storage.googleapis.com/kubernetes-release/release/v1.8.4/bin/linux/amd64/kubectl
- 1d9788b0f5420e1a219aad2cb8681823fc515e7c@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-0799f5732f2a11b329d9e3d51b9c8f2e3759f2ff.tar.gz
- f62360d3351bed837ae3ffcdee65e9d57511695a@https://kubeupv2.s3.amazonaws.com/kops/1.8.0/linux/amd64/utils.tar.gz
ClusterName: ${cluster_fqdn}
ConfigBase: s3://${config_base}
InstanceGroupName: ${instance_group}
Tags:
- _automatic_upgrades
- _aws
- _kubernetes_master
- _networking_cni
channels:
- s3://${config_base}/addons/bootstrap-channel.yaml
protokubeImage:
  hash: 1b972e92520b3cafd576893ae3daeafdd1bc9ffd
  name: protokube:1.8.0
  source: https://kubeupv2.s3.amazonaws.com/kops/1.8.0/images/protokube.tar.gz

As it can be seen in the above file, the packages and images are getting downloaded from the specified sources. It is fairly easy to customize the sources and point it to internal storage for all the packages that are mentioned in the file. But the nodeup and protokube seem to call the docker images from google's docker repository gcr.io which needs internet egress.

We tried to configure the manifests to point to a private docker repository hosted in the network where all the required images are stored. But the images to don't get pulled from the private repository and the cluster doesn't come up. This is the only blocker we are facing to achieve internet-free cluster. I'm assuming that the images are hard-coded inside the binaries. If that is the case, the only solution I can see is building a custom nodeup executable.

We are looking for other solutions or suggestions on the approach taken. Are the images hard-coded in the nodeup binary or is there some kind of a hash validation that happens? The nodeup source code is not well documented. So it is really difficult to pinpoint where exactly the images are being called.