I can create a regular GKE cluster and pull the docker image I need and get it running. When I create the GKE cluster with a routing rule through a NAT my user no longer has permission to pull the docker image.
I start the cluster with these settings:
resources:
######## Network ############
- name: gke-nat-network
type: compute.v1.network
properties:
autoCreateSubnetworks: false
######### Subnets ##########
######### For Cluster #########
- name: gke-cluster-subnet
type: compute.v1.subnetwork
properties:
network: $(ref.gke-nat-network.selfLink)
ipCidrRange: 172.16.0.0/12
region: us-east1
########## NAT Subnet ##########
- name: nat-subnet
type: compute.v1.subnetwork
properties:
network: $(ref.gke-nat-network.selfLink)
ipCidrRange: 10.1.1.0/24
region: us-east1
########## NAT VM ##########
- name: nat-vm
type: compute.v1.instance
properties:
zone: us-east1-b
canIpForward: true
tags:
items:
- nat-to-internet
machineType: https://www.googleapis.com/compute/v1/projects/{{
env["project"] }}/zones/us-east1-b/machineTypes/f1-micro
disks:
- deviceName: boot
type: PERSISTENT
boot: true
autoDelete: true
initializeParams:
sourceImage:
https://www.googleapis.com/compute/v1/projects/debian-
cloud/global/images/debian-7-wheezy-v20150423
networkInterfaces:
- network: projects/{{ env["project"] }}/global/networks/gke-nat-
network
subnetwork: $(ref.nat-subnet.selfLink)
accessConfigs:
- name: External NAT
type: ONE_TO_ONE_NAT
metadata:
items:
- key: startup-script
value: |
#!/bin/sh
# --
# ---------------------------
# Install TCP DUMP
# Start nat; start dump
# ---------------------------
apt-get update
apt-get install -y tcpdump
apt-get install -y tcpick
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
nohup tcpdump -e -l -i eth0 -w /tmp/nat.pcap &
nohup tcpdump -e -l -i eth0 > /tmp/nat.txt &
echo 1 | tee /proc/sys/net/ipv4/ip_forward
########## FIREWALL RULES FOR NAT VM ##########
- name: nat-vm-firewall
type: compute.v1.firewall
properties:
allowed:
- IPProtocol : tcp
ports: []
sourceTags:
- route-through-nat
network: $(ref.gke-nat-network.selfLink)
- name: nat-vm-ssh
type: compute.v1.firewall
properties:
allowed:
- IPProtocol : tcp
ports: [22]
sourceRanges:
- 0.0.0.0/0
network: $(ref.gke-nat-network.selfLink)
########## GKE CLUSTER CREATION ##########
- name: nat-gke-cluster
type: container.v1.cluster
metadata:
dependsOn:
- gke-nat-network
- gke-cluster-subnet
properties:
cluster:
name: nat-gke-cluster
initialNodeCount: 1
network: gke-nat-network
subnetwork: gke-cluster-subnet
nodeConfig:
machineType: n1-standard-4
tags:
- route-through-nat
zone: us-east1-b
########## GKE MASTER ROUTE ##########
- name: master-route
type: compute.v1.route
properties:
destRange: $(ref.nat-gke-cluster.endpoint)
network: $(ref.gke-nat-network.selfLink)
nextHopGateway: projects/{{ env["project"]
}}/global/gateways/default-internet-gateway
priority: 100
tags:
- route-through-nat
########## NAT ROUTE ##########
- name: gke-cluster-route-through-nat
metadata:
dependsOn:
- nat-gke-cluster
- gke-nat-network
type: compute.v1.route
properties:
network: $(ref.gke-nat-network.selfLink)
destRange: 0.0.0.0/0
description: "route all other traffic through nat"
nextHopInstance: $(ref.nat-vm.selfLink)
tags:
- route-through-nat
priority: 800
When I try to pull and start a docker image I get:
ImagePullBackOff error Google Kubernetes Engine
When I do kubectl describe pod I get:
Failed to pull image : rpc error: code = Unknown desc = unauthorized: authentication required
Edit:
I have found out that the gcloud console command has changed since v1.10 https://cloud.google.com/kubernetes-engine/docs/how-to/access-scopes
Basically certain roles are not allowed by default for these clusters which includes pulling an image from google storage.
I am still having trouble figuring out how assign these roles while using
gcloud deployment-manager deployments create gke-with-nat --config gke-with-nat-route.yml
So the reason the container images were not pulling is because gcloud clusters have changed how they handle permissions. It used to grant the 'storage-ro' role to new clusters allowing them to pull container images from the container registry. As per https://cloud.google.com/kubernetes-engine/docs/how-to/access-scopes .
I had to add scopes to the YML cluster deployment as I create my deployment using
gcloud deployment-manager deployments create gke-with-nat --config gke-with-nat-route.yml
The new YML included these settings
nodeConfig:
serviceAccount: thisuser@project-id.iam.gserviceaccount.com
oauthScopes:
- https://www.googleapis.com/auth/devstorage.read_only
If you are using cluster create I think you can use
gcloud container clusters create example-cluster --scopes scope1,scope2
If you are using the website UI I think you can choose to use the legacy setting using a checkbox in the UI. I am not sure how long this will be supported.