I have a brand new Kubernetes v1.7.6 master controller.

Nodes are able to join, and nodes that are labeled to receive pods from deployments that use public container repositories work fine (i.e. Prometheus etc.).

My API service and Controller service are both started with --cloud-provider=aws, however I was told on the #kubernetes slack channel that I did not need to set this in the kubelet's systemd unit and that in fact that would cause the kubelet to not run. That has also been my experience so it is not configured there (contrary to the Kubernetes documentation, which I believe is not updated for v1.7.6).

My AWS node is launched with an IAM role with the following permissions:

DescribeRepositories ListImages BatchCheckLayerAvailability BatchGetImage DescribeImages GetAuthorizationToken GetDownloadUrlForLayer GetRepositoryPolicy

This same IAM role worked just fine with my previous Kubernetes v1.6.3 master controller.

The error I am seeing when describing a pod with kubectl is:

Failed to pull image ":" : rpc error: code = 2 desc = unauthorized: authentication required

uname -a on my node is:

Linux ip-X-X-X-X 4.4.0-1022-aws #31-Ubuntu SMP Tue Jun 27 11:27:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

uname -a on my master controller is:

Linux ip-X-X-X-X 4.4.0-96-generic #119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

IAM Policy JSON (as requested by Thomas Khalil):

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ecr:GetAuthorizationToken", "ecr:BatchCheckLayerAvailability", "ecr:GetDownloadUrlForLayer", "ecr:GetRepositoryPolicy", "ecr:DescribeRepositories", "ecr:ListImages", "ecr:DescribeImages", "ecr:BatchGetImage" ], "Resource": "*" } ] }

I am at a complete loss as to what to do next. Any help is greatly appreciated.