I can't create a kubernetes cluster when using KOPS, the error message I get is:
error reading cluster configuration "fraxxxx-k8s-devopsxxx": error reading s3://devopsxxx-k8s-learn.s3.amazonaws.com/fraxxx-k8s-devopsbcn/config: Could not retrieve location for AWS bucket devopsxxx-k8s-learn.s3.amazonaws.com
I did create my s3 bucket, I though it was an issue like the bucket was not created, so I browsed my AWS bucket and it was in there.
Executing the command below always displays an error for me:
$ sudo kops create cluster fraxxxx-k8s-devopsxxx --zones eu-west-3a --yes
error reading cluster configuration "fraxxxx-k8s-devopsxxx": error reading s3://devopsxxx-k8s-learn.s3.amazonaws.com/fraxxx-k8s-devopsbcn/config: Could not retrieve location for AWS bucket devopsxxx-k8s-learn.s3.amazonaws.comI did configure my Access Keys:
:~$ aws configure
AWS Access Key ID [****************IYFQ]:
AWS Secret Access Key [****************+SXJ]:
Default region name [eu-west-3]:
Default output format [None]: Even exported the keys as an env vars:
~$ echo $AWS_ACCESS_KEY_ID $AWS_SECRET_ACCESS_KEY
XXXXXXXXXXXXXXXIYFQ XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX+SXJ
The bucket name was exported as env var as well, and I double checked it via GUI and it seems fine.
~$ echo $KOPS_STATE_STORE
s3://devxxxxxx-k8s-learn.s3.amazonaws.comExecuted it again the create command but now adding the "state" flag, and still it is failing.
$ sudo kops create cluster fraxxxx-k8s-devxxxxx --zones eu-west-3a --yes --state s3://devxxxxxx-k8s-learn.s3.amazonaws.com
error reading cluster configuration "fraxxxx-k8s-devxxxxxx": error reading s3://devxxxxxx-k8s-learn.s3.amazonaws.com/fraxxxx-k8s-devopsbcn/config: Could not retrieve location for AWS bucket devopsbcn-k8s-learn.s3.amazonaws.comI expect to run the create command and have a working kubernetes cluster.
Answering to myself As per AWS official documentation,after KOPS ver 1.6.1 you require:
1) Top level domain to create the kubernetes cluster.
2) IAM User with specific permission, because using the root AWS Secret Keys is complicated, or now limited.
I use these permission for my IAM user:
AmazonEC2FullAccess IAMFullAccess AmazonEC2ContainerRegistryFullAccess AmazonS3FullAccess AWSElasticBeanstalkFullAccess AmazonVPCFullAccess AmazonRoute53FullAccess
3) And of course you should use the Access Key and Secret from your IAM user.
4) Then an AWS route53 hosted zone. Steps in the link below
5) The "create" cluster command that worked for me: $ sudo kops create cluster --name cluster.kubernetes-aws.io --zones eu-west-3a --state s3://xxxxx-kops-state-store --master-size=t2.micro --yes
Mind the flags --name and --state
Please use this documentation to accomplish the requirements above https://aws.amazon.com/blogs/compute/kubernetes-clusters-aws-kops/
Beside those points listed above, I had to figure out few more things, like:
A) Adding my api DNS entry from my AWS route53 to /etc/hosts, i.e api.cluster.kubernetes-aws.io 32.56.87.41, Because when I ran this command kops validate cluster, it was trying to look locally on my PC for the cluster, see below:
wrong
$ sudo kops validate cluster --name cluster.kubernetes-aws.io --state s3://xxxxx-kops-state-store Validating cluster cluster.kubernetes-aws.io unexpected error during validation: unable to resolve Kubernetes cluster API URL dns: lookup api.cluster.xxxxxxxxx.com on 127.0.0.53:53: no such host
right
$ sudo kops validate cluster --name cluster.kubernetes-aws.io --state s3://xxxxx-kops-state-store
[sudo] password for prometheus:
Validating cluster cluster.kubernetes-aws.io
INSTANCE GROUPS
NAME ROLE MACHINETYPE MIN MAX SUBNETS
master-eu-west-3a Master t2.micro 1 1 eu-west-3a
nodes Node t2.medium 2 2 eu-west-3a
NODE STATUS
NAME ROLE READY
ip-172-XX-XX-XX.eu-west-3.xxxxx.internal master True
VALIDATION ERRORS
KIND NAME MESSAGE
Machine i-05755f2ba8b9ebea0 machine "i-05755f2ba8b9ebea0" has not yet joined cluster
Machine i-05d0a12acf5434e26 machine "i-05d0a12acf5434e26" has not yet joined cluster
Pod kube-system/kube-dns-57dd96bb49-q6zwc kube-system pod "kube-dns-57dd96bb49-q6zwc" is pending
Pod kube-system/kube-dns-autoscaler-867b9fd49d-hgpf8 kube-system pod "kube-dns-autoscaler-867b9fd49d-hgpf8" is pendingB) Again mind the flags --name and --state when trying to validate the cluster, otherwise you get errors.
C) Important, when creating the cluster, you have to add the machine type otherwise you get the error below.
error assigning default machine type for masters: error finding default machine type: could not find a suitable supported instance type for the instance group "master-eu-west-3a" (type "Master") in region "eu-west-3"
D) My s3 bucket was not being picked from the env vars (or something like that), and I just passed it as flag with --state. For point C and D I created the cluster succesfully as follows:
$ sudo kops create cluster --name cluster.kubernetes-aws.io --zones eu-west-3a --state s3://xxxxx-kops-state-store --master-size=t2.micro --yes