External-secrets not able to access the AWS STS from a private cluster

11/19/2021

from the last couple of days, I have been trying to figure a way to read the Secrets from AWS secret manager from my private EKS cluster with FARGATE.

For doing so I have attempted to use an open-source tool https://github.com/external-secrets/external-secrets but I couldn't make it work with the private cluster.

I can read the passwords in my local by running the source-code of the above-link when I am connected to internet. But when I deploy the pod to AWS private cluster it always gives an timeout error as shown in the below code snippets.

vel":"error","ts":1637220916.9525745,"logger":"controllers.ExternalSecret","msg":"could not reconcile ExternalSecret. Please check!!!","ExternalSecret":"commons/example","SecretStore":"commons/secretstore-sample","error":"could not get secret data from provider: key \"myTestSecret/random\" from ExternalSecret \"example\": WebIdentityErr: failed to retrieve credentials\ncaused by: RequestError: send request failed\ncaused by: Post \"https://sts.amazonaws.com/\": dial tcp 209.54.180.124:443: i/o timeout","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/cloudberry/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.3/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/cloudberry/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.3/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/cloudberry/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.3/pkg/internal/controller/controller.go:214"}

I have been struggling to make it work, I have also attempted to change the source-code by changing the URLs in the code as I found it hard-coded at one place but still the error doesn't go away.

Any hints of how to make external-secrets work in a private cluster will be very much helpful.

Thanks in advance.

-- Parth Kansara
amazon-web-services
aws-secrets-manager
kubernetes

1 Answer

12/20/2021

After so many days I finally manage to resolve the issue by creating my custom Code in Java.

What I did ? 1. Created a IAM profile and role in AWS that can access Secret Manager. 2. Created a service-account in K8 using which my Pod can access the secret-manager. Service-account can hold IAM role details (created in step-1). 2. Using AWS SDK and sample code fetched the secrets from secret manager. 3. Used the below gradle dependency to create K8 secrets:

implementation 'io.fabric8:kubernetes-client:5.10.1'

Git repo for step 3 Git reference is : https://github.com/fabric8io/kubernetes-client/blob/master/doc/CHEATSHEET.md

-- Parth Kansara
Source: StackOverflow