Helm fetch errors out with "x509: certificate signed by unknown authority" from inside the pod

11/29/2019

I am trying to write a helm operator using client-go and want to fetch the chart from my controller using the RepoURL, and chartname. I have written a sample piece of code which is working perfectly fine on my local machine. But the same code throws error when I run the same code inside the container. Please find the following piece of code:

func FetchURL() error {
    repoURL := "https://kubernetes-charts.storage.googleapis.com"
    username := ""
    password := ""
    var settings environment.EnvSettings
    flags := pflag.NewFlagSet("helm-env", pflag.ContinueOnError)
    settings.AddFlags(flags)
    settings.Init(flags)
    getters := getter.All(settings)

    getters := getter.All(settings)

    var err error
    destDir, err := ioutil.TempDir("", "helm-")
    if err != nil {
        return fmt.Errorf("Failed to untar: %s", err)
    }
    defer os.RemoveAll(destDir)

    chartURL, err := repo.FindChartInAuthRepoURL(repoURL, username, password, chartRef, "", "", "", "", getter.All(settings))
    if err != nil {
        return fmt.Errorf("Error finding the Chart URL: %s", err)
    }
    fmt.Println(chartURL)
    return nil
}

The above function is throwing following error while getting the chartURL:

Fetching Chart
Error finding the Chart URL: Looks like "https://kubernetes-charts.storage.googleapis.com" is not a valid chart repository or cannot be reached: Get https://kubernetes-charts.storage.googleapis.com/index.yaml: x509: certificate signed by unknown authority

I get that it is asking for some ca certificate, but I am not sure which ca certificate as I have initialised helm without tls certificates.

I also tried to copy the helm binary inside the pod and tried to run:

helm init --client-only
helm repo update
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Unable to get an update from the "stable" chart repository (https://kubernetes-charts.storage.googleapis.com):
    Get https://kubernetes-charts.storage.googleapis.com/index.yaml: x509: certificate signed by unknown authority
...Unable to get an update from the "bitnami" chart repository (https://charts.bitnami.com/bitnami):
    Get https://charts.bitnami.com/bitnami/index.yaml: x509: certificate signed by unknown authority
Update Complete.

I am facing the same issue there also. Has anyone faced similar issue?

-- Prafull Ladha
client-go
kubernetes
kubernetes-helm

1 Answer

11/30/2019

No, I don't see this problem.

Notice: the cert of domain kubernetes-charts.storage.googleapis.com is issued by known CA (Google trust services). So as that of charts.bitnami.com (issued by Amazon CA), So you just need to debug a little bit to see:

  1. which CA issues cert to this domain when accessing within the pod. E.g., run command curl -vI https://charts.bitnami.com/ If the cert issuer is what it should be, you may need to update the base image of the pod to include updated known CA certs. If not (e.g., the cert of charts.bitnami.com is self-signed), this is typical DNS poisoning (names server for the pod resolves above domains to different ones)
  2. Check names servers for your pod. if you don't trust k8s worker DNS, you can customize names servers for your pod. see https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
-- shawmzhu
Source: StackOverflow