Istio Gateway + Cert-Manager + letsencrypt certificate

6/14/2019

Based on the guide

I'm using GKE 1.13.6-gke.6 + Istio 1.1.3-gke.0 installed from cluster addon.

Follow the same steps to install cert_manager and created Issuer and Certificate I need:

ISSUER

$ kubectl describe issuer letsencrypt-prod -n istio-system
Name:         letsencrypt-prod
Namespace:    istio-system
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"certmanager.k8s.io/v1alpha1","kind":"Issuer","metadata":{"annotations":{},"name":"letsencrypt-prod","namespace":"istio-system"},"spec":{...
API Version:  certmanager.k8s.io/v1alpha1
Kind:         Issuer
Metadata:
  Creation Timestamp:  2019-06-14T03:11:17Z
  Generation:          2
  Resource Version:    10044939
  Self Link:           /apis/certmanager.k8s.io/v1alpha1/namespaces/istio-system/issuers/letsencrypt-prod
  UID:                 131f1cdd-8e52-11e9-9ba7-42010a9801a6
Spec:
  Acme:
    Email:  ---obscured---@---.net
    Http 01:
    Private Key Secret Ref:
      Name:  prod-issuer-account-key
    Server:  https://acme-v02.api.letsencrypt.org/directory
Status:
  Acme:
    Uri:  https://acme-v02.api.letsencrypt.org/acme/acct/59211199
  Conditions:
    Last Transition Time:  2019-06-14T03:11:18Z
    Message:               The ACME account was registered with the ACME server
    Reason:                ACMEAccountRegistered
    Status:                True
    Type:                  Ready
Events:                    <none>

CERTIFICATE

$ kubectl describe certificate dreamy-plum-bee-certificate -n istio-system
Name:         dreamy-plum-bee-certificate
Namespace:    istio-system
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"certmanager.k8s.io/v1alpha1","kind":"Certificate","metadata":{"annotations":{},"name":"dreamy-plum-bee-certificate","namespace":"istio-s...
API Version:  certmanager.k8s.io/v1alpha1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2019-06-14T03:24:43Z
  Generation:          3
  Resource Version:    10048432
  Self Link:           /apis/certmanager.k8s.io/v1alpha1/namespaces/istio-system/certificates/dreamy-plum-bee-certificate
  UID:                 f3ed9f15-8e53-11e9-9ba7-42010a9801a6
Spec:
  Acme:
    Config:
      Domains:
        dreamy-plum-bee.somewhere.net
      Http 01:
        Ingress Class:  istio
  Common Name:          dreamy-plum-bee.somewhere.net
  Dns Names:
    dreamy-plum-bee.somewhere.net
  Issuer Ref:
    Name:       letsencrypt-prod
  Secret Name:  dreamy-plum-bee-certificate
Status:
  Conditions:
    Last Transition Time:  2019-06-14T03:25:12Z
    Message:               Certificate is up to date and has not expired
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Not After:               2019-09-12T02:25:10Z
Events:                    <none>

GATEWAY

$ kubectl describe gateway dreamy-plum-bee-gtw -n istio-system
Name:         dreamy-plum-bee-gtw
Namespace:    istio-system
Labels:       k8s-app=istio
Annotations:  <none>
API Version:  networking.istio.io/v1alpha3
Kind:         Gateway
Metadata:
  Creation Timestamp:  2019-06-14T06:08:13Z
  Generation:          1
  Resource Version:    10084555
  Self Link:           /apis/networking.istio.io/v1alpha3/namespaces/istio-system/gateways/dreamy-plum-bee-gtw
  UID:                 cabffdf1-8e6a-11e9-9ba7-42010a9801a6
Spec:
  Selector:
    Istio:  ingressgateway
  Servers:
    Hosts:
      dreamy-plum-bee.somewhere.net
    Port:
      Name:      https
      Number:    443
      Protocol:  HTTPS
    Tls:
      Credential Name:     dreamy-plum-bee-certificate
      Mode:                SIMPLE
      Private Key:         sds
      Server Certificate:  sds
Events:                    <none>

$ kubectl get gateway dreamy-plum-bee-gtw -n istio-system -o yaml
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  creationTimestamp: 2019-06-14T06:08:13Z
  generation: 1
  labels:
    k8s-app: istio
  name: dreamy-plum-bee-gtw
  namespace: istio-system
  resourceVersion: "10084555"
  selfLink: /apis/networking.istio.io/v1alpha3/namespaces/istio-system/gateways/dreamy-plum-bee-gtw
  uid: cabffdf1-8e6a-11e9-9ba7-42010a9801a6
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - dreamy-plum-bee.somewhere.net
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: dreamy-plum-bee-certificate
      mode: SIMPLE
      privateKey: sds
      serverCertificate: sds

Now with the current setup, if I test with openssl command:

$ $ openssl s_client -connect dreamy-plum-bee.somewhere.net:443
CONNECTED(00000005)
write:errno=54
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 0 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Start Time: 1560492782
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---

In Chrome browser, it fails to visit the page with ERR_CONNECTION_RESET error message.

However, if I change Gateway's tls setting with self-signed filesystem based certificate like:

    tls:
      mode: PASSTHROUGH
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key

The site is reachable. Hence, I'm suspecting something is not right with credentialName setting. The Gateway doesn't seem to be able to pick up Certificate resource to initiate the connection.

Any advice would be appreciated like things to check/debug etc...

-- skim-so
cert-manager
google-kubernetes-engine
istio
kubernetes
ssl-certificate

1 Answer

6/16/2019

Eventually I figured out and Envoy SDS: Fortifying Istio Security - Yonggang Liu & Quanjie Lin, Google was very helpful.

  • Installed Istio from scratch (v1.1.8) instead of using addon (v1.1.3)
  • Make sure --set gateways.istio-ingressgateway.sds.enabled=true is used during the installation.
  • Enable istio-injection=enabled on the namespace for envoy proxy to be created.
  • Increase the node capacity to host Istio properly. Google suggest that at least a 4 node cluster with the 2 vCPU machine type is required.
  • Finally, remove manual TLS certificate from NodeApp I was deploying as Istio handles TLS and mTLS was not enabled yet.
-- skim-so
Source: StackOverflow