Portworx to ETCD connection doesn't work (http vs https)

7/2/2018

Done according to this article.

I installed Kubernetes. Then installed etcd cluster that works via HTTPS and listens to the localhost interface only (reachable from inside any Docker container). Now I need persistent volume to install DB cluster. Chose Portworx. It generated daemonset YAML-config. Here is the description of installed daemonset:

# kubectl describe daemonset portworx --namespace=kube-system

Name:           portworx
Selector:       name=portworx
Node-Selector:  <none>
Labels:         name=portworx
Annotations:    kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"extensions/v1beta1","kind":"DaemonSet","metadata":{"annotations":{"portworx.com/install-source":"http://install.portworx.com/?c=bp_clust...
                portworx.com/install-source=http://install.portworx.com/?c=bp_cluster&k=etcd:https://127.0.0.1:2379&kbver=1.11.0&s=/dev/xvda1&d=ens3&m=ens3&stork=false&ca=/etc/kubernetes/pki/etcd/ca.crt%%20&cert=/etc...
Desired Number of Nodes Scheduled: 2
Current Number of Nodes Scheduled: 2
Number of Nodes Scheduled with Up-to-date Pods: 2
Number of Nodes Scheduled with Available Pods: 0
Number of Nodes Misscheduled: 0
Pods Status:  2 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           name=portworx
  Service Account:  px-account
  Containers:
   portworx:
    Image:      portworx/oci-monitor:1.3.4
    Port:       <none>
    Host Port:  <none>
    Args:
      -k
      etcd:https://127.0.0.1:2379
      -c
      bp_cluster
      -d
      ens3
      -m
      ens3
      -s
      /dev/xvda1
      -ca
      /etc/kubernetes/pki/etcd/ca.crt 
      -cert
      /etc/kubernetes/pki/etcd/server.crt
      -key
      /etc/kubernetes/pki/etcd/server.key
      -x
      kubernetes
    Liveness:   http-get http://127.0.0.1:9001/status delay=840s timeout=1s period=30s #success=1 #failure=3
    Readiness:  http-get http://127.0.0.1:9015/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      PX_TEMPLATE_VERSION:  v3
    Mounts:
      /etc/pwx from etcpwx (rw)
      /etc/systemd/system from sysdmount (rw)
      /host_proc/1/ns from proc1nsmount (rw)
      /opt/pwx from optpwx (rw)
      /var/run/dbus from dbusmount (rw)
      /var/run/docker.sock from dockersock (rw)
  Volumes:
   dockersock:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/docker.sock
    HostPathType:  
   etcpwx:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/pwx
    HostPathType:  
   optpwx:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/pwx
    HostPathType:  
   proc1nsmount:
    Type:          HostPath (bare host directory volume)
    Path:          /proc/1/ns
    HostPathType:  
   sysdmount:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/systemd/system
    HostPathType:  
   dbusmount:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/dbus
    HostPathType:  
Events:
  Type    Reason            Age   From                  Message
  ----    ------            ----  ----                  -------
  Normal  SuccessfulCreate  22m   daemonset-controller  Created pod: portworx-67w7m
  Normal  SuccessfulCreate  22m   daemonset-controller  Created pod: portworx-mxtr8

But in the log of portworx I see that it is trying to connect to etcd via plain HTTP and obviously get error because cannot interpreter the response wrapped to SSL:

# kubectl logs -f pod/portworx-67w7m --namespace=kube-system

<some logs are erased du to lack of relevance>

Jul 02 13:19:25 ip-172-31-18-91 px-runc[25417]: time="2018-07-02T13:19:25Z" level=error msg="Could not load config file /etc/pwx/config.json due to: Error in obtaining etcd version: Get http://127.0.0.1:2379/version: net/http: HTTP/1.x transport connection broken: malformed HTTP response \"\\x15\\x03\\x01\\x00\\x02\\x02\".  Please visit http://docs.portworx.com for more information."
Jul 02 13:19:25 ip-172-31-18-91 px-runc[25417]: PXPROCS: px daemon exited with code: 1
Jul 02 13:19:25 ip-172-31-18-91 px-runc[25417]: 2107
Jul 02 13:19:25 ip-172-31-18-91 px-runc[25417]: 2018-07-02 13:19:25,474 INFO exited: pxdaemon (exit status 1; not expected)

What I am doing wrong?

-- Michael A.
kubernetes
persistent-volumes

1 Answer

7/3/2018

I have no idea why they didn't surface the "cannot read the -cert file" error, but you specified /etc/kubernetes/pki/etcd/server.crt in the options but did not volume mount /etc/kubernetes/pki into the container. For obvious reasons, kubernetes will not automatically volume mount its pki directory, thus, you must specify it.

If that DaemonSet was generated for you (as it appears based on the annotation), then what happened is that they are expecting the certs to live in /etc/pwx/etcdcerts (it's in their manual provisioning docs also), so when you provided a non-/etc path, the two worlds collided.

-- mdaniel
Source: StackOverflow