Iam trying to perform kubernetes webhook authentication using pamhook. But I get a TLS handshake error at the webhook when apiserver tries to contact it.

Following are the steps I followed:

apiserver IP: 192.168.20.30 pam webhook server IP: 192.168.20.50

1. Created a server certificate for my pam webhook server using the ca.crt present in /etc/kubernetes/pki/ (in master node)

> openssl genrsa -out ubuntuserver.key 2048 openssl req -new -key
> ubuntuserver.key -out ubuntuserver.csr -config myconf.conf openssl
> x509 -req -in ubuntuserver.csr -CA ca.crt -CAkey ca.key
> -CAcreateserial -out ubuntuserver.crt  -days 10000

2. Started my pam webhook server using this certificate.

 ./pam_hook-master -cert-file /root/newca/ubuntuserver.crt -key-file /root/newca/ubuntuserver.key -signing-key rootroot -bind-port 6000  

I1109 07:21:41.388836 3882 main.go:327] Starting pam_hook on :6000

3. Created a kubeconfig file as follows:

 $cat webhook-config.yaml
    apiVersion: v1
    clusters:
    - cluster:
        certificate-authority: /etc/kubernetes/pki/ca.crt
        server: https://192.168.20.50:6000/authenticate
      name: 192.168.20.50
    users:
      - name: root
        user:
          client-certificate: /etc/kubernetes/pki/client.crt
          client-key: /etc/kubernetes/pki/client.key
    current-context: 192.168.20.50
    contexts:
    - context:
        cluster: 192.168.20.50
        user: root
      name: 192.168.20.50

4. Configured api server manifest file in

   /etc/kubernetes/manifest/kube-apiserver.yaml

   > ...
   >     - --authentication-token-webhook-config-file=/etc/kubernetes/pki/webhook-config.yaml
   >     - --runtime-config=authorization.k8s.io/v1beta1=true ...

5. apiserver gets restarted.

6. Obtain token by requesting pamhook server. (I did this from my master node)

   $ curl https://192.168.20.50:6000/token --cacert /etc/kubernetes/pki/ca.crt -u root                                    
   
   Enter host password for user 'root':
   eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiIiLCJleHAiOjE1MTAyMjY3ODksImlhdCI6MTUxMDIyNjE4OSwiaXNzIjoiIiwidXNlcm5hbWUiOiJyb290In0.3LmHBy_anjR62WNqKICCx_b8YWFpF4HSKMWLmyORU0M

7. Made a request to apiserver using this token, which in turn should communicate with webhook to provide authentication. But Iam getting a 401 Error.

$ curl -vvv --insecure -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiIiLCJleHAiOjE1MTAyMjY3ODksImlhdCI6MTUxMDIyNjE4OSwiaXNzIjoiIiwidXNlcm5hbWUiOiJyb290In0.3LmHBy_anjR62WNqKICCx_b8YWFpF4HSKMWLmyORU0M" https://192.168.20.38:6443/api/

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}

Message at the webhook server is:

> 2017/11/09 07:50:53 http: TLS handshake error from 192.168.20.38:49712: remote error: tls: bad certificate

8. Document says that api server sends a http request with the token in its payload. If I try to recreate the same call in curl using the token and ca.crt, it gets authenticated.

   >   $ curl -X POST https://192.168.20.50:6000/authenticate --cacert /etc/kubernetes/pki/ca.crt -d '{"ApiVersion":"authentication.k8s.io/v1beta1", "Kind": "TokenReview", "Spec":{"Token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiIiLCJleHAiOjE1MTAyMjY3ODksImlhdCI6MTUxMDIyNjE4OSwiaXNzIjoiIiwidXNlcm5hbWUiOiJyb290In0.3LmHBy_anjR62WNqKICCx_b8YWFpF4HSKMWLmyORU0M"}}'

   {"apiVersion":"authentication.k8s.io/v1beta1","kind":"TokenReview","status":{"authenticated":true,"user":{"username":"root","uid":"0","groups":["root"]}}}

But when it is requested by apiserver, TLS handshake gets failed.

My understanding is that the TLS verification is done by checking against the certificate-authority file mentioned in webhook-config.yaml right? If so, the TLS verification should have been successful with ca.crt. But it is failing.

Does that mean api server is performing validation using some other CA? Which CA does it use? How do I go past this TLS verification successfully?