ansible k8s module failing to connect to cluster with 503 - appends /version/openshift to non openshift cluster

2/4/2019

I'm trying to use ansible new k8s module (based ok k8_raw from 2.6) to maintain an aks k8 cluster. While I can work with the cluster with kubectl , any command with the k8s cluster fails with a 503 error.

For example this task:

- name: deploy kured daemonset
  k8s:
    state: present
    context: "{{ cluster_name}}"
    host: "redacted"# tried specifying this, but does not help
    kubeconfig: "~/.kube/config"
    src: "aks/utils/kured-ds.yaml"

And failure:

Traceback (most recent call last):
  File "/home/alonisser/.ansible/tmp/ansible-tmp-1549320815.98-157731551192134/AnsiballZ_k8s.py", line 113, in <module>
    _ansiballz_main()
  File "/home/alonisser/.ansible/tmp/ansible-tmp-1549320815.98-157731551192134/AnsiballZ_k8s.py", line 105, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/home/alonisser/.ansible/tmp/ansible-tmp-1549320815.98-157731551192134/AnsiballZ_k8s.py", line 48, in invoke_module
    imp.load_module('__main__', mod, module, MOD_DESC)
  File "/tmp/ansible_k8s_payload_IYmGFG/__main__.py", line 233, in <module>
  File "/tmp/ansible_k8s_payload_IYmGFG/__main__.py", line 229, in main
  File "/tmp/ansible_k8s_payload_IYmGFG/ansible_k8s_payload.zip/ansible/module_utils/k8s/raw.py", line 131, in execute_module
  File "/tmp/ansible_k8s_payload_IYmGFG/ansible_k8s_payload.zip/ansible/module_utils/k8s/common.py", line 172, in get_api_client
  File "/home/alonisser/.local/lib/python2.7/site-packages/openshift/dynamic/client.py", line 103, in __init__
    self.__init_cache()
  File "/home/alonisser/.local/lib/python2.7/site-packages/openshift/dynamic/client.py", line 113, in __init_cache
    self.__resources.update(self.parse_api_groups())
  File "/home/alonisser/.local/lib/python2.7/site-packages/openshift/dynamic/client.py", line 169, in parse_api_groups
    new_group[version] = self.get_resources_for_api_version(prefix, group['name'], version, preferred)
  File "/home/alonisser/.local/lib/python2.7/site-packages/openshift/dynamic/client.py", line 181, in get_resources_for_api_version
    resources_response = load_json(self.request('GET', path))['resources']
  File "/home/alonisser/.local/lib/python2.7/site-packages/openshift/dynamic/client.py", line 363, in request
    _return_http_data_only=params.get('_return_http_data_only', True)
  File "/home/alonisser/.local/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/home/alonisser/.local/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api
    _request_timeout=_request_timeout)
  File "/home/alonisser/.local/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request
    headers=headers)
  File "/home/alonisser/.local/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET
    query_params=query_params)
  File "/home/alonisser/.local/lib/python2.7/site-packages/kubernetes/client/rest.py", line 222, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (503)
Reason: Service Unavailable

Ansible version: 2.7/8(dev)

What am I missing?

UPDATE: When I've added print statement to the libs used by the module beneath I found out somewhere in the pipeline /version/openshift is appended to the host name, which of course fails, because it's a non openshift cluster Any work around for this bug?

-- alonisser
ansible
kubernetes

1 Answer

2/6/2019

Answer: turned out there were two failing requests. the first is to version/openshift is catched by the client and doesn't cause the crash. the crash actually happened because of an error with my cluster metrics server, which while not really needed by the k8 client used by ansible still fails a request to it. So if anyone bumps into it, might be helpful

-- alonisser
Source: StackOverflow