Get "unable to find api field in struct Container" with "kubectl patch" called through Ansible

2/24/2020

I'm trying to debug a problem I don't know enough about. It involves an integration of Ansible, Helm, and Kubectl, all running in a container on a CI server.

We have a Docker image that runs Ansible. I don't know if it's a custom image or not. It's called "ansible:2.2", so it seems like it's referring to version 2.2 of Ansible.

We recently made some build changes that integrate Helm. Instead of storing k8s templates in git, we use Helm to generate the templates, and fill in some additional properties with Ansible.

In two of the services I've upgraded to use the new build system, I see an Ansible error message that starts like this:

failed: [] (item=/home/aft/ansible/roles/_configrole/tasks/../templates/k8s/deployment_bg.yaml) => {"changed:" true, "cmd": ["/usr/bin/kubectl", ...], ..., "stderr: "error: error when applying patch:\n\nto:...

This is followed by large blocks of json, and finally ending with:

unable to find api field in struct Container for the json field \"$setElementOrder/env\"

In the overall large error message, seemingly as part of the "kubectl patch" error message, I see three large blocks labeled with "original", "modified", and "current", all of which are slightly different variations of a k8s Deployment object.

I can't figure out the relationship of these three "original", "modified", and "current" blocks to the text of this error message.

In the first service that we saw this error message in, one of the build admins said that he tried deleting the existing Deployment object instead of letting the process try to modify it, and that appeared to work, but I believe that is not ideal.

-- David M. Karr
ansible
kubectl
kubernetes-helm

1 Answer

3/5/2020

Just to sum up the discussion in comments.

The issue has been fixed according to: .../issues/50040 and .../pull/49834.

Additionally, documentation [ 1 ] [ 2 ] states that:

You must use a kubectl version that is within one minor version difference of your cluster. For example, a v1.2 client should work with v1.1, v1.2, and v1.3 master. Using the latest version of kubectl helps avoid unforeseen issues.

In highly-available (HA) clusters, the newest and oldest kube-apiserver instances must be within one minor version.

kubelet must not be newer than kube-apiserver, and may be up to two minor versions older.

kube-controller-manager, kube-scheduler, and cloud-controller-manager must not be newer than the kube-apiserver instances they communicate with. They are expected to match the kube-apiserver minor version, but may be up to one minor version older (to allow live upgrades).

kubectl is supported within one minor version (older or newer) of kube-apiserver

-- Nick
Source: StackOverflow