I am encountering a weird behavior when i try to configure several KUBECONFIG environment entries concatenated with : such in the example here :
export KUBECONFIG=/Users/user/Work/company/project/setup/secrets/dev-qz/users/admin.conf:/Users/user/Work/company/project/setup/secrets/dev-wer/users/admin.conf:/Users/user/Work/company/project/setup/secrets/test-wer/users/admin.conf:/Users/user/Work/company/project/setup/secrets/test/users/admin.conf:/Users/user/Work/company/project/setup/secrets/dev-jg/users/admin.conf:/Users/user/Work/company/project/setup/secrets/preprod/users/admin.conf:/Users/user/Work/company/project/setup/secrets/dev/users/admin.conf:/Users/user/Work/company/project/setup/secrets/dev-fxc/users/admin.conf:/Users/user/Work/company/project/setup/secrets/cluster-setup/users/admin.conf:/Users/user/Work/company/project/setup/secrets/test-fxc/users/admin.conf:/Users/user/Work/company/project/setup/secrets/test-jg/users/admin.conf:/Users/user/Work/company/project/setup/secrets/test-qz/users/admin.confThis is what is happening: if i choose with kubectx the cluster (not every cluster from the list, but just any), when i try kubectl get po i receive : error: You must be logged in to the server (Unauthorized) . But, if try to reach the same cluster passing it directly to the kubectl command with --kubeconfig=<path to the config> it works. I am pretty struggling with this and just wanna know if anyone else is facing this kind of issues as well and how have solved it
I agree with @Bernard. This doesn't look like anything specific to kubectx as it is just a bash script, which under the hood uses kubectl binary. You can see its code here. I guess that it will also fail in kubectl if you don't provide the
But, if try to reach the same cluster passing it directly to the kubectl command with --kubeconfig= it works.
There is a bit of inconsistency in the way you're testing it as you don't provide the specific kubeconfig file to both commands. When you use kubectx it relies on your multiple in-memory merged kubeconfig files and you compare it with working kubectl example in which you directly specify the kubeconfig file that should be used. To make this comparison consistent you should also use kubectx with this particular kubeconfig file. And what happens if you run kubectl command without specifying --kubeconfig=<path to the config> ? I guess you get similar error to the one you get when running kubectx. Please correct me if I'm wrong.
There is a really good article written by Ahmet Alp Balkan - kubectx author, which nicely explains how you can work with multiple kubeconfig files. As you can read in the article:
Tip 2: Using multiple kubeconfigs at once
Sometimes you have a bunch of small kubeconfig files (e.g. one per cluster) but you want to use them all at once, with tools like
kubectlorkubectxthat work with multiple contexts at once.To do that, you need a “merged” kubeconfig file. Tip #3 explains how you can merge the kubeconfigs into a single file, but you can also merge them in-memory.
By specifying multiple files in
KUBECONFIGenvironment variable, you can temporarily stitch kubeconfig files together and use them all in kubectl.export KUBECONFIG=file1:file2 kubectl get pods --context=cluster-1 kubectl get pods --context=cluster-2Tip 3: Merging kubeconfig files
Since kubeconfig files are structured YAML files, you can't just append them to get one big kubeconfig file, but kubectl can help you merge these files:
KUBECONFIG=file1:file2:file3 kubectl config view --merge --flatten > out.txt
Try to merge your multiple kubeconfig files to a single one like in the example above to see if it's possibly problem only with in-memory merge:
KUBECONFIG=file1:file2:file3 kubectl config view --merge --flatten > out.txt
Review all your kubeconfigs and test them individually just to make sure if they're working properly when specified in KUBECONFIG env variable separately. There might be some error in one of them which causes the issue.
Eventually i found the problem. The flatten command that suggested to me @mario, helped me to debug better the situation.
Basically, the in memory or in file merge, makes what it supposed to do: create a kubeconfig with all uniq parameters of each kubeconfig files. This works perfectly unless on or more kubeconfig has the same labels that identify the same component. In this case the last in order wins. So if you have the following example:
grep -Rn 'name: kubernetes-admin#x27; infra/secrets/*/users/admin.conf
infra/secrets/cluster1/users/admin.conf:16:- name: kubernetes-admin
infra/secrets/cluster2/users/admin.conf:17:- name: kubernetes-admin
infra/secrets/cluster3/users/admin.conf:16:- name: kubernetes-admincluster1 and cluster2 won't work, while cluster3 will work perfectly, incidentally due to the order. The solution to this problem is to avoid non uniq fields, by renaming the label that identifies the user (for the example above). Once is done this change, everything will work perfectly.