I'm using k3d with calico, and trying to use calicoctl
to set a network policy. I am able to get this working on Ubuntu, but on Windows it doesn't. To set it up I did the following:
# Download Calico YAML
# DOWNLOAD: https://raw.githubusercontent.com/rancher/k3d/main/docs/usage/guides/calico.yaml
# Create k3d cluster
k3d cluster create "dev" --k3s-server-arg '--flannel-backend=none' --volume "$(pwd)/calico.yaml:/var/lib/rancher/k3s/server/manifests/calico.yaml
# Download Windows version
# DOWNLOAD: https://github.com/projectcalico/calicoctl/releases/download/v3.18.2/calicoctl-windows-amd64.exe
# Make an Alias
Set-Alias -Name calicoctl -Value ".\calicoctl-windows-amd64.exe"
When I then tried running to apply a NetWorkPolicy
with calicoctl apply -f my-policy.yaml
and I get the error:
Failed to create Calico API client: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
So I tried setting that by using my Kubeconfig's values:
# Get the IP/Port from Kubeconfig (Gives https://0.0.0.0:60776)
$kmaster=(kubectl config view -o jsonpath="{.clusters[?(@.name=='k3d-dev')].cluster.server}")
# Set the kubernetes master env variable from what's in kube config
[Environment]::SetEnvironmentVariable(“KUBERNETES_MASTER”, $kmaster, “User”)
That still gives the same error. (Note: the ip from Kubeconfig is "https://0.0.0.0:60776")
So then I tried all the following and none of them work:
set KUBECONFIG=~/.kube/config
SET CALICO_KUBECONFIG=C:\Users\myname\.kube\config
SET KUBE_CONFIG_PATH=C:\Users\myname\.kube\config
$env:KUBERNETES_MASTER=(kubectl config view -o jsonpath="{.clusters[?(@.name=='k3d-dev')].cluster.server}")
How do I get "calicoctl-windows-amd64.exe" to find my Kubernetes cluster?
I tried the three options mentioned above and I've found that the third option works for me.
Here are the steps I've followed:
$env:KUBECONFIG="C:\Users\myusername\.kube\config"
calicoctl get ippools
Note: I am using calicoctl version v3.22.0
While the official documentation claims:
~/.kube/config
KUBERNETES_MASTER
(the error output states this)KUBECONFIG
It does NOT! None of the above works on Windows. (On Ubuntu, I did not have to do anything special and it found my config)
The only solution I found was to do the following:
calicoctl
callcalico.cfg.yaml
apiVersion: projectcalico.org/v3
kind: CalicoAPIConfig
metadata:
spec:
datastoreType: "kubernetes"
kubeconfig: "C:/users/myusername/.kube/config"
NOTE: If you pass "~/.kube/config" it will fail and say it cannot find that path
Calling it now requires:
# NOTICE THE "--config=calico.cfg.yaml"
calicoctl apply -f some-policy.yaml --config=calico.cfg.yaml
I think, judging by the config file working but throwing an error when using "~/.kube/config"
as the kubeconfig path, that it is looking in that "default" path but is using a Windows API that does not understand the ~
alias.