Has anyone tried out nested kubernetes cluster inside kubernetes, I want to emulated sandboxed a small kubernetes cluster inside my running kubernetes cluster and not again use VM's again and again for new cluster.
New clusters I want to used for testing and trail of various features of kubernetes, has anyone tried anything on it.
I tried various solution like.
I also tried https://github.com/kubernetes-sigs/kubeadm-dind-cluster and this works for me, but it starts the k8s cluster using docker containers, but I wanted this sort of solution to work inside kubernetes, so that it can be scaled across nodes I have.
If anyone has tried it or can point to solution people have tried, it can be of great help.
Edit 1:
I know k8s runs workloads as containers and this was part of my problem, "is it possible to run k8s using containers or pods or using any other k8s constructs".
I know, we can create namespace and network policies around it, but that would regulate the system, I wanted people to play with master/api-servers freely, un-regulated like we do in minikube. Wanted to have freedom to screw up master/api-servers, and to clean mess probably just need something like k8s delete cluster c1 or anything of that sort.
And for creating those clusters in sandbox cluster was just a command operation. Just didn't wanted to spin vm's again and setup cluster, if someone wants to play with it,
I think you have a fundamental misunderstanding of what kubernetes does. Kubernetes, as workloads, can only run containers. If you want to deploy a new kubernetes cluster within an existing cluster, it has to be run in containers. I don't understand how you plan to run a second cluster within the first if you don't want to provision new VMs or run in containers. Where do you see your cluster running? Here are some options as I see it:
Use case 1: Test new version of kubernetes before upgrading productioncluster
Solution: You should really be testing this on similar/identical hardware/VMs as production. You won't be able to fully validate if you spin it up within an exisitng cluster
User case 2: Test new container versions, service setups, etc.
Solution: Use a new namespace with some network policies to shield it from the rest of the cluster. You can go one step further and use node taints to only allow the namespaced pods run on specific worker nodes.
Use case 3: You are trying to let developers play around without messing with production.
Solution: I'd suggest a seperate sandbox cluster is warrented. However, if you're willing to accept the risk, see the solution for #2
Use case 4: You're deploying new kubernetes clusters for CI/CD pipelines
Solution: I'd recommend separating the jobs to testing kubernetes cluster creation seperately then the workloads it'll run. One job tests the cluster creation automation, if it passes, you can promote it to the staging cluster. Use the staging cluster to test your workload jobs.
Use case 5: You really really just want to run multiple kubernetes clusters within them.
Solution: It's quite a hack, and I don't see how you can avoid creating new VMs, but you can look into a mixture of custom automation and a virtual-kubelete to register your second cluster as a "node" on the first cluster. This will really only let you deploy pods though. Running duplicate kubernetes processes on the same VMs will be quite difficult to achieve, mainly from a networking perspective.
If none of these apply, can you better describe your usecase?
I understand that for some reason you don't want VM, but if you would change mind a bit, you could use https://github.com/Mirantis/virtlet/blob/master/examples/k8s.yaml which is an example how to start kubernetes in kubernetes, while nested nodes will be pods (to be more precise - VM pods, started using Virtlet as CRI compatible runtime), based on StatefulSet described in this single yaml.