I started looking more closely to kubernetes, containers and virtualization technologies since my employer has decided to move everything to Azure and AKS.
From what I understand, everything in AKS will be running inside VM's.
The same applies to GKE and EKS. Amazon provides some bare metal instances but I am not sure whether this works with kubernetes.
Doesn't that hurt performance? The promise of containers was that you run on a lighter virtualization layer instead on a full blown vm.
Doesn't running containers inside vm's beat the purpose?
A (Linux) container is a set of 1 or more processes that are isolated from the rest of the system through Linux namespaces. Namespaces are a feature of the Linux kernel that partitions kernel resources (mounts, network devices etc) such that one set of processes sees one set of resources while another set of processes sees a different set of resources. For example, each container sees a different root file system (the container image) and its process tree is isolated from the rest of the process tree of the host.
Apart from being isolated through namespaces, the container is also limited in how much resources it can use by cgroups, also a feature of the Linux Kernel.
In other words, containers aren't really "virtualization", since all it is using is native Kernel functionality and the overhead of using container (once started) is extremely small, once it is bootstrapped it runs just like any other process.
Kubernetes is a system for automating deployment, scaling, and management of containerized applications and it needs somewhere to host the containers. This infrastructure can be either bare metal or VM's.
Thus running containerized applications inside a VM brings the same benefits as running them on bare metal (isolation and resource consumption limits). Whether to provide compute resources to the cluster through bare metal or VM's is another questions, each having its pros and cons, but the pros and cons of VM's vs bare metal do not depend on whether you run containers or regular applications and processes on them.
Your observation that AKS; GKE and EKS are using VM's to provide compute resources to the cluster is correct. It abstracts the physical hardware away and is currently the standard way by public cloud providers to provide compute resources
This is quite a popular debate in the community and has been going on for quite some time now.
I would say that it doesn't really defeat the purpose per-se but there are advantages and disadvantages of choosing one over the other.
If we were to look at this objectively, then some of the key deciding factors are:
I've put together a couple of small tables to highlight some of the trade-offs and would like to invite the community members to add to this as well.
Bare metal
| Advantages | Disadvantages |
|---|---|
| More cost-effective as the Hyperviser licensing cost can be avoided | On the flip side, would miss out on advanced functionalities available through virtualisation |
| Better performance and more optimal use of resources | Operationally inconvenient - for example upgrades to bare metal could be long and difficult |
Virtual machines
| Advantages | Disadvantages |
|---|---|
| Workload migration could be much better between VMs is likely to be smooth due to the presence of virtualisation, hence this can be convenient operationally for carrying out major upgrades | The same virtualisation layer would consume resources and therefore not very cost-effective. It also adds an extra layer of tech support should things go wrong |
| Due to the isolation of resources through VM creation, possibility of security leaks gets reduced greatly | The downside of the same factor i.e. resource(s) isolation, is that any resources for example storage allocated to a VM could go remain unused in the times of peak demand |
The list of points above is only an indicator of the trade-offs and by no means exhaustive and it's important to consider the overall context of the infrastructure deployment for these decisions.