Unable to run VSTS Agents on Kubernetes using Azure Devops
I recently spun this chart up on kubernetes. It has been archived. These are the changes I have made to get it to run:
Image
Changed the image from microsoft/vsts-agent to mcr.microsoft.com/azure-pipelines/vsts-agent
Changed the tags from latest to ubuntu-16.04-docker-17.12.0-ce
Limits
Reduced the limits from 4 CPU and 8Gbi to 1 and 1 respectively because I'm cheap and am using weak VMs.
Api Version
Changed from apiVersion: apps/v1beta2 to apiVersion: apps/v1 to allow for Statefulset
That being said everything spun up and was healthy on the cluster. So far so good, right? Well. Not so good.
The next step was to actually use the agents. I setup Azure Devops with the agent pool and configured a very simple pipeline using the GUI.
Step 1 succeeds but step 2 fails. So this is where I'm down the rabbit hole. The chart uses a mounted docker-socket which I believe has been removed from AKS (read about this here).
I am not sure why step 1 would succeed, telling me that docker is installed, but step 2 fails citing that it cannot find docker.
The main error I am seeing: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Is there an alternative solution out there? Right now I just paid for a hosted agent on Devops just to get some builds through but I do not want to do this, I want to use Kubernetes if possible to run these agents.
Similar Questions
1 Answer
Running on (any recent version) of AKS you cannot use docker-in-docker (which is what you want to do here). See this article for the limitations that got introduced when AKS switched to containerd: https://docs.microsoft.com/en-us/azure/aks/cluster-configuration#containerd-limitationsdifferences
As an alternative I suggest using VMSS-self hosted Build Agents. Works like a charm for me. Use cloud-init to define the software you want to have and ADO takes care of scaling up and down your agents.