Getting Error Varnish returning HTTP/1.1 503 Backend fetch failed/ X-Cache: uncached when executing curl -IL (external Ip)

3/17/2020

I have created helm chart for varnish cache server which is running in kubernetes cluster , while testing with the "external IP" generated its throwing error , sharing below

HTTP/1.1 503 Backend fetch failed
Date: Tue, 17 Mar 2020 08:20:52 GMT
Server: Varnish
Content-Type: text/html; charset=utf-8
Retry-After: 5
X-Varnish: 570521
Age: 0
Via: 1.1 varnish (Varnish/6.3)
X-Cache: uncached
Content-Length: 283
Connection: keep-alive

Sharing varnish.vcl, values.yaml and deployment.yaml below . Any suggestions how to resolve as I have hardcoded the backend server as .host="www.varnish-cache.org" with port : "80". My requirement is on executing curl -IL I should get the response with cached values not as described above (directly from backend server)..

varnish.vcl:

# VCL version 5.0 is not supported so it should be 4.0 or 4.1 even though actually used Varnish version is 6
vcl 4.1;

import std;
# The minimal Varnish version is 5.0
# For SSL offloading, pass the following header in your proxy server or load balancer: 'X-Forwarded-Proto: https'



{{  .Values.varnishconfigData | indent 2 }}







sub vcl_recv {
  if(req.url == "/healthcheck") {
    return(synth(200,"OK"));
  }
}


sub vcl_deliver {
    if (obj.hits > 0) {
        set resp.http.X-Cache = "cached";
    } else {
        set resp.http.X-Cache = "uncached";
    }
}

values.yaml:

# Default values for tt.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

replicaCount: 1



#varnishBackendService: "www.varnish-cache.org"
#varnishBackendServicePort: "80"

image:
  repository: varnish
  tag: 6.3
  pullPolicy: IfNotPresent

nameOverride: ""
fullnameOverride: ""

service:
#  type: ClusterIP
  type: LoadBalancer
  port: 80
#  externalIPs: 192.168.0.10




varnishconfigData: |- 
      backend default {
         .host = "www.varnish-cache.org";
         .host = "100.68.38.132"
         .port = "80";
         .first_byte_timeout = 60s;
         .connect_timeout = 300s ;
         .probe = {
                .url = "/";
                .timeout = 1s;
                .interval = 5s;
                .window = 5;
                .threshold = 3;
           }
          }
         sub vcl_backend_response {
          set beresp.ttl = 5m;
         }




ingress:
  enabled: false
  annotations: {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  path: /
  hosts:
    - chart-example.local
  tls: []
  #  - secretName: chart-example-tls
  #    hosts:
  #      - chart-example.local

resources:
  limits:
    memory: 128Mi
  requests:
    memory: 64Mi

#resources: {}
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #  cpu: 100m
  #  memory: 128Mi
  # requests:
  #  cpu: 100m
  #  memory: 128Mi

nodeSelector: {}

tolerations: []

affinity: {}

Deployment.yaml:

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: {{ include "varnish.fullname" . }}
  labels:
    app: {{ include "varnish.name" . }}
    chart: {{ include "varnish.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: {{ include "varnish.name" . }}
      release: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app: {{ include "varnish.name" . }}
        release: {{ .Release.Name }}
    spec:
      volumes: 
        - name: varnish-config
          configMap:
             name: {{ include "varnish.fullname" . }}-varnish-config
             items:
               - key: default.vcl
                 path: default.vcl
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }} 
         # command: ["/bin/sh"]
         # args: ["-c", "while true; do service varnishd status, varnishlog;  sleep 10;done"]   
          env:
          - name: VARNISH_VCL
            value: /etc/varnish/default.vcl
          volumeMounts: 
            - name: varnish-config
              mountPath : /etc/varnish/
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
              targetPort: 80
          livenessProbe: 
            httpGet:
              path: /healthcheck
              port: http
              port: 80
            failureThreshold: 3
            initialDelaySeconds: 45
            timeoutSeconds: 10
            periodSeconds: 20
          readinessProbe:
            httpGet:
              path: /healthcheck
              port: http
              port: 80
            initialDelaySeconds: 10
            timeoutSeconds: 15
            periodSeconds: 5
          restartPolicy: "Always"
          resources:
{{ toYaml .Values.resources | indent 12 }}
    {{- with .Values.nodeSelector }}
      nodeSelector:
{{ toYaml . | indent 8 }}
    {{- end }}
    {{- with .Values.affinity }}
      affinity:
{{ toYaml . | indent 8 }}
    {{- end }}
    {{- with .Values.tolerations }}
      tolerations:
{{ toYaml . | indent 8 }}
    {{- end }}
-- Debasis Singh
kubernetes
kubernetes-helm
varnish

1 Answer

3/17/2020

The HTTP/1.1 503 Backend fetch failed error indicates that Varnish is unable to connect to the backend host and port.

I advise you to try some good old manual debugging:

  • Access the bash shell of one of the containers
  • Open /etc/varnish/default.vcl and check the exact hostname & port that were parsed into the backend definition
  • Make sure curl is installed and try to curl the hostname on that specific port
  • Maybe even install telnet and try to see if the port of the hostname is accepting connections

Basically you'll try to figure out if there is a network configuration that is prohibiting you from making an outbound connection, or if there's something else preventing Varnish from making the fetch.

-- Thijs Feryn
Source: StackOverflow