Ingress backend REST error: The server encountered a temporary error and could not complete your request. Please try again in 30 seconds

3/2/2018

I am deploying application at google kubernetes engine. Applicaion has 2 services. There is also Ingress wich i am trying to use to expose one service and ingress also used for https support. I have 1 NodePort service "gateway" and ClusterIp service "internal". "Internal" should be accessed from gateway. Here is services config:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: x-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: x-x-ip
    kubernetes.io/tls-acme: "true"
  labels:
    app: gateway
spec:
  tls:
    - secretName: secret
      hosts:
      - x.x.com
  backend:
    serviceName: gateway
    servicePort: 80
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: x-x
spec:
  acme:
    server: https://acme-v01.api.letsencrypt.org/directory
    email: x@x.com
    privateKeySecretRef:
      name: x-x
    http01: {}
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
  name: gateway
  labels:
    app: gateway
spec:
  type: NodePort
  ports:
  - port: 80
    name: gateway
    targetPort: 8080
  selector:
    app: gateway
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
  name: internal
  labels:
    app: internal
spec:
  ports:
  - port: 8082
    name: internal
    targetPort: 8082
  selector:
    app: internal

Gateway serve static content and REST resources. Static content is served ok, so i see html and images and scripts. But when i try to call REST endpoint i got The server encountered a temporary error and could not complete your request. Please try again in 30 seconds. Gateway forward REST request to "internal" service and return response from internal. Gateway access internal service with url http://internal:8082/some/rest. I got errors when i call any request wich should be forwarded to "internal".

Actualy i have same scheme without Ingress and it works. "Gateway" is LoadBalancer service and "internal" is NodePort. I need Ingress for https.

UPD: I noticed i don't have any forwading rules related to 8082 port, only 80 and 443( i have used gcloud compute forwarding-rules list and gcloud compute forwarding-rules describe commands).

Here is output of kubectl describe svc

Name:                     gateway
Namespace:                default
Labels:                   app=gateway
Annotations:              service.alpha.kubernetes.io/tolerate-unready-endpoints=true
Selector:                 app=gateway
Type:                     NodePort
IP:                       *
Port:                     gateway  80/TCP
TargetPort:               8080/TCP
NodePort:                 gateway  31168/TCP
Endpoints:                *:8080
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>
---
Name:              internal
Namespace:         default
Labels:            app=internal
Annotations:       service.alpha.kubernetes.io/tolerate-unready-endpoints=true
Selector:          app=internal
Type:              ClusterIP
IP:                *
Port:              internal  8082/TCP
TargetPort:        8082/TCP
Endpoints:         *:8082
Session Affinity:  None
Events:            <none>

UPD2: Here is output of curl -v to problem url:

*   Trying *.*.*.*...
* TCP_NODELAY set
* Connected to *.*.*.* port 80 (#0)
> GET /internal/ping HTTP/1.1
> Host: *
> User-Agent: curl/7.52.1
> Accept: */*
> 
< HTTP/1.1 502 Bad Gateway
< Content-Type: text/html; charset=UTF-8
< Referrer-Policy: no-referrer
< Content-Length: 332
< Date: Sun, 04 Mar 2018 05:19:59 GMT
< 
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>502 Server Error</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Server Error</h1>
<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>
<h2></h2>
</body></html>
* Curl_http_done: called premature == 0
* Connection #0 to host trial.qurasense.com left intact

When this url requested, nothing happens in gateway logs.

-- Zufar Muhamadeev
google-cloud-platform
google-kubernetes-engine
kubernetes

1 Answer

3/9/2018

Problem was in gateway application and Ingress combination, i upgraged gateway application version and it started to work

-- Zufar Muhamadeev
Source: StackOverflow