I have a wierd phenomena, where on of my services stops responding (although the corresponding pod is still up) and the nginx-ingress starts a cycle of spamming the service.

Those checks are very rapidly after each other and each failed check triggers a log message. those log messages take up memory up until the is no more memory (and no more swap) and the whole server reboots.

i am unable to figure out the reason for this behaviour. I was able to determine, that the server behaves correctly if i change the service port, save and change it back, thus forcing the ingress to reload.

a bit more on setup: the ingress is setup to send traffic on specific ports to specific services (pure tcp, no http).

apiVersion: v1
data:
  "25": mail/postfix:25
  "143": mail/dovecot:143
  "465": mail/postfix:465
  "587": mail/postfix:587
  "993": mail/dovecot:993
kind: ConfigMap
metadata:
  creationTimestamp: "2019-09-13T15:25:03Z"
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.17.1
    component: controller
    heritage: Tiller
    release: nginx-ingress
  name: nginx-ingress-tcp
  namespace: ingress
  resourceVersion: "36494639"
  selfLink: /api/v1/namespaces/ingress/configmaps/nginx-ingress-tcp
  uid: a871d451-d63a-11e9-995d-d43d7e4a45d7

the service is nothing special, the postfix behind is nothing special. the whole k8s is setup on bare metal and currently running version 1.13.2

i am well aware, that it must have something to do with this method of passing through tcp-traffic, as the logs show many many many entries like this:

{"log":"2019/10/24 10:45:09 [error] 43#43: *132090 connect() failed (111: Connection refused) while connecting to upstream, client: 10.244.0.1, server: 0.0.0.0:465, upstream: \"10.244.0.9:465\", bytes from/to client:0/0, bytes from/to upstream:0/0\n","stream":"stderr","time":"2019-10-24T10:45:09.806192849Z"}
{"log":"2019/10/24 10:45:09 [error] 43#43: *132090 connect() failed (111: Connection refused) while connecting to upstream, client: 10.244.0.1, server: 0.0.0.0:465, upstream: \"10.244.0.9:465\", bytes from/to client:0/0, bytes from/to upstream:0/0\n","stream":"stderr","time":"2019-10-24T10:45:09.806230273Z"}
{"log":"2019/10/24 10:45:09 [error] 43#43: *132090 connect() failed (111: Connection refused) while connecting to upstream, client: 10.244.0.1, server: 0.0.0.0:465, upstream: \"10.244.0.9:465\", bytes from/to client:0/0, bytes from/to upstream:0/0\n","stream":"stderr","time":"2019-10-24T10:45:09.806263649Z"}
{"log":"2019/10/24 10:45:09 [error] 43#43: *132090 connect() failed (111: Connection refused) while connecting to upstream, client: 10.244.0.1, server: 0.0.0.0:465, upstream: \"10.244.0.9:465\", bytes from/to client:0/0, bytes from/to upstream:0/0\n","stream":"stderr","time":"2019-10-24T10:45:09.80630235Z"}
{"log":"2019/10/24 10:45:09 [error] 43#43: *132090 connect() failed (111: Connection refused) while connecting to upstream, client: 10.244.0.1, server: 0.0.0.0:465, upstream: \"10.244.0.9:465\", bytes from/to client:0/0, bytes from/to upstream:0/0\n","stream":"stderr","time":"2019-10-24T10:45:09.806336273Z"}
{"log":"2019/10/24 10:45:09 [error] 43#43: *132090 connect() failed (111: Connection refused) while connecting to upstream, client: 10.244.0.1, server: 0.0.0.0:465, upstream: \"10.244.0.9:465\", bytes from/to client:0/0, bytes from/to upstream:0/0\n","stream":"stderr","time":"2019-10-24T10:45:09.806371894Z"}
{"log":"2019/10/24 10:45:09 [error] 43#43: *132090 connect() failed (111: Connection refused) while connecting to upstream, client: 10.244.0.1, server: 0.0.0.0:465, upstream: \"10.244.0.9:465\", bytes from/to client:0/0, bytes from/to upstream:0/0\n","stream":"stderr","time":"2019-10-24T10:45:09.806406819Z"}
{"log":"2019/10/24 10:45:09 [error] 43#43: *132090 connect() failed (111: Connection refused) while connecting to upstream, client: 10.244.0.1, server: 0.0.0.0:465, upstream: \"10.244.0.9:465\", bytes from/to client:0/0, bytes from/to upstream:0/0\n","stream":"stderr","time":"2019-10-24T10:45:09.806447214Z"}
{"log":"2019/10/24 10:45:09 [error] 43#43: *132090 connect() failed (111: Connection refused) while connecting to upstream, client: 10.244.0.1, server: 0.0.0.0:465, upstream: \"10.244.0.9:465\", bytes from/to client:0/0, bytes from/to upstream:0/0\n","stream":"stderr","time":"2019-10-24T10:45:09.806482653Z"}

as this (while i was writing) just happened again, i have the log lines while changing the port forth and back:

2019/10/24 11:43:15 [error] 46#46: *52192 connect() failed (111: Connection refused) while connecting to upstream, client: 10.244.0.1, server: 0.0.0.0:465, upstream: "10.244.0.47:465", bytes from/to client:0/0, bytes from/to upstream:0/0
2019/10/24 11:43:15 [error] 46#46: *52192 connect() failed (111: Connection refused) while connecting to upstream, client: 10.244.0.1, server: 0.0.0.0:465, upstream: "10.244.0.47:465", bytes from/to client:0/0, bytes from/to upstream:0/0
2019/10/24 11:43:15 [error] 46#46: *52192 connect() failed (111: Connection refused) while connecting to upstream, client: 10.244.0.1, server: 0.0.0.0:465, upstream: "10.244.0.47:465", bytes from/to client:0/0, bytes from/to upstream:0/0
2019/10/24 11:43:15 [crit] 46#46: *52192 connect() to 0.0.0.1:1234 failed (22: Invalid argument) while connecting to upstream, client: 10.244.0.1, server: 0.0.0.0:465, upstream: "0.0.0.1:1234", bytes from/to client:0/0, bytes from/to upstream:0/0
[24/Oct/2019:11:43:15 +0000]TCP50200312.875
I1024 11:43:27.562564       6 event.go:258] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress", Name:"nginx-ingress-tcp", UID:"a871d451-d63a-11e9-995d-d43d7e4a45d7", APIVersion:"v1", ResourceVersion:"41186759", FieldPath:""}): type: 'Normal' reason: 'UPDATE' ConfigMap ingress/nginx-ingress-tcp
I1024 11:43:27.564237       6 controller.go:133] Configuration changes detected, backend reload required.
I1024 11:43:27.677561       6 controller.go:149] Backend successfully reloaded.

My question then would be: how can i stabilize this?

Edit Further research suggests this is a nginx problem, not a kubernetes or ingress problem.

best regards, scones