After 30-45 minutes, chunked HTTP connection to API server is dropped:
Transmission Control Protocol, Src Port: http-alt (8080), Dst Port: 55782 (55782), Seq: 751, Ack: 88, Len: 0
.... 0000 0001 0001 = Flags: 0x011 (FIN, ACK)This happens regardless of the activity level, i.e. it happens for connections that were idle for a long time but also for the ones that had notifications coming for the whole duration of the connection. HTTP 1.0 (with Connection: Keep-Alive header) just ends the original request, while HTTP 1.1, which is keepalive by default, sends 400 Bad Request before dropping the connection.
Is it possible to get a watch connection which remains alive for a long period of time?
Once you're certain your client properly handles disconnections, you can use the following kube-apiserver flag to control how long apiserver lets the watches stay open:
https://github.com/kubernetes/kubernetes/blob/release-1.1/docs/admin/kube-apiserver.md
--min-request-timeout=1800: An optional field indicating the minimum number of seconds a handler must keep a request open before timing it out. Currently only honored by the watch request handler, which picks a randomized value above this number as the connection timeout, to spread out load.
Test with a small value, run in production with a large value.
Watches are supposed to drop periodically - they are just long HTTP GET operations underneath, with timeouts. It's intentional. Is it causing a problem?