I am writing a Golang based L7 proxy that sits in front of k8s api server. Something like this,

kubectl <----> gateway/proxy <----> AWS EKS API server

Common kubectl commands like get pods, exec, run, apply, create etc work fine. However, for kubectl delete pod <name>, kubectl get stuck/hangs.

Generally, these are cases, where kubectl adds watch=true flag. In this scenario, k8s server does not send END_STREAM so my proxy gets stuck in 'read frame WINDOW_UPDATE'

Has anyone seen this? I was looking at client-go or kubernetes repoa to see how does kubectl handle response when watch=true is set. Would appreciate any pointers.

Below are my debug logs --

2019/10/11 00:50:19 http2: decoded hpack field header field ":authority" = "myproxy.dev.net:8443"
2019/10/11 00:50:19 http2: decoded hpack field header field ":method" = "GET"
2019/10/11 00:50:19 http2: decoded hpack field header field ":path" = "/api/v1/namespaces/guestbook/pods?fieldSelector=metadata.name%3Dshell-test&resourceVersion=1313006&watch=true"
2019/10/11 00:50:19 http2: decoded hpack field header field ":scheme" = "https"
2019/10/11 00:50:19 http2: decoded hpack field header field "accept" = "application/json"
2019/10/11 00:50:19 http2: decoded hpack field header field "user-agent" = "kubectl/v1.15.2 (darwin/amd64) kubernetes/f627830"
2019/10/11 00:50:19 http2: decoded hpack field header field "authorization" = "Bearer LS<REDACTED>
2019/10/11 00:50:19 http2: decoded hpack field header field "accept-encoding" = "gzip"
2019/10/11 00:50:19 http2: server read frame HEADERS flags=END_STREAM|END_HEADERS stream=5 len=86
2019/10/11 00:50:51 http2: server encoding header ":status" = "200"
2019/10/11 00:50:51 http2: server encoding header "audit-id" = "aaa-9682-d7b95a6daa0e"
2019/10/11 00:50:51 http2: server encoding header "content-type" = "application/json"
2019/10/11 00:50:51 http2: server encoding header "date" = "Fri, 11 Oct 2019 00:50:19 GMT"
2019/10/11 00:50:51 http2: Framer 0xc0004beee0: wrote HEADERS flags=END_HEADERS stream=5 len=32
2019/10/11 00:50:51 http2: Framer 0xc0004beee0: wrote DATA stream=5 len=4096 data="{\"type\":\"MODIFIED\",\"object\":{\"kind\":\"Pod\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"shell-test\",\"namespace\":\"guestbook\",\"selfLink\":\"/api/v1/namespaces/guestbook/pods/shell-test\",\"uid\":\"d2baa702-ebc0-11e9-06652f016772\",\"resourceVersion\":\"1313054\",\"creation" (3840 bytes omitted)
2019/10/11 00:50:51 http2: Framer 0xc0004beee0: wrote DATA stream=5 len=4096 data="0}],\"priority\":0,\"enableServiceLinks\":true},\"status\":{\"phase\":\"Running\",\"conditions\":[{\"type\":\"Initialized\",\"status\":\"True\",\"lastProbeTime\":null,\"lastTransitionTime\":\"2019-10-11T00:48:21Z\"},{\"type\":\"Ready\",\"status\":\"False\",\"lastProbeTime\":null,\"lastTransit" (3840 bytes omitted)
2019/10/11 00:50:51 http2: Framer 0xc0004beee0: read WINDOW_UPDATE stream=5 len=4 incr=6291
2019/10/11 00:50:51 http2: server read frame WINDOW_UPDATE stream=5 len=4 incr=6291  

Proxy code is written in Golang using NewSingleHostReverseProxy and it terminates TLS, patches new connection to k8s api server.