How to get real time update from s3

6/30/2018

Circleci and many other ci tools show real time logs during the job running. It like the tail -f in linux but it also show all the previous logs.

I am trying to sync specific logs from kubernetes to s3 and then move the update to the browse it that possible?

-- aisensiy
amazon-s3
kubernetes
logging
websocket

1 Answer

6/30/2018

I am trying to sync specific logs from kubernetes to s3 and then move the update to the browse it that possible?

IMHO you would want to have a "tee" mechanism to divert the log streams to each destination, since those two destinations have vastly different access (and retry!) mechanisms. AFAIK kubernetes allows unlimited(?) numbers of kubectl logs -f connections, so you would want one process that connects to the pod and relays those bytes out to the browser, and a separate process that connects and relays the bytes to S3. You could have one process that does both, but that runs the risk of a single bug wiping out both streams, making everyone unhappy.

I used kubectl logs -f as a "shortcut" for this answer, but I am super positive there's an API endpoint for doing that, I just don't remember what it is offhand -- kubectl --v=100 logs -f $pod will show it to you.

-- mdaniel
Source: StackOverflow