I am new to Kubernetes and NGINX Ingress on Microsoft Azure. I got an issue about analyzing log of NGINX Ingress.
Here is the log in my NGINX Ingress pods:
duc@Azure:~$ kubectl logs ducphuongkhang-ingress-nginx-ingress-controller-869b8b966-877bq -n kube-system | grep 'lua'
2018/11/06 16:36:55 [warn] 961#961: *10059 [lua] log.lua:52: {"timestamp":1541522215,"method":"GET","uri":"\/vulnerabilities\/sqli\/","id":"dba39b7d7dc8646b779e","client":"10.244.0.1","alerts":[{"match":1,"msg":"SQL String Termination","id":41003},{"match":1,"msg":"SQL probing attempt","id":41032},{"logdata":8,"match":8,"msg":"Request score greater than score threshold","id":99001}]} while logging request, client: 10.244.0.1, server: dvwa.thesis.analyticsvn.com, request: "GET /vulnerabilities/sqli/?id=%27&Submit=Submit HTTP/2.0", host: "dvwa.thesis.analyticsvn.com", referrer: "https://dvwa.thesis.analyticsvn.com/vulnerabilities/sqli/?id=1%3D1&Submit=Submit"
2018/11/06 16:37:02 [warn] 961#961: *10059 [lua] log.lua:52: {"timestamp":1541522222,"method":"GET","uri":"\/vulnerabilities\/sqli\/","id":"4ac4e0dfe317dcd86346","client":"10.244.0.1","alerts":[{"match":1,"msg":"SQL String Termination","id":41003},{"match":1,"msg":"SQL probing attempt","id":41032},{"logdata":8,"match":8,"msg":"Request score greater than score threshold","id":99001}]} while logging request, client: 10.244.0.1, server: dvwa.thesis.analyticsvn.com, request: "GET /vulnerabilities/sqli/?id=%27&Submit=Submit HTTP/2.0", host: "dvwa.thesis.analyticsvn.com", referrer: "https://dvwa.thesis.analyticsvn.com/vulnerabilities/sqli/?id=1%3D1&Submit=Submit"
2018/11/06 16:37:02 [warn] 961#961: *10059 [lua] log.lua:52: {"timestamp":1541522222,"method":"GET","uri":"\/vulnerabilities\/sqli\/","id":"d0eae7d54dc99773ecc0","client":"10.244.0.1","alerts":[{"match":1,"msg":"SQL String Termination","id":41003},{"match":1,"msg":"SQL probing attempt","id":41032},{"logdata":8,"match":8,"msg":"Request score greater than score threshold","id":99001}]} while logging request, client: 10.244.0.1, server: dvwa.thesis.analyticsvn.com, request: "GET /vulnerabilities/sqli/?id=%27&Submit=Submit HTTP/2.0", host: "dvwa.thesis.analyticsvn.com", referrer: "https://dvwa.thesis.analyticsvn.com/vulnerabilities/sqli/?id=1%3D1&Submit=Submit"
2018/11/06 16:37:03 [warn] 961#961: *10059 [lua] log.lua:52: {"timestamp":1541522223,"method":"GET","uri":"\/vulnerabilities\/sqli\/","id":"be18d7e7800e86789d5d","client":"10.244.0.1","alerts":[{"match":1,"msg":"SQL String Termination","id":41003},{"match":1,"msg":"SQL probing attempt","id":41032},{"logdata":8,"match":8,"msg":"Request score greater than score threshold","id":99001}]} while logging request, client: 10.244.0.1, server: dvwa.thesis.analyticsvn.com, request: "GET /vulnerabilities/sqli/?id=%27&Submit=Submit HTTP/2.0", host: "dvwa.thesis.analyticsvn.com", referrer: "https://dvwa.thesis.analyticsvn.com/vulnerabilities/sqli/?id=1%3D1&Submit=Submit"And here is my Azure Analytics Log query, which return no value:
ContainerLog | where LogEntry contains "lua"I want to collect logs that generated by NGINX Ingress (with lua-resty-waf enable) with Azure Log Analytics. Please help me to get there.
Thank you.
After doing the research, I found out that the Azure Log Analytics with OMS Agent in Kubernetes don't support to collect data in the namespace 'kube-system'. Deploy the Ingress in another namespace will allow the Log Analytics to collect log.
Reference: https://github.com/Azure/AKS/issues/293
$ kubectl describe deployments omsagent-rs -n kube-system
Pod Template:
Labels: rsName=omsagent-rs
Annotations: agentVersion=1.6.0-42
dockerProviderVersion=2.0.0-3
Service Account: omsagent
Environment:
DISABLE_KUBE_SYSTEM_LOG_COLLECTION: true