K
Q

Set limit_req for specific location in Nginx Ingress

1/4/2019

I'm trying to setup rate limiting option limit_req for specific path in Kubernetes ingress-nginx to prevent brute-forcing authentication.

I've defined limit_req_zone using ConfigMap:

http-snippet: |
      limit_req_zone $the_real_ip zone=authentication_ratelimit:10m rate=1r/s;

Next, I'm using annotation to add a custom location block:

nginx.ingress.kubernetes.io/configuration-snippet: |
  location ~* "^/authenticate
quot; {
    limit_req zone=authentication_ratelimit nodelay;
    more_set_headers "x-test: matched";
  }

This produces nginx.conf:

server {
# - - 8< - -

  location / {
    # - - 8< - -

    location ~* "^/authenticate
quot; {
      limit_req zone=authentication_ratelimit nodelay;
      more_set_headers "x-test: matched";
    }

    proxy_pass http://upstream_balancer;

    proxy_redirect                          off;
}

The result is that /authenticate always returns HTTP 503 (with x-test header). Message from ingress access logs:

<ip> - [<ip>] - - [04/Jan/2019:15:22:07 +0000] "POST /authenticate HTTP/2.0" 503 197 "-" "curl/7.54.0" 172 0.000 [-] - - - - 1a63c9825c9795be1378b2547e29992d

I suspect this might be because of conflict between nested location block and proxy_pass (but this is just a wild guess).

What other options have I tried?

  • use server-snippet annotation instead of configuration-snippet - /authenticate returns 404 because proxy_pass is not configured
  • use nginx.ingress.kubernetes.io/limit-rpm annotation - forces ratelimit on whole application which is not what I want.

Question is why custom location block responds with 503? How can I debug this? Will increasing nginx logging level give more details about 503? Or more general question: can I inject custom location blocks in ingress-nginx?

-- jazgot
kubernetes
kubernetes-ingress
nginx
nginx-ingress

Similar Questions

Prometheus many-to-many problem for kube cronjobs
Kubernetes: The code change does not appear, is there a way to sync?
GCP k8s - from a cluster to another cluster with mongodb
AKS Upgrade "Operation failed with status: 200. Details: Resource state Failed"

2 Answers

1/7/2019

It might sound funny, but could you try just to create annotation like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-with-annotations
  annotations:
    nginx.org/server-snippets: |
      limit_req_zone $binary_remote_addr zone=mylimit:10m rate=10r/s;
      location ~* "^/authenticate
quot; {
         limit_req zone=mylimit burst=10 nodelay;
         more_set_headers "x-test: matched";
      }

Because, it looks like you forgot to define limit_req_zone before you started using it in directive location. I used the official documentation about limit_req

-- Nick Rak
Source: StackOverflow
2/28/2019

This can be done by using map and that fact that Requests with an empty key value are not accounted.

http-snippets: |
  map $uri $with_limit_req {
    default 0;
    "~*^/authenticate
quot; 1;
  }
  map $with_limit_req $auth_limit_req_key {
    default '';
    '1'     $binary_remote_addr; # the limit key
  }
  limit_req_zone $auth_limit_req_key zone=authentication_ratelimit:10m rate=1r/s;

And use annotation to add a custom location block:

nginx.ingress.kubernetes.io/configuration-snippet: |
  limit_req zone=authentication_ratelimit nodelay;

Or if you use ingress from nginxinc

nginx.org/location-snippets:
  limit_req zone=authentication_ratelimit nodelay;

in this case check if requests need to be ratelimited processed on map level.

And my opinion: better to limit requests on app level as if you made rate limit on ingress level, it depends on count of ingress pods.

-- Dmitry
Source: StackOverflow