K
Q

A workaround this kubernetes error: The Ingress "generated-ingress" is invalid: metadata.annotations: Too long: must have at most 262144 characters

10/22/2019

I'm trying to replicate what we have set up in our bare-metal infrastructure using AWS. I'm admittedly not well-versed with K8S, but a colleague of mine had started to use it in AWS so I've been piggybacking on some of the work he's done. He's been able to answer most of my questions, but this one seems to have us a little stumped.

In our bare-metal setup, there are about 20k unique sites with individual SSL certificates. In doing a small scale test with a few sites and certs, it works. However, at full scale, it does not.

I was able to script importing the certificates into K8S and create a voyager ingress configuration yaml file from it, but when I try to apply it, I get the following error message:

bash-5.0# kubectl apply -f generated-ingress.yaml
The Ingress "generated-ingress" is invalid: metadata.annotations: Too long: must have at most 262144 characters

It seems that the ingress configuration is too large.

The generated ingress looks something like this:

---
apiVersion: voyager.appscode.com/v1beta1
kind: Ingress
metadata:
  name: generated-ingress
  namespace: default
spec:
  tls:
    - secretName: ssl-cert-eample-domain.com
      hosts:
        - example-domain.com
    # Repeated for each domain using SSL
  rules:
    - host: '*'
      http:
        paths:
        - backend:
            serviceName: the-backend
            servicePort: '80'
    - host: example-domain.com
      http:
        paths:
        - backend:
            serviceName: the-backend
            servicePort: '80'
            headerRules:
            - X-HTTPS YES 
    # Repeated for each domain using SSL

FYI the X-HTTPS header is needed for our backend to generate absolute urls correctly using https instead of http. It would be nice if this could be specified for all SSL connections instead of having to specify it for each individual domain, making the configuration more lean.

In any case, is there a way to increase the limit for the configuration? Or is there a better way to configure the ingress for numerous SSL certificates?

-- cue8chalk
amazon-eks
aws-eks
haproxy-ingress
kubernetes

Similar Questions

Active/Passive routing a headless service with mutliple backends
Kubernetes pod stuck in state=Terminating after node goes to status = NotReady?
how can I find the related image version with the kubernetes version?
Spring boot jar not starting on GKE

1 Answer

10/23/2019

I believe you are getting bitten by kubectl trying to record the previous state of affairs in an {"metadata":{"annotation":{"kubectl.kubernetes.io/last-applied-configuration": "GINORMOUS string"}}} as described here. It's not the Ingress itself that is causing you problems.

In your case, because the input object is so big, I believe you will have to use kubectl replace instead of apply, since (a) you appear to have the whole declaration in-hand anyway (b) I don't think kubectl replace tries to capture that annotation unless one explicitly provides --save-config=true to replace

Separate from your question, if you are not already aware, one need not declare every single Ingress in one resource -- it is much, much, much, much, much easier to manage and reason about Ingress resources when they are distinct

So your config of:

apiVersion: voyager.appscode.com/v1beta1
kind: Ingress
metadata:
  name: generated-ingress
  namespace: default
spec:
  tls:
    - secretName: ssl-cert-eample-domain.com
      hosts:
        - example-domain.com
    # Repeated for each domain using SSL
  rules:
    - host: '*'
      http:
        paths:
        - backend:
            serviceName: the-backend
            servicePort: '80'
    - host: example-domain.com
      http:
        paths:
        - backend:
            serviceName: the-backend
            servicePort: '80'
            headerRules:
            - X-HTTPS YES

would become:

apiVersion: voyager.appscode.com/v1beta1
kind: Ingress
metadata:
  name: star-domain
  namespace: default
spec:
  rules:
    - host: '*'
      http:
        paths:
        - backend:
            serviceName: the-backend
            servicePort: '80'

and then repeat this structure for every one of the domains:

apiVersion: voyager.appscode.com/v1beta1
kind: Ingress
metadata:
  name: example-domain
  namespace: default
spec:
  tls:
    - secretName: ssl-cert-example-domain.com
      hosts:
        - example-domain.com
  rules:
    - host: example-domain.com
      http:
        paths:
        - backend:
            serviceName: the-backend
            servicePort: '80'
            headerRules:
            - X-HTTPS YES

it's also redundantly redundant to put ingress in the metadata: name: of an Ingress resource, as when one is typing its name you will need to say kubectl describe ingress/generated-ingress

-- mdaniel
Source: StackOverflow