Getting 504 errors on API calls after migration to Kubernetes with Ambassador

1/29/2022

The backend (APIs) of our app is built on Grails 5.1.1 and it has been running for several years now on DiginalOcean. We are now trying to migrate the infrastructure to Kubernetes (also on DO). For the ingress controller, we went with Ambassador Edge Stack (for various reasons).

Now, the entire cluster is up and running except that we are having issues with CORS.

Note: The same codebase exactly is running without issues on DO through a regular LoadBalancer (even the same image).

When we switch our APIs to the Kubernetes cluster, we are getting errors of missing CORS headers.

Before the details, a bit of history...

At first, the OPTIONS request was rejected because, apparently, spring-security is denying these requests. For some reason, we are facing the issue only when switched to Kubernetes.

We worked around this by adding another filter (Interceptors are no good for this case, as they are executed after the security filters)...

CoprsFilter

class CorsFilter extends OncePerRequestFilter {

    private allowedOrigins = ["http://localhost:4200",
                              "https://localhost:4200",
                              "https://app.priz.guru"]

    @Override
    protected void doFilterInternal(HttpServletRequest req, HttpServletResponse resp, FilterChain chain)
            throws ServletException, IOException {

        String origin = req.getHeader('Origin')
        if (allowedOrigins.contains(origin)) {
            resp.addHeader("Access-Control-Allow-Methods", "GET, POST, PUT, DELETE, OPTIONS")
            resp.addHeader("Access-Control-Max-Age", "3600")
//            resp.addHeader("Access-Control-Allow-Credentials", "true")
            resp.addHeader("Access-Control-Allow-Origin", origin)
            resp.addHeader("Access-Control-Expose-Headers", "*")
        }

        if (req.getMethod() == "OPTIONS") {
            resp.status = 200
        } else {
            chain.doFilter(req, resp)
        }
    }
}

Adding into resource.groovy:

optionsCorsFilter(CorsFilter)

And register in BootStrap.groovy:

SpringSecurityUtils.clientRegisterFilter(
                "optionsCorsFilter",
                SecurityFilterPosition.FIRST.order - 1
        )

After that, the OPTIONS requests started working as expected. By the way, the built-in cors filter is still enabled. We can technically remove it at this point.

Once this was resolved, we are now facing the next challenge. The actual API calls do not return CORS headers.


More Details

On the current environment. Docker droplet on DO with managed LoadBalancer.

OPTIONS Request: enter image description here

Same GET request: enter image description here

As you can see, CORS headers are in place for both. Everything is happy.

On Kubernetes with Ambassador Literally the same docker image.

Ambassador Mapping:

---
apiVersion: getambassador.io/v2
kind: Mapping
metadata:
  name: priz-api
  namespace: ambassador
spec:
  prefix: /
  host: api.priz.guru
  service: priz-api.backend
  cors:
    origins:
      - https://app.priz.guru
    methods: GET, POST, PUT, PATCH, OPTIONS, DELETE
    headers: "*"
#    credentials: true
    exposed_headers: "*"
    max_age: "86400"

Note: Even if I completely remove cors config, I am still facing the same problem.

OPTIONS Request: enter image description here

Matching GET request: enter image description here

-- Shurik Agulyansky
ambassador
digital-ocean
grails
kubernetes
spring-security

0 Answers