You can't do it at Ingress Controller level. You would do it through RBAC rules.

Something like this:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  namespace: namespace1
  name: deployment-manager
rules:
- apiGroups: ["", "extensions", "apps"]
  resources: ["deployments", "replicasets", "pods"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: deployment-manager-binding
  namespace: namespace1
subjects:
- kind: User
  name: user1
  apiGroup: ""
roleRef:
  kind: Role
  name: deployment-manager
  apiGroup: ""

At Ingress Controller level (and depending on Ingress Controller), you could reject requests based on the host, the path; the user is trying to reach, headers and other attributes, but not based on the identity of the user.

Note that the Ingress Controller itself is going to be running in a namespace, so your user needs to first reach it (after being authenticated and authorized), and then be rejected, which doesn't really sound right.

You need to do it in the application itself using audience claim from the JWT token provided by the oauth provider.

If you don't want to do it in the application and okay with bringing in the complexity of a service mesh then you could use a service mesh such as istio to do it. Here is the doc for istio.