is it possible to deploy an ingress controller (nginx) without a public ip address?

Without question, yes, if the Ingress controller's Service is of type: NodePort then the Ingress controller's private IP address is every Node's IP address, on the port(s) pointing to :80 and :443 of the Service. Secretly, that's exactly what is happening anyway with type: LoadBalancer, just with the extra sugar coating of the cloud provider mapping between the load balancer's IP address and the binding to the Node's ports.

So, to close that loop: if you wished to have a 100% internal Ingress controller, then use a hostNetwork: true and bind the Ingress controller's ports: to be the host's port 80 and 443; then, make a DNS (A record|CNAME record) for each virtual-host that resolve to the address of every Node in the cluster, and poof: 100% non-Internet-facing Ingress controller.

Internal IP ingress in Google Kubernetes Engine

Assuming you wanna deploy an ingress controller (nginx) without a public ip address in GKE. Below is what worked for me.

1. Install Nginx-Ingress controller with appropriate annotations

Use stable/nginx-ingress helm chart to install ingress-nginx controller in out GKE cluster.

As per this GCP document we can create a Load Balancer resource with cloud.google.com/load-balancer-type: "Internal" annotation to create an internal Load Balancer. Run the below command to add the controller to GKE.

helm install --name ingress-controller stable/nginx-ingress \
--set controller.service.annotations."cloud\.google\.com/load-balancer-type"="Internal"

2. Deploy ingress resources using this controller

To make Ingress resources use the controller, add the kubernetes.io/ingress.class: nginx annotation to your ingress resources.

An example Ingress resource using nginx-ingress controller looks something like below:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  name: nginx-test
spec:
  rules:
    - host: www.example.com
      http:
        paths:
        - backend:
            serviceName: my-service-1
            servicePort: 80
          path: /tasks
        - backend:
            serviceName: my-service-2
            servicePort: 80
          path: /

Now using kubectl command you can see the assigned IP to your ingress resource is an internal IP address.

3. In case you wanna use TLS too (Optional)

Add the below to your YAML manifest:

  ...
  rules:
  ...
  tls:
  - hosts:
    - www.example.com
    secretName: my-certs

In the above example my-certs is a Kubernetes secret containing the server key, certificate and CA certificate created using the below command:

kubectl create secret generic my-certs --from-file=tls.crt=server.crt --from-file=tls.key=server.key --from-file=ca.crt=ca.crt

For an example above keys and certificates are created with a sample hostname referring to this Medium Article.

Hope this helps.