I have created a AKS and deployed a simple web server on it with following yaml.
Azure LoadBalancer gives a public IP address to it automatically and works fine.

Now I would like to limit the source IP address so I can access it from a specify IP address only.
I've tried adding a Azure Firewall to the virtual network of AKS (aks-vnet-XXXXXXX) with some network rule but doesn't work.

Creating a NAT rule in Firewall and redirects packets to the LoadBalancer works
but I can still access the pod with the Public IP address of the LoadBalancer.

Any suggestions?

apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  type: LoadBalancer
  selector:
    app: nginx
  ports:
    - name: http
      port: 80
      targetPort: 80

---
apiVersion: apps/v1
kind: Deployment
(skipped something not important)
    spec:
      containers:
      - name: nginx
        image: nginx:1.17.6
        ports:
        - containerPort: 80