Is it possible to use secrets on a containerPort?

For example:

spec:
  containers:
    - name: postgres
      image: testproject/postgres
      ports:
        - containerPort: 5432

Doing something like:

  containers:
    - name: postgres
      image: testproject/postgres
      ports:
        - containerPort:
          valueFrom:
            secretKeyRef:
              name: app-secrets
              key: PGPORT

Tested it and it doesn't work so seeing if there is a different syntax.

I'm probably being overly cautious, but figured it would be more secure to not have the port in plain text somewhere, even though that port wouldn't be available publicly.