It seems like the kustomize command adds a label to the selector which fails because deployment selector labels cannot be changes on deploy

I am trying to update the command in the pytorch-operator deployment in kubeflow to add a CLI argument. For that I am trying to patch using kustomize.

Below is my kustomization.yaml

resources:
  - ../.cache/manifests/manifests-1.2.0/pytorch-job/pytorch-operator/base
patches:
  - path: patch_pytorch_operator_command.yaml
    target:
      kind: Deployment

And the patch_command.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pytorch-operator
spec:
  template:
    spec:
      containers:
      - name: pytorch-operator
        command:
        - /pytorch-operator.v1
        - --alsologtostderr
        - -v=1
        - --monitoring-port=8443
        - --init-container-image={image_location}

Running kustomize build generates below

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    kustomize.component: pytorch-operator
  name: pytorch-operator
  namespace: kubeflow
spec:
  replicas: 1
  selector:
    matchLabels:
      kustomize.component: pytorch-operator
      name: pytorch-operator
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
      labels:
        kustomize.component: pytorch-operator
        name: pytorch-operator
    spec:
      containers:
      - command:
        - /pytorch-operator.v1
        - --alsologtostderr
        - -v=1
        - --monitoring-port=8443
        - --init-container-image={image_location}
        env:
        - name: MY_POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        image: gcr.io/kubeflow-images-public/pytorch-operator:vmaster-g518f9c76
        name: pytorch-operator
      serviceAccountName: pytorch-operator

If you see the kustomize.component label gets and hence I get the error when I apply the manifest

The Deployment "pytorch-operator" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"kustomize.component":"pytorch-operator", "name":"pytorch-operator"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable