I am trying to run a workflow (https://github.com/argoproj/argo-workflows/blob/master/examples/memoize-simple.yaml) with limited permissions in Argo. I am specifying a serviceaccount with the requisite permissions in the execution command and in the workflow itself, but the workflow controller logs show a different serviceaccount.

This is the execution command

argo submit -n argo --serviceaccount dma --watch whalesay.yaml

Here are the Configmaps

apiVersion: v1
data:
  containerRuntimeExecutor: emissary
  workflowDefaults: '{"annotations": {"workflows.argoproj.io/version": ">= 3.1.0"},
    "metadata": { }, "spec": {"artifactRepositoryRef": {"configMap": "my-config",
    "key": "whalesay-cache"}, "entrypoint": "entrypoint", "parallelism": 3, "podGC":
    {"strategy": "OnWorkflowSuccess"}, "securityContext": {"fsGroup": 2000, "runAsGroup":
    3000, "runAsUser": 1000}, "ttlStrategy": {"secondsAfterSuccess": 5}}}'
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"containerRuntimeExecutor":"emissary","workflowDefaults":"{\"annotations\": {\"workflows.argoproj.io/version\": \"\u003e= 3.1.0\"}, \"metadata\": { }, \"spec\": {\"artifactRepositoryRef\": {\"configMap\": \"my-config\", \"key\": \"whalesay-cache\"}, \"entrypoint\": \"entrypoint\", \"parallelism\": 3, \"podGC\": {\"strategy\": \"OnWorkflowSuccess\"}, \"securityContext\": {\"fsGroup\": 2000, \"runAsGroup\": 3000, \"runAsUser\": 1000}, \"ttlStrategy\": {\"secondsAfterSuccess\": 5}}}"},"kind":"ConfigMap","metadata":{"annotations":{},"name":"workflow-controller-configmap","namespace":"argo"}}
  creationTimestamp: "2021-10-15T11:53:11Z"
  name: workflow-controller-configmap
  namespace: argo
  resourceVersion: "1928507"
  uid: bc8c16b8-e5cd-4a31-b354-1627cdf3296c

and the workflow itself

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: memoized-simple-workflow-
spec:
  entrypoint: whalesay
  serviceAccountName: dma
  artifactRepositoryRef:
    configMap: my-config # default is "artifact-repositories"
    key: whalesay-cache # default can be set by the annotation
  arguments:
    parameters:
    - name: message
      value: test-6
  templates:
  - name: whalesay
    inputs:
      parameters:
      - name: message
    memoize:
      key: whalesay-cache
      maxAge: "10s"
      cache:
        configMap:
          name: my-config
    container:
      image: docker/whalesay:latest
      command: [sh, -c]
      args: ["cowsay {{inputs.parameters.message}} > /tmp/hello_world.txt"]
    outputs:
      parameters:
      - name: hello
        valueFrom:
          path: /tmp/hello_world.txt

but the logs show a different serviceaccount name when the workflow is actually run

time="2021-10-13T14:32:27.424Z" level=info msg="Update leases 200"
time="2021-10-13T14:32:32.441Z" level=info msg="Get leases 200"
time="2021-10-13T14:32:32.457Z" level=info msg="Update leases 200"
time="2021-10-13T14:32:32.505Z" level=info msg="Processing workflow" namespace=argo workflow=memoized-simple-workflow-c5bp5
time="2021-10-13T14:32:32.511Z" level=info msg="Updating node memoized-simple-workflow-c5bp5 exit code 0" namespace=argo workflow=memoized-simple-workflow-c5bp5
time="2021-10-13T14:32:32.511Z" level=info msg="Setting node memoized-simple-workflow-c5bp5 outputs: {\"parameters\":[{\"name\":\"hello\",\"value\":\" ________ \\n\\u003c test-6 \\u003e\\n -------- \\n    \\\\\\n     \\\\\\n      \\\\     \\n                    ##        .            \\n              ## ## ##       ==            \\n           ## ## ## ##      ===            \\n       /\\\"\\\"\\\"\\\"\\\"\\\"\\\"\\\"\\\"\\\"\\\"\\\"\\\"\\\"\\\"\\\"___/ ===        \\n  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~   \\n       \\\\______ o          __/            \\n        \\\\    \\\\        __/             \\n          \\\\____\\\\______/   \",\"valueFrom\":{\"path\":\"/tmp/hello_world.txt\"}}]}" namespace=argo workflow=memoized-simple-workflow-c5bp5
time="2021-10-13T14:32:32.511Z" level=info msg="Updating node memoized-simple-workflow-c5bp5 status Pending -> Succeeded" namespace=argo workflow=memoized-simple-workflow-c5bp5
time="2021-10-13T14:32:32.511Z" level=info msg="Saving ConfigMap cache entry" key=whalesay name=my-config namespace=argo nodeId=memoized-simple-workflow-c5bp5
time="2021-10-13T14:32:32.516Z" level=info msg="Get configmaps 200"
time="2021-10-13T14:32:32.519Z" level=info msg="Update configmaps 403"
time="2021-10-13T14:32:32.521Z" level=error msg="Failed to save node outputs to cache" error="error creating cache entry: configmaps \"my-config\" is forbidden: User \"system:serviceaccount:argo:argo\" cannot update resource \"configmaps\" in API group \"\" in the namespace \"argo\". Please check out this page for help: https://argoproj.github.io/argo-workflows/memoization/#faqs" namespace=argo nodeID=memoized-simple-workflow-c5bp5 workflow=memoized-simple-workflow-c5bp5
time="2021-10-13T14:32:32.521Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=memoized-simple-workflow-c5bp5
time="2021-10-13T14:32:32.521Z" level=info msg=reconcileAgentPod namespace=argo workflow=memoized-simple-workflow-c5bp5

my permissions can be found here

https://github.com/wdma/Hippocampus-Analytics/blob/main/manifests/policies.yaml

The first few lines of the workflow-controller pod logs show that the Configmap was loaded correctly

time="2021-10-15T19:08:12Z" level=info msg="index config" indexWorkflowSemaphoreKeys=true
time="2021-10-15T19:08:12Z" level=info msg="cron config" cronSyncPeriod=10s
time="2021-10-15T19:08:12.689Z" level=info msg="not enabling pprof debug endpoints"
time="2021-10-15T19:08:12.703Z" level=info msg="config map" name=workflow-controller-configmap
time="2021-10-15T19:08:12.746Z" level=info msg="Get configmaps 200"
time="2021-10-15T19:08:12.764Z" level=info msg="Configuration:\nartifactRepository: {}\ncontainerRuntimeExecutor: emissary\ninitialDelay: 0s\nmetricsConfig: {}\nnodeEvents: {}\npodSpecLogStrategy: {}\ntelemetryConfig: {}\nworkflowDefaults:\n  metadata:\n    creationTimestamp: null\n  spec:\n    arguments: {}\n    artifactRepositoryRef:\n      configMap: my-config\n      key: whalesay-cache\n    entrypoint: entrypoint\n    parallelism: 3\n    podGC:\n      strategy: OnWorkflowSuccess\n    securityContext:\n      fsGroup: 2000\n      runAsGroup: 3000\n      runAsUser: 1000\n    ttlStrategy:\n      secondsAfterSuccess: 5\n  status:\n    finishedAt: null\n    startedAt: null\n"
time="2021-10-15T19:08:12.766Z" level=info msg="Persistence configuration disabled"
time="2021-10-15T19:08:12.771Z" level=info msg="Starting Workflow Controller" version=v3.2.0
time="2021-10-15T19:08:12.772Z" level=info msg="Workers: workflow: 32, pod: 32, pod cleanup: 4"
time="2021-10-15T19:08:12.784Z" level=info msg="List workflows 200"

Can anyone tell me how to fix this?