What is the output of loop task in argo?

8/25/2020

As per the Argo DAG template documentation.

tasks.<TASKNAME>.outputs.parameters: When the previous task uses 'withItems' or 'withParams', this contains a JSON array of the output parameter maps of each invocation

When trying with the following simple workflow:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: test-workflow-
spec:
  entrypoint: start
  templates:
  - name: start
    dag:
      tasks:
        - name: with-items
          template: hello-letter
          arguments:
            parameters:
            - name: input-letter
              value: "{{item}}"
          withItems:
          - A
          - B
          - C
        - name: show-result
          dependencies:
          - with-items
          template: echo-result
          arguments:
            parameters:
            - name: input
              value: "{{tasks.with-items.outputs.parameters}}"

  - name: hello-letter
    inputs:
      parameters:
      - name: input-letter
    outputs:
      parameters:
      - name: output-letter
        value: "{{inputs.parameters.input-letter}}"
    script:
      image: alpine
      command: ["sh"]
      source: |
        echo "{{inputs.parameters.input-letter}}"

  - name: echo-result
    inputs:
      parameters:
      - name: input
    outputs:
      parameters:
      - name: output
        value: "{{inputs.parameters.input}}"
    script:
      image: alpine
      command: ["sh"]
      source: |
        echo {{inputs.parameters.input}}

I get the following error : Failed to submit workflow: templates.start.tasks.show-result failed to resolve {{tasks.with-items.outputs.parameters}}

Argo version (running in a minikube cluster)

argo: v2.10.0+195c6d8.dirty
  BuildDate: 2020-08-18T23:06:32Z
  GitCommit: 195c6d8310a70b07043b9df5c988d5a62dafe00d
  GitTreeState: dirty
  GitTag: v2.10.0
  GoVersion: go1.13.4
  Compiler: gc
  Platform: darwin/amd64

Same error in Argo 2.8.1, although, using .result instead of .parameters in the show-result task worked fine (result was [A,B,C]), but doesn't work in 2.10 anymore

        - name: show-result
          dependencies:
          - with-items
          template: echo-result
          arguments:
            parameters:
            - name: input
              value: "{{tasks.with-items.outputs.result}}"

The result:

STEP                                TEMPLATE      PODNAME                                     DURATION  MESSAGE
  test-workflow-parallelism-xngg4  start                                                                                                                     
 ├-✔ with-items(0:A)                hello-letter  test-workflow-parallelism-xngg4-3307649634  6s                                                              
 ├-✔ with-items(1:B)                hello-letter  test-workflow-parallelism-xngg4-768315880   7s                                                              
 ├-✔ with-items(2:C)                hello-letter  test-workflow-parallelism-xngg4-2631126026  9s                                                              
 └-⚠ show-result                    echo-result                                                         invalid character 'A' looking for beginning of value

I also tried to change the show-result task as:

        - name: show-result
          dependencies:
          - with-items
          template: echo-result
          arguments:
            parameters:
            - name: input
              value: "{{tasks.with-items.outputs.parameters.output-letter}}"

Executes without no errors:

STEP                                TEMPLATE      PODNAME                                     DURATION  MESSAGE
  test-workflow-parallelism-qvp72  start                                                                 
 ├-✔ with-items(0:A)                hello-letter  test-workflow-parallelism-qvp72-4221274474  8s          
 ├-✔ with-items(1:B)                hello-letter  test-workflow-parallelism-qvp72-112866000   9s          
 ├-✔ with-items(2:C)                hello-letter  test-workflow-parallelism-qvp72-1975676146  6s          
 └-✔ show-result                    echo-result   test-workflow-parallelism-qvp72-3460867848  3s 

But the parameter is not replaced by the value:

argo logs test-workflow-parallelism-qvp72
test-workflow-parallelism-qvp72-1975676146: 2020-08-25T14:52:50.622496755Z C
test-workflow-parallelism-qvp72-4221274474: 2020-08-25T14:52:52.228602517Z A
test-workflow-parallelism-qvp72-112866000: 2020-08-25T14:52:53.664320195Z B
test-workflow-parallelism-qvp72-3460867848: 2020-08-25T14:52:59.628892135Z {{tasks.with-items.outputs.parameters.output-letter}}

I don't understand what to expect as the output of a loop! What did I miss? Is there a way to find out what's happening?

-- jose
argo-workflows
argoproj
kubernetes

2 Answers

1/20/2022

This is quite a common problem I've faced, I have not come across this in any bug report or feature documentation so far so it's yet to be determined if this is a feature or bug. However argo is clearly not capable in performing a "map-reduce" flow OOB.

The only "real" workaround I've found is to attach an artifact, write the with-items task output to it, and pass it along to your next step where you'll do the "reduce" yourself in code/script by reading values from the artifact.

---- edit -----

As mentioned by another answer this was indeed a bug which was resolved for the latest version, this resolves the usage of parameters as you mentioned as an option but outputs.result still causes an error post bugfix.

-- Tom Slabbaert
Source: StackOverflow

1/20/2022

There was a bug which caused this error before Argo version 3.2.5. Upgrade to latest and try again.

It looks like the problem was in the CLI only. I submitted the workflow with kubectl apply, and it ran fine. The error only appeared with argo submit.

The argo submit error was resolved when I upgraded to 3.2.6.

-- crenshaw-dev
Source: StackOverflow