As per the Argo DAG template documentation.
tasks.<TASKNAME>.outputs.parameters
: When the previous task uses 'withItems' or 'withParams', this contains a JSON array of the output parameter maps of each invocation
When trying with the following simple workflow:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: test-workflow-
spec:
entrypoint: start
templates:
- name: start
dag:
tasks:
- name: with-items
template: hello-letter
arguments:
parameters:
- name: input-letter
value: "{{item}}"
withItems:
- A
- B
- C
- name: show-result
dependencies:
- with-items
template: echo-result
arguments:
parameters:
- name: input
value: "{{tasks.with-items.outputs.parameters}}"
- name: hello-letter
inputs:
parameters:
- name: input-letter
outputs:
parameters:
- name: output-letter
value: "{{inputs.parameters.input-letter}}"
script:
image: alpine
command: ["sh"]
source: |
echo "{{inputs.parameters.input-letter}}"
- name: echo-result
inputs:
parameters:
- name: input
outputs:
parameters:
- name: output
value: "{{inputs.parameters.input}}"
script:
image: alpine
command: ["sh"]
source: |
echo {{inputs.parameters.input}}
I get the following error :
Failed to submit workflow: templates.start.tasks.show-result failed to resolve {{tasks.with-items.outputs.parameters}}
Argo version (running in a minikube cluster)
argo: v2.10.0+195c6d8.dirty
BuildDate: 2020-08-18T23:06:32Z
GitCommit: 195c6d8310a70b07043b9df5c988d5a62dafe00d
GitTreeState: dirty
GitTag: v2.10.0
GoVersion: go1.13.4
Compiler: gc
Platform: darwin/amd64
Same error in Argo 2.8.1, although, using .result
instead of .parameters
in the show-result task worked fine (result was [A,B,C]
), but doesn't work in 2.10 anymore
- name: show-result
dependencies:
- with-items
template: echo-result
arguments:
parameters:
- name: input
value: "{{tasks.with-items.outputs.result}}"
The result:
STEP TEMPLATE PODNAME DURATION MESSAGE
⚠ test-workflow-parallelism-xngg4 start
├-✔ with-items(0:A) hello-letter test-workflow-parallelism-xngg4-3307649634 6s
├-✔ with-items(1:B) hello-letter test-workflow-parallelism-xngg4-768315880 7s
├-✔ with-items(2:C) hello-letter test-workflow-parallelism-xngg4-2631126026 9s
└-⚠ show-result echo-result invalid character 'A' looking for beginning of value
I also tried to change the show-result task as:
- name: show-result
dependencies:
- with-items
template: echo-result
arguments:
parameters:
- name: input
value: "{{tasks.with-items.outputs.parameters.output-letter}}"
Executes without no errors:
STEP TEMPLATE PODNAME DURATION MESSAGE
✔ test-workflow-parallelism-qvp72 start
├-✔ with-items(0:A) hello-letter test-workflow-parallelism-qvp72-4221274474 8s
├-✔ with-items(1:B) hello-letter test-workflow-parallelism-qvp72-112866000 9s
├-✔ with-items(2:C) hello-letter test-workflow-parallelism-qvp72-1975676146 6s
└-✔ show-result echo-result test-workflow-parallelism-qvp72-3460867848 3s
But the parameter is not replaced by the value:
argo logs test-workflow-parallelism-qvp72
test-workflow-parallelism-qvp72-1975676146: 2020-08-25T14:52:50.622496755Z C
test-workflow-parallelism-qvp72-4221274474: 2020-08-25T14:52:52.228602517Z A
test-workflow-parallelism-qvp72-112866000: 2020-08-25T14:52:53.664320195Z B
test-workflow-parallelism-qvp72-3460867848: 2020-08-25T14:52:59.628892135Z {{tasks.with-items.outputs.parameters.output-letter}}
I don't understand what to expect as the output of a loop! What did I miss? Is there a way to find out what's happening?
This is quite a common problem I've faced, I have not come across this in any bug report or feature documentation so far so it's yet to be determined if this is a feature or bug. However argo is clearly not capable in performing a "map-reduce" flow OOB.
The only "real" workaround I've found is to attach an artifact, write the with-items
task output to it, and pass it along to your next step where you'll do the "reduce" yourself in code/script by reading values from the artifact.
---- edit -----
As mentioned by another answer this was indeed a bug which was resolved for the latest version, this resolves the usage of parameters
as you mentioned as an option but outputs.result
still causes an error post bugfix.
There was a bug which caused this error before Argo version 3.2.5. Upgrade to latest and try again.
It looks like the problem was in the CLI only. I submitted the workflow with kubectl apply
, and it ran fine. The error only appeared with argo submit
.
The argo submit
error was resolved when I upgraded to 3.2.6.