K
Q

Unable to pass output parameters from one workflowTemplate to a workflow via another workflowTemplate

1/28/2022

I've Two workflowTemplates generate-output, lib-read-outputs and One workflow output-paramter as follows

  1. generate-output.yaml
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: generate-output
spec:
  entrypoint: main
  templates:
    - name: main
      dag:
        tasks:

          # Generate Json for Outputs
          - name: read-outputs
            arguments:
              parameters:
                - name: outdata
                  value: |
                    {
                      "version": 4,
                      "terraform_version": "0.14.11",
                      "serial": 0,
                      "lineage": "732322df-5bd43-6e92-8f46-56c0dddwe83cb4",
                      "outputs": {
                        "key_alias_arn": {
                          "value": "arn:aws:kms:us-west-2:123456789:alias/tetsing-key",
                          "type": "string",
                          "sensitive": true
                        },
                        "key_arn": {
                          "value": "arn:aws:kms:us-west-2:123456789:alias/tetsing-key",
                          "type": "string",
                          "sensitive": true
                        }
                      }
                    }
            template: retrieve-outputs

    # Create Json
    - name: retrieve-outputs
      inputs:
        parameters:
          - name: outdata
      script:
        image: python
        command: [python]
        env:
          - name: OUTDATA
            value: "{{inputs.parameters.outdata}}"
        source: |
          import json
          import os
          OUTDATA = json.loads(os.environ["OUTDATA"])
          with open('/tmp/templates_lst.json', 'w') as outfile:
            outfile.write(str(json.dumps(OUTDATA['outputs'])))
        volumeMounts:
          - name: out
            mountPath: /tmp
      volumes:
        - name: out
          emptyDir: { }
      outputs:
        parameters:
          - name: message
            valueFrom:
              path: /tmp/templates_lst.json
  1. lib-read-outputs.yaml
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: lib-read-outputs
spec:
  entrypoint: main
  templates:
    - name: main
      dag:
        tasks:
          # Read Outputs
          - name: lib-wft
            templateRef:
              name: generate-output
              template: main
  1. output-paramter.yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: output-paramter-
spec:
  entrypoint: main
  templates:
    - name: main
      dag:
        tasks:
          # Json Output data task1
          - name: wf
            templateRef:
              name: lib-read-outputs
              template: main

          - name: lib-wf2
            dependencies: [wf]
            arguments:
              parameters:
                - name: outputResult
                  value: "{{tasks.wf.outputs.parameters.message}}"
            template: whalesay

    - name: whalesay
      inputs:
        parameters:
          - name: outputResult
      container:
        image: docker/whalesay:latest
        command: [cowsay]
        args: ["{{inputs.parameters.outputResult}}"]

I am trying to pass the output parameters generated in workflowTemplate generate-output to workflow output-paramter via lib-read-outputs

When I execute them, it's giving the following error - Failed: invalid spec: templates.main.tasks.lib-wf2 failed to resolve {{tasks.wf.outputs.parameters.message}}

-- Biru
argo
argo-workflows
argoproj
kubernetes
workflow

Similar Questions

How to automatically spin n-number of helm deployments from a single helm chart
How to fix "connection refused" error during filebeat start in kubernetes
Connection Refuse communicating from pod to service
Whole Kubernetes cluster down after master crash
Kubectl desribe returns 404 for Ingresses

1 Answer

1/28/2022

DAG and steps templates don't produce outputs by default

DAG and steps templates do not automatically produce their child templates' outputs, even if there is only one child template.

For example, the no-parameters template here does not produce an output, even though it invokes a template which does have an output.

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
spec:
  templates:
  - name: no-parameters
    dag:
      tasks:
      - name: get-a-parameter
        template: get-a-parameter

This lack of outputs makes sense if you consider a DAG template with multiple tasks:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
spec:
  templates:
  - name: no-parameters
    dag:
      tasks:
      - name: get-a-parameter
        template: get-a-parameter
      - name: get-another-parameter
        depends: get-a-parameter
        template: get-another-parameter

Which task's outputs should no-parameters produce? Since it's unclear, DAG and steps templates simply do not produce outputs by default.

You can think of templates as being like functions. You wouldn't expect a function to implicitly return the output of a function it calls.

def get_a_string():
    return "Hello, world!"

def call_get_a_string():
    get_a_string()

print(call_get_a_string())  # This prints nothing.

But a DAG or steps template can forward outputs

You can make a DAG or a steps template forward an output by setting its outputs field.

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: get-parameters-wftmpl
spec:
  templates:
  - name: get-parameters
    dag:
      tasks:
      - name: get-a-parameter
        template: get-a-parameter
      - name: get-another-parameter
        depends: get-a-parameter
        template: get-another-parameter
    # This is the critical part!
    outputs:
      parameters:
      - name: parameter-1
        valueFrom: 
          expression: "tasks['get-a-parameter'].outputs.parameters['parameter-name']"
      - name: parameter-2
        valueFrom: 
          expression: "tasks['get-another-parameter'].outputs.parameters['parameter-name']"
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
spec:
  templates:
  - name: print-parameter
    dag:
      tasks:
      - name: get-parameters
        templateRef:
          name: get-parameters-wftmpl
          template: get-parameters 
      - name: print-parameter
        depends: get-parameters
        template: print-parameter
        arguments:
          parameters:
          - name: parameter
            value: "{{tasks.get-parameters.outputs.parameters.parameter-1}}"

To continue the Python analogy:

def get_a_string():
    return "Hello, world!"

def call_get_a_string():
    return get_a_string()  # Add 'return'.

print(call_get_a_string())  # This prints "Hello, world!".

So, in your specific case...

  1. Add an outputs section to the main template in the generate-parameter WorkflowTemplate to forward the output parameter from the retrieve-parameters template.

    apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: generate-parameter
spec:
  entrypoint: main
  templates:
    - name: main
      outputs:
        parameters:
          - name: message
            valueFrom: 
              expression: "tasks['read-parameters'].outputs.parameters.message"
      dag:
        tasks:
          # ... the rest of the file ...

  2. Add an outputs section to the main template in the lib-read-parameters WorkflowTemplate to forward generate-parameter's parameter.

    apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: lib-read-parameters
spec:
  entrypoint: main
  templates:
    - name: main
      outputs:
        parameters:
          - name: message
            valueFrom:
              expression: "tasks['lib-wft'].outputs.parameters.message"
      dag:
        tasks:
          # ... the rest of the file ...
-- crenshaw-dev
Source: StackOverflow