I have broken down my workflow scenario into 2 separate WorkflowTemplates. outer-template would just define the steps and inner-template would hold that job definition that will spin up desired container, with all other fancy stuff. Now when I submit a request request.yaml, it does pass the parameter message down to outer and inner template and fails with this error:

    hello-59jg8-394098346:
      Boundary ID:  hello-59jg8-1953291600
      Children:
        hello-59jg8-534805352
      Display Name:    [0]
      Finished At:     2021-06-15T00:41:45Z
      Id:              hello-59jg8-394098346
      Message:         child 'hello-59jg8[0].init-step[0].step-1' errored
      Name:            hello-59jg8[0].init-step[0]
      Phase:           Error
      Started At:      2021-06-15T00:41:45Z
      Template Name:   HelloWorld
      Template Scope:  namespaced/outer-template
      Type:            StepGroup
    hello-59jg8-534805352:
      Boundary ID:   hello-59jg8-1953291600
      Display Name:  step-1
      Finished At:   2021-06-15T00:41:45Z
      Id:            hello-59jg8-534805352
      Message:       inputs.parameters.message was not supplied
      Name:          hello-59jg8[0].init-step[0].step-1
      Phase:         Error
      Started At:    2021-06-15T00:41:45Z
      Template Ref:
        Name:          inner-template
        Template:      InnerJob
      Template Scope:  namespaced/outer-template
      Type:            Skipped
  Phase:               Failed
  Started At:          2021-06-15T00:41:45Z
  Stored Templates:

Below 2 are WorkflowTemplates and third one is the request.

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: inner-template
  namespace: cali
  labels:
    workflows.argoproj.io/controller-instanceid: cali
spec:
  templates:
    - name: InnerJob
      metadata:
        annotations:
          sidecar.istio.io/inject: "false"
      inputs:
        parameters:
          - name: message
          - name: stepName
            value: ""
      resource:
        action: create
        successCondition: status.succeeded > 0
        failureCondition: status.failed > 0
        manifest: |
          apiVersion: batch/v1
          kind: Job
          metadata:
            generateName: hello-pod-
            annotations:
              sidecar.istio.io/inject: "false"
          spec:
            template:
              metadata:
                annotations:
                  sidecar.istio.io/inject: "false"
              spec:
                containers:
                  - name: hellopods
                    image: centos:7
                    command: [sh, -c]
                    args: ["echo ${message}; sleep 5; echo done; exit 0"]
                    env:
                      - name: message
                        value: "{{inputs.parameters.message}}"
                      - name: stepName
                        value: "{{inputs.parameters.stepName}}"
                restartPolicy: Never
      outputs:
        parameters:
          - name: job-name
            valueFrom:
              jsonPath: '{.metadata.name}'
          - name: job-obj
            valueFrom:
              jqFilter: '.'

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: outer-template
  namespace: cali
  labels:
    workflows.argoproj.io/controller-instanceid: cali
spec:
  entrypoint: HelloWorld
  templates:
    - name: HelloWorld
      inputs:
        parameters:
          - name: message
      steps:
        - - name: step-1
            templateRef:
              name: inner-template
              template: InnerJob
            arguments:
              parameters:
                - name: message
                - name: stepName
                  value: "this is step 1"
        - - name: step-2
            templateRef:
              name: inner-template
              template: InnerJob
            arguments:
              parameters:
                - name: message
                - name: stepName
                  value: "this is step 2"

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: hello-
  namespace: cali
  labels:
    workflows.argoproj.io/controller-instanceid: cali
spec:
  entrypoint: HelloWorld
  serviceAccountName: argo
  templates:
      - name: HelloWorld
        steps:
          - - arguments:
                parameters:
                  - name: message
                    value: "Hello World....."
              name: init-step
              templateRef:
                name: outer-template
                template: HelloWorld