Skip to content

[Bug]: Yaml Dataflow - yaml provider - composite transform cant as middle transform #37736

@ZwxwZ

Description

@ZwxwZ

What happened?

In yaml provider,
this yaml working fine with "type: chain"

- type: yaml
  transforms:
    # Define the first transform of type "RaiseElementToPower"
    RaiseElementToPower:
      config_schema:
        properties:
          n: {type: integer}
      body:
        type: chain
        transforms:
          - type: MapToFields
            config:
              language: python
              append: true
              fields:
                power: "element ** {{n}}"

If i change the type to "type: composite",

- type: yaml
  transforms:
    # Define the first transform of type "RaiseElementToPower"
    RaiseElementToPower:
      config_schema:
        properties:
          n: {type: integer}
      body:
        type: composite
        transforms:
          - type: MapToFields
            input: xxxx
            config:
              language: python
              append: true
              fields:
                power: "element ** {{n}}"

there will have error, beacause it's required an input for the transform,
"type: composite" require include the input into the sub-yaml,

success example for "composite":
https://github.com/apache/beam-starter-python-provider/blob/main/examples/multi_input_output.yaml

do composite transform must including the input into one page?

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Infrastructure
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions