[connector/routing] Ability to dynamically determine a pipeline ID

[djaglowski]

Component(s)

connector/routing

Is your feature request related to a problem? Please describe.

Scenario

• A specialized observability team manages a gateway collector
• All other teams forward their data to the gateway collector for processing
• Each team MAY provide custom processing logic for their data

In order to accomplish this, the observability team could define a simple contract which the other teams must follow:

• Annotate your telemetry with a specific attribute. e.g. resource.attributes["team"] = "team-A"
• A routing connector with a fixed name will route to a pipeline named according to the attribute.
• A forward connector with a fixed name will subscribe to data coming off of team-specific pipelines.
• Teams may provide corresponding processing configuration in a file called e.g. team-A.yaml.

# gateway-base.yaml
receivers:
  otlp: ...
  
processors:
  transform/default: ...
  
connectors:
  routing/by-team-name:
    default_pipelines: [ logs/default ]
    table:
      - context: resource
        condition: attributes["team"] != nil
        lookup_pipeline: Format("custom-%s", attributes["team"])
      - ... # can evaluate other routes as usual
  forward/out: # all teams MUST forward here when done
      
exporters:
  otlp: ...
  
service:
  pipelines:
    logs/in: 
      receivers: [ otlp ]
      exporters: [ routing/by-team-name ]
    logs/default:
      receivers: [ routing/by-team-name ]
      processors: [ transform/default ]
      exporters: [ otlp ]
    logs/out:
      receivers: [ forward/out ]
      exporters: [ otlp ]

# team-A.yaml
processors:
  transform/team-A:
  filter/team-A:

service:
  pipelines:
    logs/my-team:
      receivers: [ routing/by-team-name ]
      processors: [ transform/team-A, filter/team-A ]
      exporters: [ forward/out ]

The observability team runs the collector with a base config to define the static elements, and each team's config to pull in team specific processing rules. otelcol --config gateway-base.yaml --config team-A.yaml ...

Problem

Currently, all routes must include a pipelines field which explicitly names the pipeline(s) to which the data should be routed if it matches the route criteria. In the above example, a contract or convention is used to manage routing, but there is no way to dynamically lookup relevant pipelines.

Describe the solution you'd like

Instead of requiring each route item to specify pipelines, allow an alternative field e.g. lookup_pipeline which is a "computed" pipeline ID.

If a route uses this field, the value is interpreted as an OTTL expression which must return a string that is interpreted as a pipeline.ID. (The data type can be inferred from the type of data that is being routed.)

The routing connector can use its PipelineIDs method to check if it is a valid pipeline. If so, it routes the data to that pipeline. If not, the route is not considered a match in which case route matching moves on to the next route as usual.

[github-actions]

Pinging code owners:

• connector/routing: @jpkrohling @mwear

See Adding Labels via Comments if you do not have permissions to add labels yourself.