Provide a new label for metrics regarding destination type

[howardjohn]

I propose adding a new label to all the traffic metrics in ztunnel: traffic_type: waypoint|direct.

The motivation here is to allow querying for traffic without duplication when dealing with waypoints.

When I have a waypoint, I will get 3 timeseries: client-->waypoint, client-->server, waypoint-->server.

Logically, I know the first and last metrics are "to/from waypoint" traffic. However, there is no query to actually express that information, so I cannot filter it out. With the traffic_type we could identify this.

On outbound this is trivial. Its a bit less clear if we can 100% identify waypoint traffic on the inbound path

[stevenctl]

For Sandwich, we use find_inbound_upstream: Checks if the connection address is a workload that is part of the HBONE address's Waypoints.

The tricky part for inbound to-workload is figuring out if the traffic was originally Service or Workload addressed. Can we make a good enough guess here for telemetry purposes? For example, inbound workload has both wl-wp and svc-wp, we can look at the source address + identity and see if it matches either of these and give one of them priority.

Other option would be Waypoint adding a header with the original HBONE target that it saw.

[keithmattix]

Dumb question: wouldn't (Istio) waypoints provide some header with a value of envoy?

[howardjohn]

There are some headers, I don't know any that are guaranteed unless we add one

[jshaughn]

When I have a waypoint, I will get 3 timeseries: client-->waypoint, client-->server, waypoint-->server.

Just to make sure I'm clear... client-->waypoint and waypoint-->server are reported by ztunnel (app="ztunnel" , reporter="destination"), and the client-->server is reported by the waypoint itself (reporter="waypoint"). Right?

I think for ztunnel, traffic_type="direct" would apply for non-waypoint traffic?

I think it would be useful to add traffic_type: waypoint|direct, as long as the waypoint identification is reliable, both when it is the source and dest workload. If there isn't a way to identify the inbound situation then it wouldn't be worth the overhead.

[jshaughn]

maybe too verbose, but just an idea: traffic_type: waypoint_to|waypoint_from|direct

[howardjohn]

The 'from waypoint' is the tricky part. 'To waypoint' is easy, and direct = !towaypoint && !fromwaypoint, but there is no reliable way to know if its from a waypoint. We can apply some heuristics -- perhaps even good enough ones -- but no guarantees for sure

[jshaughn]

The unfortunate effect is that currently, to try and identify these waypoint edges, a consumer like Kiali needs to combine waypoint config and telemetry. This is a slippery slope because config is current, but telem reflects the past. So it's much better if everything can be identified in the telemetry.

Just wanted to confirm my question above, with waypoints, the client-->server telem is only reported by waypoint, right? But waypoint reporting is only for request traffic, afaics. So, for a service using a waypoint that is handling app-level TCP traffic, there is no client-server TCP telem, the only telem is disconnected, client-->waypoint and waypoint-->server. Is that right?

[howardjohn]

No, waypoint should report TCP as well if the service is TCP (same semantics as a sidecar)

[jshaughn]

No, waypoint should report TCP as well if the service is TCP (same semantics as a sidecar)

Hmmm, I'm not seeing it. I'll re-check...

[jshaughn]

Hmmm, I'm not seeing it. I'll re-check...

I think due to istio/istio#53593