[FR] Consolidations distributed traces and continues profiling

[yaroslavkasatikov]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Hey team,
I would be wonderful if we can link tracing and profiling.
I found in profiler API "trace_id" field, but when I 'm checking it from ebpf profiler, the "trace_id" field is empty.

Use case

It can be very useful to collect all information regarding the trace. we can see not only traces (request etc),but check cpu/mem usage for all of them

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[1473371932]

Hi, DeepFlow itself doesn't passthrough trace id in your business, you need to pass trace ids from your business itself for us to recognize them. This trace id can be called any name when you pass it in the business header, just configure agent-group-config via deepflow-ctl and add the name.

[yaroslavkasatikov]

Hey @1473371932 ! Thank you for the respond.
I was able to generate trace_id ( enabled opentelemery on nginx controller and it set the trace_id for me).
So I can see it in deepflow tracing now, but how can I link it to profiler?

[1473371932]

Hey @1473371932 ! Thank you for the respond. I was able to generate trace_id ( enabled opentelemery on nginx controller and it set the trace_id for me). So I can see it in deepflow tracing now, but how can I link it to profiler?

If your request passes through the nginx gateway, you need to generate an X-Request-ID in the gateway in order to correlate the request before and after the gateway, in the case of the trace id identification already configured above, you don't need to do any additional configuration, click on the request that has been identified in the above figure, it will automatically generate a flame map

[yaroslavkasatikov]

Yes, it works fine. But it's about tracing. My goal is link trace id to the Profiler. So I want to get profile for the application (for example - nginx ingress) in the moment when trace was captured.
So here is the use case example.
I have Ollama after the ingress. And have many requests to Ollama from different sources. And on requests is implemented too long.
So here is the issue investigation path which can be useful for me in this case:

1. I'm checking Deepflow continues tracing and looking up for long request's trace_id.
2. I'm going to Continues profiling and looking for sample which is generated when process (ollama for the example) works too long.
3. I'm trying to investigate which function( syscall ) was the reason of the request delay