Spring: Cannot configure Segment name in BaseAbstractXRayInterceptor

[stnor]

Hi,
I'm manually instrumenting my fairly large Spring application.

Why are you not adding the Class name to the sub-segment name?
It's quite frustrating to not have the class name in the UI when looking at the traces.

I would suggest changing:

aws-xray-sdk-java/aws-xray-recorder-sdk-spring/src/main/java/com/amazonaws/xray/spring/aop/BaseAbstractXRayInterceptor.java

Line 48 in d66d69a

Subsegment subsegment = AWSXRay.beginSubsegment(pjp.getSignature().getName());

to

    Subsegment subsegment = AWSXRay.beginSubsegment(subsegmentNameFromSignature(pjp.getSignature()));

so that the user can override the subsegment name.

[stnor]

I ended up rolling my own abstract class and removed the aws-xray-recorder-sdk-spring dep. It's more of an example than a library at this point, no offense.

[anuraaga]

Hi @stnor - thanks for filing these issues. It sounds like you are adding tracing to a new app. Did you by any chance take a look at our support for OpenTelemetry?

https://aws-otel.github.io/docs/getting-started/java-sdk

It has far more features and can provide a better experience than XRay sdk for new apps. There is a Java agent that can automatically instrument and/or ability to manually instrument by creating spans.

Our effort is mostly focused on OpenTelemetry going forward so we probably wouldn't be able to make changes for improving the experience here, though we would fix any bugs though

[stnor]

Thanks, good to know. The app is huge and 10 years old at this point. I’m thinking of migrating from AppDynamics, which it usung -javaagent. And looking at the open telemetry support st present, it does not look production ready? For instance, no supporr for lowering sample rates from 100%?

…

On Sun, 2 May 2021 at 04:12, Anuraag Agrawal ***@***.***> wrote: Hi @stnor <https://github.com/stnor> - thanks for filing these issues. It sounds like you are adding tracing to a new app. Did you by any chance take a look at our support for OpenTelemetry? https://aws-otel.github.io/docs/getting-started/java-sdk It has far more features and can provide a better experience than XRay sdk for new apps. There is a Java agent that can automatically instrument and/or ability to manually instrument by creating spans. Our effort is mostly focused on OpenTelemetry going forward so we probably wouldn't be able to make changes for improving the experience here, though we would fix any bugs though — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#281 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEJAD6LISXYGCS3ZO572MTTLSYKHANCNFSM435BSENQ> .

[anuraaga]

@stnor While it hasn't quite hit GA yet it can be used in production and a few folks are doing so.

While X-Ray centralized sampling isn't currently supported unfortunately, the sampling rate can be tweaked with a flag (traceidratio).

https://github.com/open-telemetry/opentelemetry-java/tree/main/sdk-extensions/autoconfigure#sampler

[stnor]

Thanks, I'll check it out!

…

On Sun, 2 May 2021 at 09:59, Anuraag Agrawal ***@***.***> wrote: @stnor <https://github.com/stnor> While it hasn't quite hit GA yet it can be used in production and a few folks are doing so. While X-Ray centralized sampling isn't currently supported unfortunately, the sampling rate can be tweaked with a flag (traceidratio). https://github.com/open-telemetry/opentelemetry-java/tree/main/sdk-extensions/autoconfigure#sampler — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#281 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEJAD67CIHMTPNVUF6KJO3TLUA7HANCNFSM435BSENQ> .

[stnor]

I've gotten the otel stuff to work decently well now... However I do not understand how to compliment the -javaagent's standard tracing with my own Spring AOP aspects.

I think the right approach would be to create nested spans? @WithSpan is not an option on 1000 methods :)
It's unclear to me how I do this. Get hold of the parent span? In the Xray Java SDK it's a oneliner:
AWSXRay.beginSubsegment(pjp.getSignature().getName());

I've looked here: https://opentelemetry.io/docs/java/manual_instrumentation/ and it doesn't really cover my use-case.

Some issue referred to OpenTelemetrySdk.getTracerProvider() but that class doesn't seem to exist anymore?
Edit: Intellj was acting up. But still don't understand how to get a ref to the current Tracer/Span or whatever I need to do to proceed ;)

I've added these deps:

        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-api</artifactId>
        </dependency>
        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-sdk</artifactId>
        </dependency>

Any pointers or help would be highly appreciated.

[stnor]

Also found open-telemetry/opentelemetry-java#3035

[stnor]

Sorry for the spam.
This code is being executed, but does nothing (that shows in Xray).

(1) Is it "correct"?
(2) If so, why doesn't the spans show up in Xray? The "standard" javaagent tracing is there still.

public abstract class AbstractSpringOtelInterceptor {

    private static final Log logger = LogFactory.getLog(AbstractSpringOtelInterceptor.class);
    private Tracer tracer = OpenTelemetrySdk.builder().build().getTracer("se.nomp");

    protected Object processXRayTrace(ProceedingJoinPoint pjp) throws Throwable {
        Span span = tracer.spanBuilder(generateSubsegmentName(pjp)).startSpan();

        try(Scope scope = span.makeCurrent()) {
            return conditionalProceed(pjp);
        } finally {
            span.end();
        }
    }

    protected String generateSubsegmentName(ProceedingJoinPoint pjp) {
        return "Hello " + pjp.getSignature().getDeclaringType().getSimpleName() + "." + pjp.getSignature().getName();
    }

    public static Object conditionalProceed(ProceedingJoinPoint pjp) throws Throwable {
        return pjp.getArgs().length == 0 ? pjp.proceed() : pjp.proceed(pjp.getArgs());
    }
}

-javaagent:/Users/stnor/devel/src/nomp/web/docker/opt/aws-opentelemetry-agent-1.1.0.jar
-Dotel.resource.attributes=service.name=Nomp
-Dotel.instrumentation.jdbc-datasource.enabled=true
-Dotel.traces.sampler.traceidratio=true
-Dotel.traces.sampler.arg=1

Also tried

    protected Object processXRayTrace(ProceedingJoinPoint pjp) throws Throwable {
        Span parent = Span.current();
        Span span = tracer.spanBuilder(generateSubsegmentName(pjp)).setParent(Context.current().with(parent)).startSpan();
        try(Scope scope = span.makeCurrent()) {
            return conditionalProceed(pjp);
        } finally {
            span.end();
        }
    }

[anuraaga]

Hey @stnor - just to confirm you are using the aws distro if the Java agent and a collector configured for x-ray? Are you seeing the agent's spans, but just not seeing yours?

To get a tracer when using the agent, you need to use GlobalOpenTelemetry.get() to get the instance of OpenTelemetry managed by the agent.

In fact, you should make sure not to have a dependency on opentelemetry-sdk in your app as the agent brings it in. The dependency itself doesn't hurt but can lead to the mistake of trying to use it. The only artifact needed for instrumenting code is opentelemetry-api.

Other than that, your first snippet (which correctly implicitly parents to the current span) looks good.

[stnor]

Thanks @anuraaga - that solved it!
I also had issues getting the aws-otel-collector to work since it did not pick up my credentials. had to do:

git clone https://github.com/aws-observability/aws-otel-collector.git ; \
    cd aws-otel-collector; \
    docker run --rm -p 4317:4317 -p 55680:55680 -p 8889:8888 \
      -e AWS_REGION=xxxx-e AWS_ACCESS_KEY_ID=xxxx -e AWS_SECRET_ACCESS_KEY=xxxx \
      -v "${PWD}/examples/docker/config-test.yaml":/otel-local-config.yaml \
      --name awscollector public.ecr.aws/aws-observability/aws-otel-collector:latest \
     --config otel-local-config.yaml;

after editing the Dockerfile.

[stnor]

Here's the result :) Thanks again.

[anuraaga]

Nice trace!

If you found anything, always happy for more ideas on improving instrumentation at https://github.com/open-telemetry/opentelemetry-java-instrumentation/. Instrumentation for hikaricp seems like it could be nice for us to offer natively.

[anuraaga]

Ah and I've always had the idea of injecting OpenTelemetry into the spring context which I don't think we do right now. So you could use @Inject instead of GlobalOpenTelemetry.get. In the meantime you should probably do so in your app to make sure the actual tracers can (relatively) easily work if you decide to not use the agent in the future for any reason, you'd just replace the bean with an instance of OpenTelemetrySdk you build yourself.

[stnor]

I'm attaching my code here if anyone else would like to instrument att their services and repos using OpenTelemetry.

One needs to edit the within(se.nomp..*) to your package in the @Pointcut:s and also change the INSTRUMENTATION_APP_PACKAGE in AbstractSpringOpenTelemetryInterceptor
spring-service-repo-aspects.zip

[willarmiros]

Thanks for attaching that reference @stnor! I'll go ahead and resolve this since it seems like you've resolved your issue using OpenTelemetry.