(HTTP) Exporter rotating certificate based authentication

[Rand0mF]

Is your feature request related to a problem? Please describe.

Currently certificate based authentication with the exporter is possible right now with httpAgentOptions e.g.

new OTLPMetricExporter({
   url: `${ url }`,
   httpAgentOptions: {
      key: fs.readFileSync('private-key.pem'),
      cert: fs.readFileSync('certificate.pem'),
   }
})

However, with short lived certificates (which are rotated on filesystem level) this will fail when the certificate is expired.

Describe the solution you'd like

Not entirely sure, maybe

• a (optional) callback to provide authentication for the export call. HTTP clients like got have good options for hooks before requests, those hooks could be passed as options to the OTLPMetricExporter. Not sure if the used http client library of otel has similar functionality.
• expose the method to set the http(s) agent from the outside. This could be used to change it later in time, or to provide a custom https agent when initializing

Describe alternatives you've considered

1. Creating a new nodejs https agent when the certificate is rotated and then setting it in the otel library. In older versions this could be done like that: metricExporter._otlpExporter.agent = myNewAgent
2. Recreating the exporter seems to be not so easy because of the links to the MetricReader, MeterProvider and setting those as global instances (setGlobalMeterProvider). However it could still be possible somehow I imagine.

Additional context

[pichlermarc]

Hi, thanks for reaching out with this feature-request. 🙂

Recreating the exporter seems to be not so easy because of the links to the MetricReader, MeterProvider and setting those as global instances (setGlobalMeterProvider). However it could still be possible somehow I imagine.

This is possible by implementing the PushMetricReader interface. An implementation could keep a reference to a running exporter and would forward calls to export(), selectAggregationTemporality(), shutdown(), forceFlush(),...

Once it's time to rotate certificates, a new internal exporter is created that replaces the old one. The old one can be shut down and operation resumes with the new one.

[Rand0mF]

thanks for that alternative. Do you have an example/documentation of how to register the implementation of PushMetricReader to otel?

In general do you think that's the desired solution or a workaround which is possible right now? What would be your idea of how a good solution for this (maybe not too uncommon) usecase? Maybe I can even contribute it then.

[pichlermarc]

thanks for that alternative. Do you have an example/documentation of how to register the implementation of PushMetricReader to otel?

Ah sorry, apologies - I meant the PushMetricExporter interface - you'd register it the same as any exporter.

In general do you think that's the desired solution or a workaround which is possible right now? What would be your idea of how a good solution for this (maybe not too uncommon) usecase? Maybe I can even contribute it then.

I don't think it's a desired solution, no - it is a workaround that is possible today. Ideally we could allow to reconfigure everything and even allow custom agent implementations (for i.e connect proxies) but this is especially tricky to do in a "safe" way without breaking other features, like @opentelemetry/instrumentation-http. 😞

In particular when exposing interfaces that encourage people to use the http module on SDK setup, it makes us run into a limitation on @opentelemetry/instrumentation (the base for @opentelemetry/instrumentation-http) where the load-order of modules mattes (we wrap require/hook import, which means that if something is loaded before we can instrument it, it's not instrumented). This is why we delay loading of the http module in the exporters. It's very difficult to find a one-size-fits all solution - and when introducing one I'd like to cover as many use-cases as possible while not providing a footgun to end-users.

Usually the go-to way for anything that goes beyond the default is to use an OTel collector co-deployed with the app (in Agent mode) - it's what I usually recommend if the exporter itself does not do exactly what one wants it to do. The otlphttpexporter does offer a way to hot-reload certificates as well via the reload_interval option. - along with lots of other options that may not be possible today in the SDK exporters.