[work-in-progress][all] limitations of our type exports and proposed ways forward

[pichlermarc]

Description

Note

this is a working draft, the ideas here may be still be incomplete as I work through upstreaming examples and verifying my proposals. I will present this proposal in the OTel JS SIG meeting when it is ready.

The way we currently export our types in this repository is difficult to maintain and is the root-cause of important issues that need to be addressed. This issue is intended as a

• documentation of the status-quo
• a starting point to improvement in upcoming releases

Status-quo

As it stands today, we export classes directly, including associated types, with all internal "private" properties as part of our public API. This routinely causes trouble, as we expect the exact version of a package to be used when using other packages. While breaking is expected when a packages are pinned and an exact class type is provided, this is still a source of end-user pain.

One key problem here is that experimental packages like @opentelemetry/sdk-node depends on pinned versions of other SDK packages and expect types from these packages as part of the public API. If a package used by the end-user not match that version exactly (Demo A - TODO) then that type may be rejected by the typescript compiler due to re-definition of private properties on that class.

This problem cannot be fully fixed by just un-pinning the dependency, since end-users may still have another version somewhere in their lockfile and that old version may be incompatible, even though there were no changes to the class between these versions. So while an un-pinned version allows for de-duping of the package, the break may still happen, but but it may now happen depending on the lockfile history of the package and the pre-existing node_modules directory, which makes troubleshooting extremely difficult for us, since the user would have to supply their lockfile for us to investigate (which often a level of detail they are not allowed to provide).

This sudden increase in complexity from un-pinning means that attempting to solve this through unpinning alone may immobilize us to an extent where we cannot make any notable improvements to OTel JS, as we will be busy triaging issues and troubleshooting suddenly-occuring end-user problems. Issue triage and bug handling is already a key limiting factor for the project today, even with pinned dependencies.

Keeping packages pinned, however, is also not a solution. While pinned packages make it easier to spot such incompatibilities, it does introduce another pain-point for end-users: packages that cannot be de-duped and can blow up the size of node_modules to an extent where certain limits, like the maximum size of a lambda (250 MB), are reached, which may prevent a new or exiting OTel user from deploying their app. Similar issues apply to use of OTel JS on the web, but it also drives cost significantly on server-side deployments.

Exporting class types also poses a different problem: it promotes sub-classing our exports to modify behavior, which introduces a set of users that we have to account for during pull request reviews - and that limits us in which kind of optimizations we can apply. In most cases, users are extending existing classes using inheritance even tough they're not actively trying to make use of the polymorphic structure. There, the cleaner approach would be to use composition over inheritance, as the base that is desired is always the extension interface, rather than the base class. We even do that ourselves, which leads to APIs being exposed to end-users that would otherwise stay private. An example of this is BasicTracerProvider and NodeTracerProvider. There's no real benefit of having BasicTracerProvider be part of the polymorphic structure of the NodeTracerProvider, as the base interface the user will likely be interested in is TracerProvider from @opentelemetry/api. Further, users may extend any of these through

A side note on why this is less of a problem in contrib

This problem is only minimally present in the contrib repo, as most packages there are instrumentation packages.

These often only depend on only three packages:

• @opentelemetry/api
• @opentelemetry/core
• @opentelemetry/instrumentation

Only @opentelemetry/api and @opentelemetry/instrumentation are being used in the public API of the packages. With @opentelemetry/api being a peer dependency in all of these packages, it's fairly uncommon for multiple instances of @opentelemetry/api ending up in the node_modules of the end-user. For @opentelemetry/instrumentation, this problem occurs more often, but has less impact through the Instrumentation interface which stays compatible across versions through a lack of private private properties. If a user were to consume InstrumentationAbstract, they may run into similar issues as described above.

Proposed Solution

We stop exporting classes wherever possible, and pivot to using factory-functions with separately defined return-types instead. This will cut down a large amount of our public API, and since via this approach we won't expose private properties anymore, which usually cause this conflict. This means that two subsequent feature versions may be compatible with each other (types from the the newer may get assigned to an older one, and vice-versa - as long as they're actually compatible). An example of this approach can be seen in the re-structured OTLP exporter packages that I have been working on over the past few months (example).

I further propose we adopt Semantic Versioning for TypeScript types. This specification outlines principles that I had partially already been applying independently in the re-structured exporters. These principles seemed intuitively correct based on my experience guiding end-users through resolving type-related issues.
SemVer TS goes a step further by introducing proper documentation for types that should not be constructed directly by end-users. This is something we can also adopt for @opentelemetry/api to make expectations for SDK implementers more explicit.

Moving largely to factory-functions and hiding implementation details through hand-rolled return types is a breaking change, but has significant upsides:

• we can apply significant performance and bundle-size improvements in feature-releases as we don't have to consider the internals part of the public API anymore, as anything will be hidden behind an interface.
• it forces us to clean up any tests that assert on internal properties, which slows down development today as changes in one package often break tests in another one, that bloats up PRs and makes review more difficult. It also set a better example of how tests should be written and it forces us to write cleaner tests going forward
• most crucially, it enables us to un-pin our dependencies as the types used in the public API become stable and very predictable not only for our end-users, but also for us, which gives us breathing room to apply this change without being overwhelmed by a flood of hard-to-debug typescript compile issues.

Doing so also has some downsides:

• most notably, it prevents end-users/third-party distributions from subclassing our exports, which means that that users who do this today, would need to shift towards a composition-based approach over their current inheritance-based approach.
  • it also prevent us from sub-classing and we may have to change exports that exist today to a composition-based approach. MetricReader is an example of something that would need to change.
• it breaks the end-users's setup code, so this can only be done in an SDK major release. Fortunately, the change for end-users that don't extend classes can be a fairly straight-forward search and replace operation. See feat(sdk-metrics)!: export factory functions over classes  #4932 for an example of what changes for us and the end-user.

Once we've applied these changes, I propose we start un-pinning dependencies to packages that have received these changes, which will allow more flexibility in the types accepted by packages and will allow for easier de-duping of packages without introducing an unsustainable maintenance overhead for us.