New Blog Post: Build your own OpenTelemetry backents for fun, analytics... and maybe some profit :)

[arusevm]

Hi,

We've been using OpenTelemetry in some less obvious ways and have accumulated code and content that we started sharing recently. We believe that with the help of additional open source projects it's quite easy for software developers to build their own OTEL backends - for analytics, audit logging and even usage-based billing. Thought that might be interesting to the broader community, but not hundred percent sure what fits and what doesn't, so, I'll quickly outline it here and see how it goes.

A short intro I have in mind:

So, you've instrumented your code, launched your collector pipelines and OTEL signals are now flowing to your ops dashboards.

As a software engineer - you may be thinking 'What else can I do with OTEL data?' and you're not alone. Here is how you can build your own telemetry backends for analytics, security audit and even billing.

And a very, very quick outline (as I'm not sure these topics actually fit):

With a little bit of code (and a lot of other open source projects) you can:

• Illustrate the inner workings of your apps to help other team members learn
• Build better software - collect telemetry from your tests and analyze it (yes, as part of your release pipelines)
• Implement reliable (and yes, structured) audit logs
• Run an usage-based billing platform for your SaaS

---

Depending on the topics above - a post might refer to open source from Apache (like Arrow, BookKeeper, Druid, Parquet, Superset and other...), Pandas, Jupyter notebooks and more. May also refer to popular and not-so-open-source platforms like GitHub (and its Actions) for example.

Also, might refer to open source created by us - (note: it's a work in progress, haven't shared everything yet).

---

Happy to provide more details here or on CNCF Slack.

[svrnm]

@open-telemetry/docs-approvers adding this as context, @arusevm and I talked about this as a follow up to #5419 on slack and I suggested that a blog post may be an option here.

@arusevm on the blog post, a few points:

• If we go ahead with this blog post, I think what is important to state in the intro is that "building your own observability backend" is a fun exercise, and sometimes it can be useful to learn more about your data, but that in general you should leave it to subject matter experts to build a solution for production and that if you like that topic there are existing OSS projects that might be interested in your contributions.
• let's pick 1 or 2 open source components and then go with them. I personally think that Superset could be super interesting to see from a visualization perspective. If that works well we can talk about another post with some other technology.

[arusevm]

If we go ahead with this blog post, I think what is important to state in the intro is that "building your own observability backend" is a fun exercise, and sometimes it can be useful to learn more about your data

Sounds good to me - fun is always good, aaand also, my feeling is many people prefer to learn about something (like OTEL) by looking at the data. On top of that - by showing some data we can also add a few suggestions on how to add 'own' telemetry. That is - if a developer is looking to instrument their own code - a few practical suggestions on how to approach it... sort of follow from seeing how fields in the data are populated... We do have a few of these suggestions on our repo - I can get a link to that if an example will help?

let's pick 1 or 2 open source components and then go with them. I personally think that Superset could be super interesting to see from a visualization perspective.

Yeah, I think something more focused is better. I shared a number of bullet points that I knew are too much for a single post.

Superset comes with Druid (needs a database to query), we also have a fork of the otel demo app that also deploys Druid and Superset, so, we have the extra option that people can quickly try it. Don't know if that fits, but it's there anyway...

My only worry is that the OpenTelemetry guides on writing blog posts do mention something like 'prefer CNCF open source over non-CNCF... like Prometheus/Jaeger...' and Superset/Druid might go against that?