Include note about upgrade steps for target allocator for operator v1beta1

[jaronoff97]

opentelemetry.io/content/en/blog/2024/prom-and-otel/index.md

Line 299 in 81c8e86

### Configuration

Hello! We had someone reach out on this issue mentioning that the docs are using the old version of the target allocator and collector CRDs (v1alpha1) and there is a notable breaking change between those versions for the target allocator. We have a document for the upgrade guide and it would be great to edit this blog post to mention that if you are using v1beta1 be sure to follow the upgrade guide. I'm not sure if these back-compatible notices are something we've done before, but it seems like users are being misled a bit right now.

[lenalebt]

I'm that one person having trouble setting everything up :-).

I followed https://opentelemetry.io/blog/2024/prom-and-otel/ and basically ignored the rest of the docs and upgrade guides, whereever they may be. I am specifically living in scaleway k8s right now (https://www.scaleway.com/en/kubernetes-kapsule/), where they have a managed prometheus for me, but (for reasons beyond my understanding) do not export standard metrics from kube-state-metrics nor node-exporter automatically (and they are not running by default in their clusters). Me wanting to install them and leverage otel-collector to ingest that into that managed prometheus is basically the outline of my task.

My main troubles were roughly:

• I was unsure whether I should expect a specific "targetallocator" pod or not. For some reason, that one did not get started, but I did not have any errors in logs. Mentioning that a pod should come up would help. I found the defined service, and no endpoints, which made me quite sure a pod should exist.
• The targetallocator pod, after I had it running, did not want to start because I only installed PodMonitor and ServiceMonitor CRDs (as the blog post mentioned). Through the existence of TargetAllocator missing scrape configs from ServiceMonitor/PodMonitor when using v1beta1 spec vs v1alpha1 opentelemetry-operator#3528 I learned that support for the other CRDs has been added, so I added them (and patched the ClusterRole mentioned in the post).

Still on it bringing it to life, will post an update if I had more troubles that could get mentioned in some docs...

[lenalebt]

Apparently it's really important to add this:

  podMonitorSelector: {}
  serviceMonitorSelector: {}

to the config of the targetallocator, as it won't pick up the service monitors otherwise. Did default values change? I dunno. This is the complete configuration part I applied:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: opentelemetry
spec:
  mode: statefulset
  targetAllocator:
    enabled: true
    serviceAccount: opentelemetry-targetallocator-sa
    prometheusCR:
      enabled: true
      podMonitorSelector: {}
      serviceMonitorSelector: {}
  config:
  # ...

Hope that helps :-). It is working now.

[svrnm]

@jaronoff97 @lenalebt thanks for reporting this! We treat blog posts as pages that decay over time, and they are true at the point of writing. If the blog post serves as a point of reference we should consider moving the relevant part of the content to docs.

[lenalebt]

Totally unterstand this! Still, it is a quite recent post, so I somehow expected it to still be "quite right". But right, docs would probably be better 😊. And in the end, the blog post still was really helpful.

[svrnm]

What we can do is add a note to the blog post (as initially request by @jaronoff97) that this is the case and people should go to that new resource.

At the same time it would be great if the documentation for that would also be available through https://opentelemetry.io/docs/kubernetes/operator/