Support the Go compile-time instrumentation ecosystem

[amazingakai]

What ecosystem area?

Go compile-time instrumentation (otelc)

Key questions that will be answered once the resource is created.

• How should Go compile-time instrumentations be represented in the Ecosystem Explorer?
• How should the Ecosystem Explorer express which instrumentation applies to a given Go dependency?
• How should otelc integrate with the Ecosystem Explorer to consume this metadata?
• How can third-party Go compile-time instrumentations participate in the Ecosystem Explorer?

What should be created?

Research and define a representation for Go compile-time instrumentations in the Ecosystem Explorer.

The resulting resource should allow otelc to discover compatible instrumentations without maintaining an embedded registry, while enabling third-party instrumentation authors to publish instrumentations that otelc can consume automatically.

As future work, this could also be extended to support declarative configuration, similar to the existing Java agent integration with the Ecosystem Explorer.

References (issues, articles, PRs, projects, etc.)

• Previous Registry proposal: feat: Support optional otelc metadata block and generate otelc.json registry endpoint opentelemetry.io#11060
• Registry RFC: https://docs.google.com/document/d/1pXCHgbSNQnk23W26xG3xPtfrtLNFy-NBI_zZXukoYOA/edit?usp=sharing
• Declarative configuration tracking: LFX Term 3: Per-instrumentation configuration: let users control how individual instrumentations behave opentelemetry-go-compile-instrumentation#705

[amazingakai]

For some additional context, otelc already supports using instrumentations distributed as Go modules by declaring them in an otel.instrumentation.go file (see external-configuration).

The idea is for otelc to inspect an application's Go module dependencies, consult the Ecosystem Explorer to discover compatible instrumentations, and generate the otel.instrumentation.go file automatically.

This would decouple instrumentation discovery from the otelc binary, allowing support for new and third-party instrumentations without requiring a new otelc release.

[mikeblum]

This looks really interesting @amazingakai! I saw the press release a few weeks ago and added that to the long-running golang research tracker here. There's some in-flight work to add weaver and other sorts of tooling to get at what (from a first pass) otelcappears to do a better job of what I was doing with "exemplar" apps (self-contained mini apps that would use say otelhttp and we exercise those endpoints and return that as a weaver report).

golang-watcher WIP: #665

My next phase was going to get the instrumentation.yaml generated from this work here: mikeblum@b6b5acf#diff-bd55b31b538804c7c6e88d02f00120b7cae0bc004e73c3ff2362df038eb923a8

tl;dr i think I'll put a pin in ^ and lets see if we can reformulate the instrumentation.yaml ETL process to align with the otelc tools which I think are a better implementation of the exemplar apps like I linked ^.

That you already have the weaver schema files is a fantastic starting point!

[amazingakai]

Thanks! That makes a lot of sense.

One thing I've been thinking about is that the ETL story becomes much simpler if compile-time instrumentations live in a single contrib repository and we require Weaver schemas for all of them.

The watcher could then infer metadata in much the same way it does today for opentelemetry-go-contrib. Our layout is a little different (we don't have otel<instrumentation> subdirectories like otelhttp), but otelc already performs similar inference internally.

I'm also thinking through how third-party repositories should fit into this. That's still an open question for me, so I'll discuss it with my mentors and the Go compile instrumentation SIG to get their thoughts.

[jaydeluca]

third party repositories might be a bit trickier. I think it's worth thinking it through, but I'd recommend we start with a scope of just the official otelc repo and instrumentation as a starting point. We have a similar question for the prometheus ecosystem, where most of the components are third party.

[txabman42]

I see the support of 3rd party repos as a core question.

How should otelc provide visibility for them? Right now OTeL registry afford it. We can start providing just OTeL supported instrumentation via explorers, however, I would like to have defined if 3rd party ones would be supported in the future. That would help designing next steps for otelc instrumentation discoverability design.

xy problem: most of our otelc instrumentation will be 3rd party, mainly provided by Alibaba (loongsuite-go instrumentation) (alibaba/loongsuite-go#708). We won't have enough capacity to maintain all of them, Alibaba will do it for now.

[Satya900]

Reading through this, one reframe that might help the jaydeluca/txabman42 disagreement: the choice isn't really "official repo only" vs "open to all third parties." Collector-watcher already pulls from two upstream repos (core and contrib) in one pipeline, so scoping v1 to the official otelc repo doesn't require hardcoding the watcher to a single source forever. It could just mean starting with an allow-list of one entry, then adding loongsuite-go as a second entry once the ETL shape proves out, no open self-service registration needed on day one. That keeps jaydeluca's narrow starting scope while leaving the door open for what txabman42 is worried about.

Also curious about the weaver schema files mikeblum mentioned, are those already structured per-instrumentation in a way a watcher could parse today, or still tied to the exemplar-app setup he described?

@jaydeluca @txabman42 @mikeblum

[jaydeluca]

just to be clear, i'm not against including third-party packages, its just a more difficult problem to solve, so I didn't think it would be a good idea to start there. From my experience with the java instrumentation, we evolved our metadata schema considerably over time. It was easy enough for us because the source is all in one place, but if for example, theres information scattered across 20 repos and we need them all to update, that presents some challenges. If the otelc ecosystem has a few small project like the alibaba one where things are similarly grouped, and those maintainers are willing to accept contributions to enable this and be flexible as the data evolves, then thats awesome and maybe not as big of a challenge as I thought. @Satya900 is right that we can always start small with a list of projects and expand as things mature.

The thing we probably need to start with, is a standardized, versioned metadata schema that can be followed across any project, that provides the information we need for it to be useful. The weaver files are ideal starting points if they can be used for this, and what I'd highly recommend.

On top of the things weaver supports like telemetry, we also needed to come up with a whole other set of vocabulary and information for our metadata model that is very specific to the java instrumentation. Unfortunately it doesn't use the weaver format exactly, but long term the hope is to unify on that format.

The collector similarly has their own metadata schema specific to their components.

So thats where I'd start... defining a metadata schema that encompasses information that you'd like to have for each component in this ecosystem. Then we need to start populating those files, and then we can scrape them and start building a registry

[om7057]

@mikeblum @jaydeluca @amazingakai

The weaver schema is already organized on a per-instrumentation basis. Under schemas/otelc/groups/*.yaml, each instrumented package has its own manifest with ref: and id: as top-level fields. From what I could tell, this doesn't seem to be coupled to the exemplar app, it already looks like a reasonable manifest format for tooling to consume.

I'm not sure if that fully answers what a metadata schema is, but it does provide the two key pieces of information a watcher would need:

1. the Go package being instrumented (ref:), and
2. the canonical instrumentation identifier (id:).

That seems sufficient to map a Go module dependency to its corresponding instrumentation. If there are other metadata requirements beyond that, I can definitely take a look at it more if you feel the direction is right.

[mikeblum]

There's this work in-flight to derive weaver reports for go-contrib http module specifically: open-telemetry/opentelemetry-go-contrib#8066 (comment)

The trick here being how to fully exercise each attribute a/o metric. I'm wondering if we can re-use the otelc weaver specs across go core and go-contrib or would these be separate reports and implementations? Naively I was thinking the otelc weaver would supersede what I've been trying to derive.

[om7057]

There's this work in-flight to derive weaver reports for go-contrib http module specifically: open-telemetry/opentelemetry-go-contrib#8066 (comment)

The trick here being how to fully exercise each attribute a/o metric. I'm wondering if we can re-use the otelc weaver specs across go core and go-contrib or would these be separate reports and implementations? Naively I was thinking the otelc weaver would supersede what I've been trying to derive.

Looked at otelc's http.yaml schema and open-telemetry/opentelemetry-go-contrib#8066 to compare, they're solving different problems, not one superseding the other.

otelc's schemas/otelc/groups/*.yaml is a static declared contract: each file lists the metrics/attributes an instrumentation is supposed to emit, referencing the upstream semconv registry via ref:. It's validated with weaver registry check (make lint-schema), that's schema-level conformance, checking the declaration against the pinned semconv version. It doesn't run the instrumentation or touch real OTLP data.

#8066 is proposing live-check: actually running the instrumentation, capturing what it emits over OTLP at runtime, and using weaver's live-check tool to validate the real output against semconv. That catches drift between what's declared and what the code actually does, something the static schema check can't catch on its own.

So I don't think otelc's weaver specs supersede what you're building, they're the "should emit" side, and live-check is the "actually emits" side. The two could share the same target registry though: if otelc's http.yaml and go-contrib's http instrumentation emit the same metric names/attributes (both should, if both target http.client.request.duration etc. from upstream semconv), the same registry file could plausibly serve as the reference both a schema check and a live-check validate against. Would need someone to verify the actual attribute sets line up between the two implementations, but that seems like the direction worth checking rather than assuming they diverge.

[Satya900]

Wanted to bring something concrete instead of adding more discussion, so I went through the merged #665 implementation (ecosystem-automation/golang-instrumentation-watcher/) line by line instead of relying on descriptions.

What looks reusable as-is

After reading the implementation, these pieces appear to be generic enough to reuse directly:

• repo.CheckoutAt (switching repositories is essentially a one-line configuration change)
• instrumentation.Walk (doesn't make any go-contrib-specific assumptions)
• inventory.Manager (version tracking, read/write, idempotency—would just need a different registry path)

What doesn't carry over

The main piece that appears specific to go-contrib is DeriveMetadata.

It currently hardcodes the repository layout by assuming directories such as:

• bridges/
• exporters/
• propagators/
• samplers/

and derives the target library from an instrumentation/<target>/<wrapper> path structure.

That doesn't map cleanly onto otelc, which has a different repository organization and doesn't use wrapper subdirectories.

One pleasant surprise

The existing watcher schema is already closer to what this issue needs than I initially expected.

• Attribute already supports id and ref YAML fields, matching Weaver's conventions.
• Metadata already contains:
  • Module
  • TargetModule
  • Configurations
  • Stability

Those fields already cover much of the identity, applicability, configuration, and governance metadata discussed earlier in this thread. While it wasn't designed specifically for otelc, it provides a solid foundation instead of starting from scratch.

Rough implementation phases

1. Scope + schema agreement (blocks everything else)

   • Confirm the "official repositories first" scope.
   • Decide whether to extend the existing Metadata/Attribute schema or introduce an otelc-specific schema, given how much already aligns.

2. Metadata schema

   • Extend Metadata with a target version range (today Module.Version represents a single resolved version, whereas otelc's rule.yaml uses target + version ranges).
   • Bridge telemetry metadata from the Weaver files (schemas/otelc/groups/*.yaml) into the existing Attribute / Span / Metric model.

3. Watcher

   • Reuse CheckoutAt, Walk, and inventory.Manager as-is.
   • Implement an otelc-specific replacement for DeriveMetadata.

4. Registry

   • Publish under a dedicated path (e.g. ecosystem-registry/go/compile).
   • Keep it separate from go/contrib, since they represent different instrumentation models.

5. otelc consumption

   • Generate a lean, versioned discovery manifest that otelc can fetch at build time.
   • Keep this separate from the richer per-instrumentation JSON used by the website.
   • Handle pinning/integrity appropriately since this directly influences what gets compiled into user binaries.

6. Database + frontend

   • Extend explorer-db-builder.
   • Add UI support for browsing compile-time instrumentations.

Happy to start on Phase 2 (schema extension) if the scope discussion lands where the thread seems to be heading.

@mikeblum @jaydeluca @txabman42 @amazingakai @

[Satya900]

Related work worth coordinating

While looking into this, I noticed a connection that may be worth coordinating with: opentelemetry-go-compile-instrumentation#705.

Although #705 focuses on per-instrumentation declarative configuration rather than instrumentation discovery, both efforts appear to need a very similar underlying manifest describing each instrumentation and its configurable options.

A maintainer in that discussion drew a useful distinction between the two layers:

• Instrumentation discovery/selection (this issue)
• Instrumentation behavior configuration (feat(i18n): translate hardcoded strings in java-agent config-field renderers #705)

Those seem like separate concerns and should probably remain independent. That said, both could potentially build on the same underlying metadata model.

Issue #705 references the Java agent's metadata.yaml approach as the model for declarative configuration, and this issue already mentions declarative configuration as possible future work, similar to the existing Java agent integration.

After reading the merged #665 implementation, the existing Configuration { Name, Description, Type, Default, Examples } struct in golang-instrumentation-watcher/metadata.go already looks like a reasonable first step toward that shared manifest. The Java agent's instrumentation.yaml extends a similar structure with fields such as declarative_name and declarative_schema to bridge into OpenTelemetry's declarative configuration specification, so supporting that later would seem like an incremental extension rather than an entirely new design.

@txabman42, since you're involved in both discussions, it might be worth coordinating whatever schema comes out of this issue's Phase 2 with #705. Even if the discovery and configuration layers stay separate, having them share a common metadata foundation could help avoid two slightly different manifest formats evolving for essentially the same instrumentation metadata.

[txabman42]

@jaydeluca I'm aligned with you, I think that's achievable as first step. We should provide all the metadata you need via Weaver annotations. I don't see any limitations right now. My idea is to use weaver files directly under otelc. WDYT @amazingakai?

We can start working with otelc repo ones and add Alibaba ones also, starting a repo list could work. WDYT @NameHaibinZhang, it should be aligned with our weaver alignment ideas. Any limitations from Alibaba side?

It could be interesting to have another issue under https://github.com/open-telemetry/opentelemetry-ecosystem-explorer to discuss the mid-long term vision of the 3rd party support.

@mikeblum TY for that context, its really valuable. We are planning to do sth similar, so we'll get "inspired" by that approach. Right now we only have the validation step, but the idea is to also add emit check, sth like:

• Emit    --> generated *conv types
• Declare --> only the attributes we invent (weaver registry check)
• Verify  --> real emitted telemetry (weaver registry live-check)

btw @mikeblum, we are planning to open discussions with go-contrib sig to try to add otelc instrumentation directly there. But that's a story for another day.