feat: OpenTelemetry-compatible tracing subscriber for MoFA agents

[magic-peach]

Zero-Config OpenTelemetry Subscriber for MoFA Agents

Problem

Currently, instrumenting a MoFA agent requires manually constructing Span objects and configuring exporters. There is no turnkey solution where an agent author can add one line and have all operations appear in a monitoring dashboard.

Proposed Solution

A CognitiveObservatory::init() call that:

1. Registers a global tracing_subscriber layer
2. Intercepts all tracing::span! / #[instrument] events emitted by the agent
3. Serializes them into OpenTelemetry-compatible payloads
4. Forwards them over HTTP (OTLP/JSON) to a configurable endpoint

// Any MoFA agent adds just this:
use mofa_observatory::CognitiveObservatory;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    CognitiveObservatory::init("http://localhost:7070").await?;

    // From this point, every tracing::span in the process is
    // automatically captured and sent to the Observatory dashboard.
    run_agent().await
}

Why This Matters

• Zero friction adoption: no changes to existing agent code beyond the one-line init
• Works with existing #[instrument] macros: any function already using tracing is automatically captured
• Composable: can be layered on top of existing tracing-subscriber setups
• OpenTelemetry compatible: spans use the OTLP format, so they can also be forwarded to Jaeger, Grafana Tempo, or any OTLP backend

Implementation Notes

The subscriber layer would:

• Buffer spans in a tokio::sync::mpsc channel
• Flush in background via a Tokio task (similar to BatchSpanProcessor in the existing mofa-monitoring crate)
• Handle backpressure by dropping spans when the buffer is full and logging a warning

This feature is being prototyped in the mofa-observatory crate as part of the GSoC MVP — see #1318.

Acceptance Criteria

• CognitiveObservatory::init(endpoint) registers subscriber without panics
• Spans from #[instrument] functions appear in the Observatory trace list
• Works with RUST_LOG env var filtering
• Graceful degradation when the Observatory server is unreachable