feat: capture system prompt, tools and reasoning in OTel; fail loud on setup

TELEMETRY_ACCESS.md

@@ -115,6 +115,46 @@ If Cloud Logging is unavailable, events are written to stderr as JSON:
 
 ### Status: ✅ Confirmed working — both Cloud Logging and Cloud Trace export verified end-to-end
 
+#### Full LLM capture (added 2026-08-09, branch `feat/otel-full-llm-capture`)
+
+What is now recorded for every LLM call, on top of the prompt/response content
+that already worked:
+
+- **System prompt and tool definitions.** `setup_otel()` opts into the
+  experimental GenAI semantic conventions
+  (`OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental`). Only that path
+  emits `gen_ai.system_instructions` and `gen_ai.tool_definitions`; the stable
+  path records what the model said but never what it was told or what tools it
+  had. Set the variable to `stable` to opt back out.
+- **Thinking / reasoning.** Both agents now pass
+  `thinking_config(include_thoughts=True)` (`alphoryn/agents/thinking.py`).
+  Gemini reasons either way, but only returns the summary when asked, so before
+  this there was literally nothing for OTel to capture.
+
+  ⚠️ Thought summaries arrive as extra text parts **before** the answer, flagged
+  `thought=True`. Any code reading `parts[0].text` or the first non-empty text
+  part will parse a thought as the answer. Both agents route through
+  `is_thought_part()`; any new response reader must too.
+
+#### Setup now fails loudly (exit code 4)
+
+`setup_otel()` used to catch every exception, log a warning, and let the run
+continue with no exporters at all. Three ways that lost a whole run silently:
+`google.auth.default()` failing; credentials resolving with no project ID (ADK
+returns empty hooks here rather than raising, so nothing was ever thrown); and
+a crash losing whatever was still buffered in the `BatchSpanProcessor`.
+
+It now raises `TelemetrySetupError`, the CLI reports it and exits **4**, and
+`flush_otel()` is registered with `atexit`. `alphoryn run` also prints
+`Telemetry -> GCP project '<id>'` at startup — telemetry landing in the wrong
+project (e.g. `wortcast`, gcloud's default on this box) looks identical to
+telemetry landing nowhere.
+
+This does not contradict constitution Principle IV. Principle IV governs
+per-event emission at run time, and `TelemetryLogger.emit` still falls back to
+stderr and never blocks. Setup is a preflight check, the same class of thing as
+config validation, which already exits 1.
+
 `alphoryn/telemetry/otel.py:setup_otel()` calls `get_gcp_exporters(enable_cloud_tracing=True,
 enable_cloud_logging=True)` unconditionally at every CLI startup (`cli/main.py`). Getting here
 required fixing three stacked gaps:

alphoryn/agents/feedback_agent.py

@@ -20,6 +20,7 @@
 from google.genai import types as genai_types
 
 from alphoryn.agents.prompts import FEEDBACK_AGENT_SYSTEM_PROMPT
+from alphoryn.agents.thinking import is_thought_part, thinking_enabled_config
 from alphoryn.market_data.client import MarketDataClient
 from alphoryn.memory.bank import MemoryBank
 from alphoryn.memory.schema import FeedbackEvaluation
@@ -74,6 +75,7 @@ def __init__(
             name="alphoryn_feedback_agent",
             model=_FEEDBACK_AGENT_MODEL,
             instruction=FEEDBACK_AGENT_SYSTEM_PROMPT,
+            generate_content_config=thinking_enabled_config(),
         )
 
     # ------------------------------------------------------------------
@@ -194,7 +196,11 @@ def _call_agent(
             ),
         ):
             if event.is_final_response() and event.content and event.content.parts:
-                raw_json = event.content.parts[0].text
+                for part in event.content.parts:
+                    if is_thought_part(part):
+                        continue
+                    raw_json = part.text
+                    break
 
         if raw_json is None:
             _logger.error("feedback_agent produced no final response (attempt %d)", attempt)

alphoryn/agents/main_agent.py

@@ -18,6 +18,7 @@
 from google.genai import types as genai_types
 
 from alphoryn.agents.prompts import MAIN_AGENT_SYSTEM_PROMPT
+from alphoryn.agents.thinking import is_thought_part, thinking_enabled_config
 from alphoryn.execution.agent import AssetDecision, SessionDecision
 from alphoryn.market_data.client import MarketDataClient
 from alphoryn.telemetry.logger import TelemetryLogger
@@ -59,6 +60,7 @@ def __init__(
             model=self._MODEL,
             instruction=MAIN_AGENT_SYSTEM_PROMPT,
             tools=[market_data_client.build_snapshot, SkillToolset(skills)],
+            generate_content_config=thinking_enabled_config(),
         )
 
     def decide(
@@ -108,6 +110,8 @@ def decide(
                     )
             if event.is_final_response() and event.content and event.content.parts:
                 for part in event.content.parts:
+                    if is_thought_part(part):
+                        continue
                     text = getattr(part, "text", None)
                     if text and text.strip():
                         raw_json = _strip_fences(text.strip())

alphoryn/agents/thinking.py

@@ -0,0 +1,36 @@
+"""Gemini thinking configuration, and how to read a response that has thoughts.
+
+Gemini 2.5 models reason internally whether or not you ask, but the thought
+summary is only returned when include_thoughts is set. Without it there is
+nothing for OpenTelemetry to capture, so a trace shows what the agent decided
+and never why.
+
+Turning it on changes the shape of every response: the model's reasoning
+arrives as extra text parts in the same content, flagged with thought=True and
+placed *before* the answer. Any caller that reads parts[0].text, or the first
+non-empty text part, would start parsing a thought summary as if it were the
+answer. is_thought_part exists so every such caller skips them.
+"""
+
+from typing import Any
+
+from google.genai import types as genai_types
+
+
+def thinking_enabled_config() -> genai_types.GenerateContentConfig:
+    """Return a GenerateContentConfig that asks Gemini for its thought summary."""
+    return genai_types.GenerateContentConfig(
+        thinking_config=genai_types.ThinkingConfig(include_thoughts=True),
+    )
+
+
+def is_thought_part(part: Any) -> bool:
+    """True if this response part is reasoning rather than the model's answer.
+
+    The comparison is `is True`, not a truthiness test. Gemini sets thought
+    exactly True on a summary part and leaves it None everywhere else, so
+    nothing is lost by being strict - and being strict is what keeps a test
+    double or an attribute-generating stub from reporting every part as a
+    thought and silently swallowing the model's actual answer.
+    """
+    return getattr(part, "thought", None) is True

alphoryn/cli/main.py

@@ -34,7 +34,7 @@
 from alphoryn.scheduler.scheduler import Scheduler
 from alphoryn.secrets.client import SecretsError, load_alpaca_credentials
 from alphoryn.telemetry.logger import TelemetryLogger
-from alphoryn.telemetry.otel import setup_otel
+from alphoryn.telemetry.otel import TelemetrySetupError, setup_otel
 
 _VERSION = "0.0.1"
 
@@ -75,7 +75,6 @@ def run(
     """Start a paper trading session."""
     os.environ.setdefault("GOOGLE_GENAI_USE_VERTEXAI", "1")
     os.environ.setdefault("GOOGLE_CLOUD_LOCATION", "us-central1")
-    setup_otel()
 
     # 1. Load and validate config (exit 1 on failure)
     overrides: dict = {}
@@ -100,14 +99,26 @@ def run(
         typer.echo(f"Config error: {exc}", err=True)
         sys.exit(1)
 
-    # 2. Fetch secrets from GCP Secret Manager (exit 3 on failure)
+    # 2. Telemetry preflight (exit 4 on failure). An untraced run leaves no
+    # record of why it traded the way it did, so this blocks the run - but it
+    # comes after config validation, which is local, cheap and deterministic.
+    # Reporting a credentials problem for a config file with a typo in it
+    # sends you to the wrong place entirely.
+    try:
+        otel_project = setup_otel()
+    except TelemetrySetupError as exc:
+        typer.echo(f"Telemetry error: {exc}", err=True)
+        sys.exit(4)
+    typer.echo(f"Telemetry -> GCP project '{otel_project}'")
+
+    # 3. Fetch secrets from GCP Secret Manager (exit 3 on failure)
     try:
         load_alpaca_credentials()
     except SecretsError as exc:
         typer.echo(f"Secret Manager error: {exc}", err=True)
         sys.exit(3)
 
-    # 3. Load memory bank (exit 2 on failure)
+    # 4. Load memory bank (exit 2 on failure)
     db_path = str(Path(cfg.memory_db_path).expanduser())
     try:
         bank = MemoryBank(db_path)
@@ -116,10 +127,10 @@ def run(
         typer.echo(f"Memory bank error: {exc}", err=True)
         sys.exit(2)
 
-    # 4. Warn if run_duration is not evenly divisible by candle_timeframe
+    # 5. Warn if run_duration is not evenly divisible by candle_timeframe
     _warn_fractional_sessions(cfg)
 
-    # 5. Print startup banner
+    # 6. Print startup banner
     typer.echo(f"Alphoryn v{_VERSION} — Paper Trading")
     typer.echo(
         f"Tickers: {', '.join(cfg.tickers)}"
@@ -132,7 +143,7 @@ def run(
         f" — {len(open_positions)} open position{'s' if len(open_positions) != 1 else ''} loaded"
     )
 
-    # 6. Delegate to scheduler
+    # 7. Delegate to scheduler
     _start_scheduler(cfg, bank)
 
 

alphoryn/telemetry/otel.py

@@ -3,21 +3,45 @@
 Call setup_otel() once at CLI startup, before any agent is initialized.
 Traces and spans then flow automatically to Cloud Trace and Cloud Logging.
 
-To capture full prompt/response content in traces, set:
-  OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true
+Setup is a *preflight check*, not a logging call: it raises TelemetrySetupError
+rather than warning and continuing. Constitution Principle IV ("a logging
+failure never blocks execution") governs per-event emission at run time - see
+TelemetryLogger.emit, which still falls back to stderr. Discovering at startup
+that nothing will be recorded at all is the same class of problem as an invalid
+config, and the CLI treats it the same way: report it and exit 1, rather than
+trade blind for hours and leave no record of why.
+
+Full LLM capture (system prompt, tool definitions, input/output messages) needs
+the experimental GenAI semantic conventions, which setup_otel() opts into by
+default. Override either of these to change what is captured:
+  OTEL_SEMCONV_STABILITY_OPT_IN=stable
+  OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=false
 """
 
+import atexit
 import logging
 import os
 
 import google.auth
 from google.adk.telemetry.google_cloud import get_gcp_exporters as _get_gcp_exporters
 from google.adk.telemetry.setup import maybe_set_otel_providers as _maybe_set_otel_providers
+from opentelemetry import trace
+from opentelemetry.sdk.trace import TracerProvider
 
 _logger = logging.getLogger(__name__)
 
 _SERVICE_NAME = "alphoryn"
 
+# Opt-in token for the experimental GenAI semantic conventions. Only this path
+# records gen_ai.system_instructions and gen_ai.tool_definitions; the stable
+# path emits user/system/choice log events but never the tool definitions the
+# model was given, so a trace cannot tell you what the agent could have done.
+_GENAI_EXPERIMENTAL_OPT_IN = "gen_ai_latest_experimental"
+
+
+class TelemetrySetupError(RuntimeError):
+    """Raised when OpenTelemetry cannot be wired up to export anything."""
+
 
 def _add_gcp_project_resource_attribute(project_id: str) -> None:
     """Merge gcp.project_id into OTEL_RESOURCE_ATTRIBUTES if not already set.
@@ -35,25 +59,100 @@ def _add_gcp_project_resource_attribute(project_id: str) -> None:
     os.environ["OTEL_RESOURCE_ATTRIBUTES"] = f"{existing},{attr}" if existing else attr
 
 
-def setup_otel() -> None:
-    """Configure OTel providers with GCP exporters.
+def _enable_experimental_genai_semconv() -> None:
+    """Add the experimental GenAI opt-in to OTEL_SEMCONV_STABILITY_OPT_IN.
+
+    The variable is a CSV of opt-in tokens shared with other instrumentations,
+    so an existing value is appended to rather than replaced.
+    """
+    existing = os.environ.get("OTEL_SEMCONV_STABILITY_OPT_IN", "")
+    tokens = [t.strip() for t in existing.split(",") if t.strip()]
+    if _GENAI_EXPERIMENTAL_OPT_IN in tokens or "stable" in tokens:
+        return
+    tokens.append(_GENAI_EXPERIMENTAL_OPT_IN)
+    os.environ["OTEL_SEMCONV_STABILITY_OPT_IN"] = ",".join(tokens)
+
 
-    Sets OTEL_SERVICE_NAME and wires up Cloud Trace + Cloud Logging exporters
-    via the ADK helper. Fails silently — a logging failure never blocks startup.
+def _verify_tracer_provider_installed() -> None:
+    """Fail if no real SDK TracerProvider ended up installed.
 
-    GenAI content capture is enabled by default so prompt/response reasoning
-    is visible in Cloud Logging. Override with:
-      OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=false
+    get_gcp_exporters() returns an empty OTelHooks (no exception) when it
+    cannot determine the GCP project, and maybe_set_otel_providers() only
+    installs a TracerProvider when at least one span processor exists. Without
+    this check that path completes silently and every span is dropped for the
+    whole process. The global default is a ProxyTracerProvider, so an isinstance
+    check against the SDK type is what distinguishes "wired up" from "no-op".
+    """
+    provider = trace.get_tracer_provider()
+    if not isinstance(provider, TracerProvider):
+        raise TelemetrySetupError(
+            "no OpenTelemetry TracerProvider was installed, so no spans would be "
+            "exported. This usually means the GCP project could not be resolved "
+            "from your credentials - check `gcloud auth application-default login` "
+            "and GOOGLE_CLOUD_PROJECT."
+        )
+
+
+def flush_otel() -> None:
+    """Force-export any spans still buffered in the BatchSpanProcessor.
+
+    Spans are batched in memory and shipped every few seconds. The SDK flushes
+    on a clean interpreter exit, but a crash or a hard kill loses whatever is
+    still buffered - which is exactly the tail of the run you most want to read
+    afterwards. Never raises: this runs on the way out, and a flush failure must
+    not mask the error that is already ending the run.
+    """
+    provider = trace.get_tracer_provider()
+    if not isinstance(provider, TracerProvider):
+        return
+    try:
+        provider.force_flush()
+    except Exception as exc:
+        _logger.warning("OpenTelemetry flush failed, buffered spans may be lost: %s", exc)
+
+
+def setup_otel() -> str:
+    """Configure OTel providers with GCP exporters and return the GCP project ID.
+
+    Sets OTEL_SERVICE_NAME, opts into the experimental GenAI semantic
+    conventions and full content capture, then wires up the Cloud Trace +
+    Cloud Logging exporters via the ADK helper and registers an exit flush.
+
+    Returns:
+        The GCP project ID traces will be written to. The caller is expected to
+        show this to the user - sending a run's telemetry to the wrong project
+        looks identical to sending none at all.
+
+    Raises:
+        TelemetrySetupError: if nothing would be exported. Callers should treat
+            this like a config error and exit rather than run untraced.
     """
     os.environ.setdefault("OTEL_SERVICE_NAME", _SERVICE_NAME)
     os.environ.setdefault("OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT", "true")
+    _enable_experimental_genai_semconv()
+
     try:
         _, project_id = google.auth.default()
-        if project_id:
-            _add_gcp_project_resource_attribute(project_id)
-        gcp_exporters = _get_gcp_exporters(
-            enable_cloud_tracing=True, enable_cloud_logging=True
+    except Exception as exc:
+        raise TelemetrySetupError(
+            f"could not resolve Google credentials: {exc}. "
+            "Run `gcloud auth application-default login`."
+        ) from exc
+
+    if not project_id:
+        raise TelemetrySetupError(
+            "Google credentials resolved but carry no project ID. "
+            "Set GOOGLE_CLOUD_PROJECT (this project expects `alphoryn`)."
         )
+
+    _add_gcp_project_resource_attribute(project_id)
+
+    try:
+        gcp_exporters = _get_gcp_exporters(enable_cloud_tracing=True, enable_cloud_logging=True)
         _maybe_set_otel_providers([gcp_exporters])
     except Exception as exc:
-        _logger.warning("OpenTelemetry setup failed, traces will not be exported: %s", exc)
+        raise TelemetrySetupError(f"could not build the GCP OTel exporters: {exc}") from exc
+
+    _verify_tracer_provider_installed()
+    atexit.register(flush_otel)
+    return project_id

specs/001-etf-paper-trading-agent/contracts/cli.md

@@ -57,6 +57,7 @@ not one line per ticker):
 | 1 | Config validation error |
 | 2 | Memory bank inaccessible or corrupt (hard abort) |
 | 3 | Google Secret Manager unreachable at startup |
+| 4 | OpenTelemetry could not be wired up - the run would not be traced |
 
 ---