Add CPython 3.14 support to mixed-mode debugging - #8586
Merged
Rich Chiodo (rchiodo) merged 6 commits intoJul 22, 2026
Conversation
CPython 3.13+ publishes a self-describing _Py_DebugOffsets table as the first field of the exported _PyRuntime global (PEP 768; Include/internal/ pycore_debug_offsets.h). It lets out-of-process debuggers locate interpreter field offsets (frames, code objects, thread state, builtin object layouts) without relying on the interpreter's PDB. This lays the groundwork for using it in 3.14 mixed-mode debugging: - PyDebugOffsets: a pure, testable parser that validates the "xdebugpy" cookie, reads PY_VERSION_HEX + the free-threaded flag, and exposes every offset in the 3.14 layout. It also locates _PyRuntime from the module export table (no PDB needed via TryGetExportedStaticVariableAddress) and reads the table out of the debuggee. - PythonRuntimeInfo.DebugOffsets: lazily reads and caches the table per process. - Tests lock the parser down against bytes recorded from real 3.14.6 standard and free-threaded (python3.14t) interpreters; every expected offset was verified in-process against live objects. The free-threaded vector documents how object layouts shift (e.g. code_object.linetable and pyobject.ob_type move), which is exactly why reading the table beats hard-coded offsets. No behavior change yet: nothing consumes the offsets. Wiring the frame/ code-object paths to prefer them (falling back to PDB) is a follow-up. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Wires the 3.14 self-describing _Py_DebugOffsets table into Concord's struct offset resolution for the mixed-mode hot path (frame / code object / thread state), falling back to the PDB for everything else. This makes the debugger track the free-threaded build's shifted object layouts automatically and sources authoritative offsets even when the interpreter PDB's field info is unreliable, while still requiring a PDB for struct sizes. - Add IStructFieldOffsetProvider + DebugOffsetsFieldProvider mapping the curated hot-path fields to _Py_DebugOffsets (group, field) entries. - PythonRuntimeInfo.StructFieldOffsetProvider is non-null only for 3.14, so older interpreters resolve every field from the PDB exactly as before. - StructProxy.GetStructFields consults the provider before the PDB. - Handle 3.14's _PyStackRef-typed f_executable: its low bits are a tag (Py_TAG_BITS), set for deferred/immortal references such as frozen-module code objects. PointerProxy gains an optional tag mask; PyInterpreterFrame strips it on 3.14, and the native trace helper strips it before reading co_filename for breakpoint matching. Mirrors CPython's own CLEAR_PTR_TAG. - Add provider unit tests against the recorded standard + free-threaded 3.14.6 vectors (17/17 DebuggerTests pass). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ag bits In CPython 3.14 the frame's localsplus array holds _PyStackRef values rather than plain PyObject*. Their low bits carry a deferred/immortal reference tag (Py_TAG_BITS), and PyStackRef_NULL is itself Py_TAG_DEFERRED (bits == 1), so reading a slot as a raw pointer yields a misaligned/nonzero address. That made the Locals window enumeration dereference address 0x1 (for unassigned locals) or ptr|1 (for None/bools/small ints/interned strings), throwing and aborting the entire enumeration -> no variables shown at all. Strip the tag bits from each localsplus slot in GetFrameLocals (mirroring CPython's PyStackRef_AsPyObjectBorrow) for 3.14+. Object pointers are 8-byte aligned so the mask is a no-op for untagged slots, and it is 0 for < 3.14, leaving earlier versions byte-for-byte unchanged. This also fixes simple-name Watch/hover, which reuses the locals enumeration. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Wrap each per-variable read in GetFrameLocals so a single unreadable local (e.g. an unexpected _PyStackRef slot value) surfaces as a failed evaluation entry instead of throwing and blanking out the entire Locals window. This also gives a concrete diagnostic (the exception message per variable) when a slot cannot be resolved, rather than the window silently showing nothing. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
|
🔒 Automated review in progress — Heejae Chang (@heejaechang) is auto-reviewing this PR. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds CPython 3.14 support to mixed-mode (Python/Native) debugging by sourcing key struct field offsets from CPython’s self-describing _Py_DebugOffsets table and handling _PyStackRef tagged pointers so frame/code/local reads remain correct (including free-threaded 3.14t layout shifts).
Changes:
- Introduces a managed
_Py_DebugOffsetsreader and a hot-path offset provider soStructProxycan prefer interpreter-reported offsets on 3.14. - Adds
_PyStackReftag masking support inPointerProxy<T>and applies it when readingf_executable/localsplus/f_code. - Adds unit tests that lock down parsing and provider mappings against recorded CPython 3.14.6 (standard + free-threaded) vectors.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| Python/Tests/DebuggerTests/PyDebugOffsetsTests.cs | New tests for parsing/validating the _Py_DebugOffsets table from recorded vectors. |
| Python/Tests/DebuggerTests/PyDebugOffsetsProviderTests.cs | New tests for DebugOffsetsFieldProvider hot-path mappings and fall-through behavior. |
| Python/Tests/DebuggerTests/DebuggerTests.csproj | Includes the new unit test files in the test project. |
| Python/Product/DebuggerHelper/trace.cpp | Strips _PyStackRef tag bits before dereferencing f_code in native trace helper. |
| Python/Product/Debugger.Concord/PythonRuntimeInfo.cs | Adds lazy probing/caching for DebugOffsets and a 3.14-only offset provider. |
| Python/Product/Debugger.Concord/Proxies/Structs/PyInterpreterFrame.cs | Masks f_executable tag bits on 3.14+ when exposing f_code. |
| Python/Product/Debugger.Concord/Proxies/Structs/PyDebugOffsets.cs | New managed reader that locates _PyRuntime via exports and parses the 3.14 table layout. |
| Python/Product/Debugger.Concord/Proxies/Structs/DebugOffsetsFieldProvider.cs | New provider mapping a curated set of hot-path struct fields to table entries. |
| Python/Product/Debugger.Concord/Proxies/StructProxy.cs | Prefers the offset provider before falling back to PDB field offset lookup. |
| Python/Product/Debugger.Concord/Proxies/PointerProxy.cs | Adds optional tag masking to pointer reads (WithTagMask) used for _PyStackRef. |
| Python/Product/Debugger.Concord/ExpressionEvaluator.cs | Masks localsplus slot pointers on 3.14+ and makes locals enumeration resilient per-slot. |
| Python/Product/Debugger.Concord/DkmExtensions.cs | Adds a non-throwing export probe helper (TryGetExportedStaticVariableAddress). |
| Python/Product/Debugger.Concord/Debugger.Concord.csproj | Includes the new DebugOffsets reader/provider sources in the product project. |
Heejae Chang (heejaechang)
approved these changes
Jul 22, 2026
Heejae Chang (heejaechang)
left a comment
Heejae Chang (heejaechang)
left a comment
There was a problem hiding this comment.
Approved via Review Center.
Address review nits: correct the version floor to 3.14 in the _Py_DebugOffsets doc comments (PythonRuntimeInfo.DebugOffsets and PyDebugOffsets.TryRead) and reword the f_executable _PyStackRef comment so it describes a two-low-bit tag with mask 0x3 rather than implying Py_TAG_BITS is a count of 3 bits. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Rich Chiodo (rchiodo)
enabled auto-merge (squash)
July 22, 2026 21:54
Contributor
Author
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Heejae Chang (heejaechang)
approved these changes
Jul 22, 2026
Heejae Chang (heejaechang)
left a comment
Heejae Chang (heejaechang)
left a comment
There was a problem hiding this comment.
Approved via Review Center.
This was referenced Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds CPython 3.14 support to the mixed-mode (Python/Native) debugger, building on #8582. It reads CPython 3.14's self-describing
_Py_DebugOffsetstable for the mixed-mode hot path, handles the new_PyStackReftagged-pointer representation of frame slots, and fixes empty Locals/Autos windows when stepping through 3.14 Python code.Background
CPython 3.14 changed two things that break the existing debugger:
_Py_DebugOffsets(Include/internal/pycore_debug_offsets.h) — a table of struct field offsets embedded as the first field of_PyRuntime, deliberately placed there so out-of-process debuggers can discover struct layouts without a PDB._PyRuntimeis exported, so its address is resolvable from the DLL export table with no symbols._PyStackRef— interpreter-frame slots (f_executable,localsplus, ...) are now tagged pointers, not plainPyObject*. Their low bits (Py_TAG_BITS) carry a deferred/immortal reference tag that must be masked off to recover the real pointer.What this does
Read offsets from the interpreter (3.14+):
PyDebugOffsets— a parser that locates_PyRuntimevia the export table, validates the"xdebugpy"cookie, and exposesOffset(group, field)from the debuggee's own table (works even whenpython314.pdbis unavailable, e.g. embedded/production apps).DebugOffsetsFieldProvider— maps a curated set of hot-path struct/field names to table entries:_PyInterpreterFrame,PyCodeObject, and_ts(thread state). Everything else falls back to the PDB.StructProxy.GetStructFieldsconsults the provider beforeSymbol.GetFieldOffset. The provider is null for every version before 3.14, so those paths are byte-for-byte identical to before.Handle
_PyStackReftagging (3.14+):PointerProxy<T>gains an optional tag mask (WithTagMask);Read/TryRead/IsNullstrip it. Default mask0= existing behavior everywhere.PyInterpreterFrame.f_codemasksf_executable, andtrace.cppstrips the tag before dereferencingf_code. Both are no-ops on already-aligned pointers, so older versions are unaffected.Fix empty Locals in 3.14 mixed-mode:
_PyInterpreterFrame.localsplus[]slots are_PyStackRefin 3.14. Reading them as rawPyObject*meant any local holding a deferred object (None, booleans, small ints, interned strings, ...) dereferenced a misaligned address and threw, aborting the entire locals enumeration and leaving the window blank.GetFrameLocalsnow masks each slot (Py_TAG_BITS) on 3.14+, mirroring CPython'sPyStackRef_AsPyObjectBorrow.Files
Proxies/Structs/PyDebugOffsets.cs—_Py_DebugOffsetsreader (locates_PyRuntime, validates cookie, parses offset table).Proxies/Structs/DebugOffsetsFieldProvider.cs— maps hot-path struct/field names to table entries.PythonRuntimeInfo.cs— lazyDebugOffsets+StructFieldOffsetProvider(non-null only on 3.14).Proxies/StructProxy.cs— prefer the offset provider, fall back to the PDB.Proxies/PointerProxy.cs— optional tag mask on pointer reads.Proxies/Structs/PyInterpreterFrame.cs— maskf_executableon 3.14.DebuggerHelper/trace.cpp— strip thef_codetag before dereference.ExpressionEvaluator.cs— masklocalsplusslots on 3.14 and make per-variable enumeration resilient.DkmExtensions.cs— non-throwingTryGetExportedStaticVariableAddress.Tests/DebuggerTests/PyDebugOffsetsTests.cs,PyDebugOffsetsProviderTests.cs— parser + provider tests against recorded 3.14.6 vectors (standard and free-threaded 3.14t).Testing
Examples/PythonNativeC++ app embedding CPython 3.14 in VS 18: call stack, line numbers, stepping between Python and native code, and the Locals/Autos windows (includingNoneand other immortal values) all work.Notes