Skip to content

Commit d4865f8

Browse files
committed
feat(execute): report raw worker tracebacks across Core and Execute
Transport bounded, default-on raw tracebacks through the generic worker v5 channel and the version-2 Core outcome while retaining typed failure reasons, exact publication evidence, and cleanup precedence. The selected backend supports raw_traceback=False for secret-bearing work; the owner explicitly accepted this narrow unredacted diagnostic risk. Cover setup/invocation failures, malformed and oversized diagnostics, opt-out, Ray unit transport, and real subprocess/Core regressions in the representative policy. The integrated local good-enough suite passed 3,442 tests with 30 skips; real Ray/GPU qualification remains opt-in.
1 parent da59be3 commit d4865f8

24 files changed

Lines changed: 527 additions & 102 deletions

‎docs/dispatch.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,16 @@ message includes bounded diagnostic categories and directs callers to
9191
Incomplete coverage means undiscovered calls may hide worker requirements,
9292
not that the workload failed or that no declarations were found.
9393

94+
Execute-backed Dispatch uses the backend's default `raw_traceback=True` policy
95+
for `SubProcessConfig` and `RayBackendConfig`. Generic worker failures retain a
96+
bounded **unredacted** Python traceback in `RemoteExecutionError.remote_traceback`
97+
and its displayed text. Core-owned invocation failures use the same policy on
98+
`CoreExecutionError.remote_traceback`. Set `raw_traceback=False` on the backend
99+
configuration to opt out and receive type-only errors. Raw output can expose
100+
private paths, source lines, exception values, and credentials in notebook output
101+
or logs. Failure categories and completed publication evidence remain intact.
102+
See [Execute](execute.md).
103+
94104
## Probing And Static Coverage
95105

96106
`ProbeOptions` is an immutable, inert policy with defaults

‎docs/execute.md‎

Lines changed: 24 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -115,7 +115,7 @@ encoding; an exit failure preserves an already encoded result or workload error,
115115
but leaves the Future cleanup state incomplete and `cleanup()` raises rather than
116116
claiming that unobserved worker teardown was reconciled.
117117

118-
The generic private worker protocol is version 4. After admission and `GO`,
118+
The generic private worker protocol is version 5. After admission and `GO`,
119119
setup-bearing calls send `SETUP` and wait for `SETUP_READY` before `PAYLOAD`.
120120
Bounded `OUTPUT` may arrive during setup, invocation, and teardown; terminal
121121
outcomes and final output fences complete the exchange. An incompatible worker fails before `GO`;
@@ -336,11 +336,16 @@ earlier restores nor replays the workload.
336336
Store-table-relative publication status only; it never carries a live Repo,
337337
Store, `StoreReport`, argument, or result. A delivered core-worker failure raises
338338
`CoreExecutionError` with `core execution worker failed: <reason>`, preserves
339-
only its bounded failure category and known publication evidence, and does not
340-
imply rollback, retry, or successful caller refresh. Because the core adapter
341-
returns an opaque outcome through generic Execute before coordinator recovery,
342-
this is not a generic `RemoteExecutionError`; submitted values, traceback text,
343-
and exception values are not transported.
339+
its bounded failure category and known publication evidence, and does not imply
340+
rollback, retry, or successful caller refresh. With the backend's default
341+
`raw_traceback=True`, the failed Core outcome also transports bounded, unredacted
342+
worker text in `CoreExecutionError.remote_traceback` and its displayed error.
343+
The private Core outcome codec is version 2; raw diagnostics are dropped before
344+
publication evidence or the primary failure category if the result budget is
345+
tight. With `raw_traceback=False`, Core errors remain type/category-only. No
346+
remote exception instance, submitted argument value, or Store resource is
347+
transported as part of this diagnostic. Core failures remain separate from the
348+
generic `RemoteExecutionError` used for worker setup/transport failures.
344349

345350
`PreparedCoreCall` contains only invocation bytes, frozen storage setup, and
346351
opaque update descriptors. It never retains caller Objects or refresh progress.
@@ -418,6 +423,17 @@ evidence. `AdmissionError` reports rejected environment, world, or backend
418423
admission; `BackendUnavailableError` reports an unavailable selected backend;
419424
and `RemoteExecutionError` reports a bounded worker type without reconstructing
420425
its exception class. This includes a user callable's own `TimeoutError`.
426+
By default `SubProcessConfig` and `RayBackendConfig` have `raw_traceback=True`
427+
and include up to `diagnostic_text_limit_bytes` of worker-formatted Python
428+
traceback in `RemoteExecutionError.remote_traceback` or
429+
`CoreExecutionError.remote_traceback` and their displayed error text.
430+
**This text is unredacted**: exception messages, source lines, filenames,
431+
interpreter paths, user data, and credentials may be included. Use only when
432+
that disclosure to the caller and notebook/log is acceptable. Set
433+
`raw_traceback=False` on the backend configuration for type-only errors.
434+
Traceback text is never persisted as Store authority. An oversized diagnostic
435+
falls back to the type-only failure without dropping completed publication
436+
evidence.
421437
`ExecutionDeadlineExceeded` means the worker or coordinator observed the
422438
execution deadline and owned termination was then confirmed; a worker marker
423439
alone is insufficient. `ExecutionUncertainError` means available
@@ -465,6 +481,7 @@ capture does not probe or create them.
465481
| `live_output_queue_limit_bytes` | `262,144`, positive integer bytes | Bounded coordinator live-delivery queue; overflow disables mirroring but preserves retained capture. |
466482
| `diagnostic_text_limit_bytes` | `65,536`, positive integer bytes | Bounded retained framework diagnostic text. |
467483
| `diagnostic_issue_limit` | `64`, positive integer | Maximum retained cleanup diagnostics. |
484+
| `raw_traceback` | `True`, bool | Include bounded unredacted worker traceback text in generic remote and Core invocation failures; set `False` for type-only errors. No redaction or secret-safe logging is promised. |
468485
| `discovery_candidate_limit` | `128`, positive integer | Bound on discovered environment candidates. |
469486
| `discovery_directory_entry_limit` | `1,024`, positive integer | Bound on entries examined in each discovery directory. |
470487
| `process_read_chunk_bytes` | `8,192`, positive integer bytes | Bounded chunk size while reading owned probes/process output. |
@@ -580,7 +597,7 @@ Invocation and result data remain in the submission child until qualified future
580597
cleanup; the caller-owned spool parent, current directory, existing
581598
environments, and Ray deployment are preserved.
582599

583-
The coordinator validates private protocol v4 and executes only after the worker
600+
The coordinator validates private protocol v5 and executes only after the worker
584601
handshake/admission evidence and GO gate. Setup-bearing calls add `SETUP`,
585602
setup-time `OUTPUT`, and `SETUP_READY` before `PAYLOAD`; `RESULT` is forbidden
586603
before payload transfer. A setup `ERROR` cannot resume with readiness or payload,

‎src/dryml/core/execute.py‎

Lines changed: 80 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -180,18 +180,35 @@ class CoreExecutionError(ExecutionError):
180180
phase: Core adaptation phase which failed.
181181
execution: Optional future retaining this adaptation.
182182
evidence: Publication/update/refresh facts completed before the failure.
183+
remote_traceback: Bounded unredacted worker traceback when enabled by
184+
the selected backend; absent for type-only or local failures.
183185
184186
This error does not claim rollback or retry. A future retains it directly rather
185187
than introducing a second outcome-error scheme.
186188
"""
187189

188190
def __init__(self, message: str, *, phase: str,
189-
evidence: CoreOutcomeEvidence | None = None,
190-
execution: Any = None) -> None:
191+
evidence: CoreOutcomeEvidence | None = None,
192+
execution: Any = None,
193+
remote_traceback: str | None = None) -> None:
191194
super().__init__(message)
192195
self.phase = phase
193196
self.evidence = evidence
194197
self.execution = execution
198+
self.remote_traceback = remote_traceback
199+
200+
def __str__(self) -> str:
201+
"""Display the failure category and any enabled raw worker traceback.
202+
203+
Returns:
204+
The original category summary plus available unredacted worker text.
205+
Raw text can include paths, user values, and credentials.
206+
"""
207+
208+
summary = super().__str__()
209+
if self.remote_traceback is None:
210+
return summary
211+
return f"{summary}\nRemote traceback (raw, unredacted):\n{self.remote_traceback}"
195212

196213

197214
class _RefreshLedger:
@@ -453,9 +470,32 @@ def bind_recovery(self, prepared: PreparedCoreCall, *, args: tuple[Any, ...],
453470
return _CoreRecovery.bind(prepared, args, kwargs)
454471

455472
def invoke(self, invocation: bytes, *, repo: Repo | None, update_args: bool,
456-
invocation_limit_bytes: int = 67_108_864,
457-
result_limit_bytes: int | None = None) -> bytes:
458-
"""Run one reconstructed worker-local call through its single signature boundary."""
473+
invocation_limit_bytes: int = 67_108_864,
474+
result_limit_bytes: int | None = None,
475+
raw_traceback: bool = False,
476+
traceback_limit_bytes: int = 65_536) -> bytes:
477+
"""Run one worker call, optionally retaining bounded raw failure text.
478+
479+
Args:
480+
invocation: Prepared invocation bytes.
481+
repo: Selected worker Repo; absence is an error.
482+
update_args: Whether selected arguments may publish updates.
483+
invocation_limit_bytes: Bound for decoded invocation bytes.
484+
result_limit_bytes: Bound for the tagged outcome; omitted uses the
485+
invocation bound.
486+
raw_traceback: Include unredacted worker traceback text on failure.
487+
traceback_limit_bytes: Maximum diagnostic text bytes when enabled.
488+
489+
Returns:
490+
Bounded tagged result or failed outcome retaining publication facts.
491+
492+
Raises:
493+
ValueError: If the required worker Repo is absent.
494+
CoreCallCodecError: If invocation/outcome transport fails.
495+
496+
Side Effects:
497+
Executes the selected call and may publish exact Store state.
498+
"""
459499
if repo is None:
460500
raise ValueError("SharedDirStoreStrategy requires worker Repo authority")
461501
from .execute_codec import invoke_invocation
@@ -466,7 +506,8 @@ def invoke() -> bytes:
466506
return invoke_invocation(
467507
invocation, repo=repo, invocation_limit_bytes=invocation_limit_bytes,
468508
result_limit_bytes=result_limit_bytes or invocation_limit_bytes,
469-
update_args=update_args,
509+
update_args=update_args, raw_traceback=raw_traceback,
510+
traceback_limit_bytes=traceback_limit_bytes,
470511
)
471512

472513
# Core worker setup normally establishes this selection. Direct strategy
@@ -482,7 +523,9 @@ def recover(
482523
args: tuple[Any, ...], kwargs: Mapping[str, Any], return_objects: bool,
483524
update_args: bool, _recovery: _CoreRecovery | None = None,
484525
_decoded: CoreAdaptationOutcome | None = None,
485-
result_limit_bytes: int = 67_108_864) -> Any:
526+
result_limit_bytes: int = 67_108_864,
527+
allow_raw_traceback: bool = False,
528+
traceback_limit_bytes: int = 65_536) -> Any:
486529
"""Recover one tagged outcome and optionally restore original arguments.
487530
488531
Args:
@@ -496,6 +539,9 @@ def recover(
496539
the original live argument instances.
497540
result_limit_bytes: Frozen generic transport budget used to validate
498541
this delivered outcome.
542+
allow_raw_traceback: Accept unredacted diagnostics in the failed
543+
worker outcome when explicitly selected for this call.
544+
traceback_limit_bytes: Maximum accepted diagnostic text bytes.
499545
500546
Returns:
501547
The decoded result, or its requested fresh local materialization.
@@ -514,12 +560,15 @@ def recover(
514560
raise ValueError("SharedDirStoreStrategy requires caller Repo authority")
515561
decoded = _decoded or decode_core_outcome(
516562
result, repo=repo, result_limit_bytes=result_limit_bytes,
563+
allow_raw_traceback=allow_raw_traceback,
564+
traceback_limit_bytes=traceback_limit_bytes,
517565
)
518566
outcome = decoded.value
519567
if not outcome["success"]:
520568
raise CoreExecutionError(
521569
f"core execution worker failed: {outcome['reason']}",
522570
phase="invoke", evidence=decoded.evidence,
571+
remote_traceback=outcome["traceback"],
523572
)
524573
updates = tuple((item["target"], StateRef.from_data(item["state"])) for item in outcome["updates"])
525574
recovery = _recovery or self.bind_recovery(prepared, args=args, kwargs=kwargs)
@@ -581,13 +630,17 @@ def recover(
581630

582631
def decode_core_outcome(
583632
result: bytes, *, repo: Repo, result_limit_bytes: int = 67_108_864,
633+
allow_raw_traceback: bool = False,
634+
traceback_limit_bytes: int = 65_536,
584635
) -> CoreAdaptationOutcome:
585636
"""Decode a tagged outcome into a value plus detached exact evidence.
586637
587638
Args:
588639
result: Bounded bytes returned by :class:`SharedDirStoreStrategy`.
589640
repo: Caller-owned Repo used only to validate the closed result graph.
590641
result_limit_bytes: Frozen generic outcome byte budget.
642+
allow_raw_traceback: Admit opt-in unredacted worker text.
643+
traceback_limit_bytes: Maximum accepted diagnostic bytes.
591644
592645
Returns:
593646
A future-retainable value/evidence pair. Failed outcomes retain ``value`` as
@@ -603,7 +656,11 @@ def decode_core_outcome(
603656
"""
604657
from .execute_codec import CoreCallCodecError, decode_outcome
605658

606-
outcome = decode_outcome(result, repo=repo, limit_bytes=result_limit_bytes)
659+
outcome = decode_outcome(
660+
result, repo=repo, limit_bytes=result_limit_bytes,
661+
allow_raw_traceback=allow_raw_traceback,
662+
traceback_limit_bytes=traceback_limit_bytes,
663+
)
607664
publications = []
608665
for item in outcome["publications"]:
609666
if not isinstance(item, Mapping) or set(item) != {"state", "store", "phase", "status", "path"}:
@@ -767,7 +824,7 @@ def invoke_prepared_call(invocation: bytes) -> Any:
767824

768825
def _invoke_prepared_outcome(
769826
invocation: bytes, invocation_limit_bytes: int, result_limit_bytes: int,
770-
update_args: bool,
827+
update_args: bool, raw_traceback: bool, traceback_limit_bytes: int,
771828
) -> bytes:
772829
"""Return one opaque core outcome from an already configured worker.
773830
@@ -780,6 +837,8 @@ def _invoke_prepared_outcome(
780837
invocation_limit_bytes: Captured public backend invocation bound.
781838
result_limit_bytes: Captured public backend outcome bound.
782839
update_args: Captured caller-refresh policy.
840+
raw_traceback: Captured backend policy for unredacted failure text.
841+
traceback_limit_bytes: Captured maximum diagnostic text bytes.
783842
784843
Returns:
785844
The strategy-owned tagged outcome bytes.
@@ -792,6 +851,7 @@ def _invoke_prepared_outcome(
792851
invocation, repo=context.repo, update_args=update_args,
793852
invocation_limit_bytes=invocation_limit_bytes,
794853
result_limit_bytes=result_limit_bytes,
854+
raw_traceback=raw_traceback, traceback_limit_bytes=traceback_limit_bytes,
795855
)
796856

797857

@@ -1551,6 +1611,8 @@ class CoreExecutionFuture:
15511611
recovery: Caller-object bindings created before submission returns.
15521612
return_objects: Captured result-materialization decision.
15531613
result_limit_bytes: Captured public generic result limit.
1614+
raw_traceback: Whether failed worker outcomes may retain unredacted
1615+
traceback text under the backend diagnostic byte limit.
15541616
done_callbacks: Coordinator-local callbacks receiving this facade.
15551617
one_off: Whether generic one-off cleanup should also release recovery state.
15561618
@@ -1574,6 +1636,7 @@ def __init__(
15741636
self, backend_future: ExecutionFuture[bytes], *,
15751637
prepared_storage: _PreparedSharedStorage, prepared: PreparedCoreCall,
15761638
recovery: _CoreRecovery, return_objects: bool, result_limit_bytes: int,
1639+
raw_traceback: bool = False,
15771640
done_callbacks: Sequence[CoreCallback] = (), one_off: bool = False,
15781641
) -> None:
15791642
if not isinstance(backend_future, ExecutionFuture):
@@ -1587,6 +1650,7 @@ def __init__(
15871650
self._recovery = recovery
15881651
self._return_objects = return_objects
15891652
self._result_limit_bytes = result_limit_bytes
1653+
self._raw_traceback = raw_traceback
15901654
self._condition = Condition(RLock())
15911655
self._state = "pending"
15921656
self._phase = "prepare"
@@ -1957,6 +2021,8 @@ def _backend_completed(self, _: ExecutionFuture[bytes]) -> None:
19572021
decoded = decode_core_outcome(
19582022
data, repo=self._storage.recovery_repo,
19592023
result_limit_bytes=self._result_limit_bytes,
2024+
allow_raw_traceback=self._raw_traceback,
2025+
traceback_limit_bytes=self._diagnostic_text_limit_bytes,
19602026
)
19612027
# Recovery runs in generic callback threads, which do not inherit the
19622028
# caller's ContextVars. Install only this frozen call's owned Repo and
@@ -1967,6 +2033,8 @@ def _backend_completed(self, _: ExecutionFuture[bytes]) -> None:
19672033
args=(), kwargs={}, return_objects=self._return_objects,
19682034
update_args=self._storage.update_args, _recovery=self._recovery,
19692035
_decoded=decoded, result_limit_bytes=self._result_limit_bytes,
2036+
allow_raw_traceback=self._raw_traceback,
2037+
traceback_limit_bytes=self._diagnostic_text_limit_bytes,
19702038
)
19712039
with self._condition:
19722040
self._evidence = self._recovery.ledger.evidence(decoded.evidence)
@@ -2290,6 +2358,8 @@ def _submit_frozen_core_submission(
22902358
config.invocation_limit_bytes,
22912359
frozen.result_limit_bytes,
22922360
frozen.effective.update_args,
2361+
config.raw_traceback,
2362+
config.diagnostic_text_limit_bytes,
22932363
kwargs=None,
22942364
environment=environment,
22952365
environment_spec=environment_spec,
@@ -2307,6 +2377,7 @@ def _submit_frozen_core_submission(
23072377
recovery=frozen.recovery,
23082378
return_objects=frozen.return_objects,
23092379
result_limit_bytes=frozen.result_limit_bytes,
2380+
raw_traceback=config.raw_traceback,
23102381
done_callbacks=callbacks,
23112382
one_off=one_off,
23122383
)

0 commit comments

Comments
 (0)