Piccolo matrix-free objective-constructor slice

[gennadiryan]

Piccolo matrix-free objective-constructor slice — overall summary

Branch: feat/matrix-free-obj-constr-test in harmoniqs/Piccolo.jl (ahead of feat/matrix-free-obj-constr by one commit).
Origin: part of Piccolissimo #179; scoped into two Piccolo issues:

• ../../inputs/matrix-free/257.md
  — CoherentKetInfidelityObjective slice.
• ../../inputs/matrix-free/248.md
  — full-op UnitaryInfidelityObjective slice.

Companion documents:

• ./dto-matrix-free-summary.v1.md
  — the upstream DTO Bug-ForwardDiff and sample Matrix Orientation #113 KnotHVP interface this consumes.
• ./piccolo-issue-discrepancies.md
  — review-gap analysis vs. issues Full-operator UnitaryInfidelityObjective declares ConstantLowRankHVP (build A at construction) #248 / #257.
• ./piccolo-test-addendum.md —
  outcome of the review-pass edits.
• ./piccolo-mfree-objconstr.md —
  this document.

Consumer (still pending): the Piccolissimo #179 apply-side is
tracked in ./piccolissimo-todos.md; its
work has not yet started. This document only summarizes the Piccolo
half of the coordinated PR pair.

---

1. Overall shape

Two of Piccolo's fidelity constructors now populate the DTO
knot_hvp::Union{Nothing, KnotHVP} field on their returned
KnotPointObjective, so Piccolissimo's Altissimo backend can pick
up an exact, constant, rank-2 factor and apply the per-knot Hessian
matrix-free — instead of rediscovering the same operator via the
(6 + r) nested-ForwardDiff probes + host SVD that PR #178
introduced.

The constructors affected:

• CoherentKetInfidelityObjective (multi-ket coherent fidelity)
• UnitaryInfidelityObjective on the full-operator branch (i.e.
  U_goal isa AbstractMatrix). The EmbeddedOperator branch and
  the UnitaryFreePhaseInfidelityObjective (θ-parameterized) stay
  on knot_hvp = nothing — deliberately out of scope, as their
  Hessian structures don't fit ConstantLowRankHVP.

Ket-family variants outside the coherent multi-ket case
(KetInfidelityObjective, CoherentKetFreePhaseInfidelityObjective,
KetFreePhaseInfidelityObjective), density objectives, and
UnitarySensitivityObjective are all untouched.

---

2. Files changed

File	Kind	Change
src/control/objectives.jl	source	CoherentKetInfidelityObjective builds A via new helper, threads knot_hvp through TerminalObjective; new helper _coherent_ket_lowrank_factor(goals, state_dims) (~35 lines); UnitaryInfidelityObjective (full-op branch) builds A via new helper, threads knot_hvp; new helper _unitary_lowrank_factor(U_goal) (~24 lines); trailing signature comment on U_goal::AbstractPiccoloOperator distinguishing the full-op vs. embedded arms.
src/control/objectives.jl	tests	3 new inline @testitem blocks (rank-2 factor round-trips + EmbeddedOperator fallthrough) + 1 review-pass byte-identity testitem; expanded test coverage per issue ACs.

No new files. No API surface change (only a new keyword argument
that was already threaded by DTO #113). No public-symbol renaming.
No changes to value or gradient paths.

---

3. What each constructor now does

3.1 CoherentKetInfidelityObjective (issue #257)

The coherent overlap S = (1/K) · Σᵢ ⟨gᵢ, ψᵢ⟩ is a single complex
scalar linear in z, where z = [ψ̃₁; ψ̃₂; …; ψ̃_K] is the
concatenated per-ket iso-state at the terminal knot and each ψ̃ᵢ
follows the ket_to_iso convention [Re(ψᵢ); Im(ψᵢ)] (length
d_iso = 2 · ketdim).

Because S is a scalar linear functional of z, the loss
ℓ = |1 − |S|²| factors exactly as |1 − ‖A · z‖²| with a
constant A of shape 2 × sum(state_dims):

• Row 1 of A captures Re(S) = (1/K) · Σᵢ (Re(gᵢ)ᵀRe(ψᵢ) + Im(gᵢ)ᵀIm(ψᵢ))
• Row 2 captures Im(S) = (1/K) · Σᵢ (Re(gᵢ)ᵀIm(ψᵢ) − Im(gᵢ)ᵀRe(ψᵢ))

with the 1/K factor built into every entry of A, so
‖A · z‖² = |S|² = F exactly (no extra normalization needed at
the consumer). This is the :neg2_sign link-Hessian rule for
Piccolissimo: H · v = -2 · sign(1 − F) · Aᵀ · (A · v) (with the
kink at F = 1 handled by explicit branch selection on the
Piccolissimo side).

The constructor now stashes ConstantLowRankHVP(A, :neg2_sign) via
the knot_hvp keyword forwarded to TerminalObjective.

3.2 UnitaryInfidelityObjective (issue #248), full-op branch

The overlap S = tr(U_goal† · U(z)) is a single complex scalar
linear in z, where z = Ũ⃗ follows the operator_to_iso_vec
convention (per column i of U, a length-2n block
[Re(U[:,i+1]); Im(U[:,i+1])], total length 2n²).

The fidelity is F = |S|² / n², so the loss ℓ = |1 − F| factors
exactly as |1 − ‖A · Ũ⃗‖²| with a constant A of shape
2 × 2n², with the 1/n factor built into every entry so
‖A · Ũ⃗‖² = |S|² / n² = F exactly.

The constructor now stashes ConstantLowRankHVP(A, :neg2_sign)
when U_goal isa AbstractMatrix{<:Number}. When U_goal isa EmbeddedOperator, knot_hvp = nothing is passed instead — the
embedded case has a mixed regularizer structure
(1 / (n(n+1)) · (|tr(M†M)| + |tr(M)|²)) that doesn't fit the
constant-rank-2 form and stays on the dense fallback.

3.3 Helper functions

Two new module-private helpers in objectives.jl:

• _coherent_ket_lowrank_factor(goals, state_dims) — builds
  the (2, sum(state_dims)) factor from a vector of complex goal
  kets and their per-ket iso-dimensions. Handles arbitrary K and
  arbitrary per-ket dimensions (the API supports it, though in
  practice all kets share the same Hilbert space dim).
• _unitary_lowrank_factor(U_goal) — builds the (2, 2n²)
  factor from an n × n complex operator, per the
  operator_to_iso_vec column-major layout.

Both produce Matrix{Float64} (solver-precision real iso
representation), matching the DTO ConstantLowRankHVP.A::Matrix{Float64}
field type.

---

4. Acceptance criteria mapping

4.1 Issue #257 (CoherentKetInfidelityObjective)

AC	Status	Evidence
Constructing returns objective with knot_hvp isa ConstantLowRankHVP, core === :neg2_sign (was nothing)	✅	Testitem "knot_hvp matrix-free apply matches dense Hessian — CoherentKetInfidelityObjective" asserts both directly.
A is 2 × sum(state_dims) real, 1/K scale, F = ‖A·x_k‖² = coherent fidelity to ~1e-12 for randomly sampled terminal states and goals, across ≥1 and ≥2 goal states	✅	Same testitem now loops for K in (1, 2) × for scale in (0.4, 1.2) (F<1, F>1) and asserts size(cap.A) == (2, K * ket_dim) plus atol = 1e-12 parity. Review-pass expansion in ./piccolo-test-addendum.md.
Value and gradient are byte-for-byte unchanged	✅	New testitem "knot_hvp field does not leak into KnotPointObjective value/gradient" builds two KnotPointObjectives differing only in knot_hvp and asserts elementwise equality. Existing testitems (CoherentKetInfidelityObjective, CoherentKetFreePhaseInfidelityObjective, coherent_ket_fidelity accepts generic Complex types) still pass.
[compat] admits DTO release shipping KnotHVP	🚧 deferred	Follow-up commit once DTO v0.9.7 is registered (see ./piccolo-test-addendum.md "Items deferred").

4.2 Issue #248 (UnitaryInfidelityObjective full-op)

AC	Status	Evidence
Constructing (full-op) returns objective with knot_hvp isa ConstantLowRankHVP, core === :neg2_sign (was nothing)	✅	Testitem "knot_hvp matrix-free apply matches dense Hessian — UnitaryInfidelityObjective (full-op)" asserts both.
A is 2 × 2n² real, 1/n scale, `F = ‖A·x_k‖² =	S	²/n²to~1e-12**acrossn = 2, 3, 4`**
Embedded and free-phase constructors untouched (knot_hvp = nothing)	✅	Testitem "knot_hvp returns nothing for EmbeddedOperator UnitaryInfidelityObjective" asserts the embedded case directly. UnitaryFreePhaseInfidelityObjective constructor is unchanged in this branch (verified by inspection); an inline assertion could be added if desired.
Value and gradient are byte-for-byte unchanged	✅	Shared with #257 AC 4 above — the new byte-identity testitem is objective-agnostic (uses KnotPointObjective directly, which is the layer at which knot_hvp lives).

---

5. Test coverage summary

Post-review-pass, Piccolo.jl/src/control/objectives.jl inline
testitems total 55 assertions across 8 testitems (was 12 before
this branch, and 29 before the review pass — see
./piccolo-test-addendum.md for the
per-testitem breakdown). Wall time 16.6 s in the manual driver.

The added test surface:

• Round-trip parity vs. Symmetric(get_full_hessian, :U) · v at
  F < 1 and F > 1 — for coherent-ket at K ∈ {1, 2} and for
  full-op unitary at n ∈ {2, 3, 4}.
• Fallback verification: EmbeddedOperator case returns
  knot_hvp === nothing (scope guard).
• Byte-identity of value + gradient across a knot_hvp = nothing
  vs. knot_hvp = ConstantLowRankHVP(random_A, :neg2_sign)
  construction pair.
• No-regression: all 12 pre-branch assertions still pass unchanged.

---

6. Deferred / follow-up items

Item	Reason	When it closes
[compat] DirectTrajOpt = "0.9.7" in Piccolo.jl/Project.toml	DTO v0.9.7 not yet in the registered General.jl set; Pkg.resolve() fails when the compat entry demands a version the registry doesn't have.	One-line follow-up commit once DTO v0.9.7 is tagged and registered upstream.
Free-phase knot_hvp declarers (UnitaryFreePhaseInfidelityObjective, CoherentKetFreePhaseInfidelityObjective, KetFreePhaseInfidelityObjective)	A(θ) is not constant — deferred with three candidate approaches (CustomKnotHVP closure, ParametricLowRankHVP carrier, exploit global linear constraints on θ).	Per ./todos.md §2a: filed as a future sub-issue; needs a Phase-0 audit of the per-iterate rank of A(θ) before choosing an approach.
Piccolissimo consumer (Issue #179 apply side)	Half of the coordinated PR pair; produces the code that actually reads knot_hvp(obj, traj) and applies the constant-A HVP device-side.	Next in the workstream — implementation plan at ./piccolissimo-todos.md, decisions locked in via ../todos-conflicts.md and ./piccolissimo-saturation-guard-determination.md.

---

7. Non-goals

Recorded explicitly to keep the PR scope obvious:

• Piccolissimo consumer — a separate PR (item 1 of the workstream
  in ./todos.md).
• Sunset of PR feat: add calibration_targets kwarg to pulse problem templates #178's _build_rank_r_block — item 4 in the
  workstream, not this PR.
• Density objectives, UnitarySensitivityObjective, and the
  ket-family variants other than the coherent multi-ket case —
  still knot_hvp = nothing. All either don't fit the constant-A
  rank-2 structure or are affine (zero block); their carve-outs are
  matched to the corresponding issue-scope statements.
• New core symbols beyond :neg2_sign — none in this branch.
  When a new structured family (e.g. an embedded-op declarer) is
  added later, the same slot can carry a new symbol paired with the
  corresponding apply-branch on the Piccolissimo side.

[codecov]

Codecov Report

❌ Patch coverage is 98.33333% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/quantum/operators/direct_sums.jl	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[gennadiryan]

All tests passed pre-formatting; formatter passes now; merging @jeonghun-jj-lee @Rchari1