Skip to content

Fix review findings in the Grove FHIR migration - #220

Merged
lukaskollmer merged 70 commits into
mainfrom
lukas/grove-fhir-review-fixes
Oct 30, 2026
Merged

lukaskollmer merged 70 commits into
mainfrom
lukas/grove-fhir-review-fixes

Conversation

@lukaskollmer

@lukaskollmer lukaskollmer commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

♻️ Current situation & Problem

A review of the Grove FHIR migration stack (#202 → ...) found some problems, which this PR fixes on the app side.

  • A SensorKit sensor could stop uploading permanently. Each strategy published its records in one for … try loop, so a single record that Grove refuses aborted the batch. The batch was then never acknowledged and was delivered again on every run. Separately, when SensorKit did not replay a pending batch exactly (pendingBatchMismatch), neither the fetcher nor logout cleanup could recover, because resetAllQueryAnchors swallowed the error with try?.
  • The FHIR exchange ledger cost O(n²) per HealthKit batch. The ledger is one keychain-encrypted LocalStorage value (Grove's default setting). Every sample decrypted, rewrote and re-encrypted the whole ledger, so large deliveries and month-sized historical-export batches could not finish in a background window.
  • State of Mind samples, and any HeartRisk response with a skipped optional question, lost their HealthKit data. This is fixed in Grove, and this PR takes it in by bumping the pin.

⚙️ Release Notes

  • --studyBundle defaults to .firebase(.stanford) again, instead of .bundledWithApp(.stanford). The loader still substitutes the active study variant and still falls back to the bundled definition when the hosted one fails to load.
  • SensorKit: rework processing so a single failing record doesn't cancel the whole batch (we used to have a for ... try loop, which caused the batch to never get acknowledged/finalized, causing it to get re-run on every launch)
  • SensorKit: a record whose Grove preparation fails is refused and logged. Logs contain only the sensor, ordinal and diagnostic code, no PHI. Its reservation is released and the rest of the batch still gets acknowledged. Upload, account-fence and cancellation errors still abort without acknowledging.
  • SensorKit: on pendingBatchMismatch or retryContentChanged, the fetcher calls SensorKit.discardPendingBatches(for:) and drops that sensor's retry bookkeeping, and the next run fetches the range again under fresh identities. lastSensorKitFetch is only updated when the fetch succeeded. resetAllQueryAnchors now propagates failures, and clearPendingAccountData counts the anchor reset toward success.
  • FHIR exchange ledger: HealthKit deliveries, historical export and retractions reserve a whole batch in one transaction. An exact retry only reads the ledger. Single-key reservations such as SensorKit's use one transaction without an extra read.
  • Grove is pinned to 611d0cd8 and StudyDefinitions to d1cbe285, in the project, MyHeartCountsShared, Package.resolved and the submodule.

Behavior changes to be aware of:

  • Refused SensorKit records are dropped with a log line instead of wedging the sensor.
  • After an abandoned batch, up to one batch per incident can be uploaded twice under new IDs.
  • A sensor that keeps failing leaves lastSensorKitFetch stale, so foreground launches retry it.

Out of scope, per the review triage:

  • Dashboard read-compat for pre-migration LDL, score and Timed Walk documents: dropped. The dashboard has only shipped to TestFlight, so this is a one-time loss.
  • Backend items (MyHeartCounts-Firebase), still open:
    • the LDL writer still uses loinc.org as the quantity system;
    • nothing consumes healthDeletions/;
    • the status patch has no effect on Bundle documents.

📚 Documentation

Doc comments cover the study-bundle default, the refusal rule, and the batch reservation's durability and retry guarantees.

✅ Testing

  • Compiled locally: the app and the MyHeartCounts Unit Tests plan (build-for-testing, generic iOS Simulator, Xcode 27), with packages resolved against the new pins.
  • Not run locally: the app unit tests need a simulator, so CI runs them.
  • SwiftLint strict passes on every changed file.

New tests:

  • StudyBundleLaunchOptionsTests.defaultsToFirebase
  • DeviceUsageFHIRExchangeTests.refusedRecordsDoNotFailTheirBatch and accountFenceAbortsBatchInsteadOfRefusingRecord
  • FHIRExchangeStateTests: three batch-reservation tests (input order, existing keys consume no sequence, stale account generation), healthKitBatchReservesEverySampleUpFront, and abandoningSensorBatchesDropsOnlyThatSourcesRetryState
  • QuestionnaireGroveProjectionTests.bloodPressureRefusesAnIncompleteComponentSet now expects Grove's componentIncomplete.

Not covered by tests:

  • Fetcher-level recovery (discard on mismatch, the per-sensor success flag, the cleanup verdict). It needs a live SensorKit reader and a device.
  • The refusal loops for ECG, wrist temperature, CSV and PPG. Only the device-usage strategy has a test.

Code of Conduct & Contributing Guidelines

By creating and submitting this pull request, you agree to follow our Code of Conduct and Contributing Guidelines:

🤖 Generated with Claude Code

lukaskollmer and others added 30 commits August 29, 2026 23:12
Spezi was renamed to Grove in 0.3.0 (SchmiedmayerLab/Grove, PR #43): every
product `Spezi*` -> `Grove*` and `XCTSpezi*` -> `XCTGrove*`, the core type
`Spezi` -> `Grove`, and no source-compatibility typealiases. This moves the
app onto 0.3.0-beta.6.

Package graph:
- Both manifests and the Xcode project point at SchmiedmayerLab/Grove.git.
  The old URL still resolves through GitHub's rename redirect, but SwiftPM
  derives identity from the last path component, so a graph mixing the two
  URLs resolves one repository under two identities -- they have to move
  together.
- MyHeartCounts-StudyDefinitions is pinned to a revision until its own Grove
  release is tagged (SchmiedmayerLab/MyHeartCounts-StudyDefinitions#46);
  every 0.2.x of it still requires the `spezi` identity.
- The watch target's GroveHealthKit product dependency pointed at a package
  reference for the retired StanfordSpezi/SpeziHealthKit that was not in the
  project's packageReferences at all. It now resolves against the Grove
  reference, and the orphan is gone.

Renames the compiler cannot do for you, because the replacement is not the
`Spezi` token: `.spezi(_:)` -> `.grove(_:)`, `@Application(\.spezi)` ->
`\.grove`, `SpeziAppDelegate.spezi` -> `GroveAppDelegate.grove`, and
`UTType.speziStudyBundle` -> `UTType.studyBundle` (the vendor prefix is
dropped, not swapped).

Identifiers that name stored or transmitted data:
- BGTaskSchedulerPermittedIdentifiers now lists
  `$(PRODUCT_BUNDLE_IDENTIFIER).scheduler.refresh`. Grove honours the old
  string for one release and then stops; registering an unpermitted
  identifier fails silently, so this cannot be left to drift.
- The scheduler notification userInfo is read through
  `SchedulerNotifications.taskId(fromUserInfo:)`, which accepts the current
  key and the pre-Grove one. A direct subscript still compiles and would
  silently stop matching notifications already pending on a device.
- MHC's own coding system moved off the framework's name to an authority we
  own: `https://myheartcounts.stanford.edu/fhir/CodeSystem/sampleType`.
  Reads accept the superseded `https://spezi.stanford.edu` spelling, so
  historical bloodLipids / nicotineExposure / dietMEPAScore /
  mentalWellbeingScore observations keep parsing.
- MHC's hand-rolled sourceRevision writer derives its urls from Grove's
  `FHIRExtensionURL.sourceRevision` instead of hardcoding bdh.stanford.edu,
  so resources this app writes agree with the ones Grove writes.
  The backend still emits the old spellings: MyHeartCounts-Firebase#213.

Left alone deliberately: the `mhc.spezi.stanford.edu` config host, the
"based on the Stanford Spezi Template Application" provenance headers, and
the App Store description copy.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182tQC5aCA5KdUYixuB7tvP
The `.bundledWithApp` fallback only caught `unableToDecode`, so a fetch
failure -- an offline launch, or a bucket that does not yet carry the
object under its current name -- threw `unableToFetchFromServer` straight
out of `_update(using:)`. That is not a soft landing: `update()`'s
`returnCachedBundleOnError` can only return a bundle already loaded in this
process, which on a cold launch is nothing, so the study bundle simply
fails to load.

Both failures now take the same path, still only for the firebase selector:
an explicit `--studyBundle <url>` that cannot be fetched keeps erroring
rather than quietly loading something else.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182tQC5aCA5KdUYixuB7tvP
Grove deprecates `RWLock` (pthread_rwlock is slower than the alternatives in
nearly every access pattern, and it cannot own the state it guards), which
left the app with two deprecation warnings.

`MyHeartCountsStandard.currentEnrollmentInfo` becomes a
`Mutex<CurrentEnrollmentInfo?>`, so the lock owns the value instead of
guarding a `nonisolated(unsafe) static var` beside it, and the enrollment
snapshot is built before taking the lock rather than reading MainActor state
inside it. `Lifecycle` guards a keypath write rather than a value, so it
takes a `Mutex<Void>` -- the same swap Grove made internally for
`LocalPreference`.

Behaviourally equivalent: no reader relied on read-lock concurrency, and
both values were only ever read and written whole.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182tQC5aCA5KdUYixuB7tvP
# Conflicts:
#	MyHeartCounts.xcodeproj/project.pbxproj
#	MyHeartCounts.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved
#	MyHeartCounts/Heart Health Dashboard/Health Dashboard/HealthDashboard+BloodPressure.swift
#	MyHeartCounts/Heart Health Dashboard/Health Dashboard/HealthDashboard+SleepAnalysis.swift
#	MyHeartCounts/Heart Health Dashboard/Health Dashboard/HealthDashboard.swift
#	MyHeartCounts/Heart Health Dashboard/Health Dashboard/HealthDashboardQuantityLabel.swift
#	MyHeartCounts/Heart Health Dashboard/Health Dashboard/HealthDashboardTile.swift
#	MyHeartCounts/Heart Health Dashboard/Health Dashboard/HealthStatsChart.swift
#	MyHeartCounts/Heart Health Dashboard/QuantitySample+FHIR.swift
#	MyHeartCounts/Heart Health Dashboard/QuantitySamplesQuery.swift
#	MyHeartCounts/Heart Health Dashboard/SleepSessionsQuery.swift
#	MyHeartCounts/Modules/Lifecycle.swift
#	MyHeartCounts/Modules/MHCBackgroundTasks.swift
#	MyHeartCounts/Modules/ManagedFileUpload.swift
#	MyHeartCounts/Modules/SetupTestEnvironment.swift
#	MyHeartCounts/MyHeartCountsDelegate.swift
#	MyHeartCounts/Onboarding/OnboardingDisclaimerStep.swift
#	MyHeartCounts/Root View/RootView.swift
#	MyHeartCounts/Supporting Files/Info.plist
#	MyHeartCounts/Utils/HealthKit.swift
#	MyHeartCountsShared/Package.swift
#	MyHeartCountsTests/ManagedFileUploadTests.swift
lukaskollmer and others added 13 commits September 26, 2026 18:44
…ration

Combine study variants with the FHIR migration using the published
StudyDefinitions integration and Grove's restored standard deployment targets.
Update the dependency pins and apply the required Grove API compatibility
changes while preserving existing export-state and retraction behavior.
#216 temporarily defaulted `--studyBundle` to the bundle shipped with the app. That default is not DEBUG-gated, so every TestFlight and App Store launch exported the compiled-in study definition and never fetched the hosted one: new study revisions, consent versions and sample-type lists could not reach installed apps.

The default is `.firebase(.stanford)` again. The variant is a placeholder that StudyBundleLoader replaces with the active study variant, and the Firebase path still falls back to the bundled definition when the hosted bundle is missing or can't be decoded, so the Imperial variant keeps working until its bundle is published.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every HealthKit sample and retracted deletion reserved its event with its
own LocalStorage.modify of the whole encrypted ledger, plus another full
load for its identifier and two keychain reads, while the batch's earlier
reservations stayed in the ledger. A delivery or historical-export batch
therefore cost O(n^2) ECIES work and could not finish in the background.

FHIRExchangeStateStore.events(forKeys:) now reserves a whole batch in one
transaction: existing keys come back unchanged, new keys get consecutive
sequences in input order, and a batch whose keys all exist only reads the
ledger. HealthKitConversionBatch reserves live-delivery, direct-upload and
historical-export batches up front and builds each sample's context from
the returned producer instance and a once-read event scope; retraction
drain chunks do the same via healthKitRetractions(of:). Reservations are
still durable before any upload uses their identity, and stale account
generations are still rejected on both the read and write paths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A record whose Grove preparation or conversion throws is now logged (without
content), its exchange reservation is released, and the batch is still
acknowledged. Account-fence, retry-ledger, staging and Firestore errors still
abort the batch unacknowledged. This applies to all five upload strategies,
including the single-record CSV and PPG batches.

When a fetch fails with pendingBatchMismatch or retryContentChanged, the
sensor's pending batches are discarded via Grove's discardPendingBatches(for:)
and their retry-only state is dropped, so the next run refetches the range
under fresh coordinates. lastSensorKitFetch is only bumped once every sensor
succeeded.

resetAllQueryAnchors no longer swallows errors: failures of authorized sensors
make clearPendingAccountData keep pendingAccountDataCleanupRequired set.

Requires the Grove revision that adds SensorKit.discardPendingBatches(for:).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
event(key:) went through the batch API, whose read-only probe adds a full
decrypt of the ledger before the write transaction whenever a key is new.
SensorKit reserves records one at a time and most of them are new, so that
made its per-record path slower. Single-key reservations now run one
transaction again; batches keep the probe, which spares an exact retry any
rewrite. Both share one reservation helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Grove moves to 611d0cd8 (SchmiedmayerLab/Grove#117), which adds
SensorKit.discardPendingBatches(for:) used by the SensorKit recovery above and
fixes the State of Mind, partial-extraction, breathing-disturbance, ECG timing
and DST issues. StudyDefinitions moves to d1cbe285
(SchmiedmayerLab/MyHeartCounts-StudyDefinitions#52), which only re-pins Grove,
because SwiftPM rejects two different revision requirements for Grove. Pinned
in the project, MyHeartCountsShared, Package.resolved and the submodule.

A half-answered blood-pressure panel still refuses, but Grove now reports it
as componentIncomplete instead of answerMissing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 94568aaa-b031-4220-9e03-4db9c13a8d76
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@lukaskollmer
lukaskollmer added this pull request to stack #217 September 28, 2026 15:59
Xcode 26.6, which CI uses, could not type-check the `Date + Double * 60 + 30`
arithmetic inside the untyped `map` closure in reasonable time and failed the
test build. Explicit closure types and addingTimeInterval keep the same
samples.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lukaskollmer
lukaskollmer removed this pull request from stack #217 October 30, 2026 16:54
Base automatically changed from lukas/ios27-fr to main October 30, 2026 16:57
@lukaskollmer
lukaskollmer merged commit 9c23ea3 into main Oct 30, 2026
7 of 17 checks passed
@lukaskollmer
lukaskollmer deleted the lukas/grove-fhir-review-fixes branch October 30, 2026 16:57
@github-project-automation github-project-automation Bot moved this from In Progress to Done in MyHeart Counts Project Planning Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants