Skip to content

Make Android inference lifecycle-, coroutine-, and memory-pressure-safe - #5469

Open
maziyarpanahi wants to merge 1 commit into
masterfrom
feat/issue-3647-make-android-inference-lifecycle-coroutine
Open

maziyarpanahi wants to merge 1 commit into
masterfrom
feat/issue-3647-make-android-inference-lifecycle-coroutine

Conversation

@maziyarpanahi

Copy link
Copy Markdown
Owner

Closes #3647.

Adds OpenMedRuntime (android/openmedkit/.../OpenMedRuntime.kt), a lifecycle-owned Android runtime with an explicit IDLE, LOADING, READY, BUSY, DRAINING, RELEASED state machine and a value-free status flow. Every operation holds a lease on the session it started on. Loading is single-flight behind a Mutex, so concurrent first use opens one session. close(), replaceModel, memory pressure and backgrounding close a native session only after its final lease is released. Cancelled callers get CancellationException; revoked, refused or stale work raises OpenMedRuntimeException with a stable value-free reason code and retry guidance, and never publishes a late result. Trim-memory and host lifecycle signals come in through injectable seams: OpenMedLifecycleSource, a ComponentCallbacks2 adapter registered on the application context, and a documented ProcessLifecycleOwner snippet, so core takes no androidx.lifecycle dependency. The default policy releases idle sessions at LOW pressure, also stops admission at CRITICAL for a cool-down, refuses new work while backgrounded, and bounds consecutive reloads; each reacquisition re-runs the factory, which re-verifies the cached bundle.

The OpenMedKit facade and the ONNX OnnxTokenClassifier now have reference-counted close guards (util/LeaseGate.kt) and cancellation checkpoints before tokenization, after native inference, and after decoding, merging and policy application. ModelCache gains leases, so evict and budget trimming skip a model that an open session uses (OpenMedSessionFactory.fromCache). The React Native module now owns an OpenMedHostScope that invalidate() cancels and releases (it also unregisters trim-memory callbacks). It rejects promises with value-free messages plus userInfo.reason/retry, and builds a fresh metadata map for every span: the old shared WritableMap was consumed by the first putMap. The docs section is in docs/android-integration.md.

Breaking: OpenMedKit calls made after close() now throw OpenMedRuntimeException(RUNTIME_CLOSED) instead of reaching the closed classifier, and close() defers the native release until in-flight calls finish. ModelCache.evict returns false for a leased model. React Native rejections no longer forward the underlying exception message; read error.userInfo.reason instead.

Not covered: emulator and real-device runs for the Android matrix, including peak-memory, thermal-class and leak-check evidence. Interrupting an in-flight native ONNX Runtime run is also out of scope, so the cancellation bound is one native window run. The React Native Android module is only compile-checked, against local stand-ins for the React Native APIs it uses, because the repo has no React Native Android build yet. #4478 moves that file, so one of the two PRs will need a rebase.

Validation: cd android && ./gradlew :openmedkit:testDebugUnitTest (140 tests, 0 failures; 33 new, including virtual-time cancel/close/swap/trim race and stress tests), RN module compiled with Kotlin 1.8.22 against React Native API stubs (pass), pytest tests/ -q -m "not slow and not fuzz" --ignore=tests/unit/training/synthetic/test_dicom_overlays.py (27921 passed, 271 skipped, 298 deselected, 11356 subtests passed; the ignored file needs pydicom, which the shared venv lacks), ruff check . and ruff format --check . (pass), scripts/security/check_effect_paths.py --check (pass), mypy (pass). kotlinx-coroutines-test was already a test dependency, so there are no new dependencies and no lockfile changes. Release AAR size: 565,571 to 655,245 bytes (+89,674), which is over the 589,824-byte limit in android/gradle/budgets.properties. I left the limit unchanged, so verifyAndroidBudgets will fail until it is revisited. The strict MkDocs build is left to CI.

…essure-safe

Add OpenMedRuntime, a lifecycle-owned Android runtime with an explicit
IDLE/LOADING/READY/BUSY/DRAINING/RELEASED state machine. Operations hold
leases on the session they started on, loading is single-flight, and
close, model replacement, memory pressure, and backgrounding release a
native session only after its final lease. Cancelled or revoked work
never publishes a late result; refusals raise value-free typed errors
with retry guidance.

Trim-memory and host lifecycle signals arrive through injectable
sources, including a ComponentCallbacks2 adapter, without an
androidx.lifecycle dependency. The facade and ONNX classifier gain
reference-counted close guards and stage cancellation checkpoints, and
ModelCache leases keep in-use models out of eviction and trimming.

The React Native module now owns a host scope cancelled in invalidate(),
rejects with value-free messages, and builds a fresh metadata map per
span.
@maziyarpanahi maziyarpanahi added this to the v4.0 milestone Nov 7, 2026
@maziyarpanahi maziyarpanahi added improvement Hardening / refactor of existing code P1 High roadmap-v3 OpenMed V3 roadmap backlog labels Nov 7, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

improvement Hardening / refactor of existing code P1 High roadmap-v3 OpenMed V3 roadmap backlog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make Android inference lifecycle-, coroutine-, and memory-pressure-safe

1 participant