Skip to content

[feature] Implement local biometric speaker identity — LocalBiometric provider is a stub, leaving multi-user household memory and privacy ungated #559

Description

@dale053

What problem does this solve?

In any household with more than one speaker, GenieClaw has no way to tell users apart.
SpeakerIdentityProvider::LocalBiometric is selectable in geniepod.toml and fully
documented in genie-common/src/config.rs, but voice/identity.rs contains only a
detection stub — enrollment, embedding extraction, and match scoring are never
implemented. The CLI commands genie-ctl speaker enroll <name>, speaker list, and
speaker remove <name> exist in genie-ctl/src/main.rs but return stub errors.

This causes three user-visible failures:

  1. Memory bleeds across household members. memory/policy.rs has person-scoped
    read/write gates that only fire when identity resolves to a known member. Because
    resolution always returns Unknown, personal preferences, private facts, and
    health-adjacent memories from any speaker surface in any conversation. This is also
    the root cause behind the write-side gap in [bug] memory_store writes person-scoped memories without verified identity context (#430 write-side) #454 — enforcing that gate is impossible
    without a working identity layer underneath it.

  2. Voice-triggered personalization is dead code. tools/dispatch.rs threads speaker
    context into tool calls (timer labels, preference lookups, person-targeted HA actions).
    With permanent Unknown identity, that path never fires.

  3. genie-ctl speaker enroll is a broken user-facing contract. The command is
    documented in the README and present in the CLI help text but exits with a stub error,
    which is confusing and erodes trust in the tooling.


Proposed feature

When speaker_identity.provider = "local_biometric" is set, GenieClaw should:

  • Enroll household members by capturing N audio samples (min_enrollment_samples),
    extracting speaker embeddings via a quantized on-device model (e.g. ECAPA-TDNN ONNX,
    already assumed for Jetson in the config comments), and persisting voiceprints to a
    speaker_voiceprints table in the existing conversations.db.

  • Identify the speaker at runtime for each VAD-gated audio chunk before STT decodes
    it. Emit SpeakerIdentity::Known { name, confidence } or SpeakerIdentity::Unknown
    gated on a configurable similarity_threshold.

  • Propagate the resolved identity through voice_loop.rsserver.rs into memory
    writes, recall, tool dispatch, and conversation turn tags so every subsystem sees a
    consistent Option<SpeakerIdentity> per turn.

  • Expose enrollment and management via the already-stubbed genie-ctl speaker
    subcommands.

Config surface addition:

[speaker_identity.biometric]
model_path = "/opt/geniepod/models/ecapa_tdnn_q8.onnx"
similarity_threshold = 0.82
min_enrollment_samples = 5

Estimated files touched (11):

File Change
crates/genie-core/src/voice/identity.rs Replace stub with enrollment DB + embedding extraction + match scoring
crates/genie-core/src/voice/stt.rs Feed audio buffer to identity pipeline before STT decode
crates/genie-core/src/voice_loop.rs Thread SpeakerIdentity result into conversation context
crates/genie-core/src/server.rs Pass resolved identity to memory writes and tool dispatch
crates/genie-core/src/memory/mod.rs Enforce person-scoped writes using verified identity
crates/genie-core/src/memory/policy.rs Gate person-scoped reads on confirmed SpeakerIdentity::Known
crates/genie-core/src/tools/dispatch.rs Thread speaker identity through tool calls that reference it
crates/genie-core/src/conversation.rs Tag conversation turns with speaker name
crates/genie-common/src/config.rs Add BiometricConfig sub-struct under SpeakerIdentityConfig
crates/genie-ctl/src/main.rs Implement enroll / list / remove CLI commands
crates/genie-core/src/security/audit.rs Emit audit events for identity resolution and enrollment

Alternatives considered

  • Fixed identity (provider = "fixed") — works for single-user households but
    provides no differentiation and no fix for multi-user memory gating.
  • Telegram sender ID as identity — channel-specific, doesn't help for voice or HTTP
    sessions, and bypasses the voice privacy boundary entirely.
  • Cloud speaker diarization — contradicts the local-only audio posture; all audio
    must stay on-device.

Which milestone is this aimed at?

M3 — Home runtime boundary and skill safety

Privacy / security considerations

All audio and voiceprints stay on-device. The embedding model runs locally (ONNX on
Jetson); no audio, transcript, or biometric data leaves the box. Voiceprints are stored
in the existing conversations.db under the same file-permission boundary as the rest
of the runtime data. Enrollment events and identity resolution results are logged to
security/audit.rs for operator review. The similarity_threshold config lets operators
tune the false-accept rate for their household.


Additional context

The audio pipeline (ALSA capture, Silero VAD, chunking) and the ONNX runtime are already
present on Jetson. The gap is purely the enrollment store, embedding extraction call, and
identity propagation through the conversation context — no new hardware or external
dependencies are required.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions