diff --git a/.gitignore b/.gitignore index 96ef6c0..011d73e 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,3 @@ /target Cargo.lock +/tests/eval/local/ diff --git a/Cargo.toml b/Cargo.toml index 5b0a788..99fc26f 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -4,7 +4,7 @@ version = "0.1.0" edition = "2024" license = "AGPL-3.0-only" repository = "https://github.com/GeniePod/genie-voice-runtime" -description = "External voice runtime protocol and implementation for GeniePod Home." +description = "External voice runtime protocol and implementation for NVIDIA Jetson Orin 8GB." [lib] name = "genie_voice_runtime" diff --git a/README.md b/README.md index 8c23b51..d17f54b 100644 --- a/README.md +++ b/README.md @@ -47,9 +47,21 @@ GenieClaw to own audio details. this repo. - No Home Assistant or `genie-home-runtime` device logic in this repo. - No LLM provider logic in this repo. -- Voice hardware should be portable across Jetson, Raspberry Pi, other SBCs, - Linux laptops, and development machines where possible. -- Jetson / GeniePod Home remains the flagship tested deployment. +- Voice hardware should be portable across NVIDIA Jetson Orin 8GB, Raspberry + Pi, other SBCs, Linux laptops, and development machines where possible. +- NVIDIA Jetson Orin 8GB remains the flagship tested deployment. + +## Evaluation Data + +Public datasets can be used, but only for the layer they actually measure: + +- `genie-voice-runtime` owns audio, wake/VAD, STT/TTS, transcript quality, and + noisy utterance robustness. +- `genie-claw` owns BFCL tool-call scoring, family memory retrieval, + smart-home intent routing, and deterministic device-state questions. + +See [`docs/evaluation-data.md`](docs/evaluation-data.md) for the allowed data +sources, license notes, and where each dataset belongs. ## Status diff --git a/docs/evaluation-data.md b/docs/evaluation-data.md new file mode 100644 index 0000000..6b6cdcc --- /dev/null +++ b/docs/evaluation-data.md @@ -0,0 +1,53 @@ +# Evaluation Data Sources + +This repo can use public datasets for voice-runtime evaluation, but the data +must be mapped to the right boundary. `genie-voice-runtime` measures audio and +transcript quality. `genie-claw` measures tool-call accuracy, memory retrieval, +home-state reasoning, safety policy, and BFCL score. + +Do not add large public datasets to git. Download or generate them under a +local ignored directory such as `tests/eval/local/`, keep attribution metadata +with the local copy, and commit only small adapters, manifests, or synthetic +fixtures. + +## Usability Matrix + +| Dataset | License status | Use in `genie-voice-runtime` | Use in `genie-claw` / home agent | +|---------|----------------|------------------------------|-----------------------------------| +| [Home Assistant Intents](https://github.com/OHF-Voice/intents) | CC BY 4.0 in the source repo | Use as transcript text for STT normalization and command-phrase robustness. Do not evaluate device action correctness here. | Strong fit for BFCL `home_control`, `home_status`, timer, media, weather, and slot tests. | +| [CASAS Smart Home Data Sets](https://casas.wsu.edu/datasets/) | Current Zenodo CASAS records are CC BY 4.0; verify per record before importing. | Not a voice dataset. Use only to replay context labels around transcripts if a voice-session test needs realistic presence/activity context. | Best public source for real home sensor timelines, presence, action history, and activity-state questions. | +| [REFIT Electrical Load Measurements](https://pureportal.strath.ac.uk/en/datasets/refit-electrical-load-measurements-cleaned/) | CC BY 4.0 | Not a voice dataset. | Use for deterministic energy/device-state questions such as "what is using power" and appliance activity checks. | +| [UK-DALE](https://jack-kelly.com/data/) | CC BY 4.0 | Not a voice dataset. | Use for longer appliance electricity traces, energy summaries, and power anomaly tests. | +| [SLURP](https://github.com/pswietojanski/slurp) | Text annotations are CC BY 4.0. Audio hosted on Zenodo is CC BY-NC 4.0 unless a separate license is obtained. | Use text for assistant utterance robustness. Do not put SLURP audio into product CI or commercial distribution without license clearance. | Partial fit for assistant intent mapping. Not home-specific, so it should not dominate BFCL/home fixtures. | +| [MASSIVE](https://github.com/alexa/massive) | Dataset is CC BY 4.0; repo code is Apache 2.0. | Use text for multilingual transcript robustness and slot extraction from noisy STT-like text. | Partial fit for multilingual tool-routing and slot tests after mapping intents to Genie typed tools. | +| [Google Speech Commands](https://www.tensorflow.org/datasets/catalog/speech_commands) | CC BY 4.0 for dataset content | Use for wake-word, keyword spotting, command-word false-positive checks, and Jetson latency/memory smoke tests. | Not enough semantic structure for BFCL beyond very small command-word smoke fixtures. | + +## Project Policy + +- Prefer CC BY 4.0 sources that allow product use with attribution. +- Keep source license, citation, download URL, version, and conversion script + together in the local eval manifest. +- Treat noncommercial data as research-only unless the project obtains a + separate license. This applies to SLURP audio. +- Do not train or tune agent prompts by dumping large dataset text into the + runtime prompt. Convert datasets into small typed fixtures and measure the + score. +- For Jetson Orin 8GB validation, keep generated eval subsets small enough to + run locally without memory pressure hiding runtime bugs. + +## Recommended Mapping + +1. Use Home Assistant Intents to generate GenieClaw BFCL cases for typed home + tools, then run those predictions through `genie-ctl bfcl-score`. +2. Use CASAS timelines to synthesize realistic home context before a transcript: + presence, room activity, device history, and sensor state. +3. Use REFIT and UK-DALE to build deterministic energy/state tests for home + memory and typed device-status tools. +4. Use MASSIVE and SLURP text to stress transcript normalization, misspellings, + multilingual phrasing, and assistant-style slot extraction. +5. Use Google Speech Commands only for voice-runtime wake/keyword and low-level + audio robustness, not as a home-agent correctness benchmark. + +The expected result is not a single public benchmark. It is a layered eval +suite: audio robustness in `genie-voice-runtime`, typed tool-call accuracy in +`genie-claw`, and real home state replay through CASAS/energy traces.