`MediaStream` linked to a `MediaRecorder` without an AudioContext in between results in a runaway recording

[CallMeMSL]

Hi there,
thank you for your work on this great library!

I want to use this library for a Uni Project where I do live transcription and analyze spoken sentences with an LLM.
Somehow recording the mic doesn't seem to work.
This code produces an 11 Minute .wav file that blasts your ears away with garbage:

fn main() {
    let mic = media_devices::get_user_media_sync(MediaStreamConstraints::Audio);
    let recorder = MediaRecorder::new(&mic);
    let recording: Arc<Mutex<Vec<u8>>> = Default::default();
    let arc = Arc::clone(&recording);
    recorder.set_ondataavailable(move |mut event| {
        let mut r = arc.lock().unwrap();
        r.append(&mut event.blob);
    });
    recorder.start();
    std::thread::sleep(std::time::Duration::from_secs(4));
    recorder.stop();
    std::thread::sleep(std::time::Duration::from_secs(1));
    let data = recording.lock().unwrap();
    let mut file = fs::OpenOptions::new()
        .create(true)
        .write(true)
        .open("unprocessed.wav").unwrap();
    file.write_all(&data).unwrap();
}

Is there something obvious that I am missing? The example for get_user_media_sync for the docs works without any problems and I can hear myself clearly and without any latency.

I am using Linux with PipeWire.

Kind Regards,
David

[orottier]

Thanks for the report. This is very interesting since you are combining the MediaDevices API and the MediaRecorder API, without an AudioContext in between. This is of course fine on the web, but not something we have really accounted for in our implementation - which main focus is the Web Audio API.

Tacking an AudioContext in between makes it work:

use std::sync::{Arc, Mutex};
use web_audio_api::context::{AudioContext, AudioContextLatencyCategory, AudioContextOptions};
use web_audio_api::media_devices;
use web_audio_api::media_devices::MediaStreamConstraints;
use web_audio_api::media_recorder::MediaRecorder;
use web_audio_api::node::AudioNode;

// If you are on Linux and use ALSA as audio backend backend, you might want to run
// the example with the `WEB_AUDIO_LATENCY=playback ` env variable which will
// increase the buffer size to 1024
//
// `WEB_AUDIO_LATENCY=playback cargo run --release --example recorder
fn main() {
    let latency_hint = match std::env::var("WEB_AUDIO_LATENCY").as_deref() {
        Ok("playback") => AudioContextLatencyCategory::Playback,
        _ => AudioContextLatencyCategory::default(),
    };

    let context = AudioContext::new(AudioContextOptions {
        latency_hint,
        ..AudioContextOptions::default()
    });

    // connect the mic via an AudioContext media_stream_source to an media_stream_destination
    let mic = media_devices::get_user_media_sync(MediaStreamConstraints::Audio);
    let stream_source = context.create_media_stream_source(&mic);
    let dest = context.create_media_stream_destination();
    stream_source.connect(&dest);

    // record the media_stream_destination
    let recorder = MediaRecorder::new(dest.stream());
    let recording: Arc<Mutex<Vec<u8>>> = Default::default();
    let arc = Arc::clone(&recording);
    recorder.set_ondataavailable(move |mut event| {
        let mut r = arc.lock().unwrap();
        r.append(&mut event.blob);
    });
    recorder.start();
    std::thread::sleep(std::time::Duration::from_secs(4));
    recorder.stop();
    std::thread::sleep(std::time::Duration::from_secs(1));
    let data = recording.lock().unwrap();
    let mut file = std::fs::OpenOptions::new()
        .create(true)
        .write(true)
        .open("unprocessed.wav")
        .unwrap();
    file.write_all(&data).unwrap();
}

You might wonder what's happening in your example and why it produces a large wav with garbage. The AudioContext in our library is responsible for timing of the render quantums to produce e.g. 48000 samples per second. We have taken some shortcuts in implementing the MediaStream API that results in them supplying samples on demand, instead of having an internal clock themselves. When you hook the mic directly to the recorder, the recorders polls for samples continuously. The stream is trying to provide them ASAP, and inserting silent frames whenever the underlying source (mic in this case) cannot provide them. Since there is no AudioContext clock to tame the recorder polling, the result is a giant wav file (the faster your computer, the larger the file). The noise is because it is mixing true signal with silent gaps.

In any case this is something we need to address. For now remember to actually use the Web Audio API when using our library :)

NB: the examples/recorder.rs file has a somewhat better implementation than your 1000ms delay to flush the recorder.
NB2: some audio players (e.g. iTunes) don't display the length of the wav files correctly in case of 'streaming wavs'. Don't let this confuse you, it plays just fine

[CallMeMSL]

Thank you so much for the thorough response!

the examples/recorder.rs file has a somewhat better implementation than your 1000ms delay to flush the recorder.

Thank you for the hint. I plan to send the blobs directly into the preprocessing and then into the neural network over channels. The start/stop function will just be controlled with a GUI.

One last question if you don't mind. Part of my preprocessing is, that the audio needs to be resampled to 15kHz.

The example code for resampling just uses the default audio context sample rate.
However, if I adjust the sample rate of the audio context in your example above like this:

    let context = AudioContext::new(AudioContextOptions {
        latency_hint,
        sample_rate: Some(15_000.0),
        ..AudioContextOptions::default()
    });

Construction of the audio context panics with this stack trace:

thread 'main' panicked at /home/david/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rustfft-6.1.0/src/common.rs:25:5:
assertion `left == right` failed: Input FFT buffer must be a multiple of FFT length. Expected multiple of 639, got len = 1124
  left: 485
 right: 0
stack backtrace:
   0: rust_begin_unwind
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:597:5
   1: core::panicking::panic_fmt
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/panicking.rs:72:14
   2: core::panicking::assert_failed_inner
   3: core::panicking::assert_failed
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/panicking.rs:270:5
   4: rustfft::common::fft_error_inplace
             at /home/david/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rustfft-6.1.0/src/common.rs:25:5
   5: <rustfft::avx::avx_mixed_radix::MixedRadix9xnAvx<A,T> as rustfft::Fft<T>>::process_with_scratch
             at /home/david/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rustfft-6.1.0/src/avx/mod.rs:206:21
   6: rustfft::Fft::process
             at /home/david/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rustfft-6.1.0/src/lib.rs:188:9
   7: hrtf::make_hrtf
             at /home/david/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hrtf-0.8.1/src/lib.rs:256:5
   8: hrtf::HrtfSphere::new::{{closure}}
             at /home/david/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hrtf-0.8.1/src/lib.rs:636:33
   9: core::iter::adapters::map::map_try_fold::{{closure}}
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/iter/adapters/map.rs:91:28
  10: core::iter::traits::iterator::Iterator::try_fold
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/iter/traits/iterator.rs:2461:21
  11: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/iter/adapters/map.rs:117:9
  12: <I as alloc::vec::in_place_collect::SpecInPlaceCollect<T,I>>::collect_in_place
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/alloc/src/vec/in_place_collect.rs:258:13
  13: alloc::vec::in_place_collect::<impl alloc::vec::spec_from_iter::SpecFromIter<T,I> for alloc::vec::Vec<T>>::from_iter
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/alloc/src/vec/in_place_collect.rs:182:28
  14: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/alloc/src/vec/mod.rs:2749:9
  15: core::iter::traits::iterator::Iterator::collect
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/iter/traits/iterator.rs:2053:9
  16: hrtf::HrtfSphere::new
             at /home/david/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hrtf-0.8.1/src/lib.rs:632:22
  17: hrtf::HrtfProcessor::new
             at /home/david/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hrtf-0.8.1/src/lib.rs:871:27
  18: web_audio_api::node::panner::load_hrtf_processor::{{closure}}
             at /home/david/.cargo/registry/src/index.crates.io-6f17d22bba15001f/web-audio-api-0.37.0/src/node/panner.rs:36:29
  19: std::collections::hash::map::Entry<K,V>::or_insert_with
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/collections/hash/map.rs:2560:43
  20: web_audio_api::node::panner::load_hrtf_processor
             at /home/david/.cargo/registry/src/index.crates.io-6f17d22bba15001f/web-audio-api-0.37.0/src/node/panner.rs:27:5
  21: web_audio_api::context::concrete_base::ConcreteBaseAudioContext::new
             at /home/david/.cargo/registry/src/index.crates.io-6f17d22bba15001f/web-audio-api-0.37.0/src/context/concrete_base.rs:245:13
  22: web_audio_api::context::online::AudioContext::new
             at /home/david/.cargo/registry/src/index.crates.io-6f17d22bba15001f/web-audio-api-0.37.0/src/context/online.rs:181:20
  23: mic::main
             at ./examples/mic.rs:20:19
  24: core::ops::function::FnOnce::call_once
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Is there anything I can do about this?

I already tried to just change the sample rate in pipewire, but the audio context keeps using 44.1kHz as a default.

[orottier]

Thanks for the additional report. This is actually something we have been aware of recently but we haven't put out a mitigation yet because we figured no one would use sample rates that low. See 8688906

I will release a fix for it today