STT downsampling has insufficient alias rejection since #825

[mzau]

STT downsampling has insufficient alias rejection since #825

load_audio: decoder-side downsampling has insufficient alias rejection for miniaudio-backed
input since #825

Description

Since 0.4.6 (#825), mlx_audio.stt.utils.load_audio passes the target rate to the decoder
instead of decoding natively and calling the existing Kaiser-windowed-sinc resample_audio:

# 0.4.4
audio, sample_rate = audio_read(file, always_2d=True)
if sample_rate != sr:
    audio = resample_audio(audio, sample_rate, sr)
return mx.array(audio, dtype=dtype).mean(axis=1)

# 0.4.6+
audio, _ = audio_read(file, dtype="float32", sample_rate=sr, nchannels=1)
return mx.array(audio, dtype=dtype)

For miniaudio-backed formats this selects miniaudio's default decoder-side resampling.
miniaudio does apply a low-pass before decimating, so this is not a missing anti-aliasing
filter. The problem is that its stopband attenuation is far too shallow for ASR front-end use:
a 12 kHz tone downsampled 44.1 → 16 kHz survives at −22.4 dB and lands at 4 kHz, in the middle
of the speech band.

resample_audio is unchanged and still correct. It is simply no longer on this path.

Measurement

Both versions installed side by side, same machine, same miniaudio (1.71), through the public
entry point load_audio(path, sr=16000). Input: 2 s tone, 44.1 kHz, 16-bit mono WAV. The
16-bit quantisation floor for a −6 dBFS tone is ≈ −96 dB; values there mean the tone is gone.

Stopband — this is the defect

Tones above the 8 kHz target Nyquist. Any output energy is alias.

tone	folds to	0.4.4	0.4.7
8.5 kHz	7500 Hz	−96.01 dB	−5.61 dB
9 kHz	7000 Hz	−94.81 dB	−7.60 dB
10 kHz	6000 Hz	−95.69 dB	−12.21 dB
11 kHz	5000 Hz	−97.53 dB	−17.22 dB
12 kHz	4000 Hz	−95.88 dB	−22.42 dB
14 kHz	2000 Hz	−95.77 dB	−33.41 dB
15 kHz	1000 Hz	−94.14 dB	−39.40 dB
18 kHz	2000 Hz	−94.99 dB	−61.70 dB
20 kHz	4000 Hz	−96.56 dB	−75.33 dB

The progressive rolloff confirms a filter is present and working — it is simply far too gentle
near the cutoff. On 0.4.4 every entry sits at the test file's noise floor, and the FFT peak
lands at an arbitrary frequency rather than the predicted alias, i.e. no tone survives at all.

The most damaging region is just above cutoff: −5.6 dB at 8.5 kHz means a 44.1 kHz source
with ordinary energy slightly above 8 kHz folds back at roughly half amplitude.

Level control — alias, not noise floor

Dropping the input 20 dB separates a fixed noise floor from level-proportional leakage:

	0.4.4 out/in	0.4.4 absolute	0.4.7 out/in	0.4.7 absolute
amp 0.5	−95.88 dB	−104.91 dBFS	−22.42 dB	−31.45 dBFS
amp 0.05	−75.11 dB	−104.14 dBFS	−22.41 dB	−51.44 dBFS

0.4.4 pins the absolute level at ≈ −105 dBFS regardless of input — the 16-bit file's own
quantisation noise, so its measured attenuation is bounded by the test signal, not the filter.
Called directly on float64 with no int16 round-trip, resample_audio puts the same tone at
−159.7 dB. 0.4.7 holds the ratio at −22.4 dB while the absolute level tracks input 1:1.

Passband — also changed, in both directions

tone	0.4.4	0.4.7
1 kHz	−0.00 dB	−0.02 dB
3 kHz	−0.00 dB	−0.13 dB
5 kHz	−0.00 dB	−0.42 dB
6 kHz	−0.00 dB	−0.80 dB
7 kHz	−0.00 dB	−1.77 dB
7.5 kHz	−3.10 dB	−2.67 dB
7.9 kHz	−36.19 dB	−3.66 dB

The FIR path is flat to 7 kHz and then transitions sharply; the decoder path starts drooping
around 5–6 kHz and is still passing −3.7 dB at 7.9 kHz. Both the gentler passband droop and the
weak stopband are the same trade-off — a short IIR where the ASR path previously had a long FIR.

Backend split — ffmpeg-routed containers are NOT affected

audio_io.read sends m4a/aac/ogg/opus/webm to _decode_ffmpeg, which resamples via
ffmpeg -ar. Same 12 kHz tone, encoded as ALAC in .m4a — lossless, so the codec cannot
itself remove the tone, but routed to ffmpeg by extension:

0.4.7	out/in	peak
WAV (miniaudio)	−22.42 dB	4000 Hz
M4A/ALAC (ffmpeg)	−95.54 dB	1500 Hz (noise)

So the regression is specific to the miniaudio-backed path: WAV, MP3, FLAC. Container
formats going through ffmpeg are clean.

Scope

Confirmed by measurement: downsampling of miniaudio-backed input through
mlx_audio.stt.utils.load_audio, which is what Whisper uses —
stt/models/whisper/audio.py:66, stt/models/whisper/whisper.py:760, :1342.

Not affected: input already at the target rate (no resampling happens), and ffmpeg-routed
containers as measured above.

Same mechanism, not measured here: the general loader mlx_audio/utils.py::load_audio
received the same change in #825 and now also calls
audio_read(..., sample_rate=..., nchannels=1). It is imported by convert.py,
codec/models/stepaudio2/token2wav.py, tts/generate.py and a number of TTS models, several
of which load reference audio at rates that require downsampling. Worth checking, but this
report makes no claim about it.

Reproduction

Self-contained; writes only inside a temporary directory. Run under 0.4.4 and 0.4.6+ and
compare — it prints every table in this report.

"""Alias rejection of stt.utils.load_audio when downsampling 44.1 kHz -> 16 kHz."""
import os, platform, subprocess, shutil, struct, sys, tempfile, wave
import numpy as np
from importlib.metadata import version
from mlx_audio.stt.utils import load_audio

SRC, DST = 44100, 16000          # target Nyquist = 8000 Hz


def tone_wav(path, freq, amp, seconds=2.0):
    t = np.arange(int(SRC * seconds)) / SRC
    sig = (amp * np.sin(2 * np.pi * freq * t)).astype(np.float32)
    with wave.open(path, "w") as w:
        w.setnchannels(1); w.setsampwidth(2); w.setframerate(SRC)
        w.writeframes(struct.pack("<%dh" % len(sig), *(sig * 32767).astype(np.int16)))
    return np.sqrt(np.mean(sig.astype(np.float64) ** 2))


def through_loader(path, in_rms):
    out = np.asarray(load_audio(path, sr=DST), dtype=np.float64).ravel()
    out = out[len(out) // 4: -len(out) // 4]                  # drop filter edges
    rms = np.sqrt(np.mean(out ** 2))
    spec = np.abs(np.fft.rfft(out * np.hanning(len(out))))
    peak = np.fft.rfftfreq(len(out), 1 / DST)[int(np.argmax(spec))]
    return (20 * np.log10(max(rms, 1e-15) / in_rms),
            20 * np.log10(max(rms, 1e-15)), peak)


print(f"mlx-audio {version('mlx-audio')} | miniaudio {version('miniaudio')} | "
      f"{platform.system()}-{platform.machine()} | Python {sys.version.split()[0]}")
print(f"{SRC} Hz -> {DST} Hz via stt.utils.load_audio, 16-bit mono WAV")
print("16-bit quantisation floor for a -6 dBFS tone is ~-96 dB; values there mean 'removed'\n")

with tempfile.TemporaryDirectory() as td:
    p = os.path.join(td, "tone.wav")

    print("[1] STOPBAND  tones above 8 kHz -> any output IS alias")
    print(f"    {'tone':>6}  {'alias at':>8}  {'out/in dB':>10}  {'peak':>10}")
    for f in (8500, 9000, 10000, 11000, 12000, 14000, 15000, 18000, 20000):
        rel, _, pk = through_loader(p, tone_wav(p, f, 0.5))
        print(f"    {f:6d}  {abs(DST - f) if f < 2*DST else f - 2*DST:8d}  {rel:10.2f}  {pk:7.0f} Hz")

    print("\n[2] PASSBAND  tones below 8 kHz -> should pass near 0 dB")
    print(f"    {'tone':>6}  {'out/in dB':>10}")
    for f in (1000, 3000, 5000, 6000, 7000, 7500, 7900):
        rel, _, _ = through_loader(p, tone_wav(p, f, 0.5))
        print(f"    {f:6d}  {rel:10.2f}")

    print("\n[3] LEVEL CONTROL  12 kHz at two amplitudes")
    print("    fixed ABSOLUTE level => noise floor (tone removed)")
    print("    fixed RATIO         => level-proportional alias (tone leaking)")
    print(f"    {'amp':>6}  {'out/in dB':>10}  {'absolute dBFS':>14}")
    for amp in (0.5, 0.05):
        rel, absolute, _ = through_loader(p, tone_wav(p, 12000, amp))
        print(f"    {amp:6.2f}  {rel:10.2f}  {absolute:14.2f}")

    print("\n[4] BACKEND SPLIT  same 12 kHz tone, miniaudio vs ffmpeg routing")
    in_rms = tone_wav(p, 12000, 0.5)
    rel, _, pk = through_loader(p, in_rms)
    print(f"    {'WAV (miniaudio)':<26} {rel:8.2f} dB   peak {pk:6.0f} Hz")
    if shutil.which("ffmpeg") and shutil.which("ffprobe"):
        m4a = os.path.join(td, "tone.m4a")     # ALAC: lossless, but routed to ffmpeg by extension
        subprocess.run(["ffmpeg", "-loglevel", "error", "-y", "-i", p, "-c:a", "alac", m4a],
                       check=True)
        rel, _, pk = through_loader(m4a, in_rms)
        print(f"    {'M4A/ALAC (ffmpeg)':<26} {rel:8.2f} dB   peak {pk:6.0f} Hz")
    else:
        print("    M4A/ALAC (ffmpeg)          skipped - ffmpeg/ffprobe not on PATH")

    print("\n[5] FILTER ITSELF  resample_audio called directly, float64, no int16 round-trip")
    t = np.arange(int(SRC * 2.0)) / SRC
    sig = 0.5 * np.sin(2 * np.pi * 12000.0 * t)
    from mlx_audio.utils import resample_audio
    out = np.asarray(resample_audio(sig, SRC, DST, axis=0), dtype=np.float64).ravel()
    out = out[len(out) // 4: -len(out) // 4]
    ref = np.sqrt(np.mean(sig ** 2))
    print(f"    resample_audio 12 kHz      {20*np.log10(max(np.sqrt(np.mean(out**2)),1e-18)/ref):8.2f} dB"
          f"   (unchanged in all versions; shows what the STT path used to get)")

Why CI stayed green

tests/test_dsp.py::test_resample_rejects_energy_above_target_nyquist still passes: it calls
resample_audio directly, and #825 did not change that function — it changed which code path
calls it.

def test_resample_rejects_energy_above_target_nyquist():
    """A tone just above the new Nyquist must be band-limited away, not aliased
    back into the signal. ..."""
    out = np.asarray(resample_audio(tone, orig, target))
    assert rms < 0.01  # sharp filter -> ~0; the old default left ~0.26

The property that assertion locks in no longer holds for audio loaded through the STT path,
because the assertion no longer reaches it. A regression test at the loader level would:

1. write a temporary 44.1 kHz mono WAV with a 12 kHz tone,
2. load it via stt.utils.load_audio(..., sr=16000),
3. trim the edges, then check RMS or the 4 kHz FFT bin,
4. assert against a threshold that accepts the FIR path and rejects the current ≈ −22 dB.

Extending it to 48 → 16 kHz, and to the MP3/FLAC miniaudio paths, would cover the same
mechanism; ffmpeg-routed containers need their own case rather than being assumed equivalent.

Possible fixes

1. Decode at native rate and resample through resample_audio, keeping Improve audio loading performance #825's single-pass
   channel conversion (nchannels=1) so only the rate conversion returns to the FIR path.
2. Pass the target rate to the decoder only when it equals the source rate — audio_io.read
   already knows the native rate via get_file_info for miniaudio files — and fall back to
   resample_audio otherwise. Needs care for file objects and the ffmpeg branch.
3. Raise the decoder-side resampler quality. Note this is not just an extra argument:
   pyminiaudio's decode_file() exposes only format, channels, rate and dither, so it would
   need a lower-level decoder config, and an IIR of any supported order would still have to be
   measured against the existing FIR rather than assumed equivalent.

Versions

mlx-audio 0.4.4 and 0.4.7 (behaviour introduced in 0.4.6, #825), miniaudio 1.71,
macOS/arm64, Python 3.10.18.

[linear-code]

BLA-474

[lucasnewman]

@mzau Thanks for the detailed issue! I've put up a fix in #872