AnCoGen: Analysis, Control and Generation of Speech with a Masked Autoencoder

This repository contains the code associated with the following publication:

AnCoGen: Analysis, Control and Generation of Speech with a Masked Autoencoder
Samir Sadok, Simon Leglaive, Laurent Girin, Gaël Richard, Xavier Alameda-Pineda
International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2025.

If you use this code for your research, please cite the above paper.

Useful links:

• Abstract
• Paper
• Demo website with qualitative results

Setup

• Pypi:
  • pip install -i https://test.pypi.org/simple/ ancogen --no-deps
• Install the package locally (for use on your system):
  • In the current directory: pip install -e .
• Virtual Environment:
  • conda create -n ancogen python=3.9
  • conda activate ancogen
  • In the current directory: pip install -r requirements.txt

Usage

Pretrained models

After loading the weights of the pre-trained models: speechVQVAE, HIFIGAN and AnCoGen, put them all in the NestAnCoGen class.

from src import NestAnCoGen

ancogen = NestAnCoGen(ancogen=model, hifigan=generator, vqvae=vqvae, improved=False)

Model	Link
Speech-VQVAE	link
HiFi-GAN	link
AnCoGen	link / link (Improved)

Analysis

To do analysis with AnCoGen (link, test_analyse), which correspond to the estimation of the speech attributes from a Mel-spectrogram. Please see the paper for a complete description of the attributes.

"""
Test the analyse function of AnCoGen. 
"""
PATH_AUDIO = "path_wav_signal.wav"

# Preprocess the audio
audio = ancogen.preprocess(PATH_AUDIO)

# Analyse the audio with the AnCoGen
audio, attributes = ancogen.analyse(audio, apply_max=True)

# Pitch estimation + plotting
audio, attributes = ancogen.analyse(audio, apply_max=True, attribute_name="pitch", plot_bool=True)

Generation

To do speech analysis-resynthesis mapping wih AnCoGen (link, test_generation) which are simply obtained by using AnCoGen to map a Mel-spectrogram to the corresponding speech attributes (analysis stage) and then back to the Mel-spectrogram (generation stage).

"""
Test the generation function of AnCoGen. 
"""
PATH_AUDIO = "path_wav_signal.wav"

# Generate output
generated = ancogen.generate(path=PATH_AUDIO, from_attributes=None, save_dir="wavs", return_metrics=True)

Control

To do analysis, transformation, and synthesis with AnCoGen, where the speech attributes are controlled between the analysis and generation stages in order to perform speech denoising (by increasing the SNR attribute), pitch shifting, dereverberation (by increasing the C50 attribute) or voice conversion (by controlling the speaker identity attribute).

"""
Test the control function of AnCoGen. 
"""
PATH_AUDIO = "path_wav_signal.wav"

PATH_AUDIO = "path_wav_signal.wav"

# Preprocess the audio
audio = ancogen.preprocess(PATH_AUDIO)

# Analyse the audio with the AnCoGen
audio, attributes = ancogen.analyse(audio, apply_max=True)

# Control with AnCoGen
ancogen.pitch_control(attributes, target_pitch, **kwargs) 

ancogen.content_control(attributes, target_content_index, **kwargs)

ancogen.snr_control(attributes, target_snr, **kwargs)

ancogen.c50_control(attributes, target_c50, **kwargs)

ancogen.voice_conversion(target_identity: str, source_signal: str, save_dir: str = '')

License

GNU Affero General Public License (version 3), see LICENSE.txt.