Simplify the output of -->everyvoice/run_tests.py all

[marctessier]

Description & Motivation

It would be nice if some of the output from the test results be suppressed . In some of the tests we expect an ERROR and seeing the big red ERROR scroll down the screen might be confusion for some / scroll out of window history. ( Also , when we look at the separate output log , it's hard to know which test it was running. ( see example error / outlog ) .

TEST_latest.e233147.txt
TEST_latest.o233147.txt

I even wonder if we need to print any of the outputs and just do something like we do for the first couple of tests / be consistent with the style across the board ? ex:

test_command_help_messages (tests.test_cli.CLITest) ... ok
test_commands_present (tests.test_cli.CLITest) ... ok
test_evaluate (tests.test_cli.CLITest) ... ok

This is on of the test example below where maybe we should only print "ok" if successful ?

test_preprocessing_with_wrong_config (tests.test_cli.CLITest) ... ok

This is what we see...

test_preprocessing_with_wrong_config (tests.test_cli.CLITest)
The user should have a friendly message that informs them that they used the wrong config file type. ... 2024-10-31 15:49:23.109 | WARNING  | everyvoice.config.text_config:clean_symbols:11
7 - Normalization created a duplicate or inserted '' in dataset_0-symbols=[' ', 'c', 'e', 'h', 'k', 'p', 't', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'l', 'm', 'n', 'o', 'p', 'r', 's'
, 't', 'u', 'v', 'w', 'x', 'y']. Please check your shared-text config for problems.
2024-10-31 15:49:23.116 | WARNING  | everyvoice.config.text_config:clean_symbols:117 - Normalization created a duplicate or inserted '' in dataset_0-symbols=[' ', 'c', 'e', 'h', 'k', 'p', 
't', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'l', 'm', 'n', 'o', 'p', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y']. Please check your shared-text config for problems.
2024-10-31 15:49:23.117 | ERROR    | everyvoice.base_cli.helpers:load_config_base_command:110 - We are expecting a FastSpeech2Config but it looks like you provided a DFAlignerConfig
ok

Pitch

Make the test suit easier to read and make the "good" ERROR less scary.

Could also add a "Congratulations, 191 out of 191 tests ran with success!" Making it obvious that everything passed or how many..?

Below is what we currently see

Ran 191 tests in 63.556s

OK

We do have good output when things fail.... Example below, where I removed "sox" from my system :-)

** Side note, why 191 test above VS 163 tests FAILED (errors=3) ??? what happened to the other tests ( 28)?

======================================================================
ERROR: setUpClass (tests.test_dataloader.DataLoaderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/site-packages/torchaudio/_extension/utils.py", line 207, in wrapped
    _init_sox()
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/site-packages/torchaudio/_extension/utils.py", line 70, in _init_sox
    _load_lib("libtorchaudio_sox")
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/site-packages/torchaudio/_extension/utils.py", line 64, in _load_lib
    torch.ops.load_library(path)
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/site-packages/torch/_ops.py", line 852, in load_library
    ctypes.CDLL(path)
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libsox.so: cannot open shared object file: No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/gpfs/fs3c/nrc/dt/tes001/EveryVoice/everyvoice/tests/preprocessed_audio_fixture.py", line 69, in setUpClass
    PreprocessedAudioFixture.preprocessor.preprocess(
  File "/gpfs/fs3c/nrc/dt/tes001/EveryVoice/everyvoice/preprocessor/preprocessor.py", line 1148, in preprocess
    if filelist := self.process_all_audio():
  File "/gpfs/fs3c/nrc/dt/tes001/EveryVoice/everyvoice/preprocessor/preprocessor.py", line 576, in process_all_audio
    processed_item = self.process_one_audio(item, data_dir, sox_effects)
  File "/gpfs/fs3c/nrc/dt/tes001/EveryVoice/everyvoice/preprocessor/preprocessor.py", line 498, in process_one_audio
    input_audio, save_sr = self.process_audio(
  File "/gpfs/fs3c/nrc/dt/tes001/EveryVoice/everyvoice/preprocessor/preprocessor.py", line 198, in process_audio
    audio, sr = apply_effects_tensor(
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/site-packages/torchaudio/_extension/utils.py", line 209, in wrapped
    raise RuntimeError(
RuntimeError: apply_effects_tensor requires sox extension which is not available. Please refer to the stacktrace above for how to resolve this.

======================================================================
ERROR: test_hierarchy (tests.test_preprocessing.PreprocessingHierarchyTest)
Unit tests for preprocessing steps
----------------------------------------------------------------------
joblib.externals.loky.process_executor._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/site-packages/torchaudio/_extension/utils.py", line 207, in wrapped
    _init_sox()
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/site-packages/torchaudio/_extension/utils.py", line 70, in _init_sox
    _load_lib("libtorchaudio_sox")
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/site-packages/torchaudio/_extension/utils.py", line 64, in _load_lib
    torch.ops.load_library(path)
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/site-packages/torch/_ops.py", line 852, in load_library
    ctypes.CDLL(path)
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libsox.so: cannot open shared object file: No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/site-packages/joblib/externals/loky/process_executor.py", line 463, in _process_worker
    r = call_item()
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/site-packages/joblib/externals/loky/process_executor.py", line 291, in __call__
    return self.fn(*self.args, **self.kwargs)
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/site-packages/joblib/parallel.py", line 598, in __call__
    return [func(*args, **kwargs)
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/site-packages/joblib/parallel.py", line 598, in <listcomp>
    return [func(*args, **kwargs)
  File "/gpfs/fs3c/nrc/dt/tes001/EveryVoice/everyvoice/preprocessor/preprocessor.py", line 498, in process_one_audio
    input_audio, save_sr = self.process_audio(
  File "/gpfs/fs3c/nrc/dt/tes001/EveryVoice/everyvoice/preprocessor/preprocessor.py", line 198, in process_audio
    audio, sr = apply_effects_tensor(
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/site-packages/torchaudio/_extension/utils.py", line 209, in wrapped
    raise RuntimeError(
RuntimeError: apply_effects_tensor requires sox extension which is not available. Please refer to the stacktrace above for how to resolve this.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/gpfs/fs3c/nrc/dt/tes001/EveryVoice/everyvoice/tests/test_preprocessing.py", line 693, in test_hierarchy
    preprocessor.preprocess(
  File "/gpfs/fs3c/nrc/dt/tes001/EveryVoice/everyvoice/preprocessor/preprocessor.py", line 1148, in preprocess
    if filelist := self.process_all_audio():
  File "/gpfs/fs3c/nrc/dt/tes001/EveryVoice/everyvoice/preprocessor/preprocessor.py", line 562, in process_all_audio
    processed_items = Parallel(
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/site-packages/joblib/parallel.py", line 2007, in __call__
    return output if self.return_generator else list(output)
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/site-packages/joblib/parallel.py", line 1650, in _get_outputs
    yield from self._retrieve()
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/site-packages/joblib/parallel.py", line 1754, in _retrieve
    self._raise_error_fast()
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/site-packages/joblib/parallel.py", line 1789, in _raise_error_fast
    error_job.get_result(self.timeout)
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/site-packages/joblib/parallel.py", line 745, in get_result
    return self._return_or_raise()
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/site-packages/joblib/parallel.py", line 763, in _return_or_raise
    raise self._result
RuntimeError: apply_effects_tensor requires sox extension which is not available. Please refer to the stacktrace above for how to resolve this.

======================================================================
ERROR: setUpClass (tests.test_preprocessing.PreprocessingTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/site-packages/torchaudio/_extension/utils.py", line 207, in wrapped
    _init_sox()
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/site-packages/torchaudio/_extension/utils.py", line 70, in _init_sox
    _load_lib("libtorchaudio_sox")
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/site-packages/torchaudio/_extension/utils.py", line 64, in _load_lib
    torch.ops.load_library(path)
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/site-packages/torch/_ops.py", line 852, in load_library
    ctypes.CDLL(path)
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libsox.so: cannot open shared object file: No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/gpfs/fs3c/nrc/dt/tes001/EveryVoice/everyvoice/tests/preprocessed_audio_fixture.py", line 69, in setUpClass
    PreprocessedAudioFixture.preprocessor.preprocess(
  File "/gpfs/fs3c/nrc/dt/tes001/EveryVoice/everyvoice/preprocessor/preprocessor.py", line 1148, in preprocess
    if filelist := self.process_all_audio():
  File "/gpfs/fs3c/nrc/dt/tes001/EveryVoice/everyvoice/preprocessor/preprocessor.py", line 576, in process_all_audio
    processed_item = self.process_one_audio(item, data_dir, sox_effects)
  File "/gpfs/fs3c/nrc/dt/tes001/EveryVoice/everyvoice/preprocessor/preprocessor.py", line 498, in process_one_audio
    input_audio, save_sr = self.process_audio(
  File "/gpfs/fs3c/nrc/dt/tes001/EveryVoice/everyvoice/preprocessor/preprocessor.py", line 198, in process_audio
    audio, sr = apply_effects_tensor(
  File "/gpfs/fs3c/nrc/dt/tes001/MiniForge3/envs/EveryVoice/lib/python3.10/site-packages/torchaudio/_extension/utils.py", line 209, in wrapped
    raise RuntimeError(
RuntimeError: apply_effects_tensor requires sox extension which is not available. Please refer to the stacktrace above for how to resolve this.

----------------------------------------------------------------------
Ran 163 tests in 79.850s

FAILED (errors=3)

Alternatives

Could also add a --verbose mode to keep the current style , that could be handy if one of the test failed / give a clue why in an easy way?

Additional context

None .