pytorch/audio_classification/understand_audio_data_and_concepts.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Audio data\n",
    "\n",
    "Here are some key concepts and features of audio data. As an audio signal is the vibration generated when sound passes through air. For audio, when sound is captured from a microphone, it's an analog form. The analog sound is converted to digital sound format by sampling at consistent intervals of time. The number of audio data points recorded every second is called the `sample rate`.\n",
    "* The higher sample rate, the higher the quality of the sound. However, after a certain point, the human ear can't detect the difference.\n",
    "* The average sound sample rate is 48 kilohertz(KHz) or 48,000 samples per second. The dataset we'll be using in this module was sampled at 16KHz, so our sample rate is 16,000 samples per second.\n",
    "\n",
    "When the audio is sampled, the **frequency** of the sound is the number of times per second that a sound wave repeats itself. The `amplitude` is how loud the audio is. we can take our sample rate and frequency, and represent the signal visually. This visual signal can be represented as a `waveform`, which is the signal representation over time in a graphical format. The audio can be recorded in different channel. For example, stereo recordings have two channels, right and left.\n",
    "\n",
    "Now let's take a moment to understand how we might want to parse out a file. For example, if you have longer audio files, you might want to split it out into frames, or sections, of the audio to be classified indivisually. For this dataset, we don't need to set any frames of our audio samples, because each sample is only one second and one word. Another processing step might be an offset, which means the number of frames from the start of the file to befin data loading.\n",
    "\n",
    "## Get set up with TorchAudio\n",
    "\n",
    "TorchAudio is a library that is part of the PyTorch ecosystem. It has I/O functionality, popular open datasets, and common audio transformations that we'll need to build our model. We will use this library to work weith our audio data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import torchaudio\n",
    "import IPython.display as ipd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the speech commands sample dataset, and download the full dataset in a local directory. Note that we're only using `yes` or `no` classes to create a binary classification model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget -P ./data/SpeechCommands/speech_commands_v0.02/ http://download.tensorflow.org/data/speech_commands_v0.02.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tar -xzf ./data/SpeechCommands/speech_commands_v0.02/speech_commands_v0.02.tar.gz -C ./data/SpeechCommands/speech_commands_v0.02/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainset = torchaudio.datasets.SPEECHCOMMANDS(\"./data\", download=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize the classes available in the dataset\n",
    "\n",
    "There are 36 audio classes in the speech commands dataset, each sampled at 16 KHZ, Let's print the different commands avaliable. The two classes we'll be using are `yes` and `no` commands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "default_dir='/kaggle/working'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.chdir(f'./data/SpeechCommands/speech_commands_v0.02')\n",
    "\n",
    "labels = [name for name in os.listdir('.') if os.path.isdir(name)]\n",
    "# back to default directory\n",
    "os.chdir(default_dir)\n",
    "print(f'Total Lables: {len(labels)}\\n')\n",
    "print(f'Labels: {labels}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conver the sound to tensor\n",
    "\n",
    "You likely have used a wave file before. It's one format in which we save our digital representaion of our analog audio to be shared and played. The speech commands dataset that we will be using in this tutorial is stored in wave files that are all one second or less.\n",
    "\n",
    "Let's load up one of the wave files and take a look at how the tensors for the `waveform` look. We're loading the files by using `torchaudio.load`, which loads an audio file into a `torch.Tensor` object. TorchAudio takes care of the implmentation, so you don't need to worry about it. The `torch.load` function returns the waveform as a tensor, and an `int` of the `sample_rate`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_name = './data/SpeechCommands/speech_commands_v0.02/yes/00f0204f_nohash_0.wav'\n",
    "waveform, sample_rate = torchaudio.load(file_name,num_frames=3)\n",
    "print(f'waveform tensor with 3 frames: {waveform}\\n')\n",
    "waveform, sample_rate = torchaudio.load(file_name, num_frames=3, frame_offset=2)\n",
    "print(f'waveform tensor with 2 frames offset: {waveform}\\n')\n",
    "waveform, sample_rate = torchaudio.load(file_name)\n",
    "print(f'waveform tensor with all frames: {waveform}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot the waveform\n",
    "\n",
    "Next, create a `plot_audio` function to display the waveform, and listen to a sample of each class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_audio(filename):\n",
    "    waveform, sample_rate = torchaudio.load(filename)\n",
    "    print(\"Shape of waveform: {}\".format(waveform.size()))\n",
    "    print(\"Sample rate of waveform: {}\".format(sample_rate))\n",
    "\n",
    "    plt.figure()\n",
    "    plt.plot(waveform.t().numpy())\n",
    "\n",
    "    return waveform, sample_rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The size of the audio is 1x16000. This means that it's 1 second long, at a sample rate of 16000. We can see the graphical waveform of the sound captured, and play the audio for the `yes` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = './data/SpeechCommands/speech_commands_v0.02/yes/00f0204f_nohash_0.wav'\n",
    "waveform, sample_rate = plot_audio(filename)\n",
    "ipd.Audio(waveform.numpy(), rate=sample_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's look at the waveform and audio for the `no` command. As you can see, the waveform differs from the sound pattern for the `yes` command. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = './data/SpeechCommands/speech_commands_v0.02/no/0b40aa8e_nohash_0.wav'\n",
    "waveform, sample_rate = plot_audio(filename)\n",
    "ipd.Audio(waveform.numpy(), rate=sample_rate)"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}