ML-maximum likelihood estimation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "20fce00f",
   "metadata": {},
   "source": [
    "### Machine Learning\n",
    "#### Maximum Likelihood Estimation\n",
    "Given a number of samples, say $n$, received from the environment, and knowing their joint probability function denoted by $f_n(\\boldsymbol{x};\\boldsymbol{w})$, we can define the **likelihood function** as shown below:\n",
    "<br>$L(\\boldsymbol{x},\\boldsymbol{w})=f_n(\\boldsymbol{x};\\boldsymbol{w})$  (1)\n",
    "<br>where $\\boldsymbol{w}$ is generally a vector of unknown parameters of the joint probability function $f_n(.;.)$.\n",
    "<br>Assuming the samples are *independent and identically distributed* (i.i.d.) random variables, we can simplify Eq. (1) to this:\n",
    "<br>$L(\\boldsymbol{x};\\boldsymbol{w})=\\prod_{i=1}^{n}f(x_i;\\boldsymbol{w})$ (2)\n",
    "<br>where $f(x_i;\\boldsymbol{w})$ is the probability function of $x_i$ with parameter vector $\\boldsymbol{w}$ \n",
    "<br>The **Maximum likelihood estimation** finds the optimal $\\boldsymbol{w}$ from:\n",
    "<br>$\\boldsymbol{w}=argmax_\\boldsymbol{w} L(\\boldsymbol{x};\\boldsymbol{w})$  (3)\n",
    "<br>If the likelihood function is *differentiable*, sufficient conditions for the **MLE** are:\n",
    "<br>$\\frac{\\partial L}{\\partial w_0}=0$, $\\frac{\\partial L}{\\partial w_1}=0$,...,$\\frac{\\partial L}{\\partial w_{q-1}}=0$   (4)\n",
    "<br><br>**Contents:**\n",
    "- Estimation by the MLE for samples from a **continuous uniform distribution**.\n",
    "- Estimation by the MLE for samples from a **normal distribution**.\n",
    "<hr>\n",
    "https://github.com/ostad-ai/Machine-Learning\n",
    "<br> Explanation: https://www.pinterest.com/HamedShahHosseini/Machine-Learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "84c6d039",
   "metadata": {},
   "outputs": [],
   "source": [
    "# importing the required modules\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccd9bb5d",
   "metadata": {},
   "source": [
    "It can be shown that for samples of **i.i.d.** random variables taken from a **uniform distribution**,\n",
    "say $f(x_i,;w_0,w_1)$, the **MLE** leads to the following parameter estimation:\n",
    "<br> $w_0=min(x_1,x_2,...,x_n)$ and $w_1=max(x_1,x_2,...,x_n)$\n",
    "<br> In the following example, we investigate these formulae:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5acd3169",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We get 100 samples from a uniform distribution\n",
      "The truth values of parameters: a=3 and b=10\n",
      "The estimated parameters by MLE: a=3.016066812589626,b=9.723057963740676\n"
     ]
    }
   ],
   "source": [
    "# estimation of parameters for continuous uniform distribution by the MLE\n",
    "def MLE_uniform(samples):\n",
    "    w0=np.min(samples)\n",
    "    w1=np.max(samples)\n",
    "    return w0,w1\n",
    "\n",
    "#---example\n",
    "a,b,n=3,10,100\n",
    "samples=a+(b-a)*np.random.random(120)\n",
    "w0,w1=MLE_uniform(samples)\n",
    "print(f'We get {n} samples from a uniform distribution')\n",
    "print(f'The truth values of parameters: a={a} and b={b}')\n",
    "print(f'The estimated parameters by MLE: a={w0},b={w1}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59830c63",
   "metadata": {},
   "source": [
    "For $n$ i.i.d. samples taken from a normal distribution, we cam estimate the parameters (mean and variance) by:\n",
    "<br>$mean=\\frac{1}{n}\\sum_{i=1}^{n}x_i$\n",
    "<br>$variance=\\frac{1}{n}\\sum_{i=1}^{n}(x_i-mean)^2$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "74ae0061",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We get 1000 samples from a normal distribution\n",
      "The truth values of parameters: mean=5 and variance=15\n",
      "The estimated parameters by MLE: mean=5.211299243717775,variance=15.08865066577488\n"
     ]
    }
   ],
   "source": [
    "# example of samples taken from a normal distribution\n",
    "def MLE_normal(samples):\n",
    "    mean=np.mean(samples)\n",
    "    variance=np.mean((samples-mean)**2)\n",
    "    return mean, variance\n",
    "m,var,n=5,15,1000\n",
    "samples=m+np.sqrt(var)*np.random.randn(n)\n",
    "mean,variance=MLE_normal(samples)\n",
    "print(f'We get {n} samples from a normal distribution')\n",
    "print(f'The truth values of parameters: mean={m} and variance={var}')\n",
    "print(f'The estimated parameters by MLE: mean={mean},variance={variance}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10e53b93",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}