sinewaves.ipynb_OLD

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1> Time series prediction, end-to-end </h1>\n",
    "\n",
    "This notebook illustrates several models to find the next value of a time-series:\n",
    "<ol>\n",
    "<li> Linear\n",
    "<li> DNN\n",
    "<li> CNN \n",
    "<li> RNN\n",
    "</ol>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# change these to try this notebook out\n",
    "BUCKET = 'cloud-training-demos-ml'\n",
    "PROJECT = 'cloud-training-demos'\n",
    "REGION = 'us-central1'\n",
    "SEQ_LEN = 50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['BUCKET'] = BUCKET\n",
    "os.environ['PROJECT'] = PROJECT\n",
    "os.environ['REGION'] = REGION\n",
    "os.environ['SEQ_LEN'] = str(SEQ_LEN)\n",
    "os.environ['TFVERSION'] = '1.8'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3> Simulate some time-series data </h3>\n",
    "\n",
    "Essentially a set of sinusoids with random amplitudes and frequencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import seaborn as sns\n",
    "\n",
    "def create_time_series():\n",
    "  freq = (np.random.random()*0.5) + 0.1  # 0.1 to 0.6\n",
    "  ampl = np.random.random() + 0.5  # 0.5 to 1.5\n",
    "  noise = [np.random.random()*0.3 for i in range(SEQ_LEN)] # -0.3 to +0.3 uniformly distributed\n",
    "  x = np.sin(np.arange(0,SEQ_LEN) * freq) * ampl + noise\n",
    "  return x\n",
    "\n",
    "flatui = [\"#9b59b6\", \"#3498db\", \"#95a5a6\", \"#e74c3c\", \"#34495e\", \"#2ecc71\"]\n",
    "for i in range(0, 5):\n",
    "  sns.tsplot( create_time_series(), color=flatui[i%len(flatui)] );  # 5 series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def to_csv(filename, N):\n",
    "  with open(filename, 'w') as ofp:\n",
    "    for lineno in range(0, N):\n",
    "      seq = create_time_series()\n",
    "      line = \",\".join(map(str, seq))\n",
    "      ofp.write(line + '\\n')\n",
    "\n",
    "import os\n",
    "try:\n",
    "  os.makedirs('data/sines/')\n",
    "except OSError:\n",
    "  pass\n",
    "\n",
    "np.random.seed(1) # makes data generation reproducible\n",
    "\n",
    "to_csv('data/sines/train-1.csv', 1000)  # 1000 sequences\n",
    "to_csv('data/sines/valid-1.csv', 250)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!head -5 data/sines/*-1.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3> Train model locally </h3>\n",
    "\n",
    "Make sure the code works as intended."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "DATADIR=$(pwd)/data/sines\n",
    "OUTDIR=$(pwd)/trained/sines\n",
    "rm -rf $OUTDIR\n",
    "gcloud ml-engine local train \\\n",
    "   --module-name=sinemodel.task \\\n",
    "   --package-path=${PWD}/sinemodel \\\n",
    "   -- \\\n",
    "   --train_data_path=\"${DATADIR}/train-1.csv\" \\\n",
    "   --eval_data_path=\"${DATADIR}/valid-1.csv\"  \\\n",
    "   --output_dir=${OUTDIR} \\\n",
    "   --model=linear --train_steps=10 --sequence_length=$SEQ_LEN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3> Cloud ML Engine </h3>\n",
    "\n",
    "Now to train on Cloud ML Engine with more data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import shutil\n",
    "shutil.rmtree('data/sines', ignore_errors=True)\n",
    "os.makedirs('data/sines/')\n",
    "np.random.seed(1) # makes data generation reproducible\n",
    "for i in range(0,10):\n",
    "  to_csv('data/sines/train-{}.csv'.format(i), 1000)  # 1000 sequences\n",
    "  to_csv('data/sines/valid-{}.csv'.format(i), 250)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "gsutil -m rm -rf gs://${BUCKET}/sines/*\n",
    "gsutil -m cp data/sines/*.csv gs://${BUCKET}/sines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "for MODEL in linear dnn cnn rnn rnn2 rnnN; do\n",
    "  OUTDIR=gs://${BUCKET}/sinewaves/${MODEL}\n",
    "  JOBNAME=sines_${MODEL}_$(date -u +%y%m%d_%H%M%S)\n",
    "  gsutil -m rm -rf $OUTDIR\n",
    "  gcloud ml-engine jobs submit training $JOBNAME \\\n",
    "     --region=$REGION \\\n",
    "     --module-name=sinemodel.task \\\n",
    "     --package-path=${PWD}/sinemodel \\\n",
    "     --job-dir=$OUTDIR \\\n",
    "     --scale-tier=BASIC \\\n",
    "     --runtime-version=$TFVERSION \\\n",
    "     -- \\\n",
    "     --train_data_path=\"gs://${BUCKET}/sines/train*.csv\" \\\n",
    "     --eval_data_path=\"gs://${BUCKET}/sines/valid*.csv\"  \\\n",
    "     --output_dir=$OUTDIR \\\n",
    "     --train_steps=3000 --sequence_length=$SEQ_LEN --model=$MODEL\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Monitor training with TensorBoard\n",
    "\n",
    "Use this cell to launch tensorboard. If tensorboard appears blank try refreshing after 5 minutes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from google.datalab.ml import TensorBoard\n",
    "TensorBoard().start('gs://{}/sinewaves'.format(BUCKET))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for pid in TensorBoard.list()['pid']:\n",
    "  TensorBoard().stop(pid)\n",
    "  print('Stopped TensorBoard with pid {}'.format(pid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results\n",
    "\n",
    "When I ran it, these were the RMSEs that I got for different models:\n",
    "\n",
    "| Model | Sequence length | # of steps | Minutes | RMSE |\n",
    "| --- | ----| --- | --- | --- | \n",
    "| linear | 50 | 3000 | 10 min | 0.150 |\n",
    "| dnn | 50 | 3000 | 10 min | 0.101 |\n",
    "| cnn | 50 | 3000 | 10 min | 0.105 |\n",
    "| rnn | 50 | 3000 | 11 min | 0.100 |\n",
    "| rnn2 | 50 | 3000 | 14 min |0.105 |\n",
    "| rnnN | 50 | 3000 | 15 min | 0.097 |\n",
    "\n",
    "### Analysis\n",
    "You can see there is a significant improvement when switching from the linear model to non-linear models. But within the the non-linear models (DNN/CNN/RNN) performance for all is pretty similar. \n",
    "\n",
    "Perhaps it's because this is too simple of a problem to require advanced deep learning models. In the next lab we'll deal with a problem where an RNN is more appropriate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2017 Google Inc. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}