hands-on/shell_interaction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Interacting with the shell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `sh` module"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [`sh` module](https://amoffat.github.io/sh/) is very convenient to interact with the shell.  Note that `sh` is not part of Python's standard library, if you prefer not to use extra modules, use the `subprocess` module in the standard library.  The statements below will install `sh` using `pip` is it isn't already installed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    import sh\n",
    "except ModuleNotFoundError:\n",
    "    print('installing sh using pip')\n",
    "    !pip install sh\n",
    "    import sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Any shell command can be executed by calling it as a function on the `sh` module, passing command line arguments as arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sh.ls('-l')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = sh.ls('-l', '-a')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lines = result.split('\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(lines[1:-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lines[1:-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = sh.mkdir('tmp', '-p')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for file in (line.strip() for line in sh.ls().split()):\n",
    "    print(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exit codes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When a shell command fails, an exception is thrown which contains the full command as it was run, the exit code, the standard output and error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    sh.ls('bla.txt')\n",
    "except Exception as error:\n",
    "    err_msg = error.stderr.decode(encoding='utf8').rstrip()\n",
    "    print(f'command \"{error.full_cmd}\" exited with exit code {error.exit_code} and message \"{err_msg}\"')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### I/O redirection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Redirecting output can be done using the `_out` optional argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('tmp/date_file.txt', 'w') as file:\n",
    "    for i in range(10):\n",
    "        print(f'{i} ', end='', file=file, flush=True)\n",
    "        sh.date(_out=file)\n",
    "        sh.sleep('1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the use of the `flush` optional argument in the print function.  If this is omitted, the Python interpreter will only flush the results of its own print calls after the `sh` modules has written its output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(sh.cat('tmp/date_file.txt'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Input redirection works similarly using the optional `_in` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('tmp/date_file.txt', 'r') as file:\n",
    "    print(sh.wc('-l', _in=file))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Piping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of a command can be used as the input for another command."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pipe the output of `ls` into `grep` to select only the files with names that end with `.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(sh.grep('-e', r'\\.ipynb$', _in=sh.ls('-l')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pipe the output of `cut` into `sort`.  Also use the `_iter` argument to create a generator over standard output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for line in sh.sort('-r', _in=sh.cut('-d', ' ', '-f', '5', 'tmp/date_file.txt'), _iter=True):\n",
    "    print(line.strip())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Backgrounding & time out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Long running processes can be placed in the background."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "process = sh.sleep(10, _bg=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(10):\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "process.wait()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(process.exit_code)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A time out can be specified for a command, and on time out, the resulting exit code will be the number of the signal (SIGKILL by default)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "process = sh.sleep(10, _bg=True, _bg_exc=False, _timeout=3)\n",
    "try:\n",
    "    process.wait()\n",
    "except sh.TimeoutException as error:\n",
    "    print('process timed out')\n",
    "    print(error.exit_code)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clean up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remove the `tmp` directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sh.rm('-rf', 'tmp')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `subprocess` module"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you prefer to use standard library modules only, `subprocess` is a good choice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import subprocess"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This module has a high-level function `run` that can be used for almost any processing.  The API is still being improved in subsequent releases of Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "process = subprocess.run(['ls', '-l'], stdout=subprocess.PIPE, encoding='utf8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "process.stdout.split('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that if you don't specify the `stdout` arugment, the output of the command will not be captured.  Python 3.7 makes this easier by adding a `capture_output` argument."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exit codes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `run` function returns a `CompletedProcess` object that has an attribute for the exit code returned by the process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "process = subprocess.run(['mkdir', '-p', 'tmp'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "process.returncode"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### I/O redirection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Output of a running command can be redirected to a file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('tmp/data.txt', 'w') as file:\n",
    "    for i in range(10):\n",
    "        subprocess.run(['echo', '-n', str(i) + ' '], stdout=file)\n",
    "        subprocess.run(['date'], stdout=file)\n",
    "        subprocess.run(['sleep', '1'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that mixed I/O from the `print` function and `run` doesn't work as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(subprocess.run(['cat', 'tmp/data.txt'], stdout=subprocess.PIPE,\n",
    "                     encoding='utf8').stdout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Input redirection is similar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('tmp/data.txt', 'r') as file:\n",
    "    process = subprocess.run(['wc', '-l'], stdin=file, stdout=subprocess.PIPE,\n",
    "                             encoding='utf8')\n",
    "    print(process.stdout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Piping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Piping can also be done using `subprocess`.  It is less user friendly than using the `sh` module, but it allows more control.  You will have to resort to the low-level `Popen` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p1 = subprocess.Popen(['ls', '-l'], stdout=subprocess.PIPE)\n",
    "p2 = subprocess.Popen(['grep', r'\\.ipynb$'], stdin=p1.stdout, stdout=subprocess.PIPE, encoding='utf8')\n",
    "p1.stdout.close()\n",
    "output, _ = p2.communicate()\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "p1 = subprocess.Popen(['cut', '-d', ' ', '-f', '5', 'tmp/data.txt'], stdout=subprocess.PIPE)\n",
    "p2 = subprocess.Popen(['sort', '-r'], stdin=p1.stdout, stdout=subprocess.PIPE, encoding='utf8')\n",
    "p1.stdout.close()\n",
    "output, _ = p2.communicate()\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = subprocess.run(['rm', '-r', 'tmp'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Shell file globbing and environment variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For file globbing to work in subprocesses, provide the entire command, including all arguments as a string, rather than a list to `run`. Also, set `shell` to `True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "process = subprocess.run('ls *.ipynb', stdout=subprocess.PIPE, encoding='utf8', shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "process.stdout.split()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same applies when you want environment variables to expand."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "process = subprocess.run('echo \"hello ${USER}\"', stdout=subprocess.PIPE, encoding='utf8', shell=True)\n",
    "print(process.stdout.rstrip())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you need to add or modify environment variables, it is good practice to do that on a copy of `os.environ`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "environ = os.environ.copy()\n",
    "environ['greeting'] = 'bye'\n",
    "process = subprocess.run('echo \"${greeting} ${USER}\"', stdout=subprocess.PIPE, encoding='utf8',\n",
    "                         env=environ, shell=True)\n",
    "print(process.stdout.rstrip())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clean up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remove the `tmp` directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "process = subprocess.run(['rm', '-rf', 'tmp'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "process.returncode"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}