ExampleSharedArrays.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Parallel For Loops in Julia\n",
    "\n",
    "## Setting\n",
    "\n",
    "A computational problem ends up taking the form of **iterating over dimensions a large multidimensional array**. **Some dimensions** of this array are **independent** from each others. One may use this feature to **parallelize calculations** to obtain a speed-up compared to the serial outcome.\n",
    "\n",
    "## How ?\n",
    "\n",
    "This notebook illustrates how one may **parallelize a for loop** using :\n",
    "\n",
    "1. native parallel julia \n",
    "2. `SharedArrays`\n",
    "3. `DistibutedArrays`\n",
    "\n",
    "## A Toy Problem\n",
    "\n",
    "The program takes a 3-dimensional matrix `A[:,:,:]` and sets its rows and columns to be equal to the third dimension. For instance `A[:,:,1] = ones(n1,n2)`, `A[:,:,2] = ones(n1,n2)*2`, `A[:,:,3] = ones(n1,n2)*3`, where $n1$ and $n2$ are the number of rows and columns respectively.\n",
    "\n",
    "This is a toy example, because three lines of code would suffice\n",
    "```julia\n",
    "for k=1:size(A,3)\n",
    "    A[:,:,k] = k\n",
    "end\n",
    "```\n",
    "\n",
    "Yet, here we are going to act **as if the first and the second dimensions of $A$ are dependent** (calculations are to be serially done, respecting the order):\n",
    "```julia\n",
    "# This loop can be parallelized\n",
    "for k=1:size(A,3)\n",
    "    # We cannot parallelize this level\n",
    "    for j=1:size(A,2)\n",
    "        for i=1:size(A,1)\n",
    "             A[i,j,k] = k\n",
    "         end\n",
    "      end\n",
    "    end\n",
    "end\n",
    "```\n",
    "\n",
    "This may look like a silly constraint, but this type of restriction often arises when solving economic models via backwad induction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preliminaries\n",
    "\n",
    "Let's add workers to the current session, define some constants and load the required packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3\n"
     ]
    }
   ],
   "source": [
    "using Distributed\n",
    "\n",
    "if nworkers() < 3 # Let's say we want 3 workers\n",
    "    addprocs(3)\n",
    "end\n",
    "\n",
    "println(nworkers())\n",
    "\n",
    "@everywhere using SharedArrays\n",
    "@everywhere using DistributedArrays\n",
    "@everywhere using Distributions\n",
    "@everywhere const dim1 = 100\n",
    "@everywhere const dim2 = 100\n",
    "@everywhere const dim3 = nworkers()*1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# I. The serial solution\n",
    "\n",
    "## The obvious candidate\n",
    "\n",
    "The following solution does what it should, but does not respect the constraint that calculations at the level of the first and second dimensions should be done serially:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n",
      "3\n",
      "  0.068585 seconds (270.07 k allocations: 14.209 MiB)\n"
     ]
    }
   ],
   "source": [
    "t = zeros(dim1,dim2,dim3)\n",
    "\n",
    "@time for k=1:size(t,3)\n",
    "    println(k)\n",
    "    t[:,:,k] .= k\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Taking into account the serial constraints\n",
    "\n",
    "The following solution respects the constraint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0.005491 seconds (32.03 k allocations: 1.011 MiB)\n"
     ]
    }
   ],
   "source": [
    "@time for k=1:size(t,3)\n",
    "    # We cannot parallelize this level\n",
    "    #----------------------------------\n",
    "    for j=1:size(t,2)\n",
    "        for i=1:size(t,1)\n",
    "         t[i,j,k] = k\n",
    "         end\n",
    "    end\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's spice up the problem by making each execution slower.\n",
    "Let's use `sleep(0.0)`, which (suprisingly) takes more than 0 second to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "@everywhere function give_my_id_serial!(x::Array{Float64,3}, id::Int64)\n",
    "    for indexDim2 = 1:dim2\n",
    "        for indexDim1 = 1:dim1\n",
    "            x[indexDim1, indexDim2, id] = id\n",
    "            #sleep (0.0) still take some time:\n",
    "            #0.000236 seconds (37 allocations: 800 bytes)\n",
    "            sleep(0.0)\n",
    "        end\n",
    "    end\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "wrapper (generic function with 1 method)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function wrapper(x::Array{Float64,3})\n",
    "    give_my_id_serial!(x, 1)\n",
    "    give_my_id_serial!(x, 2)\n",
    "    give_my_id_serial!(x, 3)\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 35.104333 seconds (151.98 k allocations: 4.540 MiB)\n"
     ]
    }
   ],
   "source": [
    "@time wrapper(t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As illustrated above, the task we want to accomplish takes approximately 35 seconds.\n",
    "Yet, as the code makes it obvious, 3 workers could be used in parallel to divide the execution time by\n",
    "approximately 3."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## II. Using native Julia"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's split the data evenly among the 3 workers. In do that by initializing a 3-dimensional array on each worker.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3-element Array{Array{Float64,3},1}:\n",
       " [0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0]\n",
       " [0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0]\n",
       " [0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "asyncmap(fetch, (@spawnat p eval(:(myLocalArray = zeros(dim1, dim2, 1)))) for p in workers())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following function is built so that each worker assigns a specific value equal to `id`, respecting the serial constraint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "@everywhere function give_my_id_parallel!(x::Array{Float64,3}, id::Int64)\n",
    "    for indexDim2 = 1:dim2\n",
    "        for indexDim1 = 1:dim1\n",
    "            x[indexDim1, indexDim2, 1] = id\n",
    "            sleep(0.0)\n",
    "        end\n",
    "    end\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 12.089481 seconds (393.82 k allocations: 20.636 MiB)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "3-element Array{Nothing,1}:\n",
       " nothing\n",
       " nothing\n",
       " nothing"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@time asyncmap(fetch, (@spawnat p give_my_id_parallel!(myLocalArray, p-1)) for p in workers())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's recombine the data. For this part, I get inspiration from this excellent post:\n",
    "https://stackoverflow.com/questions/39058884/julia-use-of-pmap-with-arrays-vs-sharedarrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "recombine_data (generic function with 1 method)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#inspired by: https://stackoverflow.com/questions/39058884/julia-use-of-pmap-with-arrays-vs-sharedarrays\n",
    "getfrom(p::Int, nm::Symbol; mod=Main) = fetch(@spawnat(p, getfield(mod, nm)))\n",
    "\n",
    "function recombine_data(Data::Symbol)\n",
    "    Results = zeros(dim1,dim2,dim3)\n",
    "    for (idx, pid) in enumerate(workers())\n",
    "        Results[:,:,idx] = getfrom(pid, Data)\n",
    "    end\n",
    "    return Results\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0.061616 seconds (52.64 k allocations: 3.372 MiB)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "100×100×3 Array{Float64,3}:\n",
       "[:, :, 1] =\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  …  1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  …  1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  …  1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " ⋮                        ⋮              ⋱            ⋮                   \n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  …  1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  …  1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       "\n",
       "[:, :, 2] =\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  …  2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  …  2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  …  2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " ⋮                        ⋮              ⋱            ⋮                   \n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  …  2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  …  2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       "\n",
       "[:, :, 3] =\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  …  3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  …  3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  …  3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " ⋮                        ⋮              ⋱            ⋮                   \n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  …  3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  …  3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@time k = recombine_data(:myLocalArray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## III. Using DistributedArrays\n",
    "\n",
    "The package `DistributedArrays` was created to distribute \"chunks\" of a mutlidimensional array (matrix) to workers.\n",
    "Each worker owns only a part of the multidimensional array. In the code below, I create a 3-dimensional matrix `a`. Using the function `dzeros`, I specify that I want it to be spread among all the available workers. In the last part of the command, I tell ̀`DistributedArrays` that I want the matrix to be spread among workers along the third dimension only."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#a = dzeros(dim1,dim2,dim3);\n",
    "#Split along the 3rd dimension\n",
    "#-----------------------------\n",
    "a = dzeros((dim1,dim2,dim3), workers(), [1,1,nworkers()]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the function `procs`, one can see that:\n",
    "* workers 2 owns the chunk `t[:, :, 1]` of `a`\n",
    "* workers 3 owns the chunk `t[:, :, 2]` of `a`.\n",
    "* workers 4 owns the chunk `t[:, :, 3]` o `a`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1×1×3 Array{Int64,3}:\n",
       "[:, :, 1] =\n",
       " 2\n",
       "\n",
       "[:, :, 2] =\n",
       " 3\n",
       "\n",
       "[:, :, 3] =\n",
       " 4"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "procs(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function `give_my_id_parallel!` can be sent to workers using `asyncmap` and the usual `fetch @spawnat` commands:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 11.746890 seconds (460.22 k allocations: 23.943 MiB)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1×1×3 Array{Nothing,3}:\n",
       "[:, :, 1] =\n",
       " nothing\n",
       "\n",
       "[:, :, 2] =\n",
       " nothing\n",
       "\n",
       "[:, :, 3] =\n",
       " nothing"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@time asyncmap(fetch, (@spawnat p give_my_id_parallel!(localpart(a), p-1)) for p in procs(a))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, execution time is divided by approximately 3.\n",
    "By inspecting `a`, one sees that it has the correct dimension and values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0.043735 seconds (121.68 k allocations: 6.261 MiB)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1×1×3 Array{Future,3}:\n",
       "[:, :, 1] =\n",
       " Future(2, 1, 14504, nothing)\n",
       "\n",
       "[:, :, 2] =\n",
       " Future(3, 1, 14505, nothing)\n",
       "\n",
       "[:, :, 3] =\n",
       " Future(4, 1, 14506, nothing)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      From worker 3:\t(100, 100, 1)\n",
      "      From worker 2:\t(100, 100, 1)\n",
      "      From worker 4:\t(100, 100, 1)\n"
     ]
    }
   ],
   "source": [
    "@time [@spawnat p println(size(localpart(a))) for p in procs(a)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1×1×3 Array{Tuple{Int64,Int64,Int64},3}:\n",
       "[:, :, 1] =\n",
       " (100, 100, 1)\n",
       "\n",
       "[:, :, 2] =\n",
       " (100, 100, 1)\n",
       "\n",
       "[:, :, 3] =\n",
       " (100, 100, 1)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "map(fetch, (@spawnat p size(localpart(a))) for p=procs(a) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check that we have the right values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n",
      "2.0\n",
      "3.0\n"
     ]
    }
   ],
   "source": [
    "for k=1:3\n",
    "    println(a[1,1,k])\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IV. Using SharedArrays\n",
    "\n",
    "`SharedArrays` are `DistributedArrays` for which the **chunks are equal to the full array**. That is, each worker receives the entire array.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0.493586 seconds (361.09 k allocations: 19.209 MiB, 2.90% gc time)\n",
      " 12.050383 seconds (1.26 M allocations: 62.050 MiB, 0.09% gc time)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "3-element Array{Nothing,1}:\n",
       " nothing\n",
       " nothing\n",
       " nothing"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@time b = SharedArray(zeros(dim1,dim2,dim3));\n",
    "\n",
    "@everywhere function give_my_id!(x::SharedArray{Float64,3}, indexDim3)\n",
    "    for indexDim2 = 1:dim2\n",
    "        for indexDim1 = 1:dim1\n",
    "            x[indexDim1, indexDim2, indexDim3] = indexDim3\n",
    "            sleep(0.0)\n",
    "        end\n",
    "    end\n",
    "end\n",
    "\n",
    "@time asyncmap(fetch, (@spawnat p give_my_id!(b, p)) for p=1:nworkers())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once again, execution time is approximately divided by  3.\n",
    "Inspecting the array `b`, we see that we have the correct result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n",
      "2.0\n",
      "3.0\n"
     ]
    }
   ],
   "source": [
    "for k=1:3\n",
    "    println(b[1,1,k])\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## V. Timing initialization and execution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "initialize_solve_distributed (generic function with 1 method)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function initialize_solve_serial()\n",
    "    \n",
    "    # initialization\n",
    "    k = zeros(dim1,dim2,dim3)\n",
    "    \n",
    "    # calculation\n",
    "    wrapper(k)\n",
    "    \n",
    "    return k\n",
    "end\n",
    "\n",
    "\n",
    "function initialize_solve_native()\n",
    "    \n",
    "    # initialization\n",
    "    asyncmap(fetch, (@spawnat p eval(:(myLocalArray = zeros(dim1, dim2, 1)))) for p in workers())\n",
    "\n",
    "    # calculation\n",
    "    asyncmap(fetch, (@spawnat p give_my_id_parallel!(myLocalArray, p-1)) for p in workers())\n",
    "    \n",
    "    # recombine results\n",
    "    k = recombine_data(:myLocalArray)\n",
    "end\n",
    "\n",
    "function initialize_solve_shared()\n",
    "    \n",
    "    # initialization\n",
    "    k = SharedArray(zeros(dim1,dim2,dim3));\n",
    "    \n",
    "    # calculation\n",
    "    asyncmap(fetch, (@spawnat p give_my_id!(k, p)) for p=1:nworkers())\n",
    "    \n",
    "    return k\n",
    "end\n",
    "\n",
    "function initialize_solve_distributed()\n",
    "    \n",
    "    @sync begin \n",
    "        k = dzeros((dim1,dim2,dim3), workers(), [1,1,nworkers()]);\n",
    "    end\n",
    "    \n",
    "    asyncmap(fetch, (@spawnat p give_my_id_parallel!(localpart(a), p-1)) for p in procs(a))\n",
    "    \n",
    "    return k\n",
    "    \n",
    "bend"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make the benchmark more accurate, let's use `BenchmarkTools` rather than the usual `@time` macro"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "using BenchmarkTools"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  35.596 s (151994 allocations: 4.77 MiB)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "100×100×3 Array{Float64,3}:\n",
       "[:, :, 1] =\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  …  1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  …  1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  …  1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " ⋮                        ⋮              ⋱            ⋮                   \n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  …  1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  …  1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       "\n",
       "[:, :, 2] =\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  …  2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  …  2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  …  2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " ⋮                        ⋮              ⋱            ⋮                   \n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  …  2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  …  2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       "\n",
       "[:, :, 3] =\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  …  3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  …  3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  …  3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " ⋮                        ⋮              ⋱            ⋮                   \n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  …  3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  …  3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@btime initialize_solve_serial()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  11.995 s (1723 allocations: 767.14 KiB)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "100×100×3 Array{Float64,3}:\n",
       "[:, :, 1] =\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  …  1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  …  1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  …  1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " ⋮                        ⋮              ⋱            ⋮                   \n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  …  1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  …  1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       "\n",
       "[:, :, 2] =\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  …  2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  …  2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  …  2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " ⋮                        ⋮              ⋱            ⋮                   \n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  …  2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  …  2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       "\n",
       "[:, :, 3] =\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  …  3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  …  3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  …  3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " ⋮                        ⋮              ⋱            ⋮                   \n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  …  3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  …  3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@btime initialize_solve_native()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  11.451 s (7776 allocations: 416.05 KiB)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "100×100×3 Array{Float64,3}:\n",
       "[:, :, 1] =\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  …  1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  …  1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  …  1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " ⋮                        ⋮              ⋱            ⋮                   \n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  …  1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  …  1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       "\n",
       "[:, :, 2] =\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  …  2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  …  2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  …  2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " ⋮                        ⋮              ⋱            ⋮                   \n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  …  2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  …  2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       "\n",
       "[:, :, 3] =\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  …  3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  …  3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  …  3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " ⋮                        ⋮              ⋱            ⋮                   \n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  …  3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  …  3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@btime initialize_solve_distributed()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  11.485 s (51442 allocations: 1.78 MiB)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "100×100×3 SharedArray{Float64,3}:\n",
       "[:, :, 1] =\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  …  1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  …  1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  …  1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " ⋮                        ⋮              ⋱            ⋮                   \n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  …  1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  …  1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       " 1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0     1.0  1.0  1.0  1.0  1.0  1.0  1.0\n",
       "\n",
       "[:, :, 2] =\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  …  2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  …  2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  …  2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " ⋮                        ⋮              ⋱            ⋮                   \n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  …  2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  …  2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       " 2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0     2.0  2.0  2.0  2.0  2.0  2.0  2.0\n",
       "\n",
       "[:, :, 3] =\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  …  3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  …  3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  …  3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " ⋮                        ⋮              ⋱            ⋮                   \n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  …  3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  …  3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0\n",
       " 3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0     3.0  3.0  3.0  3.0  3.0  3.0  3.0"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@btime initialize_solve_shared()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## VI. Benchmark Results\n",
    "\n",
    "\n",
    "### Serial\n",
    "* 35.596 s (151994 allocations: 4.77 MiB)\n",
    "\n",
    "### Native Julia\n",
    "* 11.995 s (1723 allocations: 767.14 KiB)\n",
    "\n",
    "### DistributedArrays\n",
    "* 11.451 s (7776 allocations: 416.05 KiB)\n",
    "\n",
    "### SharedArrays\n",
    "* 11.485 s (51442 allocations: 1.78 MiB)\n",
    "\n",
    "\n",
    "This benchmark suggests that `DistributedArrays` is as good as native Julia, if not better.\n",
    "This benchmark also illustrates that there is a small (marginal) penalty associated to using `SharedArrays`, associated with the cost of copying the entire data to each worker, but this penalty is likely to be marginal when each calculation takes more time (look at the allocations). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Appendix\n",
    "\n",
    "### Version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Julia Version 1.5.2\n",
      "Commit 539f3ce943 (2020-09-23 23:17 UTC)\n",
      "Platform Info:\n",
      "  OS: Linux (x86_64-pc-linux-gnu)\n",
      "  CPU: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz\n",
      "  WORD_SIZE: 64\n",
      "  LIBM: libopenlibm\n",
      "  LLVM: libLLVM-9.0.1 (ORCJIT, skylake)\n"
     ]
    }
   ],
   "source": [
    "versioninfo()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Safety checks\n",
    "\n",
    "Let's make sure that all the results are equal.\n",
    "For the tests to make sense, we need to convert DistributedArrays and SharedArrays to native Julia Arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "using Base.Test\n",
    "\n",
    "# Conversion\n",
    "aArray = convert(Array{Float64,3}, a)\n",
    "bArray = convert(Array{Float64,3}, b)\n",
    "\n",
    "# Tets\n",
    "println(@test t==aArray)\n",
    "println(@test t==bArray)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.5.2",
   "language": "julia",
   "name": "julia-1.5"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}