regul_evolution_tut.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Copy_of_regularized_evolution_algorithm(1)(1)(2)(1)(1)(1)(1)(1)(1).ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K_6EyDK_B3fI",
        "colab_type": "text"
      },
      "source": [
        "# AutoML step by step tutorial with evolutionary algorithm\n",
        "Author: Daniel Wiczew\n",
        "\n",
        "Contact: daniel.wiczew@pwr.edu.pl"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eizp75WbBwaB",
        "colab_type": "text"
      },
      "source": [
        "This tutorial was based on the (i.e. aging evolution) algorithm used in:\n",
        "\n",
        "[E. Real, A. Aggarwal, Y. Huang, and Q. V. Le 2018. Regularized Evolution for Image Classifier Architecture Search](https://arxiv.org/abs/1802.01548).\n",
        "\n",
        "If you want to know technical details about the algorithm, I really encourage to read it."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VUdox1uMAMqF",
        "colab_type": "text"
      },
      "source": [
        "# Optimizing CNN architecture using evolutionary algorithm and CIFAR10 dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YtacawXFAQbg",
        "colab_type": "text"
      },
      "source": [
        "[link text](https://)The tutorial aims to show the usefulness of simple evolutionary algorithms in AutoML type problems. To achieve it a simple, sequential model is employed with the aging evolution algorithm, that was recently used in Google Brain's paper (mentioned above). The algorithm performs a stochastic search using Tournament selection and mutations, where the aging process is performed by removing the oldest representative in the population. More is explained in the article associated with the notebook. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UplHS0zOaZNF",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dNpyEEM2hVoZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "from matplotlib.pyplot import imshow\n",
        "import random\n",
        "import collections\n",
        "import time\n",
        "from copy import deepcopy \n",
        "# Set mixed precision for higher performance\n",
        "dtype = 'mixed_float16'\n",
        "policy = tf.keras.mixed_precision.experimental.Policy(dtype)\n",
        "tf.compat.v2.keras.mixed_precision.experimental.set_policy(policy)"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0WWoJSQIVyrD",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "9910ff99-c455-4bb7-f770-7338e72cc84f"
      },
      "source": [
        "print('Compute dtype: %s' % policy.compute_dtype)\n",
        "print('Variable dtype: %s' % policy.variable_dtype)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Compute dtype: float16\n",
            "Variable dtype: float32\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KHAQIt8gFHun",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 391
        },
        "outputId": "e8fc341b-67fd-4203-fda9-dab95ce089e2"
      },
      "source": [
        "# Depending on the graphic card you pick up, the speed is higher of lower\n",
        "# Actually for the mixed mode, tesla T4 is one of the best\n",
        "!nvidia-smi"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Wed Jun 17 06:04:22 2020       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 450.36.06    Driver Version: 418.67       CUDA Version: 10.1     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   57C    P8    11W /  70W |      0MiB / 15079MiB |      0%      Default |\n",
            "|                               |                      |                 ERR! |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HgxGdw0YZ4nr",
        "colab_type": "text"
      },
      "source": [
        "The notebook starts with constants, their importance will be explained later in the notebook, so don't worry. \n",
        "* `EPOCHS` - Number of epochs, the more, the slower which is not better here.\n",
        "* `BATCH_SIZE` - Its a batch size, keep so that, it is multiple of validation's set size.\n",
        "* `LEARNING_RATE = 0.01` - Learning rate used while training models. It is that high, since the Adam optimizer is used (an adaptive optimizer). \n",
        "* `MODEL_SIZE_MAX` - Maximal sequential size of the model (e.g. 6 layers).\n",
        "* `MAX_NEUR` - Maximal number of filters/neurons, depending on the model you want to use. It is better to set it low and scale up it later (it will speed up search significantly). The architecture is preserved if you scale up the number of filters by 2, each layer.\n",
        "* `ACTIVATION` - Activation function used in the convolutional layers.\n",
        "* `BAD_MODEL_LOSS` - Some randomly generated models fail to compile, thus an arbitrary loss is set for this (Usually the smallest possible, that's why it's zero). \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WkrBRubzNcSl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# CONSTANTS\n",
        "EPOCHS = 25\n",
        "BATCH_SIZE = 2048\n",
        "LEARNING_RATE = 0.01\n",
        "# This is maximal number of layers\n",
        "MODEL_SIZE_MAX = 10\n",
        "# Maximal number of neurons in the dense layers\n",
        "MAX_NEUR = 16\n",
        "ACTIVATION = 'relu'\n",
        "# Bad model loss\n",
        "BAD_MODEL_LOSS = 0.0"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qwdWMTcN3ds2",
        "colab_type": "text"
      },
      "source": [
        "# Gene definition\n",
        "Here genes are defined. The genes form elements those combinations form solutions in the search space. The genes include:\n",
        "* Identity function (key: -1): it's purpose is to put an empty connection between layers. It's like putting an empty layer, that's outputs what is on its input.\n",
        "* Convolutional layer (key: 0) with strides 2 by 2 and convolution filter of size 3 by 3, where its size is defined by the `MAX_NEUR`.\n",
        "* Batch Normalization (key: 1) normalizes out of the input during forward propagation with respect to the batch dimension, read more here: https://www.tensorflow.org/api_docs/python/tf/keras/layers/BatchNormalization\n",
        "* Batch Normalization with renormalization (key: 2) is similar as above but with renomralization, read more in the link above.\n",
        "* MaxPool2D (key: 3) extracts the most profound features from the previous layers in the height and width image's dimensions. \n",
        "\n",
        "Generally, you can add a new one, but you have to change the `AnnModel` class and `random_architecture`, `mutation` methods to include those elements. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rtl605A1hV6O",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Gene dictionaries\n",
        "# Values are initialized later for convolutional layer\n",
        "available_genes = {-1: (tf.keras.layers.Lambda, lambda x: tf.identity(x)), # Empty element\n",
        "                   0: (tf.keras.layers.Conv2D, 1), # Dense layer with\n",
        "                   1: (tf.keras.layers.BatchNormalization, None), # Batch Normalization layer\n",
        "                   2: (tf.keras.layers.BatchNormalization, ('renorm', True)), # Batch Normalization with renorm\n",
        "                   3: (tf.keras.layers.MaxPool2D, (2, 2))} # Max Poooling with 2x2 strides"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "etUpMvgHoVyH",
        "colab_type": "text"
      },
      "source": [
        "# DATASET\n",
        "Here dataset is loaded and CIFAR10 is used. Since the major concern here is the speed of neural network performance evaluation, all means to speed up training are used. The dataset is truncated a bit thus (the `SIZE_TRAIN_PERCENT` and `SIZE_VALID_PERCENT` constants). Furthermore, it is turned gray (`TURN_GRAY` constant) to further speed up calculations. \n",
        "Also there is an option `HORIZ_FLIP` that extends the dataset by augumenting the data with horizontaly flipped images."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qyk9L5IuHvNk",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "d1a15a5c-c8ee-4f85-bf6a-de7494488179"
      },
      "source": [
        "# Settings\n",
        "TURN_GRAY = True\n",
        "# Create horizontal flip copy\n",
        "HORIZ_FLIP = True\n",
        "SIZE_TRAIN_PERCENT = 1.0\n",
        "SIZE_VALID_PERCENT = 1.0\n",
        "\n",
        "# Download and load CIFAR10 dataset to the memory.\n",
        "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.cifar10.load_data()\n",
        "num_classes = int(max(y_train)) + 1\n",
        "# Add channel axis, due to removal of the last axis after turning images gray\n",
        "if TURN_GRAY:\n",
        "  # Turning images gray using dot product with vector [0.07, 0.72, 0.21] (RGB respectively)\n",
        "  x_train, x_test = x_train.dot([0.07, 0.72, 0.21]), x_test.dot([0.07, 0.72, 0.21])\n",
        "\n",
        "if x_train.ndim < 4:\n",
        "  x_train, x_test = x_train[:, :, :, np.newaxis], x_test[:, :, :, np.newaxis]\n",
        "\n",
        "# Calculate statistics for standarization\n",
        "mean = np.mean(x_train)\n",
        "var = np.var(x_train)\n",
        "# Standarize:\n",
        "x_train = (x_train - mean)/(var)\n",
        "x_test = (x_test - mean)/(var)\n",
        "# Normalize between 0 and 1 using minmax\n",
        "min_feat = np.min(x_train)\n",
        "max_feat = np.max(x_train)\n",
        "x_train = (x_train - min_feat)/(max_feat - min_feat)\n",
        "x_test = (x_test - min_feat)/(max_feat - min_feat)\n",
        "\n",
        "# Horizontal flip\n",
        "if HORIZ_FLIP:\n",
        "  x_train = np.concatenate([x_train, np.flip(x_train, 2)])\n",
        "  y_train = np.concatenate([y_train, y_train])\n",
        "\n",
        "SIZE_TRAIN = int(len(x_train) * SIZE_TRAIN_PERCENT)\n",
        "SIZE_VALID = int(len(x_test) * SIZE_VALID_PERCENT)\n",
        "\n",
        "# Truncate dataset and change the vector's type to float32 (It is float64 initially)\n",
        "x_train, y_train, x_test, y_test = (x_train.astype(np.float32)[:SIZE_TRAIN], \n",
        "                                    tf.one_hot(np.squeeze(y_train).astype(np.float32)[:SIZE_TRAIN], depth=num_classes), \n",
        "                                    x_test.astype(np.float32)[:SIZE_VALID], \n",
        "                                    tf.one_hot(np.squeeze(y_test).astype(np.float32)[:SIZE_VALID], depth=num_classes))"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz\n",
            "170500096/170498071 [==============================] - 6s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zCWewC64cjyt",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 285
        },
        "outputId": "32efd6ca-e8b9-4db3-ab23-ef3e7d0bab68"
      },
      "source": [
        "# An image in the dataset\n",
        "imshow(np.squeeze(x_train[0]), cmap='gray')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.image.AxesImage at 0x7f5868082898>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD5CAYAAADhukOtAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAY70lEQVR4nO2dW2xdZXbH/yu2EwfskIsTx+RCAokCAUECVqAMQimjGQEaCZAqLg+IBzQZVYNUpOkDolKhUh+YqoB4qKhCA5OpKAwdQCCEpkOjkWB4ABIIIUAgFyXExrFzsROHcEu8+nC2K4fZ6+/jfXz2CXz/nxTl+Fv+9l7n23v5nPP9z1rL3B1CiB8+UxrtgBCiHBTsQiSCgl2IRFCwC5EICnYhEkHBLkQiNNcy2cyuB/AYgCYA/+HuD7Hfb2lp8dbW1lzbqVOnipw/tDU3x0+N2ZqamiZ8PuYHs508eTK0MUmU+R/NY8cbGRkJbVOmFHs9iI7J1rfI8cYjWn/mB3vO7HpOtoxd5HhDQ0M4ceJErpOFg93MmgD8G4CfAOgB8I6ZvezuH0VzWltbccUVV+TaBgcHw3NFi89u+o6OjtA2a9as0DZz5szQFp1v2rRp4Rx2U7Hn/O2334a22bNnh7boD8jXX38dzvnqq69C2/Tp00MbuxlPnDiRO87Wlx2P+c8CMLp3ZsyYEc45++yzQ9vUqVNDW1EfI9s333wTzonWasOGDeGcWt7GrwGwy933uPs3AJ4FcFMNxxNC1JFagn0BgP1jfu7JxoQQZyA1fWavBjNbB2AdwN/uCiHqSy2v7L0AFo35eWE2dhruvt7du929u6WlpYbTCSFqoZZgfwfAcjNbamZTAdwO4OXJcUsIMdkUfhvv7ifN7B4A/4OK9Paku3/I5nz11Vf48MP8Xzl06FA4L3r7X2RXGuA7qmwXPJIH2Y472+n+8ssvJ3wuAPjss89CW+QLe15M1WAfvYr4H+3SA3wd2U43e8cYzWNreNZZZ4U25mNROS+6R9g1i54zu99q+szu7q8CeLWWYwghykHfoBMiERTsQiSCgl2IRFCwC5EICnYhEqHu36D7ixMGMg+TwyKJrbOzM5wzb9680MYSHRiRFMLkDia5sCQTJh0yWS5K8ikqRbJsMyZ5RWvFfGcyHzsXmzc8PJw7Xo9Cq+y++uKLL0JbdG1Y8syxY8dyx9n66pVdiERQsAuRCAp2IRJBwS5EIijYhUiE0nfjI9ra2kLbueeemzvOdrPZruThw4dDG9uljRI/2E7xOeecE9rYLvLRo0dDG0tcaW9vzx2PdqUBriYwGyPamWalloru1DOlIUpOYc+rqDrB1pg9t8hHNifa3af1BEOLEOIHhYJdiERQsAuRCAp2IRJBwS5EIijYhUiEUqW3KVOmhBIKkzsiGYd19SjSTgrgiSuRjZ2rSKsmgEtDrNbZwYMHc8eZ5MVqyTFZi9Vqi87HpCGW+MGkN1bXLmo3VrT7DKsLx455/Pjx0BatP3te0XVh10uv7EIkgoJdiERQsAuRCAp2IRJBwS5EIijYhUiEmqQ3M9sLYBjAKQAn3b2b/X5TU1OYBcYkqkg+YTJZNAfg8gQjko2YrMWkGiZDFcnkAmLJrui5mI1JQ1HWHpPXWJ22SFIE4npsADB//vzc8Tlz5oRzmMx35MiR0Mb8Z9JbtFa9vX/RJ/X/ie4Bdi9Ohs7+1+4eN2oTQpwR6G28EIlQa7A7gD+a2RYzWzcZDgkh6kOtb+OvcfdeM5sH4DUz2+Hur4/9heyPwDqAV3QRQtSXml7Z3b03+38AwIsA1uT8znp373b3brYJJ4SoL4WD3czONrP20ccAfgpg+2Q5JoSYXGp5qe0E8GImpTQD+C93/wM9WXNzKHmwrLcou4rJSYwiRSUZrKgkawnEZBx2TJYRF8k1TEJj68GKerJrFhX1ZOdishxbj0svvTS0HTqULxSxNSxaJJTJa+wjbCQPsvZmUXHLnTt3hnMKB7u77wFwWdH5QohykfQmRCIo2IVIBAW7EImgYBciERTsQiRC6QUnoyKRrMhfJFswGYRJK0yymzlzZmiLZCN2PJaFxApmsn50+/fvD22RJMOKYjJ5be3ataGNSUNbtmzJHd+1a1c4h/nIZLnBwcHQNjQ0lDvOpDAml7J7jtkY0f0T9TgE4ue8d+/ecI5e2YVIBAW7EImgYBciERTsQiSCgl2IRCh1N765uRmzZ8/OtbHd52gnliWtsF1wtrPLasYVmTNr1qzQxpJTenp6Qlu04w7EtclY3Tq2i8zq/LFd8Og6z507N5zDEoPYtd6xY0doi3b42W58W1tbaGPryBJoiig2TFHq6OjIHWdp5HplFyIRFOxCJIKCXYhEULALkQgKdiESQcEuRCKUKr21tLRg3rx5ubb29vZwXiR3REkOAG/FwxIuGJGswXxn0hWrF8ZkFyaVRXXhojp+AJdrWOIKkw6jhKIiiUYAlzfZWkU2djxmY7Itu9bMFq0/e15F7mG9sguRCAp2IRJBwS5EIijYhUgEBbsQiaBgFyIRxpXezOxJAD8DMODul2RjswH8DsASAHsB3OrucQrUGCIJokiH19bW1gnPGQ+W1RRJQ0zWGhgYCG0sy4tJVEwaijK2zj///HAOk3jYdTl69Ghoi9aKXbMLLrggtC1fvjy0MXkwkjeZFMZkLZa9xtaKtcqKjsl8LEI1R/sNgOu/M3YfgE3uvhzApuxnIcQZzLjBnvVb/+5L0E0ANmaPNwK4eZL9EkJMMkXfJ3S6e1/2+AAqHV2FEGcwNX8o8MqHs/B7jma2zsw2m9nmEydO1Ho6IURBigZ7v5l1AUD2f7gL5e7r3b3b3bvZRpYQor4UDfaXAdyVPb4LwEuT444Qol5UI709A2AtgA4z6wHwAICHADxnZncD2Afg1mpONjIyEso8rEBkJE0cP348nMMysliWF8ukiz6GLFq0KJzDnhcrRjl//vzQxlplXXTRRbnjLKOMPWfmI5PRotZF7JpFvgNcimTtmg4dOpQ7zp4zyypkWW/snmOSXZEip0UYN9jd/Y7A9ONJ9kUIUUf0DTohEkHBLkQiKNiFSAQFuxCJoGAXIhFKLTjJYLJFJBtNnz49nMOyjCI5BgB6e3tDWyTZHThwIJzDst6ifmgAcM0114Q25uPSpUtzx6PeYAD3sWiByCh7sL+/f8JzAODgwYOhra+vL7RF98GMGTPCOUWzAItm0kVyHluPIuiVXYhEULALkQgKdiESQcEuRCIo2IVIBAW7EIlQqvTW3NwcSjksEy3KlGIZZYcPHw5te/funfC5gFjGOXbsWDiHZYaxbLkFCxaENuZjlLG1evXqcA6TrliGHcvkinrtsefM5CmW2cZktEjeZPJrUXmQrRXLlouu2ZdffhnOie5F2osutAghflAo2IVIBAW7EImgYBciERTsQiRCqbvxJ0+exOBgfpcotpMZwXZGi+zuA0B7e3toi6rjsjZIbFed7ZCz3eJ9+/aFtigRhikGrLUS23FnpcGjJBmW0MLUFZY0xHy84oorcsdZ661XXnkltPX09IQ2lnzFkoYiG5sTrRWbo1d2IRJBwS5EIijYhUgEBbsQiaBgFyIRFOxCJEI17Z+eBPAzAAPufkk29iCAnwMY1VHud/dXqzlh9EV99qX/iCJyBsDr3TFZLmrTwxIxrrvuutB28cUXh7YNGzaENlZ7L5Jkdu/eHc5ZtmxZaGOSF0tciRKRWJ02dg+w69LV1RXaojZUkQQM8KQb5j+jSI3FIvdwrdLbbwBcnzP+qLuvyv5VFehCiMYxbrC7++sA4m8gCCG+F9Tymf0eM9tmZk+aWdzqUwhxRlA02B8HcAGAVQD6ADwc/aKZrTOzzWa2ucjnciHE5FAo2N29391PufsIgCcArCG/u97du929m20sCSHqS6FgN7Ox25+3ANg+Oe4IIepFNdLbMwDWAugwsx4ADwBYa2arADiAvQB+UasjkawFxHIHk0FYCx+WJcWkpuidyeWXXx7Oufrqq0Mby2xj68Gksoh58+aFNrZWw8PDoY1lqUVZjOyaff7556Ft69atoY2tcWdnZ+740NBQOCeqCQcACxcuDG3subG1itaf3R/RHHZvjxvs7n5HznAsAgshzkj0DTohEkHBLkQiKNiFSAQFuxCJoGAXIhFKLTgJxNk6TJqIspBaWlrCOayA5fz580Mb++LP4sWLc8evvfbacM6FF14Y2p566qnQxgpVXnLJJaEtkppYJhf7ZiMrVNnb2xvaopZS7Dq3tbWFNiaJsnZeUdYbK5bJpEiWfceyAFlbpqhQZUdHx4T9YEVY9couRCIo2IVIBAW7EImgYBciERTsQiSCgl2IRChdeoukAZZdFUkhUe81gEsdTMY5cOBAaLvyyitzx6N+YkDxnnOsiCWT8yIZZ8uWLeEcJkMx6Y31PYsk1tbW1nDOkiVLQtuKFStCG1vjaD2YH0y23b9/f2hjRSXZ/RjFxKxZcQGoSD5mcrRe2YVIBAW7EImgYBciERTsQiSCgl2IRCh1N35kZCRMuoh2TYF455TV/GJJCSwp5LbbbgttN9xwQ+743Llzwzms7RKDqRPsmF988UXu+AsvvBDOYck/LEmG1bVrb2/PHWc7+IwooQXgakjUDompDGw3vmg5dHbMqG5cdC2BWFGqtf2TEOIHgIJdiERQsAuRCAp2IRJBwS5EIijYhUiEato/LQLwWwCdqLR7Wu/uj5nZbAC/A7AElRZQt7r74HjHi2QG1rYmktFYiyQGk966u7tDW5Rk8N5774VzmNTE5JgjR46Etp07d4a2KOGCSZFMwpw5c2ZoK5JQxK4ZkxtZnTmW/FGkVhtLsGL3KZvH1jFK5GHSW3Q9a5XeTgL4lbuvBHAVgF+a2UoA9wHY5O7LAWzKfhZCnKGMG+zu3ufu72aPhwF8DGABgJsAbMx+bSOAm+vlpBCidib0md3MlgBYDeAtAJ3uPlov+AAqb/OFEGcoVQe7mbUBeB7Ave5+2ncNvfJBIffDgpmtM7PNZraZ1eMWQtSXqoLdzFpQCfSn3X30S9b9ZtaV2bsADOTNdff17t7t7t2sOogQor6MG+xW2d7dAOBjd39kjOllAHdlj+8C8NLkuyeEmCyqyXr7EYA7AXxgZluzsfsBPATgOTO7G8A+ALfW4gir3xVJK0xOYhIEk0Feein+m9XV1ZU7zlo1MfmEZfpFWWMAl42itWIZaiwDjJ3r4MGDoS1q88Qy7FhNvnfffTe0ffLJJxP2g9ZqI1Ikax02bdq0QrYIJm2uXLkyd5yt77jB7u5/BhBVy/vxePOFEGcG+gadEImgYBciERTsQiSCgl2IRFCwC5EIpbd/irKGWAufyMYkEnY8lnn1+eefh7aBgdzvDYXyDlC8DRUrsMhkyt7e3txxmg1F1rFoS6NIAiqS3Tge7JuZ0TFZhh27d+bMmRPamEzJnlvkP2v/FEmpzHe9sguRCAp2IRJBwS5EIijYhUgEBbsQiaBgFyIRSpfeInmCZQVFshEr8DdjxozQxqQ3lm0WMTQ0FNqKyoOdnXHhH1aoctWqVbnjmzZtCucwmLzGstSi9afSEFkr1mMtKm4JxBl9TAJcvHhxaGP3B5MpDx06NOF555xzTjgnWg/1ehNCKNiFSAUFuxCJoGAXIhEU7EIkQqm78WYW7sazHeYoqYIlHrDkFLYjzGqTRYoBayfFVIH+/v7QxuranXfeeRM+5po1a8I5rJbcjh07Qtvhw4dDW7TGbIeZ7SRHCT4AsH///tAWVTRmCS3MxnbcmSrDahFGSS0dHR3hnE8//TR3nCUF6ZVdiERQsAuRCAp2IRJBwS5EIijYhUgEBbsQiTCu9GZmiwD8FpWWzA5gvbs/ZmYPAvg5gFHd5n53f5Udq6mpKay7xmq/RYkrTM5gCRxMsmOtoaJjsoQQJuWxumRvvvlmaGNJFT09PbnjLMmEJSGxtWKJSNGasIQWlqDE2hqtXr06tEWJK+z+YBJgVIcQAE6cOBHamP9RKydWo/Dtt9/OHWdrWI3OfhLAr9z9XTNrB7DFzF7LbI+6+79WcQwhRIOpptdbH4C+7PGwmX0MIP7GhxDijGRCn9nNbAmA1QDeyobuMbNtZvakmcV1b4UQDafqYDezNgDPA7jX3Y8BeBzABQBWofLK/3Awb52ZbTazzewzjRCivlQV7GbWgkqgP+3uLwCAu/e7+yl3HwHwBIDcL1+7+3p373b3brahI4SoL+MGu1W2LTcA+NjdHxkz3jXm124BsH3y3RNCTBbV7Mb/CMCdAD4ws63Z2P0A7jCzVajIcXsB/GK8A7W0tGD+/Pm5NvYWP6oxFtUXA3iNsba2ttDG5Lwo44nJOCyjjJ2LZQGyzKsoy66vr6+QH0yGmjt3bmiL1n9wcDCcw64Lyx4s0uqLzWHyIMsqY/cBq2sXxQTL9IvkV3ZvVLMb/2cAec+CaupCiDMLfYNOiERQsAuRCAp2IRJBwS5EIijYhUiEUgtONjc3h1llLDssmsMKPbJiiEw+YZJMJK2w7DVmGx4eDm1RoUSAS0ORZMeyoZhcw6Q35n+UbcYKTjL59ciRI6GNSXYs2y9i6tSpoY1lr7F7eNmyZaEtup4s83H37t2540yy1Su7EImgYBciERTsQiSCgl2IRFCwC5EICnYhEqH0Xm9RcUMmn0R58ExmYDIIKxDJpJrIRyavMTmJyWusCCSTB6NsMya9MXmNwfrpRfImuy5M8mLSG7sPItmWZagxPxisECjLLIzuxzfeeCOcE8l17JrolV2IRFCwC5EICnYhEkHBLkQiKNiFSAQFuxCJUKr0NjIyEkoGTJKJpCYmTzEbK17IMsoiG5PymBTCSmuz9WBSU7RWTFJkUhObVyQDjBUCZVIk68HHilhGNtZHjUmp+/btC20fffRRaJszZ05o6+rqyh1n8mB0D1NZObQIIX5QKNiFSAQFuxCJoGAXIhEU7EIkwri78WbWCuB1ANOy3/+9uz9gZksBPAtgDoAtAO5093jrGZVkjKgNUX9/fzgv2nlkO9Zsp7toHbSo5hpLTGEtqthuK9u1LlrzLoLtuLM1Zgk0kWLAnhfbSWYqSVNT04SPyfxg12zPnj2hjakCzP9IGViyZEk4J0o0GhgYCOdU88r+NYDr3P0yVNozX29mVwH4NYBH3X0ZgEEAd1dxLCFEgxg32L3C6J/HluyfA7gOwO+z8Y0Abq6Lh0KISaHa/uxNWQfXAQCvAdgNYMjdR2sQ9wBYUB8XhRCTQVXB7u6n3H0VgIUA1gC4sNoTmNk6M9tsZptZAr8Qor5MaDfe3YcA/AnAXwGYaWajO1MLAeQ2k3b39e7e7e7drKmDEKK+jBvsZjbXzGZmj6cD+AmAj1EJ+r/Jfu0uAC/Vy0khRO1UkwjTBWCjmTWh8sfhOXd/xcw+AvCsmf0zgPcAbBjvQFOmTAmTHViiQzSHSUZMXmNJCYwiiTBHjx4NbawNFZOGGFGtOXYulgjDZEWWkBN9ZGPPi0mRzA9WXy9qQ8WeM0vIYffcqlWrQttll10W2lasWJE7ftVVV4VzDhw4kDvOpMFxg93dtwFYnTO+B5XP70KI7wH6Bp0QiaBgFyIRFOxCJIKCXYhEULALkQhWtPVPoZOZHQQwWsSrA0DcL6c85MfpyI/T+b75cZ67z80zlBrsp53YbLO7dzfk5PJDfiToh97GC5EICnYhEqGRwb6+gecei/w4HflxOj8YPxr2mV0IUS56Gy9EIjQk2M3sejP7xMx2mdl9jfAh82OvmX1gZlvNbHOJ533SzAbMbPuYsdlm9pqZ7cz+n9UgPx40s95sTbaa2Y0l+LHIzP5kZh+Z2Ydm9nfZeKlrQvwodU3MrNXM3jaz9zM//ikbX2pmb2Vx8zszi1P38nD3Uv8BaEKlrNX5AKYCeB/AyrL9yHzZC6CjAee9FsDlALaPGfsXAPdlj+8D8OsG+fEggL8veT26AFyePW4H8CmAlWWvCfGj1DUBYADassctAN4CcBWA5wDcno3/O4C/nchxG/HKvgbALnff45XS088CuKkBfjQMd38dwJHvDN+ESuFOoKQCnoEfpePufe7+bvZ4GJXiKAtQ8poQP0rFK0x6kddGBPsCAPvH/NzIYpUO4I9mtsXM1jXIh1E63X20qP4BAJ0N9OUeM9uWvc2v+8eJsZjZElTqJ7yFBq7Jd/wASl6TehR5TX2D7hp3vxzADQB+aWbXNtohoPKXHZU/RI3gcQAXoNIjoA/Aw2Wd2MzaADwP4F53P61TQ5lrkuNH6WviNRR5jWhEsPcCWDTm57BYZb1x997s/wEAL6KxlXf6zawLALL/49YedcTd+7MbbQTAEyhpTcysBZUAe9rdX8iGS1+TPD8atSbZuSdc5DWiEcH+DoDl2c7iVAC3A3i5bCfM7Gwzax99DOCnALbzWXXlZVQKdwINLOA5GlwZt6CENbFK8bkNAD5290fGmEpdk8iPstekbkVey9ph/M5u442o7HTuBvAPDfLhfFSUgPcBfFimHwCeQeXt4LeofPa6G5WeeZsA7ATwvwBmN8iP/wTwAYBtqARbVwl+XIPKW/RtALZm/24se02IH6WuCYBLUSniug2VPyz/OOaefRvALgD/DWDaRI6rb9AJkQipb9AJkQwKdiESQcEuRCIo2IVIBAW7EImgYBciERTsQiSCgl2IRPg/lpViPKc8RaEAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FfWXTu8jcauQ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        },
        "outputId": "a43391e3-4a34-4983-e2eb-34d93f799066"
      },
      "source": [
        "x_train.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(100000, 32, 32, 1)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d0uYu1WisqLa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class AnnModel(tf.keras.Model):\n",
        "  \"\"\"A class representing a model architecture that is going to be trained in this tutorial. \n",
        "     The model is constructed based on the keys is defined in the `available_genes` dictionary. \n",
        "     Further, it's pre-defined output layer with softmax activation function, that predicts N classes, where N\n",
        "     is evaluated based on the used dataset's out classes.\n",
        "\n",
        "     How to build such a model you can find here:\n",
        "     https://towardsdatascience.com/creating-a-trashy-model-from-scratch-with-tensorflow-2-an-example-with-an-anomaly-detection-27f0d1d7bd00\n",
        "\n",
        "     Arguments:\n",
        "        gene_dictionary: Contains all possible layers that could be added.\n",
        "        model_list: List with layers that, are going to be compiled into the model,\n",
        "              it will be explained further in detail. \n",
        "                         \n",
        "  \"\"\"\n",
        "  def __init__(self, gene_dictionary=None, model_list=None): \n",
        "    super(AnnModel, self).__init__()\n",
        "    self.gene_dictionary = gene_dictionary\n",
        "    self.model_list = model_list\n",
        "    # Last layer to calculate class probabilities\n",
        "    self.last_dense = tf.keras.layers.Dense(num_classes, activation='softmax')\n",
        "    # Current model that is going to be used in the `__call__` method\n",
        "    self.current_model = []\n",
        "    # Flatten layer, to flatten 2D image's vector into an 1D vector.\n",
        "    self.flatten = tf.keras.layers.Flatten()\n",
        "    for el_ind, param in self.model_list:\n",
        "      element, _ = self.gene_dictionary[el_ind]\n",
        "      if param is None:\n",
        "        layer = element()\n",
        "      elif el_ind == 0:\n",
        "        layer = element(param, kernel_size=(3, 3), strides=(2, 2), activation=ACTIVATION)\n",
        "      elif callable(param):\n",
        "        layer = element(param)\n",
        "      elif isinstance(param, float):\n",
        "        layer = element(param)\n",
        "      elif isinstance(param[0], str):\n",
        "        layer = element(**dict([param]))\n",
        "      elif isinstance(param, tuple):\n",
        "        layer = element(param)\n",
        "      self.current_model.append(layer)\n",
        "    self.current_model.append(self.flatten)\n",
        "    self.current_model.append(self.last_dense)\n",
        "\n",
        "  def call(self, inp_x):\n",
        "     \"\"\"Call \n",
        "     \"\"\" \n",
        "     x = inp_x\n",
        "     for layer in self.current_model:\n",
        "       x = layer(x)\n",
        "     return x\n",
        "  \n",
        "def init_train_and_eval(model_list):\n",
        "  \"\"\"Train and evaluated a model passed as the model_list. The evaluation\n",
        "     is based on the highest validation accuracy, during the training. It is\n",
        "     to ommit overfiting due to too high number of epochs.\n",
        "\n",
        "     Arguments:\n",
        "        model_list: List with layers that, are going to be compiled into the model,\n",
        "              it will be explained further in detail.\n",
        "  \"\"\"\n",
        "  try:\n",
        "    model = AnnModel(gene_dictionary=available_genes,\n",
        "                     model_list=model_list)\n",
        "    learning_rate = LEARNING_RATE\n",
        "    optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)\n",
        "    model.compile(optimizer=optimizer, metrics=[\"accuracy\"], loss=\"categorical_crossentropy\")\n",
        "    history = model.fit(x_train, y_train, validation_data=(x_test, y_test),\n",
        "                        epochs=EPOCHS, batch_size=BATCH_SIZE, verbose=0,\n",
        "                        shuffle=True)\n",
        "  except ValueError:\n",
        "    val_acc_last_n = BAD_MODEL_LOSS\n",
        "    return val_acc_last_n\n",
        "  else:\n",
        "    val_acc_last_n = history.history['val_accuracy']\n",
        "    return float(val_acc_last_n[-1])\n",
        "\n"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EcYlzo-DDJKP",
        "colab_type": "text"
      },
      "source": [
        "# Model List\n",
        "Here you can see how `model_list` is constructed. The numbers in the tuple in its first position represent the type of the layer, where the second position represents its parameter (e.g. number of filters). \n",
        "\n",
        "The code line is used to estimate the time needed for a given number of cycles, e.g. you will use 600 cycles and one such calculation take 20 s, then you will need about (600 * 20 s)/3600s/h ~ 3.3h. Remember that in Google Colab you can run code for the maximum of 12h. You can decrease the number of cycles to e.g. 100 just to see the effect of the algorithm, for example, if your time is very limited."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kVBVWBJcDIS3",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "f74722a8-9f57-4f4c-dd88-54aa1b39f534"
      },
      "source": [
        "start_time = time.time()\n",
        "model_list = [(0, 16), (1, None), (0, 16), (1, None), (0, 16), (1, None), (0, 16), (1, None), (1, None), (1, None)]\n",
        "eval_val = init_train_and_eval(model_list)\n",
        "print(model_list, eval_val)\n",
        "print(\"Calculation time: {} s\".format(time.time()-start_time))"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[(0, 16), (1, None), (0, 16), (1, None), (0, 16), (1, None), (0, 16), (1, None), (1, None), (1, None)] 0.5881999731063843\n",
            "Calculation time: 19.604204654693604 s\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f3SA6aWmArnW",
        "colab_type": "text"
      },
      "source": [
        "# Search Space Traversal\n",
        "Below we define three functions:\n",
        "* `random_element` - A function that returns a tuple with random element from the `available_genes` dictionary. You can try those functions separately below and see that it generates just tuples (like in the `model_list`). The probability distribution is weighted, to lower the probability for the identity connection, which is usually not that needed.\n",
        "*`random_architecture` - Returns `model_list` with size of `MODEL_SIZE_MAX` with random elements defined in the `random_element` function.\n",
        "* `mutate arch` - Mutates one architecture by randomly substituting some elements of the architecture using the `random_element` function. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FXACEXwy201a",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def random_element():\n",
        "  \"\"\"Returns an random elements as a tuple with\n",
        "     (element's key, element's parameter).\n",
        "  \"\"\"\n",
        "  min_ind = min(list(available_genes.keys()))\n",
        "  max_ind = max(list(available_genes.keys()))\n",
        "  list_ind = list(range(min_ind, max_ind+1))\n",
        "  random_el = random.choices(population=list_ind,\n",
        "                             k=1)[0]\n",
        "  if random_el == -1:\n",
        "    return (random_el, lambda x: tf.identity(x))\n",
        "  elif random_el == 0:\n",
        "    return (random_el, MAX_NEUR)\n",
        "  elif random_el == 1:\n",
        "    return (random_el, None)\n",
        "  elif random_el == 2:\n",
        "    return (random_el, ('renorm', True))\n",
        "  elif random_el == 3:\n",
        "    return (random_el, (2, 2))\n",
        "\n",
        "def random_architecture(max_model_size=MODEL_SIZE_MAX):\n",
        "  \"\"\"Returns a random architecture as a list of tuples.\n",
        "     \n",
        "     Arguments: \n",
        "        max_model_size - numbers of layers that are going\n",
        "                         to be generated.\n",
        "     Returns: Random architecture as a list of tuples.\n",
        "  \"\"\"\n",
        "\n",
        "  rand_arch_with_params = []\n",
        "  for el in range(max_model_size):\n",
        "    rand_arch_with_params.append(random_element())\n",
        "  return rand_arch_with_params\n",
        "\n",
        "\n",
        "def mutate_arch(parent_model_list):\n",
        "  \"\"\"Mutates the input architecture by substitution. The output architecture\n",
        "     is forced to be different than the output, thus ensuring diversity.\n",
        "\n",
        "     Arguments: \n",
        "          parent_model_list: An architecture as list of tuples\n",
        "\n",
        "     Returns: An architecture different than the input.\n",
        "  \"\"\"\n",
        "  \n",
        "  parent_hash = hash(tuple(parent_model_list))\n",
        "  model_len = len(parent_model_list)\n",
        "  while True:\n",
        "    rand_int = random.randint(0, 1)\n",
        "    child_model = deepcopy(parent_model_list)\n",
        "    if rand_int == 0:\n",
        "    # Otherwise it will mutate the same object\n",
        "      rand_i = random.randint(0, model_len-1)\n",
        "      child_model[rand_i] = random_element()\n",
        "      # Check if the mutation introduces something new \n",
        "      # sometimes it doesn't if not checked\n",
        "    elif rand_int == 1:\n",
        "      while True:\n",
        "        rand_i = random.randint(0, model_len-1)\n",
        "        rand_j = random.randint(0, model_len-1)\n",
        "        if rand_i == rand_j:\n",
        "          continue\n",
        "        else:\n",
        "          temp = child_model[rand_j]\n",
        "          child_model[rand_j] = child_model[rand_i]\n",
        "          child_model[rand_i] = temp\n",
        "          break\n",
        "    if hash(tuple(child_model)) != parent_hash:\n",
        "      break\n",
        "    else:\n",
        "      del child_model\n",
        "      continue\n",
        "  return child_model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jEmMd0n_fA6p",
        "colab_type": "text"
      },
      "source": [
        "# Here you can test how these functions work"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AXS751aUpMU1",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        },
        "outputId": "d5b9b5ad-6ab6-4941-f5f1-8e5c99546215"
      },
      "source": [
        "random_element()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(3, (2, 2))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bv4v9bLdAnEy",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 93
        },
        "outputId": "c2c6508c-3f53-42fb-f213-27be53a63217"
      },
      "source": [
        "random_arch = random_architecture()\n",
        "print(\"The random architecture is: {}\".format(random_arch))\n",
        "print(\"Let's mutate it\")\n",
        "mutated_arch = mutate_arch(random_arch)\n",
        "print(\"Now the architecture is {}\".format(mutated_arch))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The random architecture is: [(0, 16), (-1, <function random_element.<locals>.<lambda> at 0x7f5860a997b8>), (3, (2, 2)), (0, 16), (3, (2, 2)), (0, 16), (3, (2, 2)), (0, 16), (2, ('renorm', True)), (0, 16)]\n",
            "Let's mutate it\n",
            "Now the architecture is [(0, 16), (-1, <function random_element.<locals>.<lambda> at 0x7f5860a997b8>), (3, (2, 2)), (0, 16), (3, (2, 2)), (0, 16), (0, 16), (0, 16), (2, ('renorm', True)), (0, 16)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T2ciQCkqBSDe",
        "colab_type": "text"
      },
      "source": [
        "# Regularized Evolution Algorithm (Aging evolution) "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aHxYBucT-x5R",
        "colab_type": "text"
      },
      "source": [
        "The regularized evolution (i.e. aging evolution) algorithm is written bellow. It's the heart of the Jupyter Notebook, it is responsible for all the work. The explanation is in the docstring and comments above code lines."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6dr709ay2ahZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def regularized_evolution(cycles, population_size, sample_size, init_population=None, init_history=None):\n",
        "  \"\"\"Algorithm for regularized evolution (i.e. aging evolution).\n",
        "     The algorithms accept a number of cycles, population size, and sample selection size as required arguments. \n",
        "     The other two are not mandatory are serve the purpose of restarting the calculations.\n",
        "     Thus immediately you can figure out what are outputs, which are history\n",
        "     population in the last cycle and the best model found during the \n",
        "     evolutionary search.\n",
        "     \n",
        "  \n",
        "     Arguments:\n",
        "          cycles: Number of evolutionary cycles, in one cycle, one mutation and \n",
        "                  one evaluation (training a model and calculation of its performance)\n",
        "                  is performed. The higher the number of cycles, the better model you\n",
        "                  should find, but the computational cost increases linearly.\n",
        "          population_size: The number of individuals held in the population. The bigger\n",
        "                           the population, the more stable is the algorithm, and usually\n",
        "                           converges quicker to better solutions. Nevertheless, the bigger\n",
        "                           it is, the initiation is slower and takes longer to remove \n",
        "                           an bad individual due to aging.\n",
        "          sample_size: The bigger it is, the stronger is evolutionary pressure. If it set\n",
        "                       to one, then individuals are selected randomly, and if it equals \n",
        "                       the population size, then the best individual is always chosen. \n",
        "                       So it should be kept always in-between.\n",
        "\n",
        "  Returns:\n",
        "    history: a list of `Model` instances, with its scores (validation accuracies),\n",
        "    population: the last population in the calculations, helpful for restarts\n",
        "    best_of_all: the best model during calculations  \n",
        "  \"\"\"\n",
        "  \n",
        "  # Best individual, set to something inferior at the beginning\n",
        "  best_of_all = ([0], 1e-16)\n",
        "  # Initiate the population as a FILO queue\n",
        "  population = collections.deque()\n",
        "  history = []  # Not used by the algorithm, only used to report results.\n",
        "\n",
        "  # Initialize the population with random models.\n",
        "  print(\"Initialization of the population, may take a while ...\")\n",
        "  # Track hashes to exlude the same models from the population\n",
        "  # It should increase diversity in the population\n",
        "  hash_pop = []\n",
        "  if init_population is None:\n",
        "    while len(population) < population_size:\n",
        "      random_model = random_architecture()\n",
        "      if hash(tuple(random_model)) in hash_pop:\n",
        "        continue\n",
        "      hash_pop.append(hash(tuple(random_model)))\n",
        "      metric = init_train_and_eval(random_model)\n",
        "      # Exclude faulty models (those which don't compile)\n",
        "      if metric == BAD_MODEL_LOSS:\n",
        "        continue\n",
        "      population.append((random_model, metric))\n",
        "      history.append((random_model, metric))\n",
        "  else:\n",
        "    population = init_population\n",
        "    for individual in population:\n",
        "      hash_pop.append(hash(tuple(individual[0])))\n",
        "  if init_history is not None:\n",
        "    history = init_history\n",
        "  # Carry out evolution in cycles. Each cycle produces a model and removes\n",
        "  # the oldest one from the population\n",
        "  while len(history) - population_size < cycles:\n",
        "    # Sample randomly from population\n",
        "    # So perform tournament selection\n",
        "    sample = []\n",
        "    while len(sample) < sample_size:\n",
        "      candidate = random.choice(list(population))\n",
        "      sample.append(candidate)\n",
        "\n",
        "    # The parent is the best model in the sample\n",
        "    parent = max(sample, key=lambda i: i[1])\n",
        "\n",
        "    # Create the child model by mutating its architecture\n",
        "    child = mutate_arch(parent[0])\n",
        "    # Train and evaluate architecture, validation accuracy\n",
        "    # is returned in the process\n",
        "    child_metric = init_train_and_eval(child)\n",
        "    # If model is not valid - try once again\n",
        "    # e.g. it failed during compilation\n",
        "    if child_metric == BAD_MODEL_LOSS:\n",
        "      continue\n",
        "    # If not failed, then add to the population\n",
        "    population.append((child, child_metric))\n",
        "    # and add to the history\n",
        "    history.append((child, child_metric))\n",
        "    # Remove the oldest model.\n",
        "    population.popleft()\n",
        "    # Best model in the current population\n",
        "    best_candidate = max(list(population), key=lambda i: i[1])\n",
        "    if best_candidate[1] > best_of_all[1]:\n",
        "      # Best model during whole calculation\n",
        "      best_of_all = best_candidate\n",
        "    print(\"Cycle {0}, the best candidate in pop has score {1}, and best in the run {2}\"\\\n",
        "          .format(len(history) - population_size, best_candidate[1], best_of_all[1]))\n",
        "    population_mean = np.mean([x for _, x in list(population)])\n",
        "    population_std = np.std([x for _, x in list(population)])\n",
        "    print(\"Mean {0} and standard deviation {1} of score in the population\".format(population_mean, \n",
        "                                                                                  population_std))\n",
        "  return history, population, best_of_all"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xoMZKGfr1-x3",
        "colab_type": "text"
      },
      "source": [
        "## Std dev of a model optimization\n",
        "The standard deviation of a model during optimization should be small enough to assess if one model is better than another. Since the model, optimization is a noisy process, and while searching for architecture the noise should be small enough to distinguish between noise and signal. Thus you should adjust the search parameter so that, the standard deviation is at least order smaller than its mean, but the smaller, the quicker is convergence. Try it for few models (e.g. 5 as below)\n",
        "\n",
        "To improve standard deviation, you can increase the number of epochs or size of the dataset. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9YkD05a21tIn",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "dbadcf22-4e64-4c24-88f8-66c045e4add2"
      },
      "source": [
        "i = 0\n",
        "while i < 5:\n",
        "  # May throw zeros, because some model may be invalid\n",
        "  random_model = random_architecture()\n",
        "  random_model_values = [init_train_and_eval(random_model) for i in range(10)]\n",
        "  if not np.mean(random_model_values) > 0.0:\n",
        "    continue\n",
        "  i += 1\n",
        "  print(\"Standard deviation for a random model {}\".format(np.std(random_model_values)))\n",
        "  print(\"Mean for a random model {}\".format(np.mean(random_model_values)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Standard deviation for a random model 0.022070837892545207\n",
            "Mean for a random model 0.6199999928474427\n",
            "Standard deviation for a random model 0.05991732791956286\n",
            "Mean for a random model 0.4378299981355667\n",
            "Standard deviation for a random model 0.00389602497855404\n",
            "Mean for a random model 0.5369899988174438\n",
            "Standard deviation for a random model 0.06138765002377174\n",
            "Mean for a random model 0.5193800032138824\n",
            "Standard deviation for a random model 0.0474611964495042\n",
            "Mean for a random model 0.5006199926137924\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fFiwVEOyPrbp",
        "colab_type": "text"
      },
      "source": [
        "# Build a model by yourself\n",
        "Before moving forward, try to build a model by yourself, using available genes. There is a blueprint of a model, you can try to modify it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ewenrm6TPqjm",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        },
        "outputId": "9c3c0c55-f6b2-4fc9-d6ef-b42dbc7214fc"
      },
      "source": [
        "model_list = [(0, 8), (2, ('renorm', True)), (0, 8), (2, ('renorm', True)), (0, 16), (2, ('renorm', True)), (0, 32), (2, ('renorm', True))]\n",
        "eval_val = init_train_and_eval(model_list)\n",
        "print(\"Validation accuracy is: {}\".format(eval_val))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Validation accuracy is: 0.5184000134468079\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0ptMGABxBV61",
        "colab_type": "text"
      },
      "source": [
        "## Here you can finally run the algorithm\n",
        "\n",
        "Generally, the optimization starts with initiating the population, it takes longer when the initial population is bigger. Usually, the initiation of 100 beings takes the same time as evaluating 100 models. \n",
        "There are three parameters that you can adjust:\n",
        "* `cycles` - The bigger the number of cycles, the better model will find and the higher chance of convergence. By convergence, I mean that most of the models are optimal, so you will find on the plot, that most of the models have high accuracy. But usually, it takes a lot of cycles to do it so and is hard to achieve in Google colab.\n",
        "* `population_size` The bigger the population size, the more stable is the algorithm, but it takes more for the initiation. Further, bad individuals can stay longer, but for good ones.\n",
        "* `sample_size` It is a parameter for tournament selection, the size indicates how many individuals is picked randomly from the population and the one with the highest score is chosen. The tournament selection is with replacement, so it can pick up the same individual several times.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ykQshJgS-1W1",
        "colab_type": "text"
      },
      "source": [
        "We run the algorithm for 600 samples, setting the population size to 100 and sample size parameters to 25. If we consider around 19 s per cycle, then it should take around 3.3h for the whole calculation plus 100 cycles for population initialization, which gives 3.9h, which is less than the 12h limit for Google collab, so should succeed. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1izV1iKqFSgS",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "3e8d3709-8f62-4988-bfe6-b304c9bfb609"
      },
      "source": [
        "history_and_model = regularized_evolution(cycles=600, population_size=100, sample_size=25)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Initialization of the population, may take a while ...\n",
            "Cycle 1, the best candidate in pop has score 0.6399999856948853, and best in the run 0.6399999856948853\n",
            "Mean 0.37659900210797786 and standard deviation 0.16652916930544612 of score in the population\n",
            "Cycle 2, the best candidate in pop has score 0.6399999856948853, and best in the run 0.6399999856948853\n",
            "Mean 0.38187300190329554 and standard deviation 0.1660364128573448 of score in the population\n",
            "Cycle 3, the best candidate in pop has score 0.6399999856948853, and best in the run 0.6399999856948853\n",
            "Mean 0.3818510018289089 and standard deviation 0.16600861993094093 of score in the population\n",
            "Cycle 4, the best candidate in pop has score 0.6399999856948853, and best in the run 0.6399999856948853\n",
            "Mean 0.3822720019519329 and standard deviation 0.1665390382978066 of score in the population\n",
            "Cycle 5, the best candidate in pop has score 0.6399999856948853, and best in the run 0.6399999856948853\n",
            "Mean 0.3842670015990734 and standard deviation 0.16824666517881098 of score in the population\n",
            "Cycle 6, the best candidate in pop has score 0.6399999856948853, and best in the run 0.6399999856948853\n",
            "Mean 0.3876330016553402 and standard deviation 0.16967208082980204 of score in the population\n",
            "Cycle 7, the best candidate in pop has score 0.6399999856948853, and best in the run 0.6399999856948853\n",
            "Mean 0.3885920016467571 and standard deviation 0.1707922284011757 of score in the population\n",
            "Cycle 8, the best candidate in pop has score 0.6399999856948853, and best in the run 0.6399999856948853\n",
            "Mean 0.39262100160121916 and standard deviation 0.1703728362757688 of score in the population\n",
            "Cycle 9, the best candidate in pop has score 0.6402000188827515, and best in the run 0.6402000188827515\n",
            "Mean 0.39802300177514555 and standard deviation 0.1695712385273266 of score in the population\n",
            "Cycle 10, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.39966600202023983 and standard deviation 0.17116925344309064 of score in the population\n",
            "Cycle 11, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.40086600206792355 and standard deviation 0.1724400038114945 of score in the population\n",
            "Cycle 12, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.40301500223577025 and standard deviation 0.17210830578671998 of score in the population\n",
            "Cycle 13, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.4037900020927191 and standard deviation 0.17262603360012888 of score in the population\n",
            "Cycle 14, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.4082390021532774 and standard deviation 0.17230707538036558 of score in the population\n",
            "Cycle 15, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.40510000206530095 and standard deviation 0.17486619932278238 of score in the population\n",
            "Cycle 16, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.4090860018879175 and standard deviation 0.17571226994136221 of score in the population\n",
            "Cycle 17, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.40917900182306766 and standard deviation 0.1758215932090033 of score in the population\n",
            "Cycle 18, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.41034400202333926 and standard deviation 0.17684491199667254 of score in the population\n",
            "Cycle 19, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.41530500173568724 and standard deviation 0.1750183710380374 of score in the population\n",
            "Cycle 20, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.4191180017590523 and standard deviation 0.17536091932598463 of score in the population\n",
            "Cycle 21, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.4239410019665957 and standard deviation 0.17313889236495372 of score in the population\n",
            "Cycle 22, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.4277950020879507 and standard deviation 0.17345095767722474 of score in the population\n",
            "Cycle 23, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.42854600213468075 and standard deviation 0.17400160559723715 of score in the population\n",
            "Cycle 24, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.4245200020074844 and standard deviation 0.17687547837884007 of score in the population\n",
            "Cycle 25, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.4278990021348 and standard deviation 0.17758358717339717 of score in the population\n",
            "Cycle 26, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.4264260022342205 and standard deviation 0.17966261108244838 of score in the population\n",
            "Cycle 27, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.4266790024936199 and standard deviation 0.1799475414339481 of score in the population\n",
            "Cycle 28, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.43193300254642963 and standard deviation 0.1779921959806427 of score in the population\n",
            "Cycle 29, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.4321040023118258 and standard deviation 0.17814932986488333 of score in the population\n",
            "Cycle 30, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.43675600200891496 and standard deviation 0.1754701037950559 of score in the population\n",
            "Cycle 31, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.4413410021364689 and standard deviation 0.17440341775113954 of score in the population\n",
            "Cycle 32, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.44101700201630595 and standard deviation 0.17428128095906908 of score in the population\n",
            "Cycle 33, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.44133300200104714 and standard deviation 0.1745393293140953 of score in the population\n",
            "Cycle 34, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.4446870020031929 and standard deviation 0.17324879554752354 of score in the population\n",
            "Cycle 35, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.4449560019373894 and standard deviation 0.17340434848880165 of score in the population\n",
            "Cycle 36, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.4504090016335249 and standard deviation 0.17102854588198696 of score in the population\n",
            "Cycle 37, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.4508720015734434 and standard deviation 0.17126014689622462 of score in the population\n",
            "Cycle 38, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.4526140018552542 and standard deviation 0.17207354519070778 of score in the population\n",
            "Cycle 39, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.45263800166547297 and standard deviation 0.17209621856777488 of score in the population\n",
            "Cycle 40, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.4550040017813444 and standard deviation 0.17305972604449762 of score in the population\n",
            "Cycle 41, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.459690001681447 and standard deviation 0.17167467306708983 of score in the population\n",
            "Cycle 42, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.45969200141727923 and standard deviation 0.17167548040452324 of score in the population\n",
            "Cycle 43, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.46025100119411944 and standard deviation 0.17216443790990746 of score in the population\n",
            "Cycle 44, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.4624860010296106 and standard deviation 0.17269692523670258 of score in the population\n",
            "Cycle 45, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.4665170007199049 and standard deviation 0.17190965516922324 of score in the population\n",
            "Cycle 46, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.46950200088322164 and standard deviation 0.17070106436409724 of score in the population\n",
            "Cycle 47, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.4694180008023977 and standard deviation 0.17082864483647828 of score in the population\n",
            "Cycle 48, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.47016800098121164 and standard deviation 0.17111687943087311 of score in the population\n",
            "Cycle 49, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.47079200081527234 and standard deviation 0.17119533684555716 of score in the population\n",
            "Cycle 50, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.47373000107705593 and standard deviation 0.17138468021373646 of score in the population\n",
            "Cycle 51, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.4738680013269186 and standard deviation 0.17148707284872963 of score in the population\n",
            "Cycle 52, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.4758110012859106 and standard deviation 0.17228412547149843 of score in the population\n",
            "Cycle 53, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.4781250011175871 and standard deviation 0.1728986127436668 of score in the population\n",
            "Cycle 54, the best candidate in pop has score 0.6467000246047974, and best in the run 0.6467000246047974\n",
            "Mean 0.481908001229167 and standard deviation 0.17240634814223718 of score in the population\n",
            "Cycle 55, the best candidate in pop has score 0.6478000283241272, and best in the run 0.6478000283241272\n",
            "Mean 0.4866970015317202 and standard deviation 0.1702835162367657 of score in the population\n",
            "Cycle 56, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.490835001245141 and standard deviation 0.1694414576406976 of score in the population\n",
            "Cycle 57, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.491538001075387 and standard deviation 0.16992438260940493 of score in the population\n",
            "Cycle 58, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.49409300126135347 and standard deviation 0.17019233280737245 of score in the population\n",
            "Cycle 59, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.49657600112259387 and standard deviation 0.1704695310591847 of score in the population\n",
            "Cycle 60, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5007990013808012 and standard deviation 0.16887069348513678 of score in the population\n",
            "Cycle 61, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5011590015143156 and standard deviation 0.16907207414843953 of score in the population\n",
            "Cycle 62, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5020090012997389 and standard deviation 0.16924501000539796 of score in the population\n",
            "Cycle 63, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5023550014942885 and standard deviation 0.16933579265202164 of score in the population\n",
            "Cycle 64, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.4982250013947487 and standard deviation 0.17399803467333053 of score in the population\n",
            "Cycle 65, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5015630009770393 and standard deviation 0.17339857233822314 of score in the population\n",
            "Cycle 66, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5015690007805824 and standard deviation 0.1734020855037537 of score in the population\n",
            "Cycle 67, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5025010004639625 and standard deviation 0.17378416138233574 of score in the population\n",
            "Cycle 68, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5027070006728173 and standard deviation 0.17392894971445988 of score in the population\n",
            "Cycle 69, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5009350004792213 and standard deviation 0.1752000651624695 of score in the population\n",
            "Cycle 70, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.506442000195384 and standard deviation 0.17111850432719414 of score in the population\n",
            "Cycle 71, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5067250000685454 and standard deviation 0.17132042706310946 of score in the population\n",
            "Cycle 72, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5094089999049902 and standard deviation 0.17141961445460213 of score in the population\n",
            "Cycle 73, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5130869998782873 and standard deviation 0.17000955497576856 of score in the population\n",
            "Cycle 74, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5085789997130632 and standard deviation 0.17352980117435002 of score in the population\n",
            "Cycle 75, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5122449994832278 and standard deviation 0.1725656334269108 of score in the population\n",
            "Cycle 76, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5131809992343187 and standard deviation 0.17281651268025097 of score in the population\n",
            "Cycle 77, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5168489990383387 and standard deviation 0.17130260062765043 of score in the population\n",
            "Cycle 78, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.519611999168992 and standard deviation 0.1699052528242646 of score in the population\n",
            "Cycle 79, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5161259991675615 and standard deviation 0.1720576123310155 of score in the population\n",
            "Cycle 80, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5179979992657899 and standard deviation 0.17173240404753254 of score in the population\n",
            "Cycle 81, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.520023998990655 and standard deviation 0.1721929360051234 of score in the population\n",
            "Cycle 82, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5225719989091158 and standard deviation 0.17163271412258013 of score in the population\n",
            "Cycle 83, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5196899988502264 and standard deviation 0.17617079902669427 of score in the population\n",
            "Cycle 84, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5189549987763167 and standard deviation 0.17744316569999327 of score in the population\n",
            "Cycle 85, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5240669985860587 and standard deviation 0.17352179742317375 of score in the population\n",
            "Cycle 86, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5275979983061552 and standard deviation 0.17192965990303077 of score in the population\n",
            "Cycle 87, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5271929981559516 and standard deviation 0.17225547551627596 of score in the population\n",
            "Cycle 88, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5295959981530904 and standard deviation 0.17223811370232725 of score in the population\n",
            "Cycle 89, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.53127999804914 and standard deviation 0.17254252063340345 of score in the population\n",
            "Cycle 90, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5358809980005026 and standard deviation 0.16939033779556184 of score in the population\n",
            "Cycle 91, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.537903997823596 and standard deviation 0.16919504953859485 of score in the population\n",
            "Cycle 92, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5351799976080656 and standard deviation 0.17179200677519327 of score in the population\n",
            "Cycle 93, the best candidate in pop has score 0.6610999703407288, and best in the run 0.6610999703407288\n",
            "Mean 0.5372539975494146 and standard deviation 0.16993463191275385 of score in the population\n",
            "Cycle 94, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.542875997275114 and standard deviation 0.16459154486841393 of score in the population\n",
            "Cycle 95, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5440139971673489 and standard deviation 0.16483581104992862 of score in the population\n",
            "Cycle 96, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5459699971973896 and standard deviation 0.16459959641583163 of score in the population\n",
            "Cycle 97, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5467489971965551 and standard deviation 0.1626600231973373 of score in the population\n",
            "Cycle 98, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5518309973180294 and standard deviation 0.15644287150232455 of score in the population\n",
            "Cycle 99, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5561259971559047 and standard deviation 0.15253318713847774 of score in the population\n",
            "Cycle 100, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5569239972531795 and standard deviation 0.15253567206884944 of score in the population\n",
            "Cycle 101, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5522229973971844 and standard deviation 0.1570792519687242 of score in the population\n",
            "Cycle 102, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5523829977214336 and standard deviation 0.15716387146172697 of score in the population\n",
            "Cycle 103, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5495099978148937 and standard deviation 0.15905540210613636 of score in the population\n",
            "Cycle 104, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5486369977891445 and standard deviation 0.15894626927910072 of score in the population\n",
            "Cycle 105, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5483319978415966 and standard deviation 0.15882616780688508 of score in the population\n",
            "Cycle 106, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5437669977545738 and standard deviation 0.16303849832799872 of score in the population\n",
            "Cycle 107, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5438039979338646 and standard deviation 0.16305959445022986 of score in the population\n",
            "Cycle 108, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5444439980387688 and standard deviation 0.16330344954488046 of score in the population\n",
            "Cycle 109, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5441299977898598 and standard deviation 0.16314914288458088 of score in the population\n",
            "Cycle 110, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5440529975295066 and standard deviation 0.16310252591975513 of score in the population\n",
            "Cycle 111, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5441659978032112 and standard deviation 0.16317438332428696 of score in the population\n",
            "Cycle 112, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5451259979605675 and standard deviation 0.16309290322163134 of score in the population\n",
            "Cycle 113, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5417749983072281 and standard deviation 0.16622092463793914 of score in the population\n",
            "Cycle 114, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5419759982824326 and standard deviation 0.16632280101606095 of score in the population\n",
            "Cycle 115, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5472889985144138 and standard deviation 0.1607748094087949 of score in the population\n",
            "Cycle 116, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5474119986593723 and standard deviation 0.16085313066913456 of score in the population\n",
            "Cycle 117, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.547559998780489 and standard deviation 0.16092701020991942 of score in the population\n",
            "Cycle 118, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5474509985744953 and standard deviation 0.16087962793731359 of score in the population\n",
            "Cycle 119, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5480639989674091 and standard deviation 0.16118033237487786 of score in the population\n",
            "Cycle 120, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5478859986364841 and standard deviation 0.16110662743538912 of score in the population\n",
            "Cycle 121, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5433559983968734 and standard deviation 0.16635569612356862 of score in the population\n",
            "Cycle 122, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5432659983634949 and standard deviation 0.16630981812691104 of score in the population\n",
            "Cycle 123, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5418599981069565 and standard deviation 0.16647681271198192 of score in the population\n",
            "Cycle 124, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5471149983257055 and standard deviation 0.1606376920762702 of score in the population\n",
            "Cycle 125, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5417919981479645 and standard deviation 0.16644124221697068 of score in the population\n",
            "Cycle 126, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5462569983303547 and standard deviation 0.16044834953405518 of score in the population\n",
            "Cycle 127, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5456909979879856 and standard deviation 0.16021031981141862 of score in the population\n",
            "Cycle 128, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5451569981873036 and standard deviation 0.16003264606759854 of score in the population\n",
            "Cycle 129, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5454229982197285 and standard deviation 0.16015262575693526 of score in the population\n",
            "Cycle 130, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5462429983913899 and standard deviation 0.1604614140757455 of score in the population\n",
            "Cycle 131, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5465069980919361 and standard deviation 0.16061669200665785 of score in the population\n",
            "Cycle 132, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5425999980419874 and standard deviation 0.1665682979404931 of score in the population\n",
            "Cycle 133, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5431179981678724 and standard deviation 0.1668244789212179 of score in the population\n",
            "Cycle 134, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5441999983042478 and standard deviation 0.16717590830278414 of score in the population\n",
            "Cycle 135, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5444769983738661 and standard deviation 0.16722231516791905 of score in the population\n",
            "Cycle 136, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5444949983805418 and standard deviation 0.1672332634212077 of score in the population\n",
            "Cycle 137, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.545397998020053 and standard deviation 0.16755479188318761 of score in the population\n",
            "Cycle 138, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5452499978989362 and standard deviation 0.16749624045610778 of score in the population\n",
            "Cycle 139, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5449169980734587 and standard deviation 0.16738752249666647 of score in the population\n",
            "Cycle 140, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5449849980324507 and standard deviation 0.16742848666480398 of score in the population\n",
            "Cycle 141, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5450309979170561 and standard deviation 0.16745540715757623 of score in the population\n",
            "Cycle 142, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5457219981402158 and standard deviation 0.16753079535057547 of score in the population\n",
            "Cycle 143, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5456679981201887 and standard deviation 0.167501813861601 of score in the population\n",
            "Cycle 144, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5454409980028868 and standard deviation 0.16742414843769057 of score in the population\n",
            "Cycle 145, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5454989980906249 and standard deviation 0.16745526499286226 of score in the population\n",
            "Cycle 146, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5410229978710412 and standard deviation 0.1732143037789138 of score in the population\n",
            "Cycle 147, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5434639977663756 and standard deviation 0.17016760616954163 of score in the population\n",
            "Cycle 148, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5441229974478483 and standard deviation 0.17040814656953374 of score in the population\n",
            "Cycle 149, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5452019976824523 and standard deviation 0.1706137287418099 of score in the population\n",
            "Cycle 150, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5443659972399473 and standard deviation 0.17039987802171233 of score in the population\n",
            "Cycle 151, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5398499969393015 and standard deviation 0.17458897639775778 of score in the population\n",
            "Cycle 152, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5400039968639612 and standard deviation 0.17468621794994185 of score in the population\n",
            "Cycle 153, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5401679972559214 and standard deviation 0.1747862815454663 of score in the population\n",
            "Cycle 154, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5397159967571497 and standard deviation 0.1745685117600978 of score in the population\n",
            "Cycle 155, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5389679966121912 and standard deviation 0.17426377269256152 of score in the population\n",
            "Cycle 156, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5387719968706369 and standard deviation 0.17413727336915089 of score in the population\n",
            "Cycle 157, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5387849972397089 and standard deviation 0.17414509500148811 of score in the population\n",
            "Cycle 158, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5379659972339869 and standard deviation 0.17386704704315706 of score in the population\n",
            "Cycle 159, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5378899977356195 and standard deviation 0.17382474162261713 of score in the population\n",
            "Cycle 160, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5379769972711802 and standard deviation 0.17388079700515438 of score in the population\n",
            "Cycle 161, the best candidate in pop has score 0.6621999740600586, and best in the run 0.6621999740600586\n",
            "Mean 0.5379549971967935 and standard deviation 0.17387137907130612 of score in the population\n",
            "Cycle 162, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5388079974800348 and standard deviation 0.17427696219058122 of score in the population\n",
            "Cycle 163, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5397359976917505 and standard deviation 0.17465476678651645 of score in the population\n",
            "Cycle 164, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5452139979600906 and standard deviation 0.16928490764956935 of score in the population\n",
            "Cycle 165, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5453469979763031 and standard deviation 0.16936107534218425 of score in the population\n",
            "Cycle 166, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5414229980111123 and standard deviation 0.17248754663756116 of score in the population\n",
            "Cycle 167, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5418229982256889 and standard deviation 0.17271506991148913 of score in the population\n",
            "Cycle 168, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5421099981665611 and standard deviation 0.17289358257092519 of score in the population\n",
            "Cycle 169, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5457409980893135 and standard deviation 0.1713244755433786 of score in the population\n",
            "Cycle 170, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5456809982657432 and standard deviation 0.17128875409130476 of score in the population\n",
            "Cycle 171, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.54566999822855 and standard deviation 0.17128254561449258 of score in the population\n",
            "Cycle 172, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5457049986720085 and standard deviation 0.17130393136302097 of score in the population\n",
            "Cycle 173, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5456959983706474 and standard deviation 0.17129954162000288 of score in the population\n",
            "Cycle 174, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5505539982020855 and standard deviation 0.16687719325312345 of score in the population\n",
            "Cycle 175, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5506019984185696 and standard deviation 0.1669059608947358 of score in the population\n",
            "Cycle 176, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5511299984157085 and standard deviation 0.16716305949207125 of score in the population\n",
            "Cycle 177, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5512989984452724 and standard deviation 0.16724860315521198 of score in the population\n",
            "Cycle 178, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.552153998464346 and standard deviation 0.16756078438918537 of score in the population\n",
            "Cycle 179, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.555991998463869 and standard deviation 0.1646920294699665 of score in the population\n",
            "Cycle 180, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5564649982750416 and standard deviation 0.16483046466402684 of score in the population\n",
            "Cycle 181, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5548469983041286 and standard deviation 0.16460607198061203 of score in the population\n",
            "Cycle 182, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5547689984738827 and standard deviation 0.16458133855147775 of score in the population\n",
            "Cycle 183, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5603379982709885 and standard deviation 0.1585175190064105 of score in the population\n",
            "Cycle 184, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5645419982075691 and standard deviation 0.15378615626979453 of score in the population\n",
            "Cycle 185, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5644789984822274 and standard deviation 0.15375549461567933 of score in the population\n",
            "Cycle 186, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5642409989237785 and standard deviation 0.15368080533075207 of score in the population\n",
            "Cycle 187, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5668989992141724 and standard deviation 0.1525703950301981 of score in the population\n",
            "Cycle 188, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5667989993095398 and standard deviation 0.15252093381555434 of score in the population\n",
            "Cycle 189, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5667249995470047 and standard deviation 0.1524843771103314 of score in the population\n",
            "Cycle 190, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.566816999912262 and standard deviation 0.1525352479335847 of score in the population\n",
            "Cycle 191, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5666879999637604 and standard deviation 0.15249422780667363 of score in the population\n",
            "Cycle 192, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5707489997148514 and standard deviation 0.14905013095691247 of score in the population\n",
            "Cycle 193, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5724969998002052 and standard deviation 0.14908152250586013 of score in the population\n",
            "Cycle 194, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5720930001139641 and standard deviation 0.14889250741610652 of score in the population\n",
            "Cycle 195, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.572048000395298 and standard deviation 0.14887395556000207 of score in the population\n",
            "Cycle 196, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5718870005011558 and standard deviation 0.14882757299523636 of score in the population\n",
            "Cycle 197, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5754570005834103 and standard deviation 0.14352116360817938 of score in the population\n",
            "Cycle 198, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.575767000168562 and standard deviation 0.14362499406060633 of score in the population\n",
            "Cycle 199, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5759810005128384 and standard deviation 0.1437187228165836 of score in the population\n",
            "Cycle 200, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5762510006129742 and standard deviation 0.14378311670665656 of score in the population\n",
            "Cycle 201, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5803680007159709 and standard deviation 0.13786119730682683 of score in the population\n",
            "Cycle 202, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5803990004956723 and standard deviation 0.1378757151116077 of score in the population\n",
            "Cycle 203, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.583649000376463 and standard deviation 0.1351059097369392 of score in the population\n",
            "Cycle 204, the best candidate in pop has score 0.6638000011444092, and best in the run 0.6638000011444092\n",
            "Mean 0.5801740001142025 and standard deviation 0.1409064203797447 of score in the population\n",
            "Cycle 205, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5808820001780987 and standard deviation 0.14116079782401827 of score in the population\n",
            "Cycle 206, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5856370002031326 and standard deviation 0.13509485017797396 of score in the population\n",
            "Cycle 207, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5856020003557205 and standard deviation 0.13508157692780742 of score in the population\n",
            "Cycle 208, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5857020002603531 and standard deviation 0.13512424657927471 of score in the population\n",
            "Cycle 209, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.586020000576973 and standard deviation 0.1352156190985054 of score in the population\n",
            "Cycle 210, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5856560003757477 and standard deviation 0.13512146871180178 of score in the population\n",
            "Cycle 211, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5855680000782013 and standard deviation 0.13508011467311845 of score in the population\n",
            "Cycle 212, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5864179998636245 and standard deviation 0.13530210377220553 of score in the population\n",
            "Cycle 213, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5902930000424385 and standard deviation 0.13027093580915491 of score in the population\n",
            "Cycle 214, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5905069997906685 and standard deviation 0.13036371691311635 of score in the population\n",
            "Cycle 215, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5905629995465279 and standard deviation 0.1303859514796508 of score in the population\n",
            "Cycle 216, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5855149994790554 and standard deviation 0.13734482926521993 of score in the population\n",
            "Cycle 217, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5854999993741512 and standard deviation 0.1373394839077768 of score in the population\n",
            "Cycle 218, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5859929995238781 and standard deviation 0.1375220880547509 of score in the population\n",
            "Cycle 219, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5830919994413852 and standard deviation 0.13903663384540146 of score in the population\n",
            "Cycle 220, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5835369999706745 and standard deviation 0.1391797405990221 of score in the population\n",
            "Cycle 221, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5888879999518395 and standard deviation 0.13170236488165826 of score in the population\n",
            "Cycle 222, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5891029998660088 and standard deviation 0.13177638386089577 of score in the population\n",
            "Cycle 223, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.591012000143528 and standard deviation 0.13116496820197895 of score in the population\n",
            "Cycle 224, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5912600001692772 and standard deviation 0.13125335745033148 of score in the population\n",
            "Cycle 225, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5939530002325774 and standard deviation 0.1236911076498929 of score in the population\n",
            "Cycle 226, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5948330002278089 and standard deviation 0.12367337008406672 of score in the population\n",
            "Cycle 227, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5953460001200438 and standard deviation 0.1237383130920986 of score in the population\n",
            "Cycle 228, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5957929997891188 and standard deviation 0.12373390765592901 of score in the population\n",
            "Cycle 229, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5960619997233152 and standard deviation 0.12383891716874683 of score in the population\n",
            "Cycle 230, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5948180001229048 and standard deviation 0.12394374569227572 of score in the population\n",
            "Cycle 231, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5948200004547834 and standard deviation 0.12394470242645304 of score in the population\n",
            "Cycle 232, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6003130002319813 and standard deviation 0.11363682756979511 of score in the population\n",
            "Cycle 233, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6004210002720356 and standard deviation 0.11369025865151362 of score in the population\n",
            "Cycle 234, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6003370000422001 and standard deviation 0.11365521617033134 of score in the population\n",
            "Cycle 235, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6010469998419284 and standard deviation 0.11378637659468757 of score in the population\n",
            "Cycle 236, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6011450000107288 and standard deviation 0.11383020994388583 of score in the population\n",
            "Cycle 237, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6009810002148152 and standard deviation 0.11377193580871607 of score in the population\n",
            "Cycle 238, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6015050001442432 and standard deviation 0.11390614518742165 of score in the population\n",
            "Cycle 239, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.59766999989748 and standard deviation 0.12071367600164808 of score in the population\n",
            "Cycle 240, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5970349994301796 and standard deviation 0.12060790907346909 of score in the population\n",
            "Cycle 241, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5970789995789528 and standard deviation 0.12062631026537551 of score in the population\n",
            "Cycle 242, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5973509994149208 and standard deviation 0.12065919348260465 of score in the population\n",
            "Cycle 243, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5973319992423057 and standard deviation 0.1206537435691147 of score in the population\n",
            "Cycle 244, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5979239991307259 and standard deviation 0.12076885631540252 of score in the population\n",
            "Cycle 245, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5979739990830422 and standard deviation 0.12078655450838438 of score in the population\n",
            "Cycle 246, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6023679988086224 and standard deviation 0.11012378960382493 of score in the population\n",
            "Cycle 247, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6040339986979961 and standard deviation 0.10906117029528144 of score in the population\n",
            "Cycle 248, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6041359989345074 and standard deviation 0.10909849469187913 of score in the population\n",
            "Cycle 249, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.604400998800993 and standard deviation 0.10919580909923712 of score in the population\n",
            "Cycle 250, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.605277998894453 and standard deviation 0.1090801793251062 of score in the population\n",
            "Cycle 251, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6101769989728928 and standard deviation 0.09937514240680535 of score in the population\n",
            "Cycle 252, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6088269990682602 and standard deviation 0.09963430320232766 of score in the population\n",
            "Cycle 253, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6066839990019798 and standard deviation 0.10091668793572087 of score in the population\n",
            "Cycle 254, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6065829995274544 and standard deviation 0.1009268792733756 of score in the population\n",
            "Cycle 255, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6064199993014335 and standard deviation 0.10099412540675805 of score in the population\n",
            "Cycle 256, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.605995999276638 and standard deviation 0.10093494614681493 of score in the population\n",
            "Cycle 257, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.605966999232769 and standard deviation 0.10092438147421597 of score in the population\n",
            "Cycle 258, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6064319989085197 and standard deviation 0.10080244421788692 of score in the population\n",
            "Cycle 259, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.605429998934269 and standard deviation 0.10105194321450457 of score in the population\n",
            "Cycle 260, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6052029994130135 and standard deviation 0.10096736925770622 of score in the population\n",
            "Cycle 261, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.604143999516964 and standard deviation 0.10145207156004624 of score in the population\n",
            "Cycle 262, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6039509996771812 and standard deviation 0.10135671300505879 of score in the population\n",
            "Cycle 263, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6036709997057915 and standard deviation 0.10124922431309358 of score in the population\n",
            "Cycle 264, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6037069997191429 and standard deviation 0.1012655470500071 of score in the population\n",
            "Cycle 265, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6037200000882149 and standard deviation 0.10127143132165574 of score in the population\n",
            "Cycle 266, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6073720002174378 and standard deviation 0.09331585704557113 of score in the population\n",
            "Cycle 267, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6070960003137589 and standard deviation 0.09320112849611055 of score in the population\n",
            "Cycle 268, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6065010005235671 and standard deviation 0.09302699106193033 of score in the population\n",
            "Cycle 269, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6059240007400513 and standard deviation 0.09292746531268115 of score in the population\n",
            "Cycle 270, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6049230009317398 and standard deviation 0.09304344336552936 of score in the population\n",
            "Cycle 271, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6049350011348724 and standard deviation 0.09304827603147353 of score in the population\n",
            "Cycle 272, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6049970006942749 and standard deviation 0.09308174173605453 of score in the population\n",
            "Cycle 273, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6052000010013581 and standard deviation 0.09315553859065186 of score in the population\n",
            "Cycle 274, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6053760010004043 and standard deviation 0.09323223577222785 of score in the population\n",
            "Cycle 275, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6053820008039474 and standard deviation 0.0932353195956148 of score in the population\n",
            "Cycle 276, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.60514200091362 and standard deviation 0.09312884945083252 of score in the population\n",
            "Cycle 277, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6025510010123253 and standard deviation 0.09557552358154998 of score in the population\n",
            "Cycle 278, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5996110010147094 and standard deviation 0.09837920982593053 of score in the population\n",
            "Cycle 279, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.599637001156807 and standard deviation 0.09838541398623014 of score in the population\n",
            "Cycle 280, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5999210011959076 and standard deviation 0.09850807887933052 of score in the population\n",
            "Cycle 281, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6013990011811257 and standard deviation 0.0980695906367724 of score in the population\n",
            "Cycle 282, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6015630009770393 and standard deviation 0.09808600936193523 of score in the population\n",
            "Cycle 283, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.600979001224041 and standard deviation 0.0979094407176933 of score in the population\n",
            "Cycle 284, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6013350012898445 and standard deviation 0.09795867083165062 of score in the population\n",
            "Cycle 285, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6012860009074211 and standard deviation 0.09794244277236443 of score in the population\n",
            "Cycle 286, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6012530007958412 and standard deviation 0.09794319058906267 of score in the population\n",
            "Cycle 287, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.601345000565052 and standard deviation 0.09797869131592946 of score in the population\n",
            "Cycle 288, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.598792000412941 and standard deviation 0.10030703269364745 of score in the population\n",
            "Cycle 289, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5989210003614426 and standard deviation 0.10036616526500633 of score in the population\n",
            "Cycle 290, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5988590002059937 and standard deviation 0.10033298097255114 of score in the population\n",
            "Cycle 291, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5991910004615784 and standard deviation 0.100420217529115 of score in the population\n",
            "Cycle 292, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5993730008602143 and standard deviation 0.10051752823334165 of score in the population\n",
            "Cycle 293, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5990640008449555 and standard deviation 0.10037284497481647 of score in the population\n",
            "Cycle 294, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5992410010099412 and standard deviation 0.10042837317336205 of score in the population\n",
            "Cycle 295, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.59564100086689 and standard deviation 0.1055404819319096 of score in the population\n",
            "Cycle 296, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5960690009593964 and standard deviation 0.10567156984014173 of score in the population\n",
            "Cycle 297, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5972970008850098 and standard deviation 0.10566711899542727 of score in the population\n",
            "Cycle 298, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5974640011787414 and standard deviation 0.1057463790137809 of score in the population\n",
            "Cycle 299, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5974420011043549 and standard deviation 0.10573577917911718 of score in the population\n",
            "Cycle 300, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5978510010242463 and standard deviation 0.10591628036776438 of score in the population\n",
            "Cycle 301, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5986540007591248 and standard deviation 0.10609427606358714 of score in the population\n",
            "Cycle 302, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5986990010738373 and standard deviation 0.10611551281756786 of score in the population\n",
            "Cycle 303, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.5988030010461807 and standard deviation 0.10614946560854686 of score in the population\n",
            "Cycle 304, the best candidate in pop has score 0.6665999889373779, and best in the run 0.6665999889373779\n",
            "Mean 0.6035180008411407 and standard deviation 0.09746046379246495 of score in the population\n",
            "Cycle 305, the best candidate in pop has score 0.6647999882698059, and best in the run 0.6665999889373779\n",
            "Mean 0.6033950006961822 and standard deviation 0.09738850851005204 of score in the population\n",
            "Cycle 306, the best candidate in pop has score 0.6647999882698059, and best in the run 0.6665999889373779\n",
            "Mean 0.603520000576973 and standard deviation 0.09744957518456458 of score in the population\n",
            "Cycle 307, the best candidate in pop has score 0.6647999882698059, and best in the run 0.6665999889373779\n",
            "Mean 0.603468000292778 and standard deviation 0.09743409600316762 of score in the population\n",
            "Cycle 308, the best candidate in pop has score 0.6647999882698059, and best in the run 0.6665999889373779\n",
            "Mean 0.6034380000829697 and standard deviation 0.09742074844139206 of score in the population\n",
            "Cycle 309, the best candidate in pop has score 0.6647999882698059, and best in the run 0.6665999889373779\n",
            "Mean 0.600600999891758 and standard deviation 0.10038303301847797 of score in the population\n",
            "Cycle 310, the best candidate in pop has score 0.6647999882698059, and best in the run 0.6665999889373779\n",
            "Mean 0.6011640003323555 and standard deviation 0.10055040601468836 of score in the population\n",
            "Cycle 311, the best candidate in pop has score 0.6647999882698059, and best in the run 0.6665999889373779\n",
            "Mean 0.60127700060606 and standard deviation 0.10060560344488166 of score in the population\n",
            "Cycle 312, the best candidate in pop has score 0.6647999882698059, and best in the run 0.6665999889373779\n",
            "Mean 0.6005220004916191 and standard deviation 0.10041668283454336 of score in the population\n",
            "Cycle 313, the best candidate in pop has score 0.6647999882698059, and best in the run 0.6665999889373779\n",
            "Mean 0.6010050001740456 and standard deviation 0.10057570152440283 of score in the population\n",
            "Cycle 314, the best candidate in pop has score 0.6647999882698059, and best in the run 0.6665999889373779\n",
            "Mean 0.6003720000386238 and standard deviation 0.10041658577257886 of score in the population\n",
            "Cycle 315, the best candidate in pop has score 0.6647999882698059, and best in the run 0.6665999889373779\n",
            "Mean 0.5997959998250008 and standard deviation 0.10032352602643138 of score in the population\n",
            "Cycle 316, the best candidate in pop has score 0.6647999882698059, and best in the run 0.6665999889373779\n",
            "Mean 0.6047199995815754 and standard deviation 0.08970163555810651 of score in the population\n",
            "Cycle 317, the best candidate in pop has score 0.6647999882698059, and best in the run 0.6665999889373779\n",
            "Mean 0.6048179991543293 and standard deviation 0.08973858850799586 of score in the population\n",
            "Cycle 318, the best candidate in pop has score 0.6647999882698059, and best in the run 0.6665999889373779\n",
            "Mean 0.6045509989559651 and standard deviation 0.08960976774967255 of score in the population\n",
            "Cycle 319, the best candidate in pop has score 0.6647999882698059, and best in the run 0.6665999889373779\n",
            "Mean 0.6072939990460873 and standard deviation 0.08644785927787417 of score in the population\n",
            "Cycle 320, the best candidate in pop has score 0.6647999882698059, and best in the run 0.6665999889373779\n",
            "Mean 0.6070819990336895 and standard deviation 0.08636809175718886 of score in the population\n",
            "Cycle 321, the best candidate in pop has score 0.6647999882698059, and best in the run 0.6665999889373779\n",
            "Mean 0.6068789993226528 and standard deviation 0.08625668251636544 of score in the population\n",
            "Cycle 322, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6071309994161129 and standard deviation 0.08640466173551234 of score in the population\n",
            "Cycle 323, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6070949994027615 and standard deviation 0.08639029172160945 of score in the population\n",
            "Cycle 324, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.606754999011755 and standard deviation 0.08628642696412445 of score in the population\n",
            "Cycle 325, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6095629988610745 and standard deviation 0.08302049007014316 of score in the population\n",
            "Cycle 326, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6053349986672402 and standard deviation 0.09186630766350057 of score in the population\n",
            "Cycle 327, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6052219989895821 and standard deviation 0.09183497122977424 of score in the population\n",
            "Cycle 328, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6053479990363121 and standard deviation 0.09185927346575555 of score in the population\n",
            "Cycle 329, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6030809989571572 and standard deviation 0.09332497938700081 of score in the population\n",
            "Cycle 330, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.602906998693943 and standard deviation 0.09349057134649916 of score in the population\n",
            "Cycle 331, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6024299988150597 and standard deviation 0.09334923011351152 of score in the population\n",
            "Cycle 332, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6024529990553856 and standard deviation 0.09336105813724523 of score in the population\n",
            "Cycle 333, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6023749992251396 and standard deviation 0.09331452286691148 of score in the population\n",
            "Cycle 334, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6024579992890358 and standard deviation 0.09335483674225598 of score in the population\n",
            "Cycle 335, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.602264999449253 and standard deviation 0.09326136821583599 of score in the population\n",
            "Cycle 336, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6018549993634223 and standard deviation 0.09311027874479329 of score in the population\n",
            "Cycle 337, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6007549992203712 and standard deviation 0.09338166316057733 of score in the population\n",
            "Cycle 338, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6006649991869927 and standard deviation 0.0933321210068463 of score in the population\n",
            "Cycle 339, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6051869989931583 and standard deviation 0.08430926890107963 of score in the population\n",
            "Cycle 340, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6055589993298054 and standard deviation 0.08430453430761725 of score in the population\n",
            "Cycle 341, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6056139995157719 and standard deviation 0.08433510245916886 of score in the population\n",
            "Cycle 342, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6058439995348454 and standard deviation 0.08442006972136033 of score in the population\n",
            "Cycle 343, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6059459997713565 and standard deviation 0.08445655690904642 of score in the population\n",
            "Cycle 344, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6057470001280307 and standard deviation 0.08437427594009424 of score in the population\n",
            "Cycle 345, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6058060003817082 and standard deviation 0.08440250312067779 of score in the population\n",
            "Cycle 346, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6070280005037785 and standard deviation 0.08432113927287417 of score in the population\n",
            "Cycle 347, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6073050005733966 and standard deviation 0.08439759226076603 of score in the population\n",
            "Cycle 348, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6072890003025532 and standard deviation 0.08438981847042352 of score in the population\n",
            "Cycle 349, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6071660001575947 and standard deviation 0.08432533853494031 of score in the population\n",
            "Cycle 350, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6073850001394748 and standard deviation 0.08442390387416235 of score in the population\n",
            "Cycle 351, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6021740001440048 and standard deviation 0.09701114976415967 of score in the population\n",
            "Cycle 352, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6034620004892349 and standard deviation 0.09680490073166254 of score in the population\n",
            "Cycle 353, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6051510003209114 and standard deviation 0.09541205569426563 of score in the population\n",
            "Cycle 354, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6057589998841286 and standard deviation 0.09551615509437476 of score in the population\n",
            "Cycle 355, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6066440001130105 and standard deviation 0.09546748520484685 of score in the population\n",
            "Cycle 356, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6063280001282692 and standard deviation 0.09554420069372135 of score in the population\n",
            "Cycle 357, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.606429000198841 and standard deviation 0.09558644537845935 of score in the population\n",
            "Cycle 358, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6057870003581047 and standard deviation 0.0958226257727649 of score in the population\n",
            "Cycle 359, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6069290003180504 and standard deviation 0.09560119872652571 of score in the population\n",
            "Cycle 360, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6069519999623298 and standard deviation 0.09560740775059559 of score in the population\n",
            "Cycle 361, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.608401999771595 and standard deviation 0.09515473244502769 of score in the population\n",
            "Cycle 362, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6081299993395806 and standard deviation 0.09509001100693507 of score in the population\n",
            "Cycle 363, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6084429994225502 and standard deviation 0.09520864437902365 of score in the population\n",
            "Cycle 364, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6084369990229607 and standard deviation 0.09520595575001031 of score in the population\n",
            "Cycle 365, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6084549990296364 and standard deviation 0.09521401974092861 of score in the population\n",
            "Cycle 366, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6081409987807274 and standard deviation 0.09537151946403316 of score in the population\n",
            "Cycle 367, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.60836699873209 and standard deviation 0.09545500465727179 of score in the population\n",
            "Cycle 368, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6085659983754158 and standard deviation 0.09546706104870811 of score in the population\n",
            "Cycle 369, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6092349985241889 and standard deviation 0.09559276225576543 of score in the population\n",
            "Cycle 370, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6102219983935356 and standard deviation 0.09542970947527601 of score in the population\n",
            "Cycle 371, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6094959983229638 and standard deviation 0.09545373912020426 of score in the population\n",
            "Cycle 372, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6084829983115196 and standard deviation 0.09546795363977867 of score in the population\n",
            "Cycle 373, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6078659984469413 and standard deviation 0.09540281437461341 of score in the population\n",
            "Cycle 374, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6076939985156059 and standard deviation 0.09533370296290174 of score in the population\n",
            "Cycle 375, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6063579985499382 and standard deviation 0.09561672888600971 of score in the population\n",
            "Cycle 376, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6066219988465309 and standard deviation 0.09573071893217985 of score in the population\n",
            "Cycle 377, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6093279987573623 and standard deviation 0.0932252959873908 of score in the population\n",
            "Cycle 378, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6122379985451698 and standard deviation 0.08993064444046264 of score in the population\n",
            "Cycle 379, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6124099984765052 and standard deviation 0.08997018037509348 of score in the population\n",
            "Cycle 380, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.612184998691082 and standard deviation 0.08988773413851958 of score in the population\n",
            "Cycle 381, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6122379985451698 and standard deviation 0.089908862839921 of score in the population\n",
            "Cycle 382, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.612459999024868 and standard deviation 0.08995391643285752 of score in the population\n",
            "Cycle 383, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6127739986777305 and standard deviation 0.08997061074084695 of score in the population\n",
            "Cycle 384, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6125899985432625 and standard deviation 0.08994889312726677 of score in the population\n",
            "Cycle 385, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6118069985508918 and standard deviation 0.09012324542343868 of score in the population\n",
            "Cycle 386, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6120409986376762 and standard deviation 0.0901159128032317 of score in the population\n",
            "Cycle 387, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6120669987797737 and standard deviation 0.090125417848395 of score in the population\n",
            "Cycle 388, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6147589987516403 and standard deviation 0.08718571988012795 of score in the population\n",
            "Cycle 389, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6145109987258911 and standard deviation 0.08711667063943118 of score in the population\n",
            "Cycle 390, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6097649985551834 and standard deviation 0.09740187789768379 of score in the population\n",
            "Cycle 391, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6099289983510972 and standard deviation 0.09746979834336977 of score in the population\n",
            "Cycle 392, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6049439981579781 and standard deviation 0.10695975401021586 of score in the population\n",
            "Cycle 393, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6051729980111122 and standard deviation 0.10703935106087863 of score in the population\n",
            "Cycle 394, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6053109976649285 and standard deviation 0.10709240055629926 of score in the population\n",
            "Cycle 395, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6087569978833198 and standard deviation 0.10169417857785651 of score in the population\n",
            "Cycle 396, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6088379976153374 and standard deviation 0.10172989747599373 of score in the population\n",
            "Cycle 397, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6087839975953102 and standard deviation 0.10170537658471529 of score in the population\n",
            "Cycle 398, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6085859975218773 and standard deviation 0.10163270576266964 of score in the population\n",
            "Cycle 399, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6087019976973533 and standard deviation 0.10168343371456297 of score in the population\n",
            "Cycle 400, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6084269979596137 and standard deviation 0.1015684679775843 of score in the population\n",
            "Cycle 401, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6083729979395867 and standard deviation 0.10154145600150788 of score in the population\n",
            "Cycle 402, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6083829978108406 and standard deviation 0.10154570259825914 of score in the population\n",
            "Cycle 403, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6085729977488518 and standard deviation 0.10161981139429022 of score in the population\n",
            "Cycle 404, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6085429981350898 and standard deviation 0.10160822593712499 of score in the population\n",
            "Cycle 405, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6059399983286857 and standard deviation 0.10371502286736038 of score in the population\n",
            "Cycle 406, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6059569981694222 and standard deviation 0.1037235788159213 of score in the population\n",
            "Cycle 407, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6059879985451698 and standard deviation 0.10373119312179466 of score in the population\n",
            "Cycle 408, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6059859988093376 and standard deviation 0.10373043716887602 of score in the population\n",
            "Cycle 409, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.608769998550415 and standard deviation 0.10069959146149113 of score in the population\n",
            "Cycle 410, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6035789982974529 and standard deviation 0.1108497165780497 of score in the population\n",
            "Cycle 411, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6034999983012677 and standard deviation 0.11081513887390246 of score in the population\n",
            "Cycle 412, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.603733998388052 and standard deviation 0.11080750085170087 of score in the population\n",
            "Cycle 413, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6036359982192516 and standard deviation 0.11076378871785796 of score in the population\n",
            "Cycle 414, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.604241998642683 and standard deviation 0.1108767695882524 of score in the population\n",
            "Cycle 415, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6045069991052151 and standard deviation 0.11086810653290351 of score in the population\n",
            "Cycle 416, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6012489993870258 and standard deviation 0.11440498468679104 of score in the population\n",
            "Cycle 417, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6012509997189045 and standard deviation 0.11440572515938578 of score in the population\n",
            "Cycle 418, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6013839997351169 and standard deviation 0.11445213843126777 of score in the population\n",
            "Cycle 419, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.6012859995663166 and standard deviation 0.11442185297674515 of score in the population\n",
            "Cycle 420, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.600998999029398 and standard deviation 0.11438771664758539 of score in the population\n",
            "Cycle 421, the best candidate in pop has score 0.6703000068664551, and best in the run 0.6703000068664551\n",
            "Mean 0.5993549986183644 and standard deviation 0.11493509343189927 of score in the population\n",
            "Cycle 422, the best candidate in pop has score 0.661899983882904, and best in the run 0.6703000068664551\n",
            "Mean 0.5992269988358021 and standard deviation 0.11486311775948539 of score in the population\n",
            "Cycle 423, the best candidate in pop has score 0.661899983882904, and best in the run 0.6703000068664551\n",
            "Mean 0.5991879989206791 and standard deviation 0.11484999656780973 of score in the population\n",
            "Cycle 424, the best candidate in pop has score 0.661899983882904, and best in the run 0.6703000068664551\n",
            "Mean 0.5994319988787175 and standard deviation 0.11491199424893314 of score in the population\n",
            "Cycle 425, the best candidate in pop has score 0.661899983882904, and best in the run 0.6703000068664551\n",
            "Mean 0.5992959989607334 and standard deviation 0.11485998362369697 of score in the population\n",
            "Cycle 426, the best candidate in pop has score 0.661899983882904, and best in the run 0.6703000068664551\n",
            "Mean 0.5992069989442825 and standard deviation 0.11516224606020385 of score in the population\n",
            "Cycle 427, the best candidate in pop has score 0.661899983882904, and best in the run 0.6703000068664551\n",
            "Mean 0.5989429986476899 and standard deviation 0.11513284200633657 of score in the population\n",
            "Cycle 428, the best candidate in pop has score 0.661899983882904, and best in the run 0.6703000068664551\n",
            "Mean 0.5991099989414215 and standard deviation 0.11518885182551543 of score in the population\n",
            "Cycle 429, the best candidate in pop has score 0.661899983882904, and best in the run 0.6703000068664551\n",
            "Mean 0.5999739992618561 and standard deviation 0.1142448272260191 of score in the population\n",
            "Cycle 430, the best candidate in pop has score 0.661899983882904, and best in the run 0.6703000068664551\n",
            "Mean 0.6004939991235733 and standard deviation 0.11393108971071843 of score in the population\n",
            "Cycle 431, the best candidate in pop has score 0.661899983882904, and best in the run 0.6703000068664551\n",
            "Mean 0.6008659988641739 and standard deviation 0.11401079634445652 of score in the population\n",
            "Cycle 432, the best candidate in pop has score 0.661899983882904, and best in the run 0.6703000068664551\n",
            "Mean 0.600783998966217 and standard deviation 0.11397722136934396 of score in the population\n",
            "Cycle 433, the best candidate in pop has score 0.661899983882904, and best in the run 0.6703000068664551\n",
            "Mean 0.5965269987285137 and standard deviation 0.11970842090533529 of score in the population\n",
            "Cycle 434, the best candidate in pop has score 0.661899983882904, and best in the run 0.6703000068664551\n",
            "Mean 0.596502998918295 and standard deviation 0.11969755710130184 of score in the population\n",
            "Cycle 435, the best candidate in pop has score 0.661899983882904, and best in the run 0.6703000068664551\n",
            "Mean 0.5965829990804196 and standard deviation 0.1197278677307815 of score in the population\n",
            "Cycle 436, the best candidate in pop has score 0.661899983882904, and best in the run 0.6703000068664551\n",
            "Mean 0.5968159990012646 and standard deviation 0.11978789015918935 of score in the population\n",
            "Cycle 437, the best candidate in pop has score 0.661899983882904, and best in the run 0.6703000068664551\n",
            "Mean 0.5965859989821911 and standard deviation 0.11995079447692185 of score in the population\n",
            "Cycle 438, the best candidate in pop has score 0.661899983882904, and best in the run 0.6703000068664551\n",
            "Mean 0.5965019987523555 and standard deviation 0.11991797725388732 of score in the population\n",
            "Cycle 439, the best candidate in pop has score 0.661899983882904, and best in the run 0.6703000068664551\n",
            "Mean 0.5963269989192486 and standard deviation 0.11984990139342215 of score in the population\n",
            "Cycle 440, the best candidate in pop has score 0.661899983882904, and best in the run 0.6703000068664551\n",
            "Mean 0.596449999064207 and standard deviation 0.11988341672961011 of score in the population\n",
            "Cycle 441, the best candidate in pop has score 0.661899983882904, and best in the run 0.6703000068664551\n",
            "Mean 0.5952259992063046 and standard deviation 0.1199021809623467 of score in the population\n",
            "Cycle 442, the best candidate in pop has score 0.661899983882904, and best in the run 0.6703000068664551\n",
            "Mean 0.5952829991281032 and standard deviation 0.11992879753590473 of score in the population\n",
            "Cycle 443, the best candidate in pop has score 0.661899983882904, and best in the run 0.6703000068664551\n",
            "Mean 0.5953639988601208 and standard deviation 0.11996251307643897 of score in the population\n",
            "Cycle 444, the best candidate in pop has score 0.661899983882904, and best in the run 0.6703000068664551\n",
            "Mean 0.5955739985406399 and standard deviation 0.12004271544684533 of score in the population\n",
            "Cycle 445, the best candidate in pop has score 0.661899983882904, and best in the run 0.6703000068664551\n",
            "Mean 0.5941899983584881 and standard deviation 0.1202153223123152 of score in the population\n",
            "Cycle 446, the best candidate in pop has score 0.6610000133514404, and best in the run 0.6703000068664551\n",
            "Mean 0.5941659985482692 and standard deviation 0.12020204111472893 of score in the population\n",
            "Cycle 447, the best candidate in pop has score 0.6610000133514404, and best in the run 0.6703000068664551\n",
            "Mean 0.5935399989783764 and standard deviation 0.12010228412567218 of score in the population\n",
            "Cycle 448, the best candidate in pop has score 0.6610000133514404, and best in the run 0.6703000068664551\n",
            "Mean 0.5935939989984036 and standard deviation 0.1201277445737331 of score in the population\n",
            "Cycle 449, the best candidate in pop has score 0.6610000133514404, and best in the run 0.6703000068664551\n",
            "Mean 0.5934229992330075 and standard deviation 0.12006617529191846 of score in the population\n",
            "Cycle 450, the best candidate in pop has score 0.6610000133514404, and best in the run 0.6703000068664551\n",
            "Mean 0.5933819995820522 and standard deviation 0.12004542971657678 of score in the population\n",
            "Cycle 451, the best candidate in pop has score 0.6610000133514404, and best in the run 0.6703000068664551\n",
            "Mean 0.5977939993143082 and standard deviation 0.11048028565914933 of score in the population\n",
            "Cycle 452, the best candidate in pop has score 0.6610000133514404, and best in the run 0.6703000068664551\n",
            "Mean 0.5975599992275238 and standard deviation 0.11039060802616942 of score in the population\n",
            "Cycle 453, the best candidate in pop has score 0.6610000133514404, and best in the run 0.6703000068664551\n",
            "Mean 0.5979599994421005 and standard deviation 0.11050555803160314 of score in the population\n",
            "Cycle 454, the best candidate in pop has score 0.6610000133514404, and best in the run 0.6703000068664551\n",
            "Mean 0.5977099996805191 and standard deviation 0.11041080507749494 of score in the population\n",
            "Cycle 455, the best candidate in pop has score 0.6610000133514404, and best in the run 0.6703000068664551\n",
            "Mean 0.5977519994974136 and standard deviation 0.11042965931378652 of score in the population\n",
            "Cycle 456, the best candidate in pop has score 0.6610000133514404, and best in the run 0.6703000068664551\n",
            "Mean 0.5980429995059967 and standard deviation 0.11038789067941965 of score in the population\n",
            "Cycle 457, the best candidate in pop has score 0.6610000133514404, and best in the run 0.6703000068664551\n",
            "Mean 0.5980359989404679 and standard deviation 0.11038452881479058 of score in the population\n",
            "Cycle 458, the best candidate in pop has score 0.6610000133514404, and best in the run 0.6703000068664551\n",
            "Mean 0.598879998922348 and standard deviation 0.11025096417039734 of score in the population\n",
            "Cycle 459, the best candidate in pop has score 0.6610000133514404, and best in the run 0.6703000068664551\n",
            "Mean 0.5988539987802506 and standard deviation 0.1102404144171017 of score in the population\n",
            "Cycle 460, the best candidate in pop has score 0.6610000133514404, and best in the run 0.6703000068664551\n",
            "Mean 0.5990809988975525 and standard deviation 0.11033567525457944 of score in the population\n",
            "Cycle 461, the best candidate in pop has score 0.6610000133514404, and best in the run 0.6703000068664551\n",
            "Mean 0.599005998969078 and standard deviation 0.11030331032003685 of score in the population\n",
            "Cycle 462, the best candidate in pop has score 0.6610000133514404, and best in the run 0.6703000068664551\n",
            "Mean 0.5994249993562698 and standard deviation 0.11045148801122347 of score in the population\n",
            "Cycle 463, the best candidate in pop has score 0.6610000133514404, and best in the run 0.6703000068664551\n",
            "Mean 0.5993059992790222 and standard deviation 0.11039255536022342 of score in the population\n",
            "Cycle 464, the best candidate in pop has score 0.6664999723434448, and best in the run 0.6703000068664551\n",
            "Mean 0.5994629991054535 and standard deviation 0.1104768103254309 of score in the population\n",
            "Cycle 465, the best candidate in pop has score 0.666700005531311, and best in the run 0.6703000068664551\n",
            "Mean 0.5996099990606308 and standard deviation 0.11055636926868533 of score in the population\n",
            "Cycle 466, the best candidate in pop has score 0.666700005531311, and best in the run 0.6703000068664551\n",
            "Mean 0.600676999092102 and standard deviation 0.1105371278207893 of score in the population\n",
            "Cycle 467, the best candidate in pop has score 0.666700005531311, and best in the run 0.6703000068664551\n",
            "Mean 0.599974998831749 and standard deviation 0.11041401876881528 of score in the population\n",
            "Cycle 468, the best candidate in pop has score 0.666700005531311, and best in the run 0.6703000068664551\n",
            "Mean 0.6002539992332458 and standard deviation 0.11051008725183264 of score in the population\n",
            "Cycle 469, the best candidate in pop has score 0.666700005531311, and best in the run 0.6703000068664551\n",
            "Mean 0.6001659989356994 and standard deviation 0.11046573183950423 of score in the population\n",
            "Cycle 470, the best candidate in pop has score 0.666700005531311, and best in the run 0.6703000068664551\n",
            "Mean 0.59556799903512 and standard deviation 0.117894174495837 of score in the population\n",
            "Cycle 471, the best candidate in pop has score 0.666700005531311, and best in the run 0.6703000068664551\n",
            "Mean 0.5965109987556935 and standard deviation 0.11806610498380789 of score in the population\n",
            "Cycle 472, the best candidate in pop has score 0.666700005531311, and best in the run 0.6703000068664551\n",
            "Mean 0.5973039992153645 and standard deviation 0.11806437548802516 of score in the population\n",
            "Cycle 473, the best candidate in pop has score 0.666700005531311, and best in the run 0.6703000068664551\n",
            "Mean 0.594908999055624 and standard deviation 0.1206420464751607 of score in the population\n",
            "Cycle 474, the best candidate in pop has score 0.666700005531311, and best in the run 0.6703000068664551\n",
            "Mean 0.5945029990375041 and standard deviation 0.12056632306967369 of score in the population\n",
            "Cycle 475, the best candidate in pop has score 0.666700005531311, and best in the run 0.6703000068664551\n",
            "Mean 0.5944559989869594 and standard deviation 0.12059626959920944 of score in the population\n",
            "Cycle 476, the best candidate in pop has score 0.666700005531311, and best in the run 0.6703000068664551\n",
            "Mean 0.5942569987475872 and standard deviation 0.12050268129839398 of score in the population\n",
            "Cycle 477, the best candidate in pop has score 0.666700005531311, and best in the run 0.6703000068664551\n",
            "Mean 0.5941889987885952 and standard deviation 0.1204698474078747 of score in the population\n",
            "Cycle 478, the best candidate in pop has score 0.666700005531311, and best in the run 0.6703000068664551\n",
            "Mean 0.5941979990899563 and standard deviation 0.12047424449677066 of score in the population\n",
            "Cycle 479, the best candidate in pop has score 0.666700005531311, and best in the run 0.6703000068664551\n",
            "Mean 0.5939359991252422 and standard deviation 0.1203993384093503 of score in the population\n",
            "Cycle 480, the best candidate in pop has score 0.666700005531311, and best in the run 0.6703000068664551\n",
            "Mean 0.5937589989602565 and standard deviation 0.12035331152685455 of score in the population\n",
            "Cycle 481, the best candidate in pop has score 0.666700005531311, and best in the run 0.6703000068664551\n",
            "Mean 0.589820999354124 and standard deviation 0.12478673317645461 of score in the population\n",
            "Cycle 482, the best candidate in pop has score 0.666700005531311, and best in the run 0.6703000068664551\n",
            "Mean 0.589859999269247 and standard deviation 0.12480354927136915 of score in the population\n",
            "Cycle 483, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5902049992978573 and standard deviation 0.12497017633088732 of score in the population\n",
            "Cycle 484, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5905399994552135 and standard deviation 0.12507863487342444 of score in the population\n",
            "Cycle 485, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5915789996087552 and standard deviation 0.12519396681350853 of score in the population\n",
            "Cycle 486, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5909659992158413 and standard deviation 0.12519946291138925 of score in the population\n",
            "Cycle 487, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5910409991443157 and standard deviation 0.12523482934916813 of score in the population\n",
            "Cycle 488, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5904239992797374 and standard deviation 0.12508882640013608 of score in the population\n",
            "Cycle 489, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5907739989459515 and standard deviation 0.12523815373437985 of score in the population\n",
            "Cycle 490, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.594429998844862 and standard deviation 0.11818202786903712 of score in the population\n",
            "Cycle 491, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5943959985673427 and standard deviation 0.11816410690890926 of score in the population\n",
            "Cycle 492, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.599197998791933 and standard deviation 0.11003750535999024 of score in the population\n",
            "Cycle 493, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5988679988682271 and standard deviation 0.10992298382179357 of score in the population\n",
            "Cycle 494, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5988749988377094 and standard deviation 0.10992647209730694 of score in the population\n",
            "Cycle 495, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5989639987051487 and standard deviation 0.1099437405869555 of score in the population\n",
            "Cycle 496, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5984339989721775 and standard deviation 0.10978695293112282 of score in the population\n",
            "Cycle 497, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5981399990618229 and standard deviation 0.10968162559710404 of score in the population\n",
            "Cycle 498, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5980529989302158 and standard deviation 0.1096549281963549 of score in the population\n",
            "Cycle 499, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5930029986798764 and standard deviation 0.11804404288008258 of score in the population\n",
            "Cycle 500, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5929309986531734 and standard deviation 0.11801919548620991 of score in the population\n",
            "Cycle 501, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5930179987847805 and standard deviation 0.1180692206135581 of score in the population\n",
            "Cycle 502, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5931359986960888 and standard deviation 0.1181339878037156 of score in the population\n",
            "Cycle 503, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5929989986121654 and standard deviation 0.11806707442997531 of score in the population\n",
            "Cycle 504, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5930269984900951 and standard deviation 0.11808004286855604 of score in the population\n",
            "Cycle 505, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5951789985597133 and standard deviation 0.11638197280339814 of score in the population\n",
            "Cycle 506, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5950239984691144 and standard deviation 0.1163071689782573 of score in the population\n",
            "Cycle 507, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5948659984767437 and standard deviation 0.116266197024203 of score in the population\n",
            "Cycle 508, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5933969984948635 and standard deviation 0.1165499001150601 of score in the population\n",
            "Cycle 509, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5935959987342357 and standard deviation 0.11663823186338598 of score in the population\n",
            "Cycle 510, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5987989985942841 and standard deviation 0.10752825495748856 of score in the population\n",
            "Cycle 511, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5983319985866546 and standard deviation 0.10741447422281 of score in the population\n",
            "Cycle 512, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5983729988336564 and standard deviation 0.10742035131275018 of score in the population\n",
            "Cycle 513, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5980469989776611 and standard deviation 0.10731806006646687 of score in the population\n",
            "Cycle 514, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5980099987983704 and standard deviation 0.1072990885572452 of score in the population\n",
            "Cycle 515, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5974029988050461 and standard deviation 0.10737857778421427 of score in the population\n",
            "Cycle 516, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.6007969990372658 and standard deviation 0.10378795398231244 of score in the population\n",
            "Cycle 517, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.599823999106884 and standard deviation 0.10383725876808539 of score in the population\n",
            "Cycle 518, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5992259994149208 and standard deviation 0.1037308065046873 of score in the population\n",
            "Cycle 519, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.5995459994673729 and standard deviation 0.10388005189930997 of score in the population\n",
            "Cycle 520, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.600101999938488 and standard deviation 0.10403181876534516 of score in the population\n",
            "Cycle 521, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.6005550000071526 and standard deviation 0.1036047362408281 of score in the population\n",
            "Cycle 522, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.6004979994893074 and standard deviation 0.10357495467601141 of score in the population\n",
            "Cycle 523, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.6006209990382194 and standard deviation 0.10362421464795894 of score in the population\n",
            "Cycle 524, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.6006249991059304 and standard deviation 0.10362576939524407 of score in the population\n",
            "Cycle 525, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.6006599989533424 and standard deviation 0.10363847063653549 of score in the population\n",
            "Cycle 526, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.6045739990472794 and standard deviation 0.09567687245634585 of score in the population\n",
            "Cycle 527, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.6052529990673066 and standard deviation 0.09587351081522022 of score in the population\n",
            "Cycle 528, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.6049279987812042 and standard deviation 0.09578988139437375 of score in the population\n",
            "Cycle 529, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.6058279985189438 and standard deviation 0.09538522436046376 of score in the population\n",
            "Cycle 530, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.6067529988288879 and standard deviation 0.09535961466420152 of score in the population\n",
            "Cycle 531, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.6064959985017776 and standard deviation 0.09529430304018276 of score in the population\n",
            "Cycle 532, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.605538998246193 and standard deviation 0.09539936576243536 of score in the population\n",
            "Cycle 533, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.6095719982683658 and standard deviation 0.08757902824903413 of score in the population\n",
            "Cycle 534, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.6085569979250431 and standard deviation 0.08769848708469051 of score in the population\n",
            "Cycle 535, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.6086529977619648 and standard deviation 0.08774455461273127 of score in the population\n",
            "Cycle 536, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.608900997787714 and standard deviation 0.08786550553159016 of score in the population\n",
            "Cycle 537, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.6104389978945255 and standard deviation 0.08729529957700367 of score in the population\n",
            "Cycle 538, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.6105609984695911 and standard deviation 0.0873439211904757 of score in the population\n",
            "Cycle 539, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.6107549984753132 and standard deviation 0.0874179458892573 of score in the population\n",
            "Cycle 540, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.6105199982225895 and standard deviation 0.08738349601651363 of score in the population\n",
            "Cycle 541, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.6115079982578755 and standard deviation 0.08705710168689296 of score in the population\n",
            "Cycle 542, the best candidate in pop has score 0.6675999760627747, and best in the run 0.6703000068664551\n",
            "Mean 0.6115459980070591 and standard deviation 0.0870765116381391 of score in the population\n",
            "Cycle 543, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6117969979345799 and standard deviation 0.08722103225458851 of score in the population\n",
            "Cycle 544, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6118579979240895 and standard deviation 0.08725111580383764 of score in the population\n",
            "Cycle 545, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6131599982082844 and standard deviation 0.08670159722523593 of score in the population\n",
            "Cycle 546, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6131629981100559 and standard deviation 0.08670320572662116 of score in the population\n",
            "Cycle 547, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6139249975979328 and standard deviation 0.0867581697953746 of score in the population\n",
            "Cycle 548, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6090139977633953 and standard deviation 0.09763115166145606 of score in the population\n",
            "Cycle 549, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6092759977281094 and standard deviation 0.09771740372053124 of score in the population\n",
            "Cycle 550, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6091259972751141 and standard deviation 0.09766304959335073 of score in the population\n",
            "Cycle 551, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6100379978120327 and standard deviation 0.09768469326389394 of score in the population\n",
            "Cycle 552, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6101739977300167 and standard deviation 0.09771948456928671 of score in the population\n",
            "Cycle 553, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6101739977300167 and standard deviation 0.09771948456928671 of score in the population\n",
            "Cycle 554, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6103949974477291 and standard deviation 0.09778271011693984 of score in the population\n",
            "Cycle 555, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6103269974887371 and standard deviation 0.09775792293560641 of score in the population\n",
            "Cycle 556, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.610984997600317 and standard deviation 0.09788467811826783 of score in the population\n",
            "Cycle 557, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6110129980742931 and standard deviation 0.09789643458891079 of score in the population\n",
            "Cycle 558, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6109989981353283 and standard deviation 0.09789479084458039 of score in the population\n",
            "Cycle 559, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6110249982774257 and standard deviation 0.09790344560195031 of score in the population\n",
            "Cycle 560, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6109219984710217 and standard deviation 0.09786085282374969 of score in the population\n",
            "Cycle 561, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6108979986608029 and standard deviation 0.09785330146208704 of score in the population\n",
            "Cycle 562, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.606671998500824 and standard deviation 0.1045708784839954 of score in the population\n",
            "Cycle 563, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6063959985971451 and standard deviation 0.10449756832860423 of score in the population\n",
            "Cycle 564, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6061619991064071 and standard deviation 0.10438885959957742 of score in the population\n",
            "Cycle 565, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6059869992733001 and standard deviation 0.10430185806434836 of score in the population\n",
            "Cycle 566, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6015999993681908 and standard deviation 0.11128736541790366 of score in the population\n",
            "Cycle 567, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6021299996972084 and standard deviation 0.11133133788716344 of score in the population\n",
            "Cycle 568, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6020109996199607 and standard deviation 0.11128421209875675 of score in the population\n",
            "Cycle 569, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6013589999079705 and standard deviation 0.11118330589922892 of score in the population\n",
            "Cycle 570, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6061429996788502 and standard deviation 0.1030991347346127 of score in the population\n",
            "Cycle 571, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6060960002243518 and standard deviation 0.10307349793941967 of score in the population\n",
            "Cycle 572, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6062140001356602 and standard deviation 0.10311475450411615 of score in the population\n",
            "Cycle 573, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6085900001227855 and standard deviation 0.09982002157671459 of score in the population\n",
            "Cycle 574, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6089740000665188 and standard deviation 0.099848169327094 of score in the population\n",
            "Cycle 575, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6103270004689694 and standard deviation 0.09948570415852498 of score in the population\n",
            "Cycle 576, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6104870007932186 and standard deviation 0.09954791368285174 of score in the population\n",
            "Cycle 577, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6105300007760525 and standard deviation 0.09956546733244769 of score in the population\n",
            "Cycle 578, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6102860008180141 and standard deviation 0.0994897330435679 of score in the population\n",
            "Cycle 579, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6093850009143352 and standard deviation 0.09984668150704579 of score in the population\n",
            "Cycle 580, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.609785001128912 and standard deviation 0.09995364834097592 of score in the population\n",
            "Cycle 581, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6122860012948513 and standard deviation 0.09404704231167052 of score in the population\n",
            "Cycle 582, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6124660013616086 and standard deviation 0.09412782171841062 of score in the population\n",
            "Cycle 583, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6099460016191006 and standard deviation 0.09597323110573022 of score in the population\n",
            "Cycle 584, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6096930019557476 and standard deviation 0.0959069604754652 of score in the population\n",
            "Cycle 585, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6093510018289089 and standard deviation 0.0957989296461937 of score in the population\n",
            "Cycle 586, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6089760021865368 and standard deviation 0.09606635728106688 of score in the population\n",
            "Cycle 587, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6085830019414424 and standard deviation 0.09596251236121521 of score in the population\n",
            "Cycle 588, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6092350016534328 and standard deviation 0.0960520943395496 of score in the population\n",
            "Cycle 589, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6089060018956661 and standard deviation 0.0959287774429717 of score in the population\n",
            "Cycle 590, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6099260018765926 and standard deviation 0.09573808890111739 of score in the population\n",
            "Cycle 591, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6096630023419857 and standard deviation 0.09564998958282062 of score in the population\n",
            "Cycle 592, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6085470019280911 and standard deviation 0.09589591696466486 of score in the population\n",
            "Cycle 593, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6087990020215511 and standard deviation 0.09596061121794836 of score in the population\n",
            "Cycle 594, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6082110022008419 and standard deviation 0.09586193823170472 of score in the population\n",
            "Cycle 595, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6085480020940304 and standard deviation 0.0959784773427192 of score in the population\n",
            "Cycle 596, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6091300015151501 and standard deviation 0.09612971801670883 of score in the population\n",
            "Cycle 597, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.609257001131773 and standard deviation 0.09615621154514123 of score in the population\n",
            "Cycle 598, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6095350013673305 and standard deviation 0.09624840597665148 of score in the population\n",
            "Cycle 599, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6140260012447833 and standard deviation 0.08466393015781706 of score in the population\n",
            "Cycle 600, the best candidate in pop has score 0.6743999719619751, and best in the run 0.6743999719619751\n",
            "Mean 0.6143460012972355 and standard deviation 0.08478446816221127 of score in the population\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5YUKkL0_-UMH",
        "colab_type": "text"
      },
      "source": [
        "# Plotting the results\n",
        "We plot the progress of the experiment, showing how the accuracy improved over the generations (cycles). "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZXKj2rddziLV",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 532
        },
        "outputId": "3a057613-acc1-4387-d543-399f149433f1"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import matplotlib.ticker as ticker\n",
        "import seaborn as sns\n",
        "# Here we extract the history, the population and the best model\n",
        "history, population, model_best = history_and_model\n",
        "sns.set_style('white')\n",
        "xvalues = range(len(history))\n",
        "yvalues = [i[1] for i in history]\n",
        "ax = plt.gca()\n",
        "ax.scatter(\n",
        "    xvalues, yvalues, marker='.', facecolor=(0.0, 0.0, 0.0),\n",
        "    edgecolor=(0.0, 0.0, 0.0), linewidth=1, s=1)\n",
        "ax.xaxis.set_major_locator(ticker.LinearLocator(numticks=2))\n",
        "ax.xaxis.set_major_formatter(ticker.ScalarFormatter())\n",
        "ax.yaxis.set_major_locator(ticker.LinearLocator(numticks=2))\n",
        "ax.yaxis.set_major_formatter(ticker.ScalarFormatter())\n",
        "fig = plt.gcf()\n",
        "fig.set_size_inches(8, 6)\n",
        "fig.tight_layout()\n",
        "ax.tick_params(\n",
        "    axis='x', which='both', bottom='on', top='off', labelbottom='on',\n",
        "    labeltop='off', labelsize=14, pad=10)\n",
        "ax.tick_params(\n",
        "    axis='y', which='both', left='on', right='off', labelleft='on',\n",
        "    labelright='off', labelsize=14, pad=5)\n",
        "plt.xlabel('Number of Models Evaluated', labelpad=-16, fontsize=16)\n",
        "plt.ylabel('Accuracy', labelpad=-30, fontsize=16)\n",
        "plt.xlim(0, 600)\n",
        "sns.despine()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.\n",
            "  import pandas.util.testing as tm\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAngAAAHKCAYAAACQfTbZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXwU9eH/8XfCpVHkDCBH2+C3CQohCYhAAVvSIogHElRQRMELj+LPiiCKQqsIfvEsp4LWesQQMAE88KgK9UDlSrAqbRVBwSuEUwwSSOb3B99ds8lmd3Z3Znd29vV8PHy0bGZnPnN95j2fz2dmkwzDMAQAAADXSI51AQAAAGAtAh4AAIDLEPAAAABchoAHAADgMgQ8AAAAlyHgAQAAuAwBDwAAwGUSOuDl5+crNzdXmZmZysvL04YNG2JdJAAA4EdZWZluu+029enTR5mZmRo6dKjWrVvn/bthGJo7d6769++v7t27a8yYMfrss8985rF//35NmjRJPXv2VM+ePTVp0iQdOHAg2qsSFQkb8FatWqWZM2fquuuu04oVK5STk6NrrrlG33zzTayLBgAAajhw4IAuueQSGYahRYsWadWqVbrrrrvUqlUr7zSLFy/W3/72N9111116/vnn1bJlS40bN04HDx70TjNx4kR9+umnevzxx/X444/r008/1eTJk2OxSrZLStRfsrjooouUkZGhGTNmeD8766yzNHjwYE2cODGGJQMAADU99NBDWrdunZYsWeL374ZhaMCAARo9erSuv/56SdJPP/2kvn376rbbbtOoUaO0detWDR06VM8995x69uwpSdqwYYNGjx6tV155RZ07d47a+kRDQrbgVVZW6pNPPlG/fv18Pu/Xr59KSkpiVCoAAODPG2+8oaysLN18883q27evhg0bpmeffVaeNqqdO3dq165dPtf14447Tr169fJe10tKSpSSkqIePXp4p+nZs6dSUlJcee1PyIC3d+9eVVVVqXXr1j6ft2rVSrt27YpRqQAAgD87duzQc889p06dOumJJ57Q5ZdfrgcffFD5+fmS5L12+7uul5eXS5LKy8vVsmVLJSUlef+elJSkli1beqdxk4axLgAAAEAghmGoW7du3iFUp512mr788kvl5+frsssui3HpnCkhW/BatGihBg0a1Ensu3fvVmpqaoxKBQAA/ElNTdUpp5zi81nnzp317bffev8uye913dOq17p1a+3Zs0c1Hz0wDEN79uyp0/LnBgkZ8Bo3bqyuXbtq7dq1Pp+vXbtWOTk5MSoVAADwp0ePHtq2bZvPZ9u3b1f79u0lSR07dlRqaqrPdf3w4cPasGGD97qek5OjiooKn/F2JSUlqqiocOW1v8Gf//znP8e6ELFw4oknau7cuUpNTdVxxx2nBQsWaMOGDZo5c6ZOOumkWBcPAAD8n5NPPlnz589XcnKy2rRpo/fff1+PPPKIxo8fr+7duyspKUlHjx7VokWLlJaWpqqqKt13333atWuX7r77bjVu3FgtW7bU5s2b9dJLL+nUU0/Vd999p+nTp3vfmec2CfuaFOnYi46feOIJlZWVKT09Xbfffrt69eoV62IBAIBa1qxZo4ceekjbtm1T+/btNXr0aI0ZM8b70IRhGJo3b54KCwu1f/9+ZWVladq0aUpPT/fOY//+/brnnnv01ltvSZJyc3M1bdo0VzbsJHTAAwAAcKOEHIMHAADgZgQ8AAAAlyHgAQAAuAwBDwAAwGUIeAAAAC5DwAMAAHAZAh4AAIDLEPAAAABchoAHAADgMgQ8AAAAlyHgAQAAuIzpgJefn6/c3FxlZmYqLy9PGzZsqHfaKVOmKCMjo85/2dnZPtOtW7dOeXl5yszM1O9//3sVFBTUWeZ5552nHj16qEePHho5cqTWrFnjM83rr7+uq666Sn369FFGRoY+/PBDs6sEAADgyoxjKuCtWrVKM2fO1HXXXacVK1YoJydH11xzjb755hu/00+dOlXvvvuuz3+dOnXS2Wef7Z1mx44duvbaa5WTk6MVK1Zo/PjxmjFjhl577TXvNG3bttWtt96q5cuXq6ioSH369NGNN96of//7395pKioqlJOToylTpphe6ZoKCwvD+h4AAHAus9d312Ycw4QLL7zQmDp1qs9ngwYNMh544AEzXzc2bNhgpKenGxs3bvR+Nnv2bGPQoEE+091xxx3GxRdfHHBevXr1MgoKCup8vnv3biM9Pd344IMPTJXJY/jw4SFNDwAAnM/s9d2tGSdoC15lZaU++eQT9evXz+fzfv36qaSkxFSIXLZsmX7961+rR48e3s9KS0vrzLN///76+OOPdeTIkTrzqKqq0ssvv+xNswAAAJFwc8ZpGGyCvXv3qqqqSq1bt/b5vFWrVlq7dm3QBfzwww965ZVXdMstt/h8Xl5err59+/p81rp1ax09elR79+5VmzZtJEn/+c9/NGrUKB0+fFgpKSmaN2+eMjIygi43kMLCQm/TbVlZWUTzAgAAznPKKacoLy/P+++RI0dq5MiRPtO4MeN4BA14kXrhhRdUXV2tYcOGhfX9tLQ0rVixQj/88INee+013XbbbXrmmWeUnp4edplq7uSaOx8AALjD/fffb/synJhxPIJ20bZo0UINGjRQeXm5z+e7d+9Wampq0AUsXbpUZ511lpo3b+7zeevWrbV7926fz8rLy9WwYUO1aNHC+1njxo31y1/+Ut26ddPEiRN16qmn6u9//3vQ5QIAAATi5owTNOA1btxYXbt2rdNUuXbt2qD9xB999JH+/e9/6+KLL67zt+zsbL/z7Natmxo1alTvPKurq1VZWRms2AAAAAG5OeOYek3KuHHjtHz5ci1btkxbt27VjBkzVFZWplGjRkmSJk+erMmTJ9f5XmFhoX71q1+pd+/edf42atQoff/997r33nu1detWLVu2TMuXL9eVV17pneaBBx7Qhg0btHPnTv3nP//Rgw8+qHXr1um8887zTrNv3z5t2bJFn332mSTpq6++0pYtW7Rr167QtgQAAEg4bs04psbgDR06VHv37tXChQtVVlam9PR0LVq0SB06dJAkffvtt3W+c/DgQa1atUo33HCD33l26tRJixYt0qxZs1RQUKA2bdpo6tSpGjx4sHea8vJyTZo0Sbt27VLTpk2VkZGhxYsXa8CAAd5p3nrrLd1+++3ef995552SpD/+8Y+aMGGCmdUDAAAJyq0ZJ8kwDMP8ZnCfvLw8FRcXx7oYAAAAluG3aAEAAFyGgAcAAOAyBDwAAACXIeABAAC4DAEPAADAZQh4AAAALkPAAwAAcBkCHgAAgMsQ8AAAAFyGgAcAAOAyBDwAAACXIeABAAC4DAEPABAzhmGotLRUhmHEuiiAqxDwAAAxs3nzZo0YMUKbN2+OdVEAVyHgAUg4tBo5R1ZWloqKipSVlRXrosBi1dXVKiwsVHV1dVjf5zyNDAEPQMKh1cg5kpKSlJ2draSkpFgXBRZbtmyZLrvsMi1btiys70dynnrCYSIj4AGwnb878VjendfXahSsTLQoHMN2gBkXXXSRnn32WV100UU+n5s9fiJp3d28ebPy8vJC/p6bEPCAGEuEi6W/O3E7WtHMbsv6Wo2Clclsmd2wTwOtg+fiuXTpUsetoxu2vVskJydr5MiRSk72jRpmz6NIWnezsrI0a9askL/nKkaCGz58eKyLgBirrq42SkpKjOrq6pgsv6SkxOjcubNRUlISk+VHg79tHM52D/adSLdlsHKaLbMb9mmgdaiurjaWLFniyHUMd9uHcozGus6Id5Fsv1C+m+j7h4BHwEt4dlyMQ62EEvFiEWi96/tbsH1lx7YM5/hwwz4Ntg7h7L9oCHfZ/vZzffs+1GMinG3llGPIjhuxSLjh5ilaCHgEvITnlFBgpWhfHMJZXqBtVN/forVe4bTa+ZuuqqrKWLJkiVFVVRXSMq1gx7YyG1Rq7j+nBJVg7GzB83c8e+axadMmv9vKKTcW4ZTDyvqv9jrFy/HkBAQ8Ah5sEKtKyN9FIxrMVOihVNSev1VVVYUdriJh1UVtyZIlRsOGDY2CgoKg5bP6psCOkLxp0yajQ4cOxqZNmwIur7q62ti0aZP3P7uOxUhaGe1edrBpPdtr06ZN3mO9Zre31TdN4QpnPc2et2bE+mY5nhHwCHiIY7Ur39oXjWgFo5oX9PpaO8K50Jut3K2+CITbLVV7G3gu2hs3bgw5AEfKU56NGzf6lCmSbVVfwPO37qEei8HWJZwu+3DX1cy+iLSFzd+5m5aWZixZsiTs89aKYyiSeURybAVbp5qhMZwW9kREwCPgIY7VrlAj6TqKtKKsr3I3e6EPVpEH4pRKPtZdy/7K06FDB6NDhw5hHyM1hRK0rFzn+sJPKC14oRxLZh4gCXRTY2Y9ardyhTI/O1u1Au3LYC1zZrvwzS63vr/X9//hi4BHwEOInBImrChLzYo70qciw2lV8DfmyKlPZ5oJC5F220VSpvr+HmoIiXWXnL9lbNq0ySgoKAj7ZqZ2aKtvX3qmM9uSFuzmpXYZa5bD33FuNqzYWQf526fBzk0z5/imTZsCbtdww7qT6mOnIeAR8BAiN94x1teyYEfl6S/U1fy31WHBqnUw22pg9kJV3+D6UMtU30XT6nn6m18o3YrBBGsdrB2iQmk1rF3OQK1BoayPv33pr+xmW8OiFVbMHKP+ArGnzLX/N9Dwi5r7L9DNG0HNegQ8Al7CC7Viqa+yi1arjZX8XWxqjrOyOswGakmxS7B1CKXrzkzwNdvVVDOwhLudA3UnWj1Pf+UOtPxQ1Vdef9u6Zsukv5am2mHQX0uala1BVrToml2Gv67dUJdh5hgNFHRrt+j5a8H0V95Qzxun1ZfxhoBHwIsaJ56s/i5QZssZrNuivu+kpaUZBQUFIXWb2cVfpVoz4Fm9z6xs8TEr2AXcjhBrtqvJbBnDWZ6ZbrNwyli7tcaqYySU+dTXAhesRS2arA6L9dU34dzA1AzIkRwfgW5uQz2v/IVwpw7XiBcEPAJe1Fh9IfWI5ALjL3CYLaeZSs7fd5YsWVJn4HusBAsb9U0TznzNLi/Y5+Es18NfGKivJTYarTLBylifSMsQybkYTjgMd15mvudvHlVVVUZBQYGxceNGy49ts8LdxrW/F+5xGqwcdtXHHlYco5GM2fNMk8gIeAS8qPF312hFBevvoh1sfoEqTbMVR7itLMHuniNZjh3CuRCE8h0rL0ChtGbUnjbYv2suw0wrZLjHkZnvhdNqY8VyzQhUttovfrYrZJSU1H162N/y7AzTVn0vUBlDPcZq1j12jXW1ar7hBtfa0yQyAh4BL6rsqGBrX7TNXHzDKUeg71s9faTfs1I4YTaUJzfrm96OFrz6pg3l5sNs15HZC1A4T01acfEL9h07gk3tFz/bFQbMHlORhulgLYlWCDTf+spW33fqC75W8JQlnG5Vu851WvASHAHPOoFOuPru7qxorai9HLPvsAq1HIG+b/X0Zr4X6TpEKtBFxM4Wv1BZfYyZnaeVLcF2lC/YdzzLtOJFxR7+XvxsxXEbq+On5nJjcTMW6jlYO/ha2eoWybyccCPrRgQ8Ap5lAp2kkZzA4YapUF/KGWrLUyzULGPtgeTRqCRrt5Za9VLfSAJwsH1mRStxrMWifJ5lBntgIdL9HW4rZqRlCEegm6pIymB1+UO9cYj1wwxOP//iFQGPgGcZJ1TANQW7cNT+u53dF8GEUiF7ylj7dRXRCKc1t5kd+9TsPGuGj2D7zM5jLx5uCiIVbPtFemNhVyumHawug9kQHc48zbSkWdmCF04Z3XrOOAUBj4Bnq1ieyMGWbbYFz6q79EBCacXwV0a7g1fN5du5P81uh5KSn183U/u3VkMR6fpYfVNg1/Z10jERSWttrEOBvzJEUi6z3eChLCPcVrlo3qw4IawnAgIeAc9W4ZzIVlbkVo/vsatiirSc4XR5RbN8Vi+nutqad2QF21ZmbhKCXRTDuThbfXwFmm+0g5OTLu5W1w9Wh91wWvjCbZWLZg+GE8J6IiDgEfBsFc6JbPauNpR5RVJhRaMFz0qhdnOGGzzMfD9WLUdWbQMrjp9Qtl242zTU1ura5bP6584CcVIrpdX1g9Xh1Y4WvvokwnCDREPAI+A5jpXjUvx1w1p5cYmHwFefUC5G/tbTzPdj1Vpj1XKtunCGs+3q4++7kcwvUGtoqPO16nyIVlhzen1gdn5OahV1gniul61EwCPgOZbV410Mw/qK0KldttFYhh0teFbtcyta9+wUSRn8dcHZtS9DnW+kNw3hzCfcsrpJtOsLp29rAu8xBDwCnqXsrgQiPXHtvMO2ct6JWkFZ3TpldhnxyInrEWm3fzjzcTq3rEvN/RWoWz+W/N38JDICHgHPUnZVAvFw4ga74IZS0Tu968gudrTamllGNMWyGzPW626mLE4qoxWcGMTDUftmNtbvzvNXNitfN+MGBDwCnqXCrQSCDfCNh0oy2IXJygAYarmcVBnHgpNCQyyP5Xg4j6zq5o1ULIN4PKi9XrFcT09jQkFBAQ+J1EDAI+DZKpRBwoEe0XdDJRmsQrTr4uvU7hSzrBjH56RgE8tjOR7OI6u6eSPlpGMmHsRye3ET6x8Bj4AXc57Wu0heWBsNVl8ca1eIdrbgOf2iHkioFw5/08f7NrCDG7aJU1vw3LBtQxWLdY6nBz9igYBHwIs5K+78onFyW32HSoVkjhUteKiLFir7sG2jg+0cWLJgOcMwVFpaKsMwYl2UuJCVlaWioiJlZWWFPY/NmzcrLy9PS5cutW27W1HOmpKSkpSdna2kpCRL5udWoW6nWGzXSM/5WNQZVh/PkYh1nWn18p20beNNKPuC7RwYAc8Gmzdv1ogRI7R58+ZYF8Ur1hVobTXLY8UFOSsrS7NmzdIdd9xh23YnkKE+kZ7zsagznHQ8m11/u+oxq7d/sG3rtPrYKQzD0NKlS03vCycdw44UzeZCJ7Kji9ZsF1E0upKc+vi4HU3rVj7VRTcfQhHp8ZLox1soD2M5/cXiZuYVL8NSoi3eHwhzGgJeDMfgRWP8gGcZVvyuayiCVT5OHzMXz2M7nFjxO7FMiD/xcByZqTvCXY+a34vnOqo+8bB/4wkBL4yAF0/vR4rVCeOEyifWLXiJvO1rc2KZ6sNFBpGw8/ipeR5xnCIYAl4YAS+eLlaxQuUT/nFiZtsFmsaJ2z6ebmY4v93JiedFqNywDogeHrIIA0/uBOfkwa9GlAY4h3ucmBnwHWgaJ277aJSp9jYJdz9zfjuHleeqEx9+C5UTz204FwEvDJxk8c2Oit7fhSjc48RMwDAzTbSCrFPU3ibh7mfOb+ew8lwluCPREPAQN6wKLOFU9MGWbeWFyEzAMDNNJGWKx3BYe5sk2gU9HvdZMFbuQ4K7e7nx2LcCAc+lAh3w8XoyWBWiwqnogy3biWEikjI5sTsr1OM21P3s9PPCjpsMp68zoQxmOLG+cgT7h/k5m1t/qizQQPF4HUTOj7RHjxWvcbCa3U8QOv28CFa+cLaJ09cZ5iVaHVVTIq97IAQ8lwa8eHvKMlRuWAc3sjMw2P0OMKceU55yVVVVWV4+p64zQlffOcE+Tlx00bpUoK4NN3R7uL1J3nBY15nZ8tjZVe05bj3lef755y1djlPPC8+x/tFHH1levprb1EnHG0JX37nn5LrSafWc2xDw4AihnuhOHPNmJX+VciwrQ7MXiWi9DuXCCy9UUlKS48KYHaJxrDs5BMCc+s49u4+fSOoljjt7JRkJHp3z8vJUXFwc62IEZBiGNm/erKysLNde0EpLSzVixAgVFRV5WxQSmb99Hstt5KRj0EllcQu7tyn7zL0iqZc4LuxFwMvLU1FRkaMPskQIP7VPdE78utgmCMapx0gi1GGJyqnHHOiileT8ZuJ4644Mp8m+dveC0/dJLMR6jBjjZaIv1G3u1PMm3uowmBfregn1I+DJ+ZVPvJ1AVlxknL5PEpFTw4ObhbrNnXrexFsdBmfiJjM0BDzZV/kk6sFoxUWGC4LzODU8uFmo29xp503NOjDe68N4L78bcJMZGgKejeLlYLS64nLaRQbmBToW2K+RC/Vci/dtXrMOjJf6sD7xXn434CYzNAQ8G8XLwUjFlVgChQyOBXsl2vatWQfGS31Yn3gvfyxZ1YgQ7zc80UbAs1G8/BamkyouukHsFyhk1DwW2BfWc9K5Fg0168B4vzjHe/ljKdFubJyCgOcgsToJnFRxuaUicHI4ChQyah4LbtkXHk7YJ0461wAzrDhvEu3GxikIeA7CSeCebeDkcGQ2ZLhlX3g4eZ8ATmXFecONTWzwouM4+CULM3jZpLUi3Z7R2h/sd/PYVkDoOG/iFy14LkHrhLUi3Z7RumNlv5vn5lYEJ3Q/w53cfN64HQHPJZzcnRaPFx8nb8+a4qWcsBdBH0BtBLw4VTs0OfkuyykXn1CCppO3Z01OK2c8hnk3IOgDqI2AF6ecEprMcMrFJ562md3sCmJs49hwWtAHEHsEvP8Tby0PTglNZjjl4hONbRYvx5FdQSyejkvArHg5r4GaCHj/x+6WB34OLPaisc3ipQXLriDGcQmnC6cujpfzGqiJgPd/7G55oIJIDPHSgkUQQ6IKpy6Ol/M6GmjNjB+8By9K78HjXULWYDsCiAR1SGRKS0s1YsQIFRUVKTs7O9bFQQC04EUJLSbWiGZLKHeqgPtQF0eG1sz4QcBDXIlm5UK3OgD4IiDHDwIe4kp9lYsdrW3cqQJA7NGbEh4CHuKa58T3jAuxsrWNO1UAiD16U8JDwEPUWXk35jnxJdHa5jDcdUeObegMbtgP8bwO9KaEh4CHqLPybsxz4mdnZ9Pa5jDcdUeObegMbtgP8bwO9KaEh9ekROk1KfgZrylIDOznyLENnSGU/eDUfebUcsE+tOAh6rgbSwzs58ixDZ0hlP3g1JYyjqXEQ8ADAMAijBeDUzSMdQEAAHALT0sZEGu04MH14vnpMatZvS3YtgDgTAQ8uJ5Tx8TEgtXbgm0Lt+GmBW5BwEPcMlsRMybmZ1ZvC7at8xBQIsNNC9yCgIe4ZbYi5umxn1m9Ldi2zkNAiQw3LXAL3oPHe/DiFu91AurivICbcXybRwteLXRvxA9aj+AGVtc5nBfmUNfHJ1qozSPg1cLBAyCaqHNig+0en+hCN48u2lpdtDT/Aogm6pzYYLvD7WjBq4XuDcA94qEbjjonNtjucDsCHgDXohsOMC8ebohgHgEvyjiBgOhx23gd6g/YiRsidyHgRVl9J5AVFTeV/8/YFpDc1w3HBRh2ctsNUaIj4EVZfSeQFRU3lf/P2BZwIy7AsJPbbogSHU/ROuRFx1Y80VV7Hon8lJjb1t1t6wMAsBcteA5hxZ1T7XkkciuW2+5EE3lfAgBCR8ALIp7HxtGdU1e8js1jXzqflcdWvB6n0cC2Acwh4AURz2Pj7GzFitdKNl5bwtzWIulGVh5b8XqcRgPbBjCHMXhBxuDZMTbODUpLSzVixAgVFRUpOzs71sUxzY37As5g5bHFcVq/WG+bWC8fMIuAZ9FDFol20ifa+gKAFJ83t9TXiYkuWoskWrdB7S7DeO2yBYBQxON42ES7PuEYAp5F4vGktxIVCIBEEI/jYZ18faJxwD4EPIuEetLbdVDzxC4AoCYnh1IaB+xDwIsRuw5qs/O1Ogg6uQIBAIQuGg0GNA7Yh4AXI3Yd1GbnG27ApDkdABJDNFrXaBywD0/ROuSnyqIt3Keq4vEJMgBA6Hj6Nr7RghemeG/JCveuieZ0AEgMtK7FNwJemBJ1YCgnPBBb8X5zaVairCdgFwJemPy1ZFEhAbBbotxcJsp6AnYh4IXJX0sWFRIAuyXKMIlEWU/ALgQ8C1EhAbBbogyTSJT1jDV6ntyLgGchqyskTjwAgJ3oeXIvAp6DceIBx3CzA9iDnif3IuA5GCcecAw3O4A96Ap3LwKegznl922BWONmBwBCQ8BzEVo54Fa0MgBAaAh4LpIorRy0VAIAEBgBz0USpZWDlkoAAAIj4CHuuKmlktZIAIAdCHiIO25qqaQ1EgBgBwIeEENuao0EADgHAS8Aus/swXb9mZtaIwEAzkHAC4DuM3uwXQEAsBcBL4Ca3We0OlmHbkkAsB7XKdREwAugZvdZIrc6WV1p0C0JANZL5OsU6iLgmZTIrU5UGgDgfIl8nUJdSUaCt+Xm5eWpuLg41sVwNMMwtHnzZmVlZdHqBgBAHGgY6wLA+TxdqgAAID7QRQsAAOAyBDwAAACXIeABAAC4DAEPAADAZQh4LseLLwEASDwEPJfjHXYAACQeAp7DRdoCx4svAQBIPAQ8h4u0BY6fBQMAIPEQ8ByOFjgAABAqAp7DRbMFjgcyAABwBwIevHggAwAAdyDgwYvuYPeidRYAEgsBD148kOFetM4CQGIh4AEJwOrWWVoEAcDZCHhxggsqImF16ywtggDgbAS8OMEFFU7CeE0AcDYCXoyE2iLHBRVOwnhNAHA2Al6MhNoixwUVAACYRcCLEVrkAACAXWwPeBMnTtSGDRvsXkzcoUUOQCA8WAUgErYHvNLSUo0ZM0bnnHOOnn76aR04cMDuRQJA3OPBKgCRSDKicHv4zjvvqLCwUGvWrFHDhg119tlna+TIkcrOzrZ70UHl5eWpuLg41sUAAB+GYWjz5s3KysqipR9AyKIS8Dx27dqlpUuXqqioSN9++60yMjI0cuRInX/++TrhhBOiVQwfBDwAAOA2UQ14Ht9//71uvfVWrV+/XpKUkpKikSNHasKECUpJSYlqWQh4AADAbRpGc2Hvv/++lixZojfffFMnnHCCxo4dqyFDhmj16tV6+umntXPnTs2dOzeaRQIAAHAd2wPe3r17VVxcrKVLl+qrr77Saaedpr/85S8699xz1aRJE0lSdna20tPTNXXqVLuLAwBwOMYfApGzPeCdeeaZSk5O1tChQ3X//fere/fufqfr3LmzWrZsaXdxAAAO53mCuKioyBEP4wHxyPYxeE8++aTy8vLUrFkzOxcTtppj8LhrDA3bC4AdqFuAyNn+Hrxx48Y5NtzVxnunQsP2AmAHXgQPRM72gDdz5kxNmjTJ73WH4koAACAASURBVN8mTZqk//3f/7W7CKbx82GhYXsBAOBMtge8t956S/369fP7t/79++vNN9+0uwimBbtr5KeDfHGXDQCAM9ke8L7//nu1b9/e79/atWun77//3u4iWIYuSQAAEA9sD3jNmjXTl19+6fdvX331VdRfbByJRO+SpAUTAID4YHvA69u3rxYuXKjy8nKfz8vLy/Xoo4/qN7/5jd1FsEyid0nSggkAQHyw/TUpO3fu1IUXXqjKykoNHDhQbdu21ffff681a9aoSZMmKiwsVKdOnewsQkD8VJl5vLoAAID4YPuLjjt27Kjnn39ec+bM0dq1a7Vv3z41b95cgwYN0oQJE9ShQwe7iwCLeFowAQCAs0Xlt2g7duyo2bNnR2NRAAAACc/2MXgAAACIrqi04O3evVsvvfSStm3bpsOHD/v8LSkpSTNnzoxGMRAGxt0BABB/bA94X3zxhUaNGqWjR4/q0KFDatGihfbv36+qqio1a9ZMJ554ot1FcI1YhK3S0lKdd955evHFF5WTkxOVZQIAgMjY3kU7e/ZsZWZmau3atTIMQ4sXL9bmzZs1Y8YMHXfccZo/f77dRXC0UN4tx2tKAACAGbYHvI8//liXXHKJGjduLEmqrq5Ww4YNdeGFF2r06NG699577S6Co4US2mLxouXs7Gy99NJLPD0LAEAcsT3g/fjjj2revLmSk5PVtGlT7d271/u3zMxMffzxx3YXwdFCCW2xeNFyor/cGQCAeGR7wOvYsaN27dolSUpLS9Orr77q/duaNWvUtGlTu4vgaAQoAABgNdsfsvjNb36jtWvX6uyzz9bYsWN1yy23aOPGjWrYsKG++OILXXfddXYXAQAAIKHYHvAmTpyoyspKSdLQoUN13HHHadWqVfrpp590+eWX6+KLL7a7CAAAAAnF1oBXVVWlL774Qm3atPF+lpubq9zcXDsXCwAAkNBsHYOXlJSkESNG6NNPP7VzMQAAuEYor88C6mNrwEtOTla7du106NAhOxcDC1ChAIAz8M5TWMH2p2hHjhypp556yjsOD4HFKmh5KpTS0lKCHgDEUCzeeQr3sf0hix9//FFfffWV/vCHP2jAgAFKTU31eSVIUlKSbrrpJruLETc8QauoqMjUy4Wt+vkyT4ViGEZIywcAWMvz+iwgEkmGzU01Xbp0CVyApCRt2bLFziIElJeXp+Li4pgtv7ZQA1tpaamlgSwWv3cLAACsZXvAc7qzzz5bq1atitswQyADAAC12T4Gz+m2bt0a1wNZ+SUMAGbwIBWQWBI+4J1yyilxO5CVChuAWTyZCSQW2x+y6NKlS9DWpViOwTv++OPjtvUr1AcyACQuM09mMuQDcA/bA96NN95Yp6LYt2+f3n33XVVWViovL8/uIrhWqI/SU3kDicvMk5ncNALuYXvAmzBhgt/Pq6qqdN111+nEE0+0uwiuFeqj9FTeAALh/WuAe8RsDF6DBg106aWX6umnn45VERIOlTeAQHhoC3AP21vwAqmsrNS+fftiWYSEwsszAQBIDLYHvG+++abOZ0eOHNFnn32mBx98UN26dbO7CAAAAAnF9oCXm5vrt7nfMAz94he/0PTp0+0uAgAAQEKxPeDNnDmzTsBr0qSJ2rdvr8zMTDVo0MDuIgAAACQU2wMer0EBjuE1Ne7C/gTgZLY/Rbtt2zatW7fO79/Wr1+v7du3210EwBH4JYH45e9XY9ifAJzM9oA3c+ZMrV692u/fVq9erVmzZtldBMAReE1N/PIX5tifAJzM9oD38ccf6/TTT/f7t169eulf//qX3UUAHIF3jMUvf2GO/QnAyWwPeD/++KOaNGni928NGzbUDz/8YHcRACAibglz/rqaAbiT7QGvU6dOev/99/3+7YMPPlCHDh3sLgIAQIwbBBKJ7QFv2LBheuqpp5Sfn6/KykpJx37BIj8/X0899ZSGDx9udxEAAGLcIJBIkgyb2+qrqqr0pz/9Sa+//rqSk5PVrFkz7d+/X9XV1TrrrLP0yCOPKDk5Zj+Jq7y8PBUXF8ds+QDgNrxCBog929+D16BBA82ZM0fvv/++1q5dq3379qlFixbq16+fevfubffiEQIqZQBW8HQFFxUV8fvXQIzY3oLndLTg/ay0tJRKGUDEuFkEYs/2vtHVq1fr2Wef9fu3/Px8/fOf/7S7CDCJ8TkArOCWp46BeGZ7wFuwYIEqKir8/u2nn37SggUL7C4CTKJSBgDAHWwPeF988YW6du3q92+nnnqqtm7dancRAAAAEortAa+6urreFrwff/xRR48etbsIAADAQXjptv1sD3hdunTRiy++6PdvL774ojIyMuwuAgAAcBBeum0/2wPelVdeqddff1033XST3n33XX3++ed67733dNNNN+kf//iHrrrqKruLAAAAHISH+uwXldekPPPMM3r44Yd16NAhSceaZlNSUjRx4kSNHj3a7sUHxGtSAACA20TtPXgHDx7Upk2btH//frVo0UI5OTn65JNPtHz5cs2aNSsaRfCLgAcAANwmar8RduKJJ+rMM89U9+7dtXHjRp133nm64oor9Oqrr0arCAAAAAnB9p8qk6QffvhBq1at0vLly70DKrt06aJrr71W5557bjSKAAAAkDBsC3jV1dV65513tHz5cq1evVqHDx9WmzZtNHr0aOXn5+uOO+5Qr1697Fq8q/CzPwAAIBS2BLz77rtPL730knbv3q0mTZroD3/4g4YPH67f/OY3OnjwYL0/XQb/+OFuAAAQClsC3t///nclJSXpt7/9rWbNmqUWLVp4/0YLVOh4nBwAAITClocsLrzwQp1wwglas2aNhgwZorvvvlsfffSRHYtKCPxGLAAACIUtLXgzZszQXXfdpX/84x9avny5CgsLVVBQoF/96lcaNGgQQQUAAMBGUXkPXllZmVauXKmVK1fq888/lyRlZ2frkksu0ZAhQ9SkSRO7i1Av3oMHAADcJmovOvb417/+pRUrVujll1/Wvn371LRpU61fvz6aRfBBwAMAAG4Tlffg1ZSZmanMzExNmTJFa9as0YoVK6JdBAAAAFeLesDzaNSokQYNGqRBgwbFqggAAACuFLWfKgMAAEB0EPAAAABchoAHAECcMQxDpaWlivJzkogjBDwAAOKM5ycsN2/eHOuiwKEIeAAAxBl+whLBxOwpWgAAEB7PT1gC9aEFDwAAwGUIeAAAAC5DwAMAAHAZAh4AAIDLEPAAAABchoAHOBwvNAUAhIqA50IEAnfhhaYAgFAR8FyIQOAuvNAUABAqXnTsQgQCd+GFpgCAUBHwXIhAAABAYqOLFgAAwGUIeBaLtwcc4q28AAAgOAKexeLtAYd4Ky8AAAiOgGexeHvAId7KCwAAgiPg1RJpl6XnAYekpCSLS2aPeCsvAAAIjoBXC12WCIQxiwCAeEDAq4UuSwTCDQAAIB7wHrxaeIccAuEGAAAQDwh4QAi4AQAAxAO6aBMUY8kAAHAvAl6CimQsGeEQAABnI+AlqEjGkvGgAQAAzsYYvAQVyVgyHjQAAMDZCHgIGQ8aAADgbHTRAgAAuAwBD/XiYQoAEnUBEI8IeKgXD1MAkKgLgHhEwDPBSXev0SwLD1MAkKgLgHhEwDPBSXev0SyL52GKpKQk25cFwLmoC4D4Q8AzwUl3r04qCwAAcCZek2KCk14L4qSyAAAAZ6IFL8E4aTwhAACwBwEvwdQcw0fYAwDAnQh4CabmGD4nPTwCAACsQ8BLMDWfhuOBDQCA09C7ZA0CXgLj1QcAAKehd8kaBDwAAOAY9C5Zg9ekAAAAx+B1YNagBQ91MP4BAID4RsCzmBXhKNYBi/EPAADENwKeBWoGMivCkZl52BkCGf8AAEB8I+CFyF+wqhnIrAhHZuZhZysbT9cCABDfkowEH2iVl5en4uJi09OXlpZqxIgRKioq8g4C9bTcZWVlRS0UxWKZAAAgPtCCF6KarWue1jxJUW/xopUNAADUh4AXoprBiocRAACAExHwIsDDCAAAwIkIeBGgmxT+hPKEc6xfiQMAcCcCHmCxULru6eYHANiBgAdYLJSue7r5AQB24LdoAYuF8juK/OYiAMAOtOABAAC4DAHPYRh0DwCwE9eZxEDAcxgG3QMA7MR1JjEQ8ByGQfcAADtxnUkMPGThMAy6BwDYietMYqAFzwUYT5HY2P8AgNoIeC7AeIrExv4HANRGwHMBxlMkNvY/AKA2xuC5AOMpEhv7HwBQGy14AAAALkPAcygGzgMAgHAR8ByKgfMAEg03toB1CHgOxcB5uBkXcvjDjS1gHQKeQ3kGziclJcW6KIDluJDDH25sAesQ8KKIVgvgGC7k8IcbW8A6BLwootUidgjXzsKFHADsRcCLIlotYodwDSDRcGOb2Ah4UUSrRewQrgEkGm5sExu/ZIGEwK89AEg03NgmNlrwACAK6C5DtNFrlNgIeLAEFy8gMLrLAEQTAQ+W4OIFBEZ3GYBoYgweLMHFCwiMcaAAoomAB0tw8QIAwDnoogUAAHAZAh4AAIDLEPAAAABchoAHU3gNCgAA8YOAB1N4DQoAAPGDgAdTeA0KACBS9AZFDwEPpvCTNwCASNEbFD0EPAAAEBX0BkUPLzoGAABRwUvxo4cWPMBmoYw5YXxKdLCdAbgdAQ+wWShjThifEh1sZ0QTNxSIBQIeYLNQxpwwPiU62M6IJm4oEAtJRoLfUuTl5am4uDimZTAMQ5s3b1ZWVhZPqQKAy1DHIxZowfPDX3O6nU3s3N0BgHvxminEAgHPD3+BK5IQFiwchtpdxHiOxFB7P7PfAQBmEfD88Be4IhmzEywchnp3R4tfYqi9n9nvAACzGIMXhTF4Vo+/YDxHYqi9n9nvAACzaMGLAqvHXzCeIzHU3s/sdyB+McQC0UbAAwDAZgyxQLQR8AAAsBnvXkS08Vu0AADYjN9gRbTRggcAAOAyBDwAEWHwOAA4DwEPQEQYPA7AKbjh/BkBD0BEGDwOwCm44fwZAQ+IUKLfMbr5/XyJvm+BeMMN588IeECEuGN0L/YtEF/cfMMZKgIeECHuGN2LfQsgXvEePCBCvN/Kvdi3AOIVLXgAAAAuQ8ADAABwGQIeAACAyxDwAABwMF7Xg3AQ8AAAcDBe14NwEPAAAHAwXteDcPCaFAAAHIzX9SActOABAAC4DAEPfjGoFwCA+EXAg18M6nUHgjoAJCYCHvxiUK87ENQBIDHxkAX8YlCvOxDUASAx0YIHuJgnqCclJcW6KEBMMEwBiYqABwBwLYYpIFER8AAArsUwBSQqxuABAFyL8cRIVLTgAQAAuAwBDwAAwGUIeAAAAC5DwAMAAHAZAh4AAIDLEPAAAABchoAHAADgMgQ8xAw/IQQAgD0IeIgZfkIIAAB7EPAQM/yEEAAA9iDgIWY8PyGUlJQU66LYhm5oAEAsEPDgaPEekNzYDR3v+wQAEgEBT/J7sTJzETMMQyUlJSopKan3+9XV1XXmE+0LpNXLqzm/QNvAiuUsXbo0pgEp0m3npG7o+tYl1HV0Y2gFANcxEtyQIUOMzp07GyUlJT6fl5SU+P289jQdOnQwOnToUO/3lyxZUmc+ZuZtJauXV3N+gbaBFctJS0szlixZYlRXV1s671DKEM19Zaf61iXUdayurjZKSkpitk8AAMElGUZi97Pk5eVp2rRpysrK8hkLZhiGNm/eXOfzmoz/a/mQVGcsmef73bt310cffeQzHzPztpLVy6s5P0n1bgMrlxOrcXpOKINV6lsXN60jAOAYAl5enoqLi2NdDAAAAMswBg8AAMBlCHgAAAAuQ8ADAABwGQIeAACAyxDwAAAAXIaABwAA4DIEPAAAAJch4AEAALgMAQ8AAMBlCHgAAAAuQ8ADAABwGdMBLz8/X7m5ucrMzFReXp42bNgQcPp169YpLy9PmZmZ+v3vf6+CggKfv+fm5iojI6POf9dee613mvXr1+u6667TgAEDlJGR4fc3Yx955BENGTJE2dnZ6tWrl6644gpt2rTJ7GoBAIAE58aMYyrgrVq1SjNnztR1112nFStWKCcnR9dcc42++eYbv9Pv2LFD1157rXJycrRixQqNHz9eM2bM0Guvvead5vnnn9e7777r/W/58uVKSkrS2Wef7Z2moqJC6enpmjp1qo477ji/y0pLS9P06dP14osv6rnnnlPHjh119dVXq7y83NQGAAAAicu1Gccw4cILLzSmTp3q89mgQYOMBx54wO/0s2fPNgYNGuTz2R133GFcfPHF9S5jwYIFRs+ePY1Dhw75/Xt2drZRVFQUtKw//PCDkZ6ebrz99ttBpzUMwxg+fLip6QAAgPu4NeM0DBYAKysr9cknn+jKK6/0+bxfv34qKSnx+53S0lL169fP57P+/ftrxYoVOnLkiBo1alQ7ZOr555/X+eefX2+KNaOyslKFhYU68cQTdeqpp9Y7XWFhoQoLCyVJ//3vf5WXlxf2MgEAgPOUlZWpTZs23n+PHDlSI0eO9JnGjRnHI2jA27t3r6qqqtS6dWufz1u1aqW1a9f6/U55ebn69u3r81nr1q119OhR7d2712eDS9J7772nnTt36uKLLw5aYH9Wr16tW265RYcOHVJqaqqefPLJOuWtqeZOzsvL89vvDQAA4peZ67sbM46HI56iXbp0qTIzM9WlS5ewvt+7d2+tWLFCS5Ys0YABA3TzzTerrKzM4lICAACEJlYZJ2jAa9GihRo0aFBnQN/u3buVmprq9zutW7fW7t27fT4rLy9Xw4YN1aJFizrzeeutt8JOtpKUkpKiX/7yl8rOztbMmTPVsGFDLVu2LOz5AQAA93Nzxgka8Bo3bqyuXbvWaapcu3atcnJy/H4nOzvb7/TdunWr0zddXFysRo0a6ZxzzglaWLOqq6tVWVlpatra/fEAACD+mbm+uznjBB2DJ0njxo3T5MmT1b17d/Xo0UMFBQUqKyvTqFGjJEmTJ0+WJM2ePVuSNGrUKOXn5+vee+/VqFGjtGnTJi1fvlwPPvigz3w9Aw/POeccnXDCCXWW++OPP+qrr77yrtA333yjLVu2qFmzZmrfvr0OHjyoxYsXKzc3V6mpqdqzZ4/y8/P13Xff+TyKHAgBDwAA9zF7fXdrxjEV8IYOHaq9e/dq4cKFKisrU3p6uhYtWqQOHTpIkr799luf6Tt16qRFixZp1qxZKigoUJs2bTR16lQNHjzYZ7oPP/xQ27dv1/333+93uR9//LEuv/xy77/nzp2ruXPnavjw4brvvvvUoEEDff755yoqKtK+ffvUvHlzZWZmKj8/P+y+bgAAkDjcmnGSDMMwQt0YAAAAcC5HPEULAAAA6xDwAAAAXIaABwAA4DIEPAAAAJch4AEAALgMAQ8Iori4WBkZGTr99NO1f/9+n78dPXpUGRkZmjt3btTLNXfuXGVkZOjo0aNRX3Yoqqurde+996p///7q0qWLbrjhhnqnzc3NVUZGhiZOnOj372PGjFFGRoYuueQSy8o3ZcoU5ebmhvy9Dz/8UBkZGfrwww8jWr5nPvX9d+DAgYjmX5/c3FxNmTLFlnlL0pYtWzR37lzt27fP0vl6zsedO3daOl/AbUy9Bw+A9MMPP2jx4sW69dZbY12UuPLqq6/q6aef1pQpU5Sdna3mzZsHnP6EE07QG2+8oYMHD+rEE0/0fv71119r/fr1fl8Y6gZ33nmnMjMz63wer+u7ZcsWzZs3T+eff37QfQ7AegQ8wKT+/fvr2Wef1dixY9W6detYFycqKisr1bhx44jm8cUXX0iSrrjiCiUnB+806Nevn9577z29/vrrysvL836+cuVKdejQQSeffLKqqqoiKpMTnXLKKcrOzo51MQC4REJ30ebn5ys3N1eZmZnKy8vThg0bYl0kONj1118vSVq4cGHA6Txdp7XV7grcuXOnMjIyVFBQoAcffFD9+vVTTk6Obr31Vh06dEhffvmlrrrqKuXk5GjQoEFavny53+Vt3bpVY8aMUVZWlvr376+//vWvqq6u9plmz549mjZtmgYMGKBu3bppyJAhKiws9JnG0/W1fv163XTTTTr99NN10UUXBVzXt99+WyNHjlT37t3Vs2dP3XDDDd5AJx3rBvR0X5966qnKyMhQcXFxwHk2adJEgwcP1sqVK30+X7lypYYNG6akpKQ63ykrK9PkyZPVu3dvdevWTeedd16d70vS+++/r+HDhyszM1N/+MMftGTJEr9lOHTokO6//37l5uaqW7duys3N1cKFC+ts19reeecdjRo1Sj179lROTo4GDx6sefPmBfyOGbt27dJpp52mp59+us7fFi9erK5du2rPnj2SpHfffVfXXHON+vfvr6ysLJ177rn629/+FjQUmz1uJWnOnDkaPny4evTood69e+vyyy9XaWmp9+/FxcW6/fbbJUlnnXWWt7vZ06169OhRPfbYYxoyZIi6deum/v3767777tPhw4d9lrNjxw5de+21ysrKUp8+fTRjxgzTvzMO9ykrK9Ntt92mPn36KDMzU0OHDtW6deu8fzcMQ3PnzlX//v3VvXt3jRkzRp999pnPPPbv369JkyapZ8+e6tmzpyZNmmTbMIhYS9gWvFWrVmnmzJmaPn26evbsqeeee07XXHONXn75ZbVv3z7WxYMDpaamavTo0Xrqqad05ZVXen/GJlKLFi3SGWecofvuu09bt27V/fffr+TkZG3ZskUXXXSRrrzyShUUFOj2229Xt27d9Otf/9rn+zfeeKNGjBih8ePH691339WCBQuUnJysCRMmSJIOHjyoSy65RIcPH9aECRPUsWNHvfPOO/rzn/+syspKjRkzxmd+t956q8455xzNmTMn4Pi+t99+W+PHj1efPn308MMPq6KiQnPmzNGll16qlStXqm3btpo3b56eeeYZFRcXewPlL37xi6Db5IILLtDYsWP13XffqV27diotLdX27dt1wQUXaP369T7TVlRUaMyYMdq/f79uueUWtWvXTi+88IImT56sn376yft7lFu3btU111yjbt266eGHH1ZlZaXmzp2riooKNWjQwDu/o0eP6qqrrtLWrVt1/fXXKyMjQ6WlpVqwYIH2799f77i1HTt26Prrr9fgwYN1ww03qFGjRvryyy+1Y8eOoOsrHRurWHt7JyUlqUGDBkpNTVXfvn31wgsv+Py0kSS98MILGjBggFq2bOktR9++fXXZZZepSZMm+vjjjzV37lzt2bPHsuEF33//va644gq1a9dOhw4d0gsvvKDLLrtMRUVFysjI0O9+9ztdf/31Wrhwof7617+qXbt2kqQ2bdpIkiZNmqTVq1fr6quvVo8ePbR161b99a9/1ddff+29IaisrNS4ceP0008/adq0aWrVqpWWLFmif/zjH5asA+LLgQMHdMkll6hnz55atGiRWrRooZ07d6pVq1beaRYvXqy//e1vuu+++5SWlqb58+dr3LhxevXVV73DPSZOnKhvv/1Wjz/+uKRjQyMmT56sRx99NCbrZSsjQV144YXG1KlTfT4bNGiQ8cADD8SoRHCqoqIiIz093di+fbuxd+9eo2fPnsaUKVMMwzCMI0eOGOnp6cacOXO808+ZM8dIT0+vM5/bbrvNGDhwoPffO3bsMNLT040xY8b4THfjjTca6enpxooVK7yf7du3zzj11FONuXPn1lnOY4895vP9qVOnGtnZ2cb+/fsNwzCMefPmGd26dTO2bdtWZ7ozzjjDOHLkiM963nvvvaa2y/Dhw41BgwZ5v28YhvHVV18Zp512mjFz5kzvZw899JDf7eHPwIEDjYkTJxrV1dXGwIEDves2ffp0Y+TIkYZhGMZll11mjBo1yvudZ555xkhPTzc++OADn3ldccUVRp8+fYyjR48ahmEYt9xyi3HGGWcYP/74o3eab775xujatavPflm+fLmRnp5urFu3zmd+CxYsMLp27WqUl5cbhmEYH3zwgc9yX3nlFSM9Pd344YcfTK2rh2c+/v4755xzvNOtXLnSSE9PN7Zu3er97NNPPzXS09ONl19+2e+8q6urjSNHjhgLFiwwTj/9dKOqqsr7t4EDBxq33Xab999mj9vajh49ahw5csQ466yzjHvuucf7ec3zpqb169cb6enpxvLly30+96zfp59+ahiGYRQWFhrp6elGSUmJd5qqqipj6NChRnp6urFjx456ywT3efDBB711gD/V1dVGv379jAULFng/O3TokJGdnW0UFBQYhmEYn3/+uZGenm5s2LDBO43neKx5XrlFQnbRVlZW6pNPPlG/fv18Pu/Xr59KSkpiVCrEg+bNm2vcuHFauXKlT1dkJM4880yff3fu3FmSNGDAAO9nzZo1U8uWLev86LUknX322T7/Puecc1RRUaH//ve/ko51G2ZlZaljx446evSo97/+/ftr3759+vzzz32+P2jQoKBlrqio0Keffqqzzz5bDRv+3BHQqVMn9ejRo04rW6iSkpK83ayVlZV65ZVXdMEFF/iddv369Wrbtq169+7t8/n555+vPXv2eNevtLRUv/3tb5WSkuKd5uSTT1ZOTo7P99555x116NBBOTk5PturX79+OnLkiE9XZE2nnnqqGjVqpD/96U969dVXtXv37pDWedq0aXr++ed9/nv44Ye9fx80aJBSUlJ8up5Xrlyppk2b6ve//733s7KyMk2bNk0DBw5Ut27d1LVrVz3yyCM6cOBAyGWqz9q1azVmzBj17t1bp512mrp27art27dr27ZtQb/7zjvvqFGjRho8eHCd41GS99gpKSnRySef7DMuMTk5uc7xjsTwxhtvKCsrSzfffLP69u2rYcOG6dlnn5VhGJKODXnZtWuXz3X9uOOOU69evbzX9ZKSEqWkpKhHjx7eaXr27KmUlBRXXvsTsot27969qqqqqjNQvlWrVlq7dm2MSoV4MXbsWD377LOaM2eOHnjggYjns57Z+AAACNxJREFU16xZM59/N2rUSJJ00kkn+XzeuHHjOmOUJPl0UdT8d1lZmaRj4+++/PJLde3a1e/ya7/GIjU1NWiZDxw4IMMwvF1uNbVu3Vpff/110HkEc8EFF+jRRx/V/PnzVVFRoaFDh/qdbv/+/X7L7Dm/Pa+22bVrV51t5a+8e/bs0ddff216e3n88pe/1OOPP67Fixdr8uTJqqysVPfu3XXrrbfqjDPOCLyyktLS0vw+Retx/PHHa/DgwXrxxRd18803q7q6Wi+99JKGDBmiJk2aSDrWzXv99derrKxMEyZMUOfOndWkSRO98cYbevTRR/0eP6H65JNPdO2116p///669957lZqaquTkZN15552mxsft3r1bR44cqfeBEs/2rW9/+fsM7rdjxw4999xzGjt2rK699lpt2bJFM2bMkCRddtll2rVrlyT5va576sLy8nK1bNnSZxxvUlKSWrZsqfLy8iitSfQkZMADInHCCSdo/Pjxuu+++3TVVVfV+bvnYlv7CVSr3wfmsXv3bp9WKU8rjSd8NW/eXC1bttTUqVP9fj8tLc3n3/4eYqjtpJNOUlJSkrdSram8vNyS12KkpaUpKytLixYt0qBBg+oEXo9mzZr5bTnyVNieAJ2amuq3Bat2xd68eXN17NhRjzzyiN/lBRp72adPH/Xp00eVlZXauHGj5syZo/Hjx+vNN9/0jpGLxLBhw7R8+XJt3LhRP/30k3bt2qVhw4Z5//7VV1/p448/1uzZs30+X716ddB5mz1uX3/9dTVo0EBz58713oxIx0J/ffuopubNm6tJkybKz8/3+3fPcZuamlqndVmSZa2QiC+GYahbt27ed2Sedtpp+vLLL5Wfn6/LLrssxqVzpoTsom3RooUaNGhQp2LfvXu3qdYL4NJLL1Xbtm39hgDPQzo1n946cOCAbV0Ar7zyis+/X375ZaWkpHifiBwwYIC2bdum9u3bKzMzs85/Nd81Z1ZKSoq6du2qV1991efpzK+//lolJSWmWqzMuPrqqzVw4MCAFfgZZ5yh7777Ths3bvT5/KWXXlKrVq30P//zP5Kk7Oxs/fOf/1RFRYV3mm+//bbOfhkwYIC+++47paSk+N1eZoJa48aN1bdvX1199dWqqKiw7KW8vXv3Vrt27bRy5Urva2NOP/10799/+uknSfIJXkeOHNGLL74YdN5mj9tDhw4pOTnZ50bg/fff1zfffOMznSckesrkMWDAAB0+fFgHDx70u33btm0rScrJydG3337r0yVeXV1d53hHYkhNTdUpp5zi81nnzp29w1Y8125/13VPq17r1q21Z88eb7eudCw47tmzx5WvvkrIFrzGjRura9euWrt2rc94jrVr1+qss86KYckQLxo3bqwbb7xRd911V52/nXnmmWratKnuuusuTZgwQZWVlXr88cd9WtmstHTpUlVXVyszM1Pvvvuuli1bpgkTJqhp06aSjnUpr1q1SpdeeqnGjh2rtLQ0HTp0SF988YU2bNgQ9LUv9fl//+//afz48Ro/frwuvfRSVVRUaO7cuTrxxBM1btw4S9btrLPOCnpODh8+XE8//bQmTJigP/3pT2rbtq1efPFFvffee7r77ru9T8jecMMNeu2113TllVfq6quvVmVlpebNm1eny++8885TcXGxxo4dqyuvvFJdunRRZWWlduzYobfeekvz58/X8ccfX6ccBQUF2rBhg84880ydfPLJ2rt3rx577DG1adNG6enpQdd169atfo+R9PR07+fJyck677zzVFhYqKNHj+qKK67wCVqdO3dWhw4d9PDDDys5OVkNGzbUU089FXTZkvnjdsCAAXrqqac0ZcoUjRgxQtu2bdOCBQu8wczDE6zz8/M1fPhwNWzYUBkZGerdu7fOPfdc3XTTTRo7dqy6d++u5ORkff311/rnP/+pW2+9VWlpabrgggu0aNEi/fGPf9Qtt9yiVq1aqaCgQAcPHjS1PnCXHj161Gmp3759u/fGpGPHjkpNTdXatWvVvXt3SdLhw4e1YcMGTZ48WdKxm4aKigqVlJR4x+GVlJSooqKizlhcN0jIgCdJ48aN0+TJk9W9e3f16NFDBQUFKisr06hRo2JdNMSJvLw8PfHEE9q+fbvP5yeddJIeffRRzZo1SzfffLPatWunG264Qe+//77PO5ussmDBAt1zzz1asGCBmjZtquuvv97n58CaNm2qJUuWaP78+Vq8eLHKysrUtGlTpaWlRXRDc+aZZ+qxxx7T/PnzdfPNN6tRo0Y644wzNGnSpDoXezulpKTomWee0f33368HHnhAP/74o9LS0up0U55yyilatGiRZs+erZtvvllt27bVNddco9LSUp/90qhRIz3xxBNatGiRCgsLtXPnTqWkpKhTp0763e9+59M6VlOXLl309ttv66GHHtLu3bvVvHlz9ejRQw888ICOO+64oOvhGU9U2/PPP+8zNm/YsGFavHix9//X1LhxY82fP1933323brvtNjVr1kwjRoxQ+/btdeeddwZcvtnjdsCAAbrzzjv15JNP6vXXX9evf/1rzZ49u86NQpcuXTRhwgQVFhZq2bJlqq6u1ptvvqmOHTvq/vvv1zPPPKOioiI9+uijaty4sTp06KD+/ft7W1IaN26sJ598Unfffbf+8pe/6Pjjj9e5556r3/3ud5o+fXrQ7Ql3ueKKK3TJJZdo4cKFGjp0qD799FM988wzuuWWWyQdG1py+eWX67HHHlPnzp31q1/9SgsXLlRKSorOPfdcScfqgAEDBmj69Om6++67JUnTp0/XwIEDvQ+3uUmSUbOtMsHk5+friSeeUFlZmdLT03X77berV69esS4WAACoZc2aNXrooYe8Q05Gjx6tMWPGeFuxDcPQvHnzVFhYqP379ysrK0vTpk3zaUHfv3+/7rnnHr311luSjr2Mfdq0aabGj8abhA54AAAAbpSQD1kAAAC4GQEPAADAZQh4AAAALkPAAwAAcBkCHgAAgMsQ8AAAAFyGgAcAAOAyBDwAAACX+f+aZX8sQaBuqAAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 576x432 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XVkOpZur6d9F",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 223
        },
        "outputId": "d1f1ce7b-1217-46f2-d05f-23684c77448f"
      },
      "source": [
        "model_best"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "([(1, None),\n",
              "  (0, 16),\n",
              "  (0, 16),\n",
              "  (-1, <function __main__.random_element.<locals>.<lambda>>),\n",
              "  (1, None),\n",
              "  (0, 16),\n",
              "  (2, ('renorm', True)),\n",
              "  (0, 16),\n",
              "  (-1, <function __main__.random_element.<locals>.<lambda>>),\n",
              "  (1, None)],\n",
              " 0.6743999719619751)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qj-VvVYmowwr",
        "colab_type": "text"
      },
      "source": [
        "# Run continuation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X490iOzwTFEL",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "64f78de9-3132-4e68-9909-534c5065f128"
      },
      "source": [
        "model_best_scaled = []\n",
        "scaling_factor = 2\n",
        "scaling_exp = 1\n",
        "for el in model_best[0]:\n",
        "  if el[0] == 0:\n",
        "    new_el = (el[0], el[1]*scaling_factor**scaling_exp)\n",
        "    scaling_exp += 1\n",
        "  else:\n",
        "    new_el = el\n",
        "  model_best_scaled.append(new_el)\n",
        "model = AnnModel(gene_dictionary=available_genes, model_list=model_best_scaled)\n",
        "learning_rate = LEARNING_RATE\n",
        "optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)\n",
        "model.compile(optimizer=optimizer, metrics=['accuracy'], loss='categorical_crossentropy')\n",
        "history_test = model.fit(x_train, y_train, batch_size=2048, epochs=60,\n",
        "                         verbose=2, validation_data=(x_test, y_test), shuffle=True)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/60\n",
            "49/49 - 1s - loss: 1.8549 - accuracy: 0.3708 - val_loss: 4.8713 - val_accuracy: 0.1678\n",
            "Epoch 2/60\n",
            "49/49 - 1s - loss: 1.3190 - accuracy: 0.5324 - val_loss: 10.3075 - val_accuracy: 0.1244\n",
            "Epoch 3/60\n",
            "49/49 - 1s - loss: 1.0494 - accuracy: 0.6290 - val_loss: 11.7640 - val_accuracy: 0.1860\n",
            "Epoch 4/60\n",
            "49/49 - 1s - loss: 0.8641 - accuracy: 0.6969 - val_loss: 11.7456 - val_accuracy: 0.1850\n",
            "Epoch 5/60\n",
            "49/49 - 1s - loss: 0.7325 - accuracy: 0.7438 - val_loss: 6.3978 - val_accuracy: 0.2946\n",
            "Epoch 6/60\n",
            "49/49 - 1s - loss: 0.6389 - accuracy: 0.7772 - val_loss: 2.1457 - val_accuracy: 0.4932\n",
            "Epoch 7/60\n",
            "49/49 - 1s - loss: 0.5460 - accuracy: 0.8092 - val_loss: 1.1410 - val_accuracy: 0.6433\n",
            "Epoch 8/60\n",
            "49/49 - 1s - loss: 0.4689 - accuracy: 0.8369 - val_loss: 1.0231 - val_accuracy: 0.6805\n",
            "Epoch 9/60\n",
            "49/49 - 1s - loss: 0.4000 - accuracy: 0.8597 - val_loss: 1.0789 - val_accuracy: 0.6843\n",
            "Epoch 10/60\n",
            "49/49 - 1s - loss: 0.3449 - accuracy: 0.8779 - val_loss: 1.2043 - val_accuracy: 0.6860\n",
            "Epoch 11/60\n",
            "49/49 - 1s - loss: 0.2901 - accuracy: 0.8980 - val_loss: 1.2727 - val_accuracy: 0.6838\n",
            "Epoch 12/60\n",
            "49/49 - 1s - loss: 0.2498 - accuracy: 0.9113 - val_loss: 1.3500 - val_accuracy: 0.6911\n",
            "Epoch 13/60\n",
            "49/49 - 1s - loss: 0.1922 - accuracy: 0.9335 - val_loss: 1.4073 - val_accuracy: 0.6918\n",
            "Epoch 14/60\n",
            "49/49 - 1s - loss: 0.1697 - accuracy: 0.9408 - val_loss: 1.6869 - val_accuracy: 0.6935\n",
            "Epoch 15/60\n",
            "49/49 - 1s - loss: 0.1677 - accuracy: 0.9400 - val_loss: 1.6773 - val_accuracy: 0.6788\n",
            "Epoch 16/60\n",
            "49/49 - 1s - loss: 0.1323 - accuracy: 0.9532 - val_loss: 1.6618 - val_accuracy: 0.6904\n",
            "Epoch 17/60\n",
            "49/49 - 1s - loss: 0.1024 - accuracy: 0.9655 - val_loss: 1.7356 - val_accuracy: 0.6885\n",
            "Epoch 18/60\n",
            "49/49 - 1s - loss: 0.0867 - accuracy: 0.9706 - val_loss: 1.7950 - val_accuracy: 0.6968\n",
            "Epoch 19/60\n",
            "49/49 - 1s - loss: 0.0805 - accuracy: 0.9719 - val_loss: 1.7783 - val_accuracy: 0.6964\n",
            "Epoch 20/60\n",
            "49/49 - 1s - loss: 0.0756 - accuracy: 0.9737 - val_loss: 1.9232 - val_accuracy: 0.6891\n",
            "Epoch 21/60\n",
            "49/49 - 1s - loss: 0.0721 - accuracy: 0.9749 - val_loss: 2.0234 - val_accuracy: 0.6891\n",
            "Epoch 22/60\n",
            "49/49 - 1s - loss: 0.0745 - accuracy: 0.9736 - val_loss: 2.1089 - val_accuracy: 0.6908\n",
            "Epoch 23/60\n",
            "49/49 - 1s - loss: 0.0766 - accuracy: 0.9731 - val_loss: 2.2544 - val_accuracy: 0.6792\n",
            "Epoch 24/60\n",
            "49/49 - 1s - loss: 0.0786 - accuracy: 0.9719 - val_loss: 2.0389 - val_accuracy: 0.6965\n",
            "Epoch 25/60\n",
            "49/49 - 1s - loss: 0.0726 - accuracy: 0.9744 - val_loss: 2.1160 - val_accuracy: 0.6996\n",
            "Epoch 26/60\n",
            "49/49 - 1s - loss: 0.0596 - accuracy: 0.9792 - val_loss: 2.2152 - val_accuracy: 0.6959\n",
            "Epoch 27/60\n",
            "49/49 - 1s - loss: 0.0483 - accuracy: 0.9835 - val_loss: 2.1470 - val_accuracy: 0.6972\n",
            "Epoch 28/60\n",
            "49/49 - 1s - loss: 0.0345 - accuracy: 0.9890 - val_loss: 2.1031 - val_accuracy: 0.7032\n",
            "Epoch 29/60\n",
            "49/49 - 1s - loss: 0.0205 - accuracy: 0.9939 - val_loss: 2.1688 - val_accuracy: 0.7055\n",
            "Epoch 30/60\n",
            "49/49 - 1s - loss: 0.0130 - accuracy: 0.9965 - val_loss: 2.1607 - val_accuracy: 0.7079\n",
            "Epoch 31/60\n",
            "49/49 - 1s - loss: 0.0072 - accuracy: 0.9986 - val_loss: 2.2617 - val_accuracy: 0.7057\n",
            "Epoch 32/60\n",
            "49/49 - 1s - loss: 0.0033 - accuracy: 0.9997 - val_loss: 2.1350 - val_accuracy: 0.7096\n",
            "Epoch 33/60\n",
            "49/49 - 1s - loss: 0.0012 - accuracy: 1.0000 - val_loss: 2.1944 - val_accuracy: 0.7112\n",
            "Epoch 34/60\n",
            "49/49 - 1s - loss: 6.5297e-04 - accuracy: 1.0000 - val_loss: 2.2043 - val_accuracy: 0.7123\n",
            "Epoch 35/60\n",
            "49/49 - 1s - loss: 5.0596e-04 - accuracy: 1.0000 - val_loss: 2.2157 - val_accuracy: 0.7112\n",
            "Epoch 36/60\n",
            "49/49 - 1s - loss: 4.2332e-04 - accuracy: 1.0000 - val_loss: 2.2399 - val_accuracy: 0.7122\n",
            "Epoch 37/60\n",
            "49/49 - 1s - loss: 3.9332e-04 - accuracy: 1.0000 - val_loss: 2.2542 - val_accuracy: 0.7116\n",
            "Epoch 38/60\n",
            "49/49 - 1s - loss: 4.7002e-04 - accuracy: 1.0000 - val_loss: 2.2597 - val_accuracy: 0.7120\n",
            "Epoch 39/60\n",
            "49/49 - 1s - loss: 3.2566e-04 - accuracy: 1.0000 - val_loss: 2.2804 - val_accuracy: 0.7130\n",
            "Epoch 40/60\n",
            "49/49 - 1s - loss: 2.7708e-04 - accuracy: 1.0000 - val_loss: 2.2905 - val_accuracy: 0.7127\n",
            "Epoch 41/60\n",
            "49/49 - 1s - loss: 2.4978e-04 - accuracy: 1.0000 - val_loss: 2.3002 - val_accuracy: 0.7127\n",
            "Epoch 42/60\n",
            "49/49 - 1s - loss: 2.3419e-04 - accuracy: 1.0000 - val_loss: 2.3135 - val_accuracy: 0.7130\n",
            "Epoch 43/60\n",
            "49/49 - 1s - loss: 2.2987e-04 - accuracy: 1.0000 - val_loss: 2.3271 - val_accuracy: 0.7126\n",
            "Epoch 44/60\n",
            "49/49 - 1s - loss: 2.0779e-04 - accuracy: 1.0000 - val_loss: 2.3260 - val_accuracy: 0.7128\n",
            "Epoch 45/60\n",
            "49/49 - 1s - loss: 1.8771e-04 - accuracy: 1.0000 - val_loss: 2.3404 - val_accuracy: 0.7133\n",
            "Epoch 46/60\n",
            "49/49 - 1s - loss: 1.7347e-04 - accuracy: 1.0000 - val_loss: 2.3498 - val_accuracy: 0.7130\n",
            "Epoch 47/60\n",
            "49/49 - 1s - loss: 1.6342e-04 - accuracy: 1.0000 - val_loss: 2.3596 - val_accuracy: 0.7129\n",
            "Epoch 48/60\n",
            "49/49 - 1s - loss: 1.5458e-04 - accuracy: 1.0000 - val_loss: 2.3678 - val_accuracy: 0.7127\n",
            "Epoch 49/60\n",
            "49/49 - 1s - loss: 1.4804e-04 - accuracy: 1.0000 - val_loss: 2.3752 - val_accuracy: 0.7127\n",
            "Epoch 50/60\n",
            "49/49 - 1s - loss: 1.4107e-04 - accuracy: 1.0000 - val_loss: 2.3822 - val_accuracy: 0.7124\n",
            "Epoch 51/60\n",
            "49/49 - 1s - loss: 1.3230e-04 - accuracy: 1.0000 - val_loss: 2.3893 - val_accuracy: 0.7127\n",
            "Epoch 52/60\n",
            "49/49 - 1s - loss: 1.2656e-04 - accuracy: 1.0000 - val_loss: 2.3983 - val_accuracy: 0.7123\n",
            "Epoch 53/60\n",
            "49/49 - 1s - loss: 1.1973e-04 - accuracy: 1.0000 - val_loss: 2.4045 - val_accuracy: 0.7128\n",
            "Epoch 54/60\n",
            "49/49 - 1s - loss: 1.1487e-04 - accuracy: 1.0000 - val_loss: 2.4111 - val_accuracy: 0.7125\n",
            "Epoch 55/60\n",
            "49/49 - 1s - loss: 1.0981e-04 - accuracy: 1.0000 - val_loss: 2.4186 - val_accuracy: 0.7131\n",
            "Epoch 56/60\n",
            "49/49 - 1s - loss: 1.0420e-04 - accuracy: 1.0000 - val_loss: 2.4276 - val_accuracy: 0.7127\n",
            "Epoch 57/60\n",
            "49/49 - 1s - loss: 9.7967e-05 - accuracy: 1.0000 - val_loss: 2.4326 - val_accuracy: 0.7128\n",
            "Epoch 58/60\n",
            "49/49 - 1s - loss: 9.5209e-05 - accuracy: 1.0000 - val_loss: 2.4380 - val_accuracy: 0.7128\n",
            "Epoch 59/60\n",
            "49/49 - 1s - loss: 9.0864e-05 - accuracy: 1.0000 - val_loss: 2.4471 - val_accuracy: 0.7131\n",
            "Epoch 60/60\n",
            "49/49 - 1s - loss: 8.8117e-05 - accuracy: 1.0000 - val_loss: 2.4549 - val_accuracy: 0.7137\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5BITdZlQ7Diy",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "bd259c40-3a4f-46fc-b499-502405af5742"
      },
      "source": [
        "# Create an training generator:\n",
        "idg = tf.keras.preprocessing.image.ImageDataGenerator(horizontal_flip=False, rotation_range=35, \n",
        "                                                      width_shift_range=0.2, height_shift_range=0.2,\n",
        "                                                      shear_range=0.1)\n",
        "model_best_scaled = []\n",
        "scaling_factor = 2\n",
        "scaling_exp = 1\n",
        "for el in model_best[0]:\n",
        "  if el[0] == 0:\n",
        "    new_el = (el[0], el[1]*scaling_factor**scaling_exp)\n",
        "    scaling_exp += 1\n",
        "  else:\n",
        "    new_el = el\n",
        "  model_best_scaled.append(new_el)\n",
        "model = AnnModel(gene_dictionary=available_genes, model_list=model_best_scaled)\n",
        "learning_rate = LEARNING_RATE\n",
        "optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)\n",
        "model.compile(optimizer=optimizer, metrics=['accuracy'], loss='categorical_crossentropy')\n",
        "history_test = model.fit(idg.flow(x_train, y_train, batch_size=2048), epochs=180, steps_per_epoch=len(x_train)/2048,\n",
        "                         verbose=2, validation_data=(x_test, y_test), shuffle=True)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/180\n",
            "49/48 - 20s - loss: 2.1596 - accuracy: 0.2577 - val_loss: 2.6698 - val_accuracy: 0.0962\n",
            "Epoch 2/180\n",
            "49/48 - 19s - loss: 1.7522 - accuracy: 0.3654 - val_loss: 9.9112 - val_accuracy: 0.1207\n",
            "Epoch 3/180\n",
            "49/48 - 20s - loss: 1.5750 - accuracy: 0.4339 - val_loss: 7.3309 - val_accuracy: 0.1158\n",
            "Epoch 4/180\n",
            "49/48 - 19s - loss: 1.4719 - accuracy: 0.4738 - val_loss: 2.5517 - val_accuracy: 0.2550\n",
            "Epoch 5/180\n",
            "49/48 - 20s - loss: 1.3851 - accuracy: 0.5068 - val_loss: 1.6542 - val_accuracy: 0.4280\n",
            "Epoch 6/180\n",
            "49/48 - 20s - loss: 1.3275 - accuracy: 0.5277 - val_loss: 1.3862 - val_accuracy: 0.4954\n",
            "Epoch 7/180\n",
            "49/48 - 20s - loss: 1.2688 - accuracy: 0.5502 - val_loss: 1.2898 - val_accuracy: 0.5375\n",
            "Epoch 8/180\n",
            "49/48 - 20s - loss: 1.2197 - accuracy: 0.5681 - val_loss: 1.1451 - val_accuracy: 0.5991\n",
            "Epoch 9/180\n",
            "49/48 - 20s - loss: 1.1783 - accuracy: 0.5843 - val_loss: 1.1459 - val_accuracy: 0.5944\n",
            "Epoch 10/180\n",
            "49/48 - 20s - loss: 1.1517 - accuracy: 0.5941 - val_loss: 1.2017 - val_accuracy: 0.5938\n",
            "Epoch 11/180\n",
            "49/48 - 19s - loss: 1.1136 - accuracy: 0.6079 - val_loss: 1.0881 - val_accuracy: 0.6276\n",
            "Epoch 12/180\n",
            "49/48 - 20s - loss: 1.0937 - accuracy: 0.6151 - val_loss: 1.1224 - val_accuracy: 0.6182\n",
            "Epoch 13/180\n",
            "49/48 - 20s - loss: 1.0707 - accuracy: 0.6240 - val_loss: 1.0930 - val_accuracy: 0.6250\n",
            "Epoch 14/180\n",
            "49/48 - 20s - loss: 1.0505 - accuracy: 0.6324 - val_loss: 1.0690 - val_accuracy: 0.6371\n",
            "Epoch 15/180\n",
            "49/48 - 20s - loss: 1.0292 - accuracy: 0.6386 - val_loss: 1.1793 - val_accuracy: 0.6265\n",
            "Epoch 16/180\n",
            "49/48 - 20s - loss: 1.0109 - accuracy: 0.6454 - val_loss: 0.9947 - val_accuracy: 0.6658\n",
            "Epoch 17/180\n",
            "49/48 - 20s - loss: 0.9918 - accuracy: 0.6531 - val_loss: 1.0658 - val_accuracy: 0.6496\n",
            "Epoch 18/180\n",
            "49/48 - 19s - loss: 0.9759 - accuracy: 0.6582 - val_loss: 0.9616 - val_accuracy: 0.6764\n",
            "Epoch 19/180\n",
            "49/48 - 19s - loss: 0.9590 - accuracy: 0.6636 - val_loss: 1.0002 - val_accuracy: 0.6674\n",
            "Epoch 20/180\n",
            "49/48 - 19s - loss: 0.9429 - accuracy: 0.6688 - val_loss: 1.0033 - val_accuracy: 0.6680\n",
            "Epoch 21/180\n",
            "49/48 - 20s - loss: 0.9359 - accuracy: 0.6721 - val_loss: 0.9161 - val_accuracy: 0.6909\n",
            "Epoch 22/180\n",
            "49/48 - 19s - loss: 0.9244 - accuracy: 0.6771 - val_loss: 0.9629 - val_accuracy: 0.6716\n",
            "Epoch 23/180\n",
            "49/48 - 20s - loss: 0.9172 - accuracy: 0.6785 - val_loss: 1.0019 - val_accuracy: 0.6614\n",
            "Epoch 24/180\n",
            "49/48 - 20s - loss: 0.9070 - accuracy: 0.6837 - val_loss: 0.9644 - val_accuracy: 0.6782\n",
            "Epoch 25/180\n",
            "49/48 - 20s - loss: 0.8940 - accuracy: 0.6862 - val_loss: 0.9021 - val_accuracy: 0.6891\n",
            "Epoch 26/180\n",
            "49/48 - 20s - loss: 0.8853 - accuracy: 0.6897 - val_loss: 0.9740 - val_accuracy: 0.6766\n",
            "Epoch 27/180\n",
            "49/48 - 20s - loss: 0.8787 - accuracy: 0.6914 - val_loss: 0.9258 - val_accuracy: 0.6915\n",
            "Epoch 28/180\n",
            "49/48 - 20s - loss: 0.8736 - accuracy: 0.6941 - val_loss: 0.8632 - val_accuracy: 0.7096\n",
            "Epoch 29/180\n",
            "49/48 - 19s - loss: 0.8697 - accuracy: 0.6953 - val_loss: 0.8847 - val_accuracy: 0.7007\n",
            "Epoch 30/180\n",
            "49/48 - 19s - loss: 0.8609 - accuracy: 0.6984 - val_loss: 0.8767 - val_accuracy: 0.7071\n",
            "Epoch 31/180\n",
            "49/48 - 19s - loss: 0.8581 - accuracy: 0.6997 - val_loss: 0.9140 - val_accuracy: 0.6960\n",
            "Epoch 32/180\n",
            "49/48 - 20s - loss: 0.8525 - accuracy: 0.7023 - val_loss: 0.8189 - val_accuracy: 0.7260\n",
            "Epoch 33/180\n",
            "49/48 - 20s - loss: 0.8408 - accuracy: 0.7065 - val_loss: 0.9080 - val_accuracy: 0.7000\n",
            "Epoch 34/180\n",
            "49/48 - 19s - loss: 0.8306 - accuracy: 0.7111 - val_loss: 0.9660 - val_accuracy: 0.6831\n",
            "Epoch 35/180\n",
            "49/48 - 19s - loss: 0.8326 - accuracy: 0.7079 - val_loss: 0.8987 - val_accuracy: 0.7047\n",
            "Epoch 36/180\n",
            "49/48 - 19s - loss: 0.8218 - accuracy: 0.7120 - val_loss: 0.8955 - val_accuracy: 0.7041\n",
            "Epoch 37/180\n",
            "49/48 - 20s - loss: 0.8172 - accuracy: 0.7149 - val_loss: 0.9186 - val_accuracy: 0.7006\n",
            "Epoch 38/180\n",
            "49/48 - 19s - loss: 0.8207 - accuracy: 0.7122 - val_loss: 0.8011 - val_accuracy: 0.7293\n",
            "Epoch 39/180\n",
            "49/48 - 19s - loss: 0.8120 - accuracy: 0.7151 - val_loss: 0.8583 - val_accuracy: 0.7146\n",
            "Epoch 40/180\n",
            "49/48 - 19s - loss: 0.8015 - accuracy: 0.7192 - val_loss: 0.9021 - val_accuracy: 0.7020\n",
            "Epoch 41/180\n",
            "49/48 - 20s - loss: 0.8051 - accuracy: 0.7181 - val_loss: 0.7977 - val_accuracy: 0.7371\n",
            "Epoch 42/180\n",
            "49/48 - 20s - loss: 0.7970 - accuracy: 0.7201 - val_loss: 0.8079 - val_accuracy: 0.7279\n",
            "Epoch 43/180\n",
            "49/48 - 20s - loss: 0.7929 - accuracy: 0.7228 - val_loss: 0.9038 - val_accuracy: 0.7101\n",
            "Epoch 44/180\n",
            "49/48 - 20s - loss: 0.7877 - accuracy: 0.7231 - val_loss: 0.8001 - val_accuracy: 0.7321\n",
            "Epoch 45/180\n",
            "49/48 - 20s - loss: 0.7858 - accuracy: 0.7234 - val_loss: 0.8373 - val_accuracy: 0.7183\n",
            "Epoch 46/180\n",
            "49/48 - 19s - loss: 0.7858 - accuracy: 0.7244 - val_loss: 0.8279 - val_accuracy: 0.7285\n",
            "Epoch 47/180\n",
            "49/48 - 19s - loss: 0.7766 - accuracy: 0.7270 - val_loss: 0.8681 - val_accuracy: 0.7128\n",
            "Epoch 48/180\n",
            "49/48 - 19s - loss: 0.7762 - accuracy: 0.7265 - val_loss: 0.8589 - val_accuracy: 0.7139\n",
            "Epoch 49/180\n",
            "49/48 - 20s - loss: 0.7680 - accuracy: 0.7304 - val_loss: 0.8126 - val_accuracy: 0.7306\n",
            "Epoch 50/180\n",
            "49/48 - 19s - loss: 0.7799 - accuracy: 0.7272 - val_loss: 0.8303 - val_accuracy: 0.7271\n",
            "Epoch 51/180\n",
            "49/48 - 19s - loss: 0.7641 - accuracy: 0.7313 - val_loss: 0.8692 - val_accuracy: 0.7209\n",
            "Epoch 52/180\n",
            "49/48 - 20s - loss: 0.7606 - accuracy: 0.7335 - val_loss: 0.8250 - val_accuracy: 0.7318\n",
            "Epoch 53/180\n",
            "49/48 - 19s - loss: 0.7621 - accuracy: 0.7325 - val_loss: 0.8333 - val_accuracy: 0.7299\n",
            "Epoch 54/180\n",
            "49/48 - 20s - loss: 0.7586 - accuracy: 0.7352 - val_loss: 0.8073 - val_accuracy: 0.7342\n",
            "Epoch 55/180\n",
            "49/48 - 20s - loss: 0.7521 - accuracy: 0.7359 - val_loss: 0.8085 - val_accuracy: 0.7359\n",
            "Epoch 56/180\n",
            "49/48 - 20s - loss: 0.7530 - accuracy: 0.7348 - val_loss: 0.8048 - val_accuracy: 0.7366\n",
            "Epoch 57/180\n",
            "49/48 - 20s - loss: 0.7490 - accuracy: 0.7381 - val_loss: 0.8208 - val_accuracy: 0.7322\n",
            "Epoch 58/180\n",
            "49/48 - 19s - loss: 0.7394 - accuracy: 0.7419 - val_loss: 0.8514 - val_accuracy: 0.7256\n",
            "Epoch 59/180\n",
            "49/48 - 19s - loss: 0.7415 - accuracy: 0.7399 - val_loss: 0.8302 - val_accuracy: 0.7293\n",
            "Epoch 60/180\n",
            "49/48 - 19s - loss: 0.7398 - accuracy: 0.7417 - val_loss: 0.7768 - val_accuracy: 0.7380\n",
            "Epoch 61/180\n",
            "49/48 - 20s - loss: 0.7340 - accuracy: 0.7429 - val_loss: 0.7634 - val_accuracy: 0.7452\n",
            "Epoch 62/180\n",
            "49/48 - 20s - loss: 0.7327 - accuracy: 0.7426 - val_loss: 0.8627 - val_accuracy: 0.7198\n",
            "Epoch 63/180\n",
            "49/48 - 20s - loss: 0.7350 - accuracy: 0.7417 - val_loss: 0.8150 - val_accuracy: 0.7394\n",
            "Epoch 64/180\n",
            "49/48 - 20s - loss: 0.7315 - accuracy: 0.7436 - val_loss: 0.8095 - val_accuracy: 0.7332\n",
            "Epoch 65/180\n",
            "49/48 - 20s - loss: 0.7313 - accuracy: 0.7442 - val_loss: 0.8023 - val_accuracy: 0.7371\n",
            "Epoch 66/180\n",
            "49/48 - 19s - loss: 0.7208 - accuracy: 0.7471 - val_loss: 0.7603 - val_accuracy: 0.7508\n",
            "Epoch 67/180\n",
            "49/48 - 19s - loss: 0.7304 - accuracy: 0.7418 - val_loss: 0.8653 - val_accuracy: 0.7168\n",
            "Epoch 68/180\n",
            "49/48 - 19s - loss: 0.7301 - accuracy: 0.7437 - val_loss: 0.8024 - val_accuracy: 0.7397\n",
            "Epoch 69/180\n",
            "49/48 - 20s - loss: 0.7216 - accuracy: 0.7472 - val_loss: 0.8177 - val_accuracy: 0.7360\n",
            "Epoch 70/180\n",
            "49/48 - 20s - loss: 0.7124 - accuracy: 0.7507 - val_loss: 0.7441 - val_accuracy: 0.7539\n",
            "Epoch 71/180\n",
            "49/48 - 19s - loss: 0.7262 - accuracy: 0.7465 - val_loss: 0.7930 - val_accuracy: 0.7411\n",
            "Epoch 72/180\n",
            "49/48 - 20s - loss: 0.7192 - accuracy: 0.7490 - val_loss: 0.8438 - val_accuracy: 0.7288\n",
            "Epoch 73/180\n",
            "49/48 - 20s - loss: 0.7127 - accuracy: 0.7496 - val_loss: 0.7809 - val_accuracy: 0.7478\n",
            "Epoch 74/180\n",
            "49/48 - 19s - loss: 0.7153 - accuracy: 0.7494 - val_loss: 0.8209 - val_accuracy: 0.7295\n",
            "Epoch 75/180\n",
            "49/48 - 20s - loss: 0.7060 - accuracy: 0.7521 - val_loss: 0.7573 - val_accuracy: 0.7517\n",
            "Epoch 76/180\n",
            "49/48 - 20s - loss: 0.7108 - accuracy: 0.7516 - val_loss: 0.7076 - val_accuracy: 0.7620\n",
            "Epoch 77/180\n",
            "49/48 - 20s - loss: 0.7056 - accuracy: 0.7532 - val_loss: 0.8079 - val_accuracy: 0.7341\n",
            "Epoch 78/180\n",
            "49/48 - 20s - loss: 0.7046 - accuracy: 0.7519 - val_loss: 0.7811 - val_accuracy: 0.7383\n",
            "Epoch 79/180\n",
            "49/48 - 20s - loss: 0.7007 - accuracy: 0.7545 - val_loss: 0.7589 - val_accuracy: 0.7475\n",
            "Epoch 80/180\n",
            "49/48 - 20s - loss: 0.7026 - accuracy: 0.7525 - val_loss: 0.8074 - val_accuracy: 0.7320\n",
            "Epoch 81/180\n",
            "49/48 - 20s - loss: 0.7010 - accuracy: 0.7536 - val_loss: 0.7450 - val_accuracy: 0.7516\n",
            "Epoch 82/180\n",
            "49/48 - 20s - loss: 0.7010 - accuracy: 0.7543 - val_loss: 0.7750 - val_accuracy: 0.7469\n",
            "Epoch 83/180\n",
            "49/48 - 20s - loss: 0.6958 - accuracy: 0.7559 - val_loss: 0.7780 - val_accuracy: 0.7496\n",
            "Epoch 84/180\n",
            "49/48 - 20s - loss: 0.6933 - accuracy: 0.7562 - val_loss: 0.8064 - val_accuracy: 0.7445\n",
            "Epoch 85/180\n",
            "49/48 - 19s - loss: 0.7005 - accuracy: 0.7541 - val_loss: 0.8165 - val_accuracy: 0.7394\n",
            "Epoch 86/180\n",
            "49/48 - 20s - loss: 0.6888 - accuracy: 0.7590 - val_loss: 0.8163 - val_accuracy: 0.7323\n",
            "Epoch 87/180\n",
            "49/48 - 20s - loss: 0.6936 - accuracy: 0.7552 - val_loss: 0.7513 - val_accuracy: 0.7531\n",
            "Epoch 88/180\n",
            "49/48 - 19s - loss: 0.6894 - accuracy: 0.7580 - val_loss: 0.7690 - val_accuracy: 0.7516\n",
            "Epoch 89/180\n",
            "49/48 - 20s - loss: 0.6967 - accuracy: 0.7560 - val_loss: 0.7296 - val_accuracy: 0.7594\n",
            "Epoch 90/180\n",
            "49/48 - 20s - loss: 0.6844 - accuracy: 0.7607 - val_loss: 0.7860 - val_accuracy: 0.7462\n",
            "Epoch 91/180\n",
            "49/48 - 20s - loss: 0.6803 - accuracy: 0.7614 - val_loss: 0.7886 - val_accuracy: 0.7425\n",
            "Epoch 92/180\n",
            "49/48 - 20s - loss: 0.6899 - accuracy: 0.7575 - val_loss: 0.7152 - val_accuracy: 0.7607\n",
            "Epoch 93/180\n",
            "49/48 - 20s - loss: 0.6834 - accuracy: 0.7595 - val_loss: 0.8327 - val_accuracy: 0.7360\n",
            "Epoch 94/180\n",
            "49/48 - 20s - loss: 0.6863 - accuracy: 0.7602 - val_loss: 0.7393 - val_accuracy: 0.7519\n",
            "Epoch 95/180\n",
            "49/48 - 19s - loss: 0.6765 - accuracy: 0.7626 - val_loss: 0.8206 - val_accuracy: 0.7367\n",
            "Epoch 96/180\n",
            "49/48 - 20s - loss: 0.6817 - accuracy: 0.7599 - val_loss: 0.8071 - val_accuracy: 0.7458\n",
            "Epoch 97/180\n",
            "49/48 - 20s - loss: 0.6774 - accuracy: 0.7618 - val_loss: 0.8008 - val_accuracy: 0.7426\n",
            "Epoch 98/180\n",
            "49/48 - 20s - loss: 0.6765 - accuracy: 0.7624 - val_loss: 0.7641 - val_accuracy: 0.7497\n",
            "Epoch 99/180\n",
            "49/48 - 20s - loss: 0.6828 - accuracy: 0.7597 - val_loss: 0.7525 - val_accuracy: 0.7513\n",
            "Epoch 100/180\n",
            "49/48 - 20s - loss: 0.6757 - accuracy: 0.7630 - val_loss: 0.7884 - val_accuracy: 0.7452\n",
            "Epoch 101/180\n",
            "49/48 - 20s - loss: 0.6714 - accuracy: 0.7639 - val_loss: 0.7511 - val_accuracy: 0.7566\n",
            "Epoch 102/180\n",
            "49/48 - 20s - loss: 0.6684 - accuracy: 0.7657 - val_loss: 0.7741 - val_accuracy: 0.7517\n",
            "Epoch 103/180\n",
            "49/48 - 20s - loss: 0.6744 - accuracy: 0.7643 - val_loss: 0.7937 - val_accuracy: 0.7467\n",
            "Epoch 104/180\n",
            "49/48 - 19s - loss: 0.6692 - accuracy: 0.7635 - val_loss: 0.6946 - val_accuracy: 0.7703\n",
            "Epoch 105/180\n",
            "49/48 - 20s - loss: 0.6700 - accuracy: 0.7643 - val_loss: 0.7712 - val_accuracy: 0.7463\n",
            "Epoch 106/180\n",
            "49/48 - 20s - loss: 0.6640 - accuracy: 0.7661 - val_loss: 0.7462 - val_accuracy: 0.7532\n",
            "Epoch 107/180\n",
            "49/48 - 20s - loss: 0.6676 - accuracy: 0.7650 - val_loss: 0.7938 - val_accuracy: 0.7437\n",
            "Epoch 108/180\n",
            "49/48 - 20s - loss: 0.6658 - accuracy: 0.7663 - val_loss: 0.7035 - val_accuracy: 0.7693\n",
            "Epoch 109/180\n",
            "49/48 - 20s - loss: 0.6647 - accuracy: 0.7676 - val_loss: 0.7550 - val_accuracy: 0.7505\n",
            "Epoch 110/180\n",
            "49/48 - 20s - loss: 0.6606 - accuracy: 0.7679 - val_loss: 0.8053 - val_accuracy: 0.7441\n",
            "Epoch 111/180\n",
            "49/48 - 20s - loss: 0.6676 - accuracy: 0.7660 - val_loss: 0.7582 - val_accuracy: 0.7553\n",
            "Epoch 112/180\n",
            "49/48 - 20s - loss: 0.6591 - accuracy: 0.7677 - val_loss: 0.7226 - val_accuracy: 0.7618\n",
            "Epoch 113/180\n",
            "49/48 - 20s - loss: 0.6629 - accuracy: 0.7679 - val_loss: 0.7357 - val_accuracy: 0.7611\n",
            "Epoch 114/180\n",
            "49/48 - 20s - loss: 0.6557 - accuracy: 0.7706 - val_loss: 0.7579 - val_accuracy: 0.7529\n",
            "Epoch 115/180\n",
            "49/48 - 20s - loss: 0.6612 - accuracy: 0.7668 - val_loss: 0.7745 - val_accuracy: 0.7446\n",
            "Epoch 116/180\n",
            "49/48 - 19s - loss: 0.6574 - accuracy: 0.7686 - val_loss: 0.7879 - val_accuracy: 0.7445\n",
            "Epoch 117/180\n",
            "49/48 - 20s - loss: 0.6495 - accuracy: 0.7726 - val_loss: 0.7903 - val_accuracy: 0.7489\n",
            "Epoch 118/180\n",
            "49/48 - 20s - loss: 0.6547 - accuracy: 0.7698 - val_loss: 0.7664 - val_accuracy: 0.7511\n",
            "Epoch 119/180\n",
            "49/48 - 20s - loss: 0.6515 - accuracy: 0.7722 - val_loss: 0.7734 - val_accuracy: 0.7479\n",
            "Epoch 120/180\n",
            "49/48 - 19s - loss: 0.6545 - accuracy: 0.7699 - val_loss: 0.7778 - val_accuracy: 0.7453\n",
            "Epoch 121/180\n",
            "49/48 - 20s - loss: 0.6581 - accuracy: 0.7686 - val_loss: 0.7556 - val_accuracy: 0.7554\n",
            "Epoch 122/180\n",
            "49/48 - 20s - loss: 0.6520 - accuracy: 0.7712 - val_loss: 0.7316 - val_accuracy: 0.7620\n",
            "Epoch 123/180\n",
            "49/48 - 20s - loss: 0.6549 - accuracy: 0.7698 - val_loss: 0.6824 - val_accuracy: 0.7730\n",
            "Epoch 124/180\n",
            "49/48 - 20s - loss: 0.6503 - accuracy: 0.7719 - val_loss: 0.7734 - val_accuracy: 0.7472\n",
            "Epoch 125/180\n",
            "49/48 - 20s - loss: 0.6538 - accuracy: 0.7709 - val_loss: 0.7738 - val_accuracy: 0.7495\n",
            "Epoch 126/180\n",
            "49/48 - 20s - loss: 0.6468 - accuracy: 0.7732 - val_loss: 0.7043 - val_accuracy: 0.7633\n",
            "Epoch 127/180\n",
            "49/48 - 20s - loss: 0.6444 - accuracy: 0.7719 - val_loss: 0.7253 - val_accuracy: 0.7589\n",
            "Epoch 128/180\n",
            "49/48 - 20s - loss: 0.6501 - accuracy: 0.7732 - val_loss: 0.7668 - val_accuracy: 0.7516\n",
            "Epoch 129/180\n",
            "49/48 - 20s - loss: 0.6490 - accuracy: 0.7709 - val_loss: 0.7336 - val_accuracy: 0.7597\n",
            "Epoch 130/180\n",
            "49/48 - 20s - loss: 0.6463 - accuracy: 0.7734 - val_loss: 0.7327 - val_accuracy: 0.7619\n",
            "Epoch 131/180\n",
            "49/48 - 19s - loss: 0.6474 - accuracy: 0.7734 - val_loss: 0.7835 - val_accuracy: 0.7480\n",
            "Epoch 132/180\n",
            "49/48 - 19s - loss: 0.6440 - accuracy: 0.7740 - val_loss: 0.7410 - val_accuracy: 0.7613\n",
            "Epoch 133/180\n",
            "49/48 - 20s - loss: 0.6452 - accuracy: 0.7734 - val_loss: 0.7581 - val_accuracy: 0.7574\n",
            "Epoch 134/180\n",
            "49/48 - 20s - loss: 0.6434 - accuracy: 0.7740 - val_loss: 0.7379 - val_accuracy: 0.7572\n",
            "Epoch 135/180\n",
            "49/48 - 20s - loss: 0.6414 - accuracy: 0.7746 - val_loss: 0.7041 - val_accuracy: 0.7645\n",
            "Epoch 136/180\n",
            "49/48 - 20s - loss: 0.6425 - accuracy: 0.7760 - val_loss: 0.7727 - val_accuracy: 0.7457\n",
            "Epoch 137/180\n",
            "49/48 - 20s - loss: 0.6405 - accuracy: 0.7746 - val_loss: 0.6789 - val_accuracy: 0.7764\n",
            "Epoch 138/180\n",
            "49/48 - 20s - loss: 0.6388 - accuracy: 0.7757 - val_loss: 0.7692 - val_accuracy: 0.7539\n",
            "Epoch 139/180\n",
            "49/48 - 20s - loss: 0.6363 - accuracy: 0.7775 - val_loss: 0.7850 - val_accuracy: 0.7508\n",
            "Epoch 140/180\n",
            "49/48 - 20s - loss: 0.6363 - accuracy: 0.7770 - val_loss: 0.7328 - val_accuracy: 0.7653\n",
            "Epoch 141/180\n",
            "49/48 - 20s - loss: 0.6361 - accuracy: 0.7766 - val_loss: 0.7557 - val_accuracy: 0.7600\n",
            "Epoch 142/180\n",
            "49/48 - 20s - loss: 0.6361 - accuracy: 0.7754 - val_loss: 0.7469 - val_accuracy: 0.7587\n",
            "Epoch 143/180\n",
            "49/48 - 21s - loss: 0.6399 - accuracy: 0.7761 - val_loss: 0.7167 - val_accuracy: 0.7640\n",
            "Epoch 144/180\n",
            "49/48 - 20s - loss: 0.6336 - accuracy: 0.7763 - val_loss: 0.7348 - val_accuracy: 0.7651\n",
            "Epoch 145/180\n",
            "49/48 - 21s - loss: 0.6349 - accuracy: 0.7779 - val_loss: 0.7305 - val_accuracy: 0.7650\n",
            "Epoch 146/180\n",
            "49/48 - 20s - loss: 0.6337 - accuracy: 0.7762 - val_loss: 0.7064 - val_accuracy: 0.7705\n",
            "Epoch 147/180\n",
            "49/48 - 21s - loss: 0.6288 - accuracy: 0.7781 - val_loss: 0.7431 - val_accuracy: 0.7605\n",
            "Epoch 148/180\n",
            "49/48 - 21s - loss: 0.6303 - accuracy: 0.7792 - val_loss: 0.7048 - val_accuracy: 0.7690\n",
            "Epoch 149/180\n",
            "49/48 - 21s - loss: 0.6294 - accuracy: 0.7793 - val_loss: 0.7515 - val_accuracy: 0.7564\n",
            "Epoch 150/180\n",
            "49/48 - 21s - loss: 0.6303 - accuracy: 0.7796 - val_loss: 0.7397 - val_accuracy: 0.7584\n",
            "Epoch 151/180\n",
            "49/48 - 21s - loss: 0.6357 - accuracy: 0.7781 - val_loss: 0.7642 - val_accuracy: 0.7532\n",
            "Epoch 152/180\n",
            "49/48 - 21s - loss: 0.6285 - accuracy: 0.7780 - val_loss: 0.7419 - val_accuracy: 0.7579\n",
            "Epoch 153/180\n",
            "49/48 - 21s - loss: 0.6322 - accuracy: 0.7776 - val_loss: 0.7726 - val_accuracy: 0.7522\n",
            "Epoch 154/180\n",
            "49/48 - 21s - loss: 0.6227 - accuracy: 0.7808 - val_loss: 0.7226 - val_accuracy: 0.7686\n",
            "Epoch 155/180\n",
            "49/48 - 21s - loss: 0.6261 - accuracy: 0.7813 - val_loss: 0.7212 - val_accuracy: 0.7643\n",
            "Epoch 156/180\n",
            "49/48 - 21s - loss: 0.6249 - accuracy: 0.7804 - val_loss: 0.7236 - val_accuracy: 0.7610\n",
            "Epoch 157/180\n",
            "49/48 - 21s - loss: 0.6244 - accuracy: 0.7804 - val_loss: 0.7401 - val_accuracy: 0.7531\n",
            "Epoch 158/180\n",
            "49/48 - 21s - loss: 0.6288 - accuracy: 0.7803 - val_loss: 0.7232 - val_accuracy: 0.7681\n",
            "Epoch 159/180\n",
            "49/48 - 21s - loss: 0.6299 - accuracy: 0.7783 - val_loss: 0.7245 - val_accuracy: 0.7640\n",
            "Epoch 160/180\n",
            "49/48 - 21s - loss: 0.6272 - accuracy: 0.7800 - val_loss: 0.7035 - val_accuracy: 0.7681\n",
            "Epoch 161/180\n",
            "49/48 - 22s - loss: 0.6171 - accuracy: 0.7829 - val_loss: 0.7229 - val_accuracy: 0.7708\n",
            "Epoch 162/180\n",
            "49/48 - 21s - loss: 0.6249 - accuracy: 0.7807 - val_loss: 0.7169 - val_accuracy: 0.7712\n",
            "Epoch 163/180\n",
            "49/48 - 22s - loss: 0.6199 - accuracy: 0.7833 - val_loss: 0.7547 - val_accuracy: 0.7508\n",
            "Epoch 164/180\n",
            "49/48 - 21s - loss: 0.6208 - accuracy: 0.7815 - val_loss: 0.7319 - val_accuracy: 0.7652\n",
            "Epoch 165/180\n",
            "49/48 - 21s - loss: 0.6170 - accuracy: 0.7847 - val_loss: 0.7190 - val_accuracy: 0.7686\n",
            "Epoch 166/180\n",
            "49/48 - 21s - loss: 0.6257 - accuracy: 0.7803 - val_loss: 0.7324 - val_accuracy: 0.7685\n",
            "Epoch 167/180\n",
            "49/48 - 21s - loss: 0.6217 - accuracy: 0.7803 - val_loss: 0.7518 - val_accuracy: 0.7572\n",
            "Epoch 168/180\n",
            "49/48 - 21s - loss: 0.6187 - accuracy: 0.7814 - val_loss: 0.7680 - val_accuracy: 0.7577\n",
            "Epoch 169/180\n",
            "49/48 - 21s - loss: 0.6132 - accuracy: 0.7851 - val_loss: 0.7543 - val_accuracy: 0.7550\n",
            "Epoch 170/180\n",
            "49/48 - 21s - loss: 0.6175 - accuracy: 0.7826 - val_loss: 0.7864 - val_accuracy: 0.7519\n",
            "Epoch 171/180\n",
            "49/48 - 22s - loss: 0.6179 - accuracy: 0.7823 - val_loss: 0.7107 - val_accuracy: 0.7677\n",
            "Epoch 172/180\n",
            "49/48 - 21s - loss: 0.6192 - accuracy: 0.7826 - val_loss: 0.7074 - val_accuracy: 0.7689\n",
            "Epoch 173/180\n",
            "49/48 - 21s - loss: 0.6127 - accuracy: 0.7842 - val_loss: 0.7913 - val_accuracy: 0.7459\n",
            "Epoch 174/180\n",
            "49/48 - 21s - loss: 0.6176 - accuracy: 0.7830 - val_loss: 0.7075 - val_accuracy: 0.7715\n",
            "Epoch 175/180\n",
            "49/48 - 22s - loss: 0.6136 - accuracy: 0.7839 - val_loss: 0.7380 - val_accuracy: 0.7653\n",
            "Epoch 176/180\n",
            "49/48 - 22s - loss: 0.6084 - accuracy: 0.7850 - val_loss: 0.6869 - val_accuracy: 0.7689\n",
            "Epoch 177/180\n",
            "49/48 - 22s - loss: 0.6137 - accuracy: 0.7840 - val_loss: 0.7292 - val_accuracy: 0.7615\n",
            "Epoch 178/180\n",
            "49/48 - 22s - loss: 0.6085 - accuracy: 0.7861 - val_loss: 0.7140 - val_accuracy: 0.7709\n",
            "Epoch 179/180\n",
            "49/48 - 21s - loss: 0.6079 - accuracy: 0.7866 - val_loss: 0.6738 - val_accuracy: 0.7809\n",
            "Epoch 180/180\n",
            "49/48 - 21s - loss: 0.6054 - accuracy: 0.7872 - val_loss: 0.7699 - val_accuracy: 0.7514\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6YV5HzGNCkkW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}