examples/notebooks/Run_your_first_transform_colab.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "922vLtOCvgJD"
      },
      "source": [
        "# Data Prep Kit - Hello World (Colab friendly)\n",
        "\n",
        "This notebook guides you through running your first data preparation transformation using the data-prep-kit. In this example, we will demonstrate a transformation that takes PDF files as input and extracts their content.\n",
        "\n",
        "**Notebook versions:**\n",
        "\n",
        "- Pure python (run locally): [Run_your_first_transform_python.ipynb](https://github.com/IBM/data-prep-kit/blob/dev/examples/notebooks/Run_your_first_transform_python.ipynb)\n",
        "- Ray version (run locally): [Run_your_first_transform_ray.ipynb](https://github.com/IBM/data-prep-kit/blob/dev/examples/notebooks/Run_your_first_transform_ray.ipynb)\n",
        "- Google Colab friendly notebook: this notebook  |  [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/IBM/data-prep-kit/blob/dev/examples/notebooks/Run_your_first_transform_colab.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qnaUEB57vgJF"
      },
      "source": [
        "## Step-1: Install data-prep-kit dependencies\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ENbZ2ZNsvqrh"
      },
      "outputs": [],
      "source": [
        "! pip install  --default-timeout=100  data-prep-toolkit-transforms[pdf2parquet]==0.2.2.dev1\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T8MnqTZ-wC2P"
      },
      "source": [
        "## Restart Runtime\n",
        "\n",
        "After installing dependencies, be sure <font color=\"red\">restart runtime</font>, so libraries will be loaded\n",
        "\n",
        "You do this by going to **`Runtime --> Restart Session`**\n",
        "\n",
        "Then continue from **Step-2** below"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IZOmU1sxvgJG"
      },
      "source": [
        "## Step-2: Get Data\n",
        "\n",
        "For this example, we will show PDF processing capabilities of DPK.  And we will download and use this PDF documents\n",
        "\n",
        "- [IBM Granite model](https://arxiv.org/abs/2405.04324)\n",
        "- [Attention is all you need](https://arxiv.org/pdf/1706.03762) - seminal paper on transformer/attention architecture\n",
        "\n",
        "The code below will download the PDF.  Feel free to try your own PDFs to test it out"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "H5yF6_rYvgJG"
      },
      "outputs": [],
      "source": [
        "INPUT_DIR = 'Input-Test-Data'\n",
        "OUTPUT_DIR = 'Output-Test-Data'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qfmO_eLnvgJH",
        "outputId": "2735f68b-8313-488e-ecef-0775ede27543"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "Input-Test-Data/Granite_code_models.pdf (1.27 MB) downloaded successfully.\n",
            "\n",
            "Input-Test-Data/attention_is_all_you_need.pdf (2.22 MB) downloaded successfully.\n"
          ]
        }
      ],
      "source": [
        "## This cell will download the input files\n",
        "\n",
        "import os\n",
        "import shutil\n",
        "import requests\n",
        "from humanfriendly import format_size\n",
        "\n",
        "def download_file(url, local_file, chunk_size=1024*1024):\n",
        "    # Check if the local file already exists\n",
        "    if os.path.exists(local_file):\n",
        "        file_size = format_size(os.path.getsize(local_file))\n",
        "        print(f\"Local file '{local_file}' ({file_size}) already exists. Skipping download.\")\n",
        "        return\n",
        "\n",
        "    # Create the directory if it doesn't exist\n",
        "    os.makedirs(os.path.dirname(local_file), exist_ok=True)\n",
        "\n",
        "    # Stream the file download\n",
        "    with requests.get(url, stream=True) as r:\n",
        "        r.raise_for_status()\n",
        "        with open(local_file, 'wb') as f:\n",
        "            for chunk in r.iter_content(chunk_size=chunk_size):\n",
        "                if chunk: # filter out keep-alive new chunks\n",
        "                    f.write(chunk)\n",
        "        print()\n",
        "        file_size = format_size(os.path.getsize(local_file))\n",
        "        print(f\"{local_file} ({file_size}) downloaded successfully.\")\n",
        "## --- end: download_file ------\n",
        "\n",
        "## setup input/output directories\n",
        "shutil.os.makedirs(INPUT_DIR, exist_ok=True)\n",
        "shutil.rmtree(OUTPUT_DIR, ignore_errors=True)\n",
        "shutil.os.makedirs(OUTPUT_DIR, exist_ok=True)\n",
        "\n",
        "## Download PDF files\n",
        "download_file (url = 'https://arxiv.org/pdf/2405.04324', local_file = os.path.join(INPUT_DIR, 'Granite_code_models.pdf' ))\n",
        "download_file (url = 'https://arxiv.org/pdf/1706.03762', local_file = os.path.join(INPUT_DIR, 'attention_is_all_you_need.pdf' ))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8J62IyS5vgJJ"
      },
      "source": [
        "## Step-3: Extract Text from PDF\n",
        "\n",
        "This code is designed to set up a data transformation process that extracts text from PDF.  We will save the output as parquet format."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "aRkeYDvuvgJJ"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "import ast\n",
        "\n",
        "# Utilities from the data-prep-kit's data-processing-lib library provide functions and classes for parameter management,\n",
        "from pdf2parquet_transform import (pdf2parquet_contents_type_cli_param, pdf2parquet_contents_types,)\n",
        "from data_processing.utils import GB, ParamsUtils\n",
        "\n",
        "\n",
        "ingest_config = {\n",
        "    pdf2parquet_contents_type_cli_param: pdf2parquet_contents_types.JSON,\n",
        "}\n",
        "\n",
        "#local_conf: A dictionary specifying the local input and output folders where the PDF files will be read from and the transformed data will be saved.\n",
        "local_conf = {\n",
        "    \"input_folder\": INPUT_DIR,\n",
        "    \"output_folder\": OUTPUT_DIR,\n",
        "}\n",
        "\n",
        "#params: A dictionary containing various runtime parameters for the transformation.\n",
        "#data_local_config: Configuration for local data access, such as input and output folders, converted into a format compatible with the transformation using ParamsUtils.convert_to_ast.\n",
        "#data_files_to_use: Specifies that only PDF files (['.pdf']) will be used as input data.\n",
        "\n",
        "params = {\n",
        "    \"data_local_config\": ParamsUtils.convert_to_ast(local_conf),\n",
        "    \"data_files_to_use\": ast.literal_eval(\"['.pdf']\"),\n",
        "}\n",
        "sys.argv = ParamsUtils.dict_to_req(d=(params | ingest_config))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7EFL1Q7kvgJJ"
      },
      "source": [
        "### 3 - Execute"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q3xy8QnpvgJJ"
      },
      "source": [
        "Now its time to run the transformation.\n",
        "\n",
        "You will notice, that the code will download models to execute the transformation.  These models will be used to process PDFs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 862,
          "referenced_widgets": [
            "4da9634138834af6b53830e77822bbee",
            "02ce2efff8214690a2d0cea6832492a4",
            "86df69d190354ddaa33adce96936a6df",
            "e93c215aa5304de39220f244ebc4b911",
            "86c48f1756cd4b0fa73125e39dcb77ba",
            "f3e425eac5734ff19306ae71f627e079",
            "691f09e257d54b499faaf90f72e44a3b",
            "30dc5f48be2d4b4da8716d2e0d23e372",
            "08be842c2e5e46589fb13deeb9c036bc",
            "ae837f1a69a74a3b863a9c19202da0e7",
            "c92ce42d31784202b1b3b210ae77aae7",
            "950921b442fc4f739fd186f051c76d81",
            "93342e0027934b22b8e5967fb59af428",
            "5dbede0d03044ce3b9a6f08f7a924c62",
            "4fff11b9bfad46e98b7c933f2994a725",
            "ae8e349ca5a741878cef03f248061f2e",
            "58207c5fd9594c988308dcd6e32775b5",
            "1f048b048640473bb0b215d7da9e1a92",
            "15263bd25c9741c3b34b073010642944",
            "91ba93a7c927481b9e5ee9793988e9de",
            "8010d3af49824ca5a6f7f9dd302494ba",
            "0e851e377db1415fbe9c62bc8bcc7d1a",
            "88a9c683544b416e8b58c54627320901",
            "71b4cb950be7449c9a63fc552fccf647",
            "f0a4d1f74e0947c780e64be7e2d33f97",
            "4b8d0c62fda9415ca77acd3b135d3fa0",
            "b7eb047535a245a2b1e46c68aa9c86d8",
            "1a02de7a971448fdb47d6b22c93476b3",
            "4d5ceb4c8b51480cb82caa36d573ca0b",
            "f5cde5c28a3e4788bbcf193dc64ebe57",
            "a1939affb9954aca86cdcd9430ab569b",
            "93232da807aa4bdfaa6df7af867ca183",
            "4912fdf719054575a6f5695fb106e74d",
            "1a1ce455b56c4c5abc61a67b04fa8983",
            "295ffb18518d4f55bed187eae1bb4a91",
            "7d1f4804da354359ba4ffa44db911ef2",
            "463d76d334c9438882ac033baa6a2a71",
            "98e88ee67f8940b0b7c1ab0f4e5ed7fe",
            "cac6ab1306d4430f91be43b5f4fc4585",
            "a4797fa4499941318dbbbdc06bd59912",
            "9d8ea99d01c940e9a10889d1639d74e8",
            "c3922d4ff33641f388ab101d7d4a981f",
            "68f49fdff2154a72b9875756c8469dba",
            "ac44b83e481e4a4482f323b6328e39ff",
            "76c2058596f04b1c9adbd456a5799f36",
            "8409eca6c9034f3f94fce36a2933e3f3",
            "d69525c97a264fea8cf50f1243aab2db",
            "fda7c65808ed440b917d7299b500b485",
            "2eeb6f5268484ef196f0ad26c77307cb",
            "dc5a5200d9f94dcf8c47c14590ff3952",
            "7e426116c9ac4a228ba1f9d94c6519bf",
            "136199f932974fc7a1b07d24dbf527bb",
            "4b2609707adc42e980fe7bc8171c0d1d",
            "1fc82f82701c4c918f39b85dab253723",
            "c3c45d72c4e74a1faef0632f2c768608",
            "e0ce578a163c40f684053b3d6f3db5ec",
            "6e04886fc4e6485a9941d67b4577bfda",
            "1196c066c5784a1bbcadc9a0202bfe02",
            "677a6dc64b124372bb44e6df8066aa38",
            "85ff85b8cc074a9caa85f9db2f669ac0",
            "3e0010c2a38243d2ae4d4f27518b503b",
            "091493736776456392e4493174f9fa0f",
            "ebe75f475b9c43aca827b1f5cf68e511",
            "30cacbb0c6674d4fa198381a502ab44e",
            "a87905e174e64986b59575a4b0a1e799",
            "c451b2d71a7d4291baabc7b59ed47b96",
            "9959454762d64b199ab209eef345e8e3",
            "657fccdce16d4e128d51b6cb06f91056",
            "266a3a063fe34782a6546beac0eecb62",
            "02192a51069f40a2a0f831ecc2809085",
            "70834764cd4242708965f339e9f1f08e",
            "e4efea17c0fe49409a9c970dbcba9a93",
            "0873f21424a3404dbcb995ec862a9f44",
            "2c22a39735d3452593db594323ed27dc",
            "402b3286fec044b7a8bd12c9e46e3286",
            "7c4a475bdb434c91a051d3408cda94d9",
            "bfe2e357e7384315a3614797e9d530ae",
            "8beac29537a4472ba369ccc91674dbf1",
            "5797c0a5b8824a3f84b72e4ce2e6de16",
            "b483da44bc634a0c9b7a6fbc9f8443db",
            "5dc4d60351e54257907737d5deaea73a",
            "904f7974a8d24a60a9b6fe65fd42f4a4",
            "a425f8aa6b9c4b8bb95cdc59f20734cf",
            "0756cd16dc96426ba54f20c75686e6e3",
            "ef3251cf7d884620a30719e0ae5b49a0",
            "0453e6e227e6466b8fb2f6f80be09b8a",
            "8916ed32df05413ebc9fa43076f03f77",
            "ee33612510194eee92a956bad872838c"
          ]
        },
        "id": "UIMqzaAtvgJK",
        "outputId": "680e8934-483d-4878-92f5-dc3d708c2b67"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "20:39:40 INFO - pdf2parquet parameters are : {'artifacts_path': None, 'contents_type': <pdf2parquet_contents_types.JSON: 'application/json'>, 'do_table_structure': True, 'do_ocr': False}\n",
            "INFO:pdf2parquet_transform:pdf2parquet parameters are : {'artifacts_path': None, 'contents_type': <pdf2parquet_contents_types.JSON: 'application/json'>, 'do_table_structure': True, 'do_ocr': False}\n",
            "20:39:40 INFO - pipeline id pipeline_id\n",
            "INFO:data_processing.runtime.execution_configuration:pipeline id pipeline_id\n",
            "20:39:40 INFO - code location None\n",
            "INFO:data_processing.runtime.execution_configuration:code location None\n",
            "20:39:40 INFO - data factory data_ is using local data access: input_folder - Input-Test-Data output_folder - Output-Test-Data\n",
            "INFO:data_processing.data_access.data_access_factory_basee9e219d6-eeb0-4e90-a0e6-12ea6b0bc0b7:data factory data_ is using local data access: input_folder - Input-Test-Data output_folder - Output-Test-Data\n",
            "20:39:40 INFO - data factory data_ max_files -1, n_sample -1\n",
            "INFO:data_processing.data_access.data_access_factory_basee9e219d6-eeb0-4e90-a0e6-12ea6b0bc0b7:data factory data_ max_files -1, n_sample -1\n",
            "20:39:40 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.pdf'], files to checkpoint ['.parquet']\n",
            "INFO:data_processing.data_access.data_access_factory_basee9e219d6-eeb0-4e90-a0e6-12ea6b0bc0b7:data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.pdf'], files to checkpoint ['.parquet']\n",
            "20:39:40 INFO - orchestrator pdf2parquet started at 2024-09-04 20:39:40\n",
            "INFO:data_processing.runtime.pure_python.transform_orchestrator:orchestrator pdf2parquet started at 2024-09-04 20:39:40\n",
            "20:39:40 INFO - Number of files is 2, source profile {'max_file_size': 2.112621307373047, 'min_file_size': 1.2146415710449219, 'total_file_size': 3.3272628784179688}\n",
            "INFO:data_processing.runtime.pure_python.transform_orchestrator:Number of files is 2, source profile {'max_file_size': 2.112621307373047, 'min_file_size': 1.2146415710449219, 'total_file_size': 3.3272628784179688}\n",
            "20:39:40 INFO - Initializing models\n",
            "INFO:pdf2parquet_transform:Initializing models\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "4da9634138834af6b53830e77822bbee",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Fetching 7 files:   0%|          | 0/7 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "950921b442fc4f739fd186f051c76d81",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              ".gitignore:   0%|          | 0.00/5.18k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "88a9c683544b416e8b58c54627320901",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "(…)del_artifacts/tableformer/tm_config.json:   0%|          | 0.00/7.09k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "1a1ce455b56c4c5abc61a67b04fa8983",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "README.md:   0%|          | 0.00/40.0 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "76c2058596f04b1c9adbd456a5799f36",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "otslp_all_fast.check:   0%|          | 0.00/146M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "e0ce578a163c40f684053b3d6f3db5ec",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "model.pt:   0%|          | 0.00/169M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "9959454762d64b199ab209eef345e8e3",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "config.json:   0%|          | 0.00/41.0 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "8beac29537a4472ba369ccc91674dbf1",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              ".gitattributes:   0%|          | 0.00/1.60k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "20:46:16 INFO - Completed 1 files (50.0%) in 5.423 min\n",
            "INFO:data_processing.runtime.pure_python.transform_orchestrator:Completed 1 files (50.0%) in 5.423 min\n",
            "20:47:47 INFO - Completed 2 files (100.0%) in 6.94 min\n",
            "INFO:data_processing.runtime.pure_python.transform_orchestrator:Completed 2 files (100.0%) in 6.94 min\n",
            "20:47:47 INFO - Done processing 2 files, waiting for flush() completion.\n",
            "INFO:data_processing.runtime.pure_python.transform_orchestrator:Done processing 2 files, waiting for flush() completion.\n",
            "20:47:47 INFO - done flushing in 0.0 sec\n",
            "INFO:data_processing.runtime.pure_python.transform_orchestrator:done flushing in 0.0 sec\n",
            "20:47:48 INFO - Completed execution in 8.119 min, execution result 0\n",
            "INFO:data_processing.runtime.pure_python.transform_launcher:Completed execution in 8.119 min, execution result 0\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "✅ Tranformation run completed successfully\n",
            "CPU times: user 10min 47s, sys: 10 s, total: 10min 57s\n",
            "Wall time: 8min 7s\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "\n",
        "from data_processing.runtime.pure_python import PythonTransformLauncher\n",
        "from pdf2parquet_transform_python import Pdf2ParquetPythonTransformConfiguration\n",
        "\n",
        "\n",
        "launcher = PythonTransformLauncher(Pdf2ParquetPythonTransformConfiguration())\n",
        "return_code = launcher.launch()\n",
        "\n",
        "if return_code == 0:\n",
        "    print (f\"✅ Tranformation run completed successfully\")\n",
        "else:\n",
        "    raise Exception (\"❌ Transformation run failed\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WCUMN8_ivgJK"
      },
      "source": [
        "## Step-4: Inspect the generated output\n",
        "\n",
        "We will use pandas to read parquet files and display.\n",
        "\n",
        "You should see one-entry per PDF input file"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "Rdo3YCokvgJL"
      },
      "outputs": [],
      "source": [
        "import glob\n",
        "import pandas as pd\n",
        "\n",
        "## Reads parquet files in a folder into a pandas dataframe\n",
        "def read_parquet_files_as_df (parquet_dir):\n",
        "    parquet_files = glob.glob(f'{parquet_dir}/*.parquet')\n",
        "\n",
        "    # read each parquet file into a DataFrame and store in a list\n",
        "    dfs = [pd.read_parquet (f) for f in parquet_files]\n",
        "\n",
        "    # Concatenate all DataFrames into a single DataFrame\n",
        "    data_df = pd.concat(dfs, ignore_index=True)\n",
        "    return data_df"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 236
        },
        "id": "Fvq-ggkdvgJL",
        "outputId": "472edfd8-f57a-4b58-d9d4-4961aa339711"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"output_df\",\n  \"rows\": 2,\n  \"fields\": [\n    {\n      \"column\": \"filename\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"Granite_code_models.pdf\",\n          \"attention_is_all_you_need.pdf\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"contents\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"{\\\"_name\\\":\\\"\\\",\\\"type\\\":\\\"pdf-document\\\",\\\"description\\\":{\\\"logs\\\":[]},\\\"file-info\\\":{\\\"filename\\\":\\\"Granite_code_models.pdf\\\",\\\"document-hash\\\":\\\"cd6f480d8c03baf8fc4fc5a874dfa726764028fd6fb032532b6807522da6df48\\\",\\\"#-pages\\\":28,\\\"page-hashes\\\":[{\\\"hash\\\":\\\"2683abedc4e9b0a7ecffbba4e696db4cdeeaaa770de105f14b46cf7d0fef935a\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":1},{\\\"hash\\\":\\\"6b67e3f7d141634b469b81edae327289f954805104902cb51eced075585ecdf5\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":2},{\\\"hash\\\":\\\"d6c88ae80a41c48e99bacf83121051d01010c0edd5489fbc1c34b1f11f7d45c8\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":3},{\\\"hash\\\":\\\"a342fd3b69659664f36d83fba38be16f7a78bd8fa31cc3216c884a856c42c842\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":4},{\\\"hash\\\":\\\"bd8ab5e38ea21b04545dd2c9a73cd82d17e6e732f497125aca4e9d8468891e7d\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":5},{\\\"hash\\\":\\\"b24778e5dd56e510671b10b05e099b3ce72178a6a49cc6241dcf52ac1aaea27a\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":6},{\\\"hash\\\":\\\"3a279bd1e60253b7dfc02e5644d353aa9f98fd0a94b45d2dbabc616f526fa494\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":7},{\\\"hash\\\":\\\"3f333f5b7d034c7933e125a454eb419f6852ed085ba3fc5b2d43bfadb590cc33\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":8},{\\\"hash\\\":\\\"4baa552c46312b9718e4373d7ddd4de85b018bb5cd07aca51ab6034cadf5103c\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":9},{\\\"hash\\\":\\\"7fd102f1856ae7c94c9bdd41bc1d39092cde1161eb6ddeaf5f2583be611af981\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":10},{\\\"hash\\\":\\\"802a0f5ad7a141042cf8cb9f9ae9d0b5b596eba4079f86caef4ea31e51dfb8b3\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":11},{\\\"hash\\\":\\\"d53f72a373976815a57835da8d1c39d186599722fe6ac8e9ecca9433ecbea04f\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":12},{\\\"hash\\\":\\\"0ada0a41017360f82e16e77a6f3d9f7e1a9602514d964878aeaf0cf456afe3af\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":13},{\\\"hash\\\":\\\"d2aaf56f2d6637e58ca064bce92b30a96602a328c858badfc0f95e432775a180\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":14},{\\\"hash\\\":\\\"541a95ae9bc0caf6ad61fed7646891662545bece1c42ee213d65ef726ccb3707\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":15},{\\\"hash\\\":\\\"eba5296396cb609e3cc72d8bc09bbf746e8b088fb85591735e478189f0a9a19a\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":16},{\\\"hash\\\":\\\"3600e2cdeed48ed0a89135ccdb1714b72f48b93bda28c6c0888af6b004c98742\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":17},{\\\"hash\\\":\\\"bb11b3c49258a01a75ac328260c5c0b3efc69971dbe4ce8deb28f34874310329\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":18},{\\\"hash\\\":\\\"2c0ca9f690b7db2254806fd04b82e5ef412605f0ac5a44294bac60f7ecfad2d3\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":19},{\\\"hash\\\":\\\"93b2188a6ce6f7d754843166b16c4d4d01d88dc2489806cf4a1b39c7cbb84ecf\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":20},{\\\"hash\\\":\\\"f97fa1c4920b87478a01946989a5364f289a1fbf17080fe4d2ab317eb6e15073\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":21},{\\\"hash\\\":\\\"eef131a14656958c06e00a8a6147fc2f40651571df483811f6d4cd43e94e8dd8\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":22},{\\\"hash\\\":\\\"cf255d8b67311ce85375764c9f5e2ba1f292c0dba1ba09e8b5e8542ad260ab56\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":23},{\\\"hash\\\":\\\"53c081b5d4cf6c254cf3c0bb045b9dcdcd590f22bf7f1c6dec875d8bd59fdf07\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":24},{\\\"hash\\\":\\\"110451f86609b1b9228fe4be038b59b770292aabdc38bd0232403716fcc57e91\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":25},{\\\"hash\\\":\\\"7e0aea21f911462e8d1672e18f26185b758aedcbbc087c8e183f46a4c0be8ab2\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":26},{\\\"hash\\\":\\\"6507aafa736989bcf287cbc354a68a7db864044f946aa72d9be3cfa673d8d380\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":27},{\\\"hash\\\":\\\"cd9cc61ba719c18ce4c24ccdbfd760ac16db36328163dc84ed662c2a61e5a3b6\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":28}]},\\\"main-text\\\":[{\\\"text\\\":\\\"arXiv:2405.04324v1 [cs.AI] 7 May 2024\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[17,238,36,576],\\\"page\\\":1,\\\"span\\\":[0,37]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,764],\\\"page\\\":1,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"Granite Code Models: A Family of Open Foundation Models for Code Intelligence\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[143,673,468,711],\\\"page\\\":1,\\\"span\\\":[0,77]}]},{\\\"text\\\":\\\"Mayank Mishra \\\\u22c6 Matt Stallone \\\\u22c6 Gaoyuan Zhang \\\\u22c6 Yikang Shen Aditya Prasad Adriana Meza Soria Michele Merler Parameswaran Selvam Saptha Surendran Shivdeep Singh Manish Sethi Xuan-Hong Dang Pengyuan Li Kun-Lung Wu Syed Zawad Andrew Coleman Matthew White Mark Lewis Raju Pavuluri Yan Koyfman Boris Lublinsky Maximilien de Bayser Ibrahim Abdelaziz Kinjal Basu Mayank Agarwal Yi Zhou Chris Johnson Aanchal Goyal Hima Patel Yousaf Shah Petros Zerfos Heiko Ludwig Asim Munawar Maxwell Crouse Pavan Kapanipathi Shweta Salaria Bob Calio Sophia Wen Seetharami Seelam Brian Belgodere Carlos Fonseca Amith Singhee Nirmit Desai David D. Cox Ruchir Puri \\\\u2020 Rameswar Panda \\\\u2020\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"meta-data\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,538,504,647],\\\"page\\\":1,\\\"span\\\":[0,658]}]},{\\\"text\\\":\\\"IBM Research \\\\u22c6 Equal Contribution \\\\u2020 Corresponding Authors ruchir@us.ibm.com, rpanda@ibm.com\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"meta-data\\\",\\\"prov\\\":[{\\\"bbox\\\":[221,484,391,529],\\\"page\\\":1,\\\"span\\\":[0,91]}]},{\\\"text\\\":\\\"Abstract\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[283,465,329,475],\\\"page\\\":1,\\\"span\\\":[0,8]}]},{\\\"text\\\":\\\"Large Language Models (LLMs) trained on code are revolutionizing the software development process. Increasingly, code LLMs are being integrated into software development environments to improve the productivity of human programmers, and LLM-based agents are beginning to show promise for handling complex tasks autonomously. Realizing the full potential of code LLMs requires a wide range of capabilities, including code generation, fixing bugs, explaining and documenting code, maintaining repositories, and more. In this work, we introduce the Granite series of decoder-only code models for code generative tasks, trained with code written in 116 programming languages. The Granite Code models family consists of models ranging in size from 3 to 34 billion parameters, suitable for applications ranging from complex application modernization tasks to on-device memory-constrained use cases. Evaluation on a comprehensive set of tasks demonstrates that Granite Code models consistently reaches state-of-the-art performance among available open-source code LLMs. The Granite Code model family was optimized for enterprise software development workflows and performs well across a range of coding tasks (e.g. code generation, fixing and explanation), making it a versatile \\\\\\\"all around\\\\\\\" code model. We release all our Granite Code models under an Apache 2.0 license for both research and commercial use.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[143,233,470,450],\\\"page\\\":1,\\\"span\\\":[0,1401]}]},{\\\"text\\\":\\\"\\\\u0089 https:\\\\/\\\\/github.com\\\\/ibm-granite\\\\/granite-code-models\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[177,211,434,222],\\\"page\\\":1,\\\"span\\\":[0,52]}]},{\\\"text\\\":\\\"1 Introduction\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,179,195,188],\\\"page\\\":1,\\\"span\\\":[0,14]}]},{\\\"text\\\":\\\"Over the last several decades, software has been woven into the fabric of every aspect of our society. As demand for software development surges, it is more critical than ever to increase software development productivity, and LLMs provide promising path for augmenting human programmers. Prominent enterprise use cases for LLMs in software development productivity include code generation, code explanation, code fixing, unit test and documentation generation, application modernization, vulnerability detection, code translation, and more.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,88,504,164],\\\"page\\\":1,\\\"span\\\":[0,541]}]},{\\\"text\\\":\\\"Recent years have seen rapid progress in LLM's ability to generate and manipulate code, and a range of models with impressive coding abilities are available today. Models range in\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,59,505,81],\\\"page\\\":1,\\\"span\\\":[0,179]}]},{\\\"text\\\":\\\"1\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[303,32,308,39],\\\"page\\\":1,\\\"span\\\":[0,1]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,764],\\\"page\\\":2,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"Figure 1: Comparison of Granite-8B-Code (Base\\\\/Instruct) with other open source (code) LLMs of similar size on HumanEvalPack (Muennighoff et al. , 2023), spanning 3 coding tasks and 6 programming languages. See Tables 3 , 10 , 11 for more details. Best viewed in color.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,358,505,391],\\\"page\\\":2,\\\"span\\\":[0,268]}]},{\\\"name\\\":\\\"Picture\\\",\\\"type\\\":\\\"figure\\\",\\\"$ref\\\":\\\"#\\\\/figures\\\\/0\\\"},{\\\"text\\\":\\\"size from single-digit billions of parameters (e.g. Llama-7B (Touvron et al. , 2023), Gemma7B (Gemma-Team et al. , 2024), etc.) to hundreds of billions: DBRX (Databricks), Arctic (Snowflake), Grok, Mixtral 8x22B (MistralAI), Command R+ (Cohere), and vary in the generality of intended use, with some models aiming to cover a range of uses outside of code, while others focus primarily on coding-related tasks (e.g. StarCoder (Li et al. , 2023a; Lozhkov et al. , 2024), CodeGen (Nijkamp et al. , 2023), CodeLlama (Roziere et al. ' ' , 2023), and CodeGemma (CodeGemma Team et al. , 2024)).\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,253,505,329],\\\"page\\\":2,\\\"span\\\":[0,587]}]},{\\\"text\\\":\\\"However, there remain important gaps in the current field of LLMs for code, especially in the context of enterprise software development. First, while very large, generalist LLMs can achieve excellent coding performance, their size makes them expensive to deploy. Smaller code-focused models (Li et al. , 2023a; Lozhkov et al. , 2024; Nijkamp et al. , 2023; Roziere et al. ' ' , 2023; CodeGemma Team et al. , 2024) can achieve excellent code generation performance in a smaller and more flexible package, but performance in coding tasks beyond generation (e.g. fixing and explanation) can lag behind code generation performance.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,170,505,247],\\\"page\\\":2,\\\"span\\\":[0,628]}]},{\\\"text\\\":\\\"In many enterprise contexts, code LLM adoption can be further complicated by factors beyond the performance of the models. For instance, even open models are sometimes plagued by a lack of transparency about the data sources and data processing methods that went into model, which can erode trust in models in mission critical and regulated contexts. Furthermore, license terms in today's open LLMs can encumber and complicate an enterprise's ability to use a model.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,98,504,164],\\\"page\\\":2,\\\"span\\\":[0,466]}]},{\\\"text\\\":\\\"Here, we present Granite Code models, a series of highly capable code LLMs, designed to support enterprise software development across a wide range of coding tasks. Granite Code models has two main variants that we release in four different sizes (3B, 8B, 20B, and 34B):\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,60,505,92],\\\"page\\\":2,\\\"span\\\":[0,270]}]},{\\\"text\\\":\\\"2\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[303,32,308,40],\\\"page\\\":2,\\\"span\\\":[0,1]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,764],\\\"page\\\":3,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"\\\\u00b7 Granite Code Base: base foundation models for code-related tasks;\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[133,698,439,708],\\\"page\\\":3,\\\"span\\\":[0,67]}]},{\\\"text\\\":\\\"\\\\u00b7 Granite Code Instruct: instruction following models finetuned using a combination of Git commits paired with human instructions and open-source synthetically generated code instruction datasets.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"List-item\\\",\\\"prov\\\":[{\\\"bbox\\\":[133,657,504,690],\\\"page\\\":3,\\\"span\\\":[0,196]}]},{\\\"text\\\":\\\"The base models in the series have been trained from scratch with a two-phase training strategy. In phase 1, our model is trained on 3 to 4 trillion tokens sourced from 116 programming languages, ensuring a comprehensive understanding of programming languages and syntax. In phase 2, our model is further trained on 500 billion tokens with a carefully designed mixture of high-quality data from code and natural language domains to improve the model's ability to reason. We use the unsupervised language modeling objective to train the base models in both the phases of training. The instruct models are derived by further finetuning the above trained base models on a combination of a filtered variant of CommitPack (Muennighoff et al. , 2023), natural language instruction following datasets (OASST (Kopf et al. \\\\u00a8 \\\\u00a8 , 2023), HelpSteer (Wang et al. , 2023)) and open-source math datasets (MathInstruct (Yue et al. , 2023) and MetaMathQA (Yu et al. , 2023)), including synthetically generated code datasets for improving instruction following and reasoning capabilities.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,514,505,645],\\\"page\\\":3,\\\"span\\\":[0,1070]}]},{\\\"text\\\":\\\"We conduct extensive evaluations of our code LLMs on a comprehensive set of benchmarks, including HumanEvalPack (Muennighoff et al. , 2023), MBPP(+) (Austin et al. , 2021; Liu et al. , 2023a), RepoBench (Liu et al. , 2023b), ReCode (Wang et al. , 2022), and more. This set of benchmarks encompasses many different kinds of coding tasks beyond just code synthesis in Python, e.g., code fixing, code explanation, code editing, code translation, etc., across most major programming languages (Python, JavaScript, Java, Go, C++, Rust, etc.).\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,442,505,508],\\\"page\\\":3,\\\"span\\\":[0,537]}]},{\\\"text\\\":\\\"Our findings reveal that among open-source models, the Granite Code models overall show very strong performance across all model sizes and benchmarks (often outperforming other open-source code models that are twice large compared to Granite). As an illustration, figure 1 (top) shows a comparison of Granite-8B-Code-Base with other open-source base code LLMs, including recent high-performing general purpose base LLMs like Mistral (Jiang et al. , 2023b) and LLama-3 (AI@Meta , 2024) on HumanEvalPack (Muennighoff et al. , 2023). While CodeGemma and StarCoder2 perform reasonably well in generating code, they perform significantly worse on the code fixing and explanation variants of HumanEvalPack. On average, Granite-8B-Code-Base outperforms the most competitive CodeGemma-8B model by almost 12 points on HumanEvalPack (33.2% vs 21.3%), despite being trained on significantly less number of tokens (4.5T vs 7.5T tokens). Besides base models, the instruction tuned variants of our Granite Code models also show strong performance on HumanEvalPack, outperforming other open-source (code) instruction models, demonstrating benefits to a wider set of coding tasks with natural language instructions (see figure 1 (bottom)).\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,283,505,436],\\\"page\\\":3,\\\"span\\\":[0,1223]}]},{\\\"text\\\":\\\"Furthermore, since reasoning is critical for solving complicated questions and tasks, we also test our Granite-8B-Code-Base model on six mathematical benchmarks, including MATH (Cobbe et al. , 2021), GSM8K (Cobbe et al. , 2021) and problem solving with access to computational tools, where our Granite 8B model achieves better performance compared to most state-of-the-art 7B or 8B LLMs. For example, Granite-8B-Code-Base outperforms Llama-3-8B-Base by \\\\u223c12 points on GSM8K and \\\\u223c6 points on MATH (see table 15).\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,211,504,276],\\\"page\\\":3,\\\"span\\\":[0,510]}]},{\\\"text\\\":\\\"The key advantages of Granite Code models include:\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,194,341,205],\\\"page\\\":3,\\\"span\\\":[0,50]}]},{\\\"text\\\":\\\"\\\\u00b7 All-rounder Code LLM: Granite Code models achieve competitive or state-of-theart performance on different kinds of code-related tasks, including code generation, explanation, fixing, editing, translation, etc., demonstrating their ability to solve diverse coding tasks;\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[133,138,506,183],\\\"page\\\":3,\\\"span\\\":[0,271]}]},{\\\"text\\\":\\\"\\\\u00b7 Trustworthy Enterprise-Grade LLM: All our models are trained on licensepermissible data collected following IBM's AI Ethics principles 1 and guided by IBM's Corporate Legal team for trustworthy enterprise usage. All the Granite Code models are released under the Apache 2.0 license.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[133,86,505,131],\\\"page\\\":3,\\\"span\\\":[0,284]}]},{\\\"text\\\":\\\"1 https:\\\\/\\\\/www.ibm.com\\\\/impact\\\\/ai-ethics\\\",\\\"type\\\":\\\"footnote\\\",\\\"name\\\":\\\"Footnote\\\",\\\"prov\\\":[{\\\"bbox\\\":[120,59,287,70],\\\"page\\\":3,\\\"span\\\":[0,38]}]},{\\\"text\\\":\\\"3\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[303,31,308,40],\\\"page\\\":3,\\\"span\\\":[0,1]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,764],\\\"page\\\":4,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"We describe our entire data collection, filtering, and preprocessing pipeline in section 2. Section 3 describes the details of model architecture, followed by training details in Section 4. Section 5 provides the details about instruction tuning, and Section 6 describes the experiments and results comparing Granite Code models with other open-source LLMs.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,664,505,708],\\\"page\\\":4,\\\"span\\\":[0,357]}]},{\\\"text\\\":\\\"2 Data Collection\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,638,211,648],\\\"page\\\":4,\\\"span\\\":[0,17]}]},{\\\"text\\\":\\\"In this section, we describe the process of crawling and filtering (Sec. 2.1), deduplication (Sec. 2.2), HAP\\\\/PII filtering (Sec. 2.3) used to prepare the code data for model training. We also provide an overview of high-quality natural language data used to enhance the model's language understanding and mathematical reasoning skills.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,582,504,626],\\\"page\\\":4,\\\"span\\\":[0,335]}]},{\\\"text\\\":\\\"2.1 Data Crawling and Filtering\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,560,259,571],\\\"page\\\":4,\\\"span\\\":[0,31]}]},{\\\"text\\\":\\\"The pretraining code data was sourced from a combination of publicly available datasets like Github Code Clean 2 , StarCoderdata 3 , and additional public code repositories and issues from GitHub. We filter raw data to retain a list of 116 programming languages out of 300+ languages, as listed in Appendix A. The assignment of data to programming languages is performed based solely on file extension, similar to StarCoder (Li et al. , 2023a). After language filtering, we apply four key filtering rules to filter out lower-quality code (Li et al. , 2023a): (1) remove files with fewer than 25% alphabetic characters, (2) except for the XSLT language, filter out files where the string \\\\\\\"<?xml version=\\\\\\\" appears within the first 100 characters, (3) for HTML files, only keep files where the visible text makes up at least 20% of the HTML code and has a minimum length of 100 characters, (4) for JSON and YAML files, only keep files that have a character count ranging from 50 to 5000 characters. We also filter GitHub issues using a set of quality metrics that include removing auto-generated text, filtering out non-English issues, excluding comments from bots, and using the number of users engaged in the conversation as an indicator of quality. We also annotate each code file with license information associated with the respective repository, found via Github APIs and only keep files with permissive licenses for model training.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,374,505,551],\\\"page\\\":4,\\\"span\\\":[0,1435]}]},{\\\"text\\\":\\\"2.2 Exact and Fuzzy Deduplication\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,352,272,362],\\\"page\\\":4,\\\"span\\\":[0,33]}]},{\\\"text\\\":\\\"We adopt an aggressive deduplication strategy including both exact and fuzzy deduplication to remove documents having (near) identical code content in our training set. For exact deduplication, we first compute SHA256 hash on the document content and remove records having identical hashes. Post exact deduplication, we apply fuzzy deduplication with the goal of removing code files that may have slight variations and thereby unbiasing the data further. We apply a two-step method for this: (1) compute MinHashes of all the documents and then utilize Locally Sensitive Hashing (LSH) to group documents based on their MinHash fingerprints, (2) measure Jaccard similarity between each pair of documents in the same bucket and annotate documents except one as duplicates based on a similarity threshold of 0.7. We apply this near-deduplication process to all programming languages including GitHub issues to enhance the richness and diversity of the training dataset.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,221,504,342],\\\"page\\\":4,\\\"span\\\":[0,965]}]},{\\\"text\\\":\\\"2.3 HAP, PII, Malware Filtering\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,199,257,210],\\\"page\\\":4,\\\"span\\\":[0,31]}]},{\\\"text\\\":\\\"To reduce the likelihood of generating hateful, abusive, or profane (HAP) language from the models, we make diligent efforts to filter HAP content from the training set. We first create a dictionary of HAP keywords and then annotate each code document with the number of occurrences of such keywords in the content including comments. We filter out documents which exceeds the HAP threshold, computed based on a distributional analysis as well as manual inspection of code files. Moreover, to protect privacy, we follow StarCoder (Li et al. , 2023a) and make diligent efforts to redact Personally Identifiable Information (PII) from the training set. Specifically, we leverage the StarPII 4 model to detect IP addresses,\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,100,505,189],\\\"page\\\":4,\\\"span\\\":[0,720]}]},{\\\"text\\\":\\\"2 https:\\\\/\\\\/huggingface.co\\\\/datasets\\\\/codeparrot\\\\/github-code-clean\\\",\\\"type\\\":\\\"footnote\\\",\\\"name\\\":\\\"Footnote\\\",\\\"prov\\\":[{\\\"bbox\\\":[119,82,381,92],\\\"page\\\":4,\\\"span\\\":[0,62]}]},{\\\"text\\\":\\\"3 https:\\\\/\\\\/huggingface.co\\\\/datasets\\\\/bigcode\\\\/starcoderdata\\\",\\\"type\\\":\\\"footnote\\\",\\\"name\\\":\\\"Footnote\\\",\\\"prov\\\":[{\\\"bbox\\\":[119,71,352,81],\\\"page\\\":4,\\\"span\\\":[0,55]}]},{\\\"text\\\":\\\"4 https:\\\\/\\\\/huggingface.co\\\\/bigcode\\\\/starpii\\\",\\\"type\\\":\\\"footnote\\\",\\\"name\\\":\\\"Footnote\\\",\\\"prov\\\":[{\\\"bbox\\\":[119,60,284,70],\\\"page\\\":4,\\\"span\\\":[0,40]}]},{\\\"text\\\":\\\"4\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[303,32,308,39],\\\"page\\\":4,\\\"span\\\":[0,1]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,764],\\\"page\\\":5,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"keys, email addresses, names, user names, and passwords found in the content. The PII redaction step replaces the PII text with the corresponding tokens \\\\u27e8NAME\\\\u27e9 , \\\\u27e8EMAIL\\\\u27e9 , \\\\u27e8KEY\\\\u27e9 , \\\\u27e8PASSWORD\\\\u27e9 and change the IP address with a synthetically generated IP address, as in Li et al. (2023a). We also scan our datasets using ClamAV 5 to identify and remove instances of malware in the source code.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,653,505,708],\\\"page\\\":5,\\\"span\\\":[0,389]}]},{\\\"text\\\":\\\"2.4 Natural Language Datasets\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,629,253,639],\\\"page\\\":5,\\\"span\\\":[0,29]}]},{\\\"text\\\":\\\"In addition to collecting code data for model training, we curate several publicly available high-quality natural language datasets for improving the model's proficiency in language understanding and mathematical reasoning. Representative datasets under this category include web documents (Stackexchange, CommonCrawl), mathematical web text (OpenWebMath; Paster et al. (2023), StackMathQA; Zhang (2024)), academic text (Arxiv, Wikipedia), and instruction tuning datasets (FLAN; Longpre et al. (2023), HelpSteer (Wang et al. , 2023)). We do not deduplicate these already preprocessed natural language datasets.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,542,505,619],\\\"page\\\":5,\\\"span\\\":[0,610]}]},{\\\"text\\\":\\\"3 Model Architecture\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,520,232,530],\\\"page\\\":5,\\\"span\\\":[0,20]}]},{\\\"text\\\":\\\"We train a series of code models of varying sizes based on the transformer decoder architecture (Vaswani et al. , 2017). The model hyperparameters for these models are given in Table 1. For all model architectures, we use pre-normalization (Xiong et al. , 2020): normalization applied to the input of attention and MLP blocks.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,463,505,508],\\\"page\\\":5,\\\"span\\\":[0,326]}]},{\\\"text\\\":\\\"Table 1: Model configurations for Granite Code models.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[183,442,428,453],\\\"page\\\":5,\\\"span\\\":[0,54]}]},{\\\"name\\\":\\\"Table\\\",\\\"type\\\":\\\"table\\\",\\\"$ref\\\":\\\"#\\\\/tables\\\\/0\\\"},{\\\"text\\\":\\\"3B: The smallest model in the Granite-code model family is trained with RoPE embedding (Su et al. , 2023) and Multi-Head Attention (Vaswani et al. , 2017). This model use the swish activation function (Ramachandran et al. , 2017) with GLU (Shazeer , 2020) for the MLP, also commonly referred to as swiglu. For normalization, we use RMSNorm (Zhang & Sennrich , 2019) since it's computationally more efficient than LayerNorm (Ba et al. , 2016). The 3B model is trained with a context length of 2048 tokens.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,219,505,285],\\\"page\\\":5,\\\"span\\\":[0,504]}]},{\\\"text\\\":\\\"8B: The 8B model has a similar architecture as the 3B model with the exception of using Grouped-Query Attention (GQA) (Ainslie et al. , 2023). Using GQA offers a better tradeoff between model performance and inference efficiency at this scale. We train the 8B model with a context length of 4096 tokens.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,169,504,213],\\\"page\\\":5,\\\"span\\\":[0,303]}]},{\\\"text\\\":\\\"20B: The 20B code model is trained with learned absolute position embeddings. We use Multi-Query Attention (Shazeer , 2019) during training for efficient downstream inference. For the MLP block, we use the GELU activation function (Hendrycks & Gimpel , 2023). For normalizing the activations, we use LayerNorm (Ba et al. , 2016). This model is trained with a context length of 8192 tokens.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,108,505,163],\\\"page\\\":5,\\\"span\\\":[0,389]}]},{\\\"text\\\":\\\"34B: To train the 34B model, we follow the approach by Kim et al. for depth upscaling of the 20B model. Specifically, we first duplicate the 20B code model with 52 layers and then\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,80,504,102],\\\"page\\\":5,\\\"span\\\":[0,179]}]},{\\\"text\\\":\\\"5 https:\\\\/\\\\/www.clamav.net\\\\/\\\",\\\"type\\\":\\\"footnote\\\",\\\"name\\\":\\\"Footnote\\\",\\\"prov\\\":[{\\\"bbox\\\":[120,59,229,70],\\\"page\\\":5,\\\"span\\\":[0,25]}]},{\\\"text\\\":\\\"5\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[303,31,308,40],\\\"page\\\":5,\\\"span\\\":[0,1]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,764],\\\"page\\\":6,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"Figure 2: An overview of depth upscaling (Kim et al. , 2024) for efficient training of Granite34B-Code. We utilize the 20B model after 1.6T tokens to start training of 34B model with the same code pretraining data without any changes to the training and inference framework.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,536,505,569],\\\"page\\\":6,\\\"span\\\":[0,274]}]},{\\\"name\\\":\\\"Picture\\\",\\\"type\\\":\\\"figure\\\",\\\"$ref\\\":\\\"#\\\\/figures\\\\/1\\\"},{\\\"text\\\":\\\"remove final 8 layers from the original model and initial 8 layers from its duplicate to form two models. Finally, we concatenate both models to form Granite-34B-Code model with 88 layers (see Figure 2 for an illustration). After the depth upscaling, we observe that the drop in performance compared to 20B model is pretty small contrary to what is observed by Kim et al.. This performance is recovered pretty quickly after we continue pretraining of the upscaled 34B model. Similar, to 20B, we use a 8192 token context during pretraining.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,456,505,522],\\\"page\\\":6,\\\"span\\\":[0,539]}]},{\\\"text\\\":\\\"4 Pretraining\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,430,188,442],\\\"page\\\":6,\\\"span\\\":[0,13]}]},{\\\"text\\\":\\\"In this section, we provide details on two phase training (Sec. 4.1), training objectives (Sec. 4.2), optimization (Sec. 4.3) and infrastructure (Sec. 4.4) used in pretraining the models.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,396,505,417],\\\"page\\\":6,\\\"span\\\":[0,187]}]},{\\\"text\\\":\\\"4.1 Two Phase Training\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,374,220,384],\\\"page\\\":6,\\\"span\\\":[0,22]}]},{\\\"text\\\":\\\"Granite Code models are trained on 3.5T to 4.5T tokens of code data and natural language datasets related to code. Data is tokenized via byte pair encoding (BPE, (Sennrich et al. , 2015)), employing the same tokenizer as StarCoder (Li et al. , 2023a). Following (Shen et al. , 2024; Hu et al. , 2024), we utilize high-quality data with two phases of training as follows.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,320,505,364],\\\"page\\\":6,\\\"span\\\":[0,370]}]},{\\\"text\\\":\\\"\\\\u00b7 Phase 1 (code only training): During phase 1, both 3B and 8B models are trained for 4 trillion tokens of code data comprising 116 languages. The 20B parameter model is trained on 3 trillion tokens of code. The 34B model is trained on 1.4T tokens after the depth upscaling which is done on the 1.6T checkpoint of 20B model.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[133,267,504,311],\\\"page\\\":6,\\\"span\\\":[0,324]}]},{\\\"text\\\":\\\"\\\\u00b7 Phase 2 (code + language training): In phase 2, we include additional high-quality publicly available data from various domains, including technical, mathematics, and web documents, to further improve the model's performance in reasoning and problem solving skills, which are essential for code generation. We train all our models for 500B tokens (80% code and 20% language data) in phase 2 training.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[133,208,505,263],\\\"page\\\":6,\\\"span\\\":[0,402]}]},{\\\"text\\\":\\\"4.2 Training Objective\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,183,216,193],\\\"page\\\":6,\\\"span\\\":[0,22]}]},{\\\"text\\\":\\\"For training of all our models, we use the causal language modeling objective and Fill-Inthe-Middle (FIM) (Bavarian et al. , 2022) objective. The FIM objective is tasked to predict inserted tokens with the given context and subsequent text. We train our models to work with both PSM (Prefix-Suffix-Middle) and SPM (Suffix-Prefix-Middle) modes, with relevant formatting control tokens, same as StarCoder (Li et al. , 2023a).\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,118,505,173],\\\"page\\\":6,\\\"span\\\":[0,423]}]},{\\\"text\\\":\\\"The overall loss is computed as a weighted combination of the 2 objectives:\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,101,438,112],\\\"page\\\":6,\\\"span\\\":[0,75]}]},{\\\"text\\\":\\\"L = \\\\u03b1L CLM + (1 - \\\\u03b1)LF IM (1)\\\",\\\"type\\\":\\\"equation\\\",\\\"name\\\":\\\"Formula\\\",\\\"prov\\\":[{\\\"bbox\\\":[246,86,505,98],\\\"page\\\":6,\\\"span\\\":[0,29]}]},{\\\"text\\\":\\\"We emperically set \\\\u03b1 = 0.5 during training and find that this works well in practice leading to SOTA performance on both code completion and code infilling tasks. It should be\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,59,505,81],\\\"page\\\":6,\\\"span\\\":[0,175]}]},{\\\"text\\\":\\\"6\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[303,31,308,39],\\\"page\\\":6,\\\"span\\\":[0,1]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,764],\\\"page\\\":7,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"noted that the FIM objective is only used during pretraining, however we drop it during instruction finetuning i.e we set \\\\u03b1 = 1.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,686,504,708],\\\"page\\\":7,\\\"span\\\":[0,128]}]},{\\\"text\\\":\\\"4.3 Optimization\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,663,192,673],\\\"page\\\":7,\\\"span\\\":[0,16]}]},{\\\"text\\\":\\\"We use AdamW optimizer (Kingma & Ba , 2017) with \\\\u03b21 = 0.9, \\\\u03b22 = 0.95 and weight decay of 0.1 for training all our Granite code models. For the phase-1 pretraining, the learning rate follows a cosine schedule starting from 3 \\\\u00d7 10 - 4 which decays to 3 \\\\u00d7 10 - 5 with an initial linear warmup step of 2k iterations. For phase-2 pretraining, we start from 3 \\\\u00d7 10 - 4 (1.5 \\\\u00d7 10 - 4 for 20B and 34B models) and adopt an exponential decay schedule to anneal it to 10% of the initial learning rate. We use a batch size of 4M-5M tokens depending on the model size during both phases of pretraining.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,570,504,652],\\\"page\\\":7,\\\"span\\\":[0,589]}]},{\\\"text\\\":\\\"To accelerate training, we use FlashAttention 2 (Dao et al. , 2022; Dao , 2023), the persistent layernorm kernel, Fused RMSNorm kernel (depending on the model) and the Fused Adam kernel available in NVIDIA's Apex library. We use a custom fork of NVIDIA's MegatronLM (Shoeybi et al. , 2019; Narayanan et al. , 2021) for distributed training of all our models. We train with a mix of 3D parallelism: tensor parallel, pipeline parallel and data parallel. We also use sequence parallelism (Korthikanti et al. , 2023) for reducing the activation memory consumption of large context length during training. We use Megatron's distributed optimizer with mixed precision training (Micikevicius et al. , 2018) in BF16 (Kalamkar et al. , 2019) with gradient all-reduce and gradient accumulation in FP32 for training stability.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,466,505,564],\\\"page\\\":7,\\\"span\\\":[0,815]}]},{\\\"text\\\":\\\"4.4 Infrastructure\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,444,194,452],\\\"page\\\":7,\\\"span\\\":[0,18]}]},{\\\"text\\\":\\\"We train the Granite Code models using IBM's two supercomputing clusters, namely Vela and Blue Vela, outfitted with NVIDIA A100 and H100 GPUs, respectively. In the Vela A100 GPU cluster, each node has 2\\\\u00d7 Intel Xeon Scalable Processors with 8\\\\u00d7 80GB A100 GPUs connected to each other by NVLink and NVSwitch. The Vela cluster adopts RoCE (RDMA over Converged Ethernet) and GDR (GPU-direct RDMA) for high-performance networking. Similarly, each node in Blue Vela cluster consists of dual 48-core Intel processors with 8\\\\u00d7 80GB H100 GPUs. Blue Vela employs 3.2Tbps InfiniBand interconnect to facilitate seamless communication between nodes, known for their high throughput and low latency. In addition, Blue Vela employs a separate, dedicated InfiniBand Storage fabric providing 800Gbps per compute node, backed by multiple ESS6000 storage appliances. Both clusters provide a scalable and efficient infrastructure for training our models over thousands of GPUs. We estimate the carbon emissions from pretraining the Granite Code models to be \\\\u223c455 tCO 2 eq, which is computed based on the total energy usage in the models and US national average carbon intensity factor of 0.423 kg CO2eq\\\\/KWh without taking the location of data centers in consideration. The Blue Vela cluster runs on 100% renewable energy to minimize the environmental impact.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,256,505,431],\\\"page\\\":7,\\\"span\\\":[0,1336]}]},{\\\"text\\\":\\\"5 Instruction Tuning\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,228,228,240],\\\"page\\\":7,\\\"span\\\":[0,20]}]},{\\\"text\\\":\\\"Finetuning code LLMs on a variety of tasks explained via instructions has been shown to improve model usability and general performance. While there has been much progress in code instruction tuning, most of them adopt synthetically generated data from OpenAI models, which limits the model use in many enterprise applications. Thus, following OctoCoder (Muennighoff et al. , 2023), we use only a combination of permissively licensed data, with an aim to enhance instruction following capabilities of our models, including logical reasoning and problem-solving skills. Specifically, Granite Code Instruct models are trained on the following types of data.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,126,504,214],\\\"page\\\":7,\\\"span\\\":[0,655]}]},{\\\"text\\\":\\\"\\\\u00b7 Code Commits Dataset: CommitPackFT (Muennighoff et al. , 2023), a filtered version of full CommitPack dataset across 92 programming languages 6 ;\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"List-item\\\",\\\"prov\\\":[{\\\"bbox\\\":[133,91,506,115],\\\"page\\\":7,\\\"span\\\":[0,147]}]},{\\\"text\\\":\\\"6 We selected 92 programming languages that are common across the original CommitPackFT and our list of 116 languages used during pretraining.\\\",\\\"type\\\":\\\"footnote\\\",\\\"name\\\":\\\"Footnote\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,59,504,80],\\\"page\\\":7,\\\"span\\\":[0,142]}]},{\\\"text\\\":\\\"7\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[303,32,308,40],\\\"page\\\":7,\\\"span\\\":[0,1]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,764],\\\"page\\\":8,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"Table 2: Summary of evaluation tasks.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[222,700,389,710],\\\"page\\\":8,\\\"span\\\":[0,37]}]},{\\\"name\\\":\\\"Table\\\",\\\"type\\\":\\\"table\\\",\\\"$ref\\\":\\\"#\\\\/tables\\\\/1\\\"},{\\\"text\\\":\\\"\\\\u00b7 Math Datasets: MathInstruct 7 (Yue et al. , 2023) and MetaMathQA (Yu et al. , 2023);\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[133,426,505,437],\\\"page\\\":8,\\\"span\\\":[0,86]}]},{\\\"text\\\":\\\"\\\\u2022 Code Instruction Datasets: Glaive-Code-Assistant-v38, Self-OSS-Instruct-SC29 , Glaive-Function-Calling-v2 10 , NL2SQL 11 and few synthetically generated API calling datasets (Basu et al. , 2024);\\\",\\\"type\\\":\\\"footnote\\\",\\\"name\\\":\\\"Footnote\\\",\\\"prov\\\":[{\\\"bbox\\\":[133,388,506,422],\\\"page\\\":8,\\\"span\\\":[0,197]}]},{\\\"text\\\":\\\"\\\\u2022 Language Instruction Datasets: High-quality datasets like HelpSteer (Wang et al. , 2023), an open license-filtered version of Platypus 12 (Lee et al. , 2023) including a collection of hardcoded prompts to ensure model generates correct outputs given inquiries about its name or developers.\\\",\\\"type\\\":\\\"footnote\\\",\\\"name\\\":\\\"Footnote\\\",\\\"prov\\\":[{\\\"bbox\\\":[133,338,505,383],\\\"page\\\":8,\\\"span\\\":[0,291]}]},{\\\"text\\\":\\\"For training, we use a cosine scheduler with 250 warmup steps, an initial learning rate 10 - 5 , and train for three epochs. Further, we add random, uniform noise with a magnitude of \\\\u221a 5 Nh , where N is the sequence length and h is the embedding dimension, to the embedding vector, as proposed by Jain et al.. The additional noise improved overall answer quality of the instruction model. We use FlashAttention 2 (Dao , 2023; Dao et al. , 2022) with a Padding-Free Transformer 13 implementation to reduce GPU memory usage and redundant FLOPs during finetuning. We also use full activation checkpointing (Korthikanti et al. , 2023), which allows us to finetune our Granite-20B-Code models with 8K context length within a single node within a few hours on 8\\\\u00d7A100 GPUs.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,220,505,327],\\\"page\\\":8,\\\"span\\\":[0,766]}]},{\\\"text\\\":\\\"6 Evaluation\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,195,185,205],\\\"page\\\":8,\\\"span\\\":[0,12]}]},{\\\"text\\\":\\\"We evaluate Granite Code models on a wide variety of tasks, including code generation, code explanation, code fixing, code editing, math reasoning, etc., as shown in Table 2. We compare our models with several open-source code LLMs: StableCode (Pinnaparaju et al. , 2024),\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,146,505,179],\\\"page\\\":8,\\\"span\\\":[0,272]}]},{\\\"text\\\":\\\"7 We removed GSM8K-RFT and Camel-Math from MathInstruct due to unknown or NC license.\\\",\\\"type\\\":\\\"footnote\\\",\\\"name\\\":\\\"Footnote\\\",\\\"prov\\\":[{\\\"bbox\\\":[120,126,497,137],\\\"page\\\":8,\\\"span\\\":[0,85]}]},{\\\"text\\\":\\\"8 https:\\\\/\\\\/huggingface.co\\\\/datasets\\\\/glaiveai\\\\/glaive-code-assistant-v3\\\",\\\"type\\\":\\\"footnote\\\",\\\"name\\\":\\\"Footnote\\\",\\\"prov\\\":[{\\\"bbox\\\":[119,115,393,126],\\\"page\\\":8,\\\"span\\\":[0,67]}]},{\\\"text\\\":\\\"9 https:\\\\/\\\\/huggingface.co\\\\/datasets\\\\/bigcode\\\\/self-oss-instruct-sc2-exec-filter-50k\\\",\\\"type\\\":\\\"footnote\\\",\\\"name\\\":\\\"Footnote\\\",\\\"prov\\\":[{\\\"bbox\\\":[118,104,439,115],\\\"page\\\":8,\\\"span\\\":[0,79]}]},{\\\"text\\\":\\\"10 https:\\\\/\\\\/huggingface.co\\\\/datasets\\\\/glaiveai\\\\/glaive-function-calling-v2\\\",\\\"type\\\":\\\"footnote\\\",\\\"name\\\":\\\"Footnote\\\",\\\"prov\\\":[{\\\"bbox\\\":[116,93,399,104],\\\"page\\\":8,\\\"span\\\":[0,70]}]},{\\\"text\\\":\\\"11 https:\\\\/\\\\/huggingface.co\\\\/datasets\\\\/bugdaryan\\\\/sql-create-context-instruction\\\",\\\"type\\\":\\\"footnote\\\",\\\"name\\\":\\\"Footnote\\\",\\\"prov\\\":[{\\\"bbox\\\":[117,82,429,92],\\\"page\\\":8,\\\"span\\\":[0,75]}]},{\\\"text\\\":\\\"12 https:\\\\/\\\\/huggingface.co\\\\/datasets\\\\/garage-bAInd\\\\/Open-Platypus\\\",\\\"type\\\":\\\"footnote\\\",\\\"name\\\":\\\"Footnote\\\",\\\"prov\\\":[{\\\"bbox\\\":[117,71,382,81],\\\"page\\\":8,\\\"span\\\":[0,61]}]},{\\\"text\\\":\\\"13 https:\\\\/\\\\/huggingface.co\\\\/blog\\\\/mayank-mishra\\\\/padding-free-transformer\\\",\\\"type\\\":\\\"footnote\\\",\\\"name\\\":\\\"Footnote\\\",\\\"prov\\\":[{\\\"bbox\\\":[117,60,414,71],\\\"page\\\":8,\\\"span\\\":[0,69]}]},{\\\"text\\\":\\\"8\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[303,32,308,40],\\\"page\\\":8,\\\"span\\\":[0,1]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,764],\\\"page\\\":9,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"Code Llama (Roziere et al. , 2023), StarCoder (Li et al. , 2023b), StarCoder2 (Lozhkov et al. , 2024), and CodeGemma 14 , including recent high-performing general purpose open LLMs like Mistral (Jiang et al. , 2023a) and LLama-3 15 . For all the benchmarks, we evaluate the baseline models (including ours) using the same script and environment for fair comparison.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,662,505,708],\\\"page\\\":9,\\\"span\\\":[0,365]}]},{\\\"text\\\":\\\"6.1 Code Generation\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,637,208,646],\\\"page\\\":9,\\\"span\\\":[0,19]}]},{\\\"text\\\":\\\"6.1.1 HumanEvalSynthesize: Multilingual Code Generation in 6 Languages\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,614,457,624],\\\"page\\\":9,\\\"span\\\":[0,70]}]},{\\\"text\\\":\\\"While most of the prior code LLMs evaluate code generation capabilities only on Python using HumanEval (Chen et al. , 2021), we adopt the challenging HumanEvalSynthesize (Muennighoff et al. , 2023) benchmark in our study, which extends Python problems of Humaneval Benchmark to five additional commonly used programming languages, namely JavaScript, Java, Go, C++, Rust. We evaluate all models in a zero-shot manner using greedy decoding with completion format for the base models, and instruction template for the instructiontuned models. In constructing prompts for instruction-tuned models, we adhere to the formats provided in their official examples. We search for a suitable prompt format in the HuggingFace model card, GitHub repository, and formal publications or technical reports.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,506,506,605],\\\"page\\\":9,\\\"span\\\":[0,790]}]},{\\\"text\\\":\\\"Table 3 shows the results on of base and instruct models HumanEvalSynthesize benchmark. Granite-3B-Code-Base is the best performing small model with +3% improvement over CodeGemma-2B. Overall, among base models, Granite Code models achieve the best average performance at 7B-8B scale, 2nd best average performance in 13B-20B size models, and is very close to the best model (falls behind StarCoder2-15B by 0.1%). While CodeLlama34B achieves better score on HumanEval Python, Granite-34B-Code-Base achieves much better performance on other languages, leading to a 4% improvement on average across 6 languages. Among the instruct models, Granite Code models consistently outperform equivalent size CodeLlama; the 3B, 8B, and 20B models even outperform CodeLlama models that are two times larger. It's worth noting that even our smaller model, Granite-3B-CodeInstruct, surpasses the performance of CodeLlama-34B-Instruct. Further, we can also see that Granite Code models outperform much larger state-of-the-art open-source general-purpose language models, including Gemma, Mixtral, and Llama 3 series models. This shows that domain-specific code models could achieve better performance and efficiency, thus making them more suitable for cost- and performance-sensitive enterprise environments.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,336,505,500],\\\"page\\\":9,\\\"span\\\":[0,1291]}]},{\\\"text\\\":\\\"6.1.2 MultiPL-E: Multilingual Code Generation in 18 Languages\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,310,405,321],\\\"page\\\":9,\\\"span\\\":[0,61]}]},{\\\"text\\\":\\\"MultiPL-E (Cassano et al. , 2023) is a canonical benchmark for evaluating code models on a more diverse set of 18 different programming languages. On MultiPL-E, we compare all the base models on 18 languages, by sampling 50 completions per prompt at temperature 0.2 with top-p 0.95, as in (Lozhkov et al. , 2024). Table 4 shows the results of different models on MultiPL-E. As can be seen from the table, there is no single model that works best at every language across all model sizes. In comparison to the similarly sized open-source model CodeLlama-7B, Granite-8B-Code-Base performs the best on 16\\\\/18 programming languages. Of the medium models, StarCoder2-15B performs best. Among the large models, Granite-34B-Code-Base does better than CodeLlama-34B on most languages, demonstrating its effectiveness in code generation across a diverse set of languages.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,192,505,302],\\\"page\\\":9,\\\"span\\\":[0,861]}]},{\\\"text\\\":\\\"6.1.3 MBPP and MBPP+: Code Generation in Python\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,167,354,178],\\\"page\\\":9,\\\"span\\\":[0,47]}]},{\\\"text\\\":\\\"MBPP (Austin et al. , 2021) and MBPP+ (Liu et al. , 2023a) are two of the most widely studied benchmarks for evaluating code models. While the prompt for each MBPP problem includes a natural language description followed by a few tests, MBPP+ consists of 35\\\\u00d7 more tests than the original benchmarks. We use greedy decoding and report the mean pass@1 for all the models. Table 5 summarizes the results of different base models. As we can see, Granite3B-Code-Base significantly outperforms CodeGemma-2B but falls short of StarCoder2-3B on\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,92,505,158],\\\"page\\\":9,\\\"span\\\":[0,536]}]},{\\\"text\\\":\\\"14 https:\\\\/\\\\/storage.googleapis.com\\\\/deepmind-media\\\\/gemma\\\\/codegemma report.pdf\\\",\\\"type\\\":\\\"footnote\\\",\\\"name\\\":\\\"Footnote\\\",\\\"prov\\\":[{\\\"bbox\\\":[116,71,455,82],\\\"page\\\":9,\\\"span\\\":[0,75]}]},{\\\"text\\\":\\\"15 https:\\\\/\\\\/github.com\\\\/meta-llama\\\\/llama3\\\",\\\"type\\\":\\\"footnote\\\",\\\"name\\\":\\\"Footnote\\\",\\\"prov\\\":[{\\\"bbox\\\":[117,60,285,70],\\\"page\\\":9,\\\"span\\\":[0,39]}]},{\\\"text\\\":\\\"9\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[303,32,308,39],\\\"page\\\":9,\\\"span\\\":[0,1]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,764],\\\"page\\\":10,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"Table 3: Pass@1 performance on HumanEvalSynthesize benchmark (Muennighoff et al. , 2023). All models are evaluated using greedy decoding with completion format for the base models, and instruction template for the instruction-tuned models.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,678,505,711],\\\"page\\\":10,\\\"span\\\":[0,239]}]},{\\\"name\\\":\\\"Table\\\",\\\"type\\\":\\\"table\\\",\\\"$ref\\\":\\\"#\\\\/tables\\\\/2\\\"},{\\\"text\\\":\\\"\\\\u22c6 We could not produce reasonable results for Python generation despite many attempts using different prompts, generation parameters, precision settings.\\\",\\\"type\\\":\\\"footnote\\\",\\\"name\\\":\\\"Footnote\\\",\\\"prov\\\":[{\\\"bbox\\\":[113,176,464,195],\\\"page\\\":10,\\\"span\\\":[0,153]}]},{\\\"text\\\":\\\"both benchmarks. At mid parameter ranges, Granite Code models beat both CodeLlama-7B and CodeLLama-13B by a margin of \\\\u223c5% and \\\\u223c15% on average respectively. Additionally, Granite-34B-Code-Base is very competitive with CodeLlama-34B with only a difference of 0.9% on average across both benchmarks.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,59,505,103],\\\"page\\\":10,\\\"span\\\":[0,296]}]},{\\\"text\\\":\\\"10\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[301,31,311,39],\\\"page\\\":10,\\\"span\\\":[0,2]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,764],\\\"page\\\":11,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"Table 4: Pass@1 results on MultiPL-E averaged over 50 samples for each problem. All models are evaluated at temperature 0.2 and top-p 0.95.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,688,504,711],\\\"page\\\":11,\\\"span\\\":[0,139]}]},{\\\"name\\\":\\\"Table\\\",\\\"type\\\":\\\"table\\\",\\\"$ref\\\":\\\"#\\\\/tables\\\\/3\\\"},{\\\"text\\\":\\\"6.1.4 DS1000: Data Science Tasks in Python\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,283,311,294],\\\"page\\\":11,\\\"span\\\":[0,42]}]},{\\\"text\\\":\\\"DS-1000 (Lai et al. , 2023) is a widely studied benchmark which offers a comprehensive collection of 1,000 data science workflows across seven different libraries, from Matplotlib to TensorFlow. We use temperature 0.2 and top-p 0.95 to generate 40 samples per each library and report mean pass@1 with code completion setting for all the models up to 8B parameters. Results on DS-1000 are summarized in Table 7. Of the small models, StarCoder2-3B performs the best. Granite-3B-Code-Base is in second place, outperforming CodeGemma-2B by more than 12 points on average across 7 libraries. Granite-8B-Code-Base achieves the best average performance of 34.5% outperforming all other models of the similar parameter sizes.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,184,505,272],\\\"page\\\":11,\\\"span\\\":[0,717]}]},{\\\"text\\\":\\\"The Granite Code models achieve relatively high accuracy across all sizes (e.g., outperforming CodeGemma at 2B-3B scale, StarCoder2 at 7B-8B scale and CodeLlama models with half of the sizes). This shows that our Granite Code models are not only capable of generating good code but also of using libraries more accurately in real data science workflows.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,134,505,178],\\\"page\\\":11,\\\"span\\\":[0,353]}]},{\\\"text\\\":\\\"6.1.5 RepoBench, CrossCodeEval: Repository-Level Code Generation\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,103,428,114],\\\"page\\\":11,\\\"span\\\":[0,64]}]},{\\\"text\\\":\\\"Code generation in practice often occurs within the context of a repository rather than in isolated files. Thus, we use RepoBench (Liu et al. , 2023b) and CrossCodeEval (Ding et al. , 2024), to evaluate repository-level code completion capabilities of different models.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,59,505,92],\\\"page\\\":11,\\\"span\\\":[0,269]}]},{\\\"text\\\":\\\"11\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[301,32,310,39],\\\"page\\\":11,\\\"span\\\":[0,2]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,764],\\\"page\\\":12,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"Table 5: Pass@1 on MBPP and MBPP+. Results in the table represent zero-shot evaluation using greedy decoding.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,678,288,711],\\\"page\\\":12,\\\"span\\\":[0,109]}]},{\\\"text\\\":\\\"Table 6: Average exact match (EM) and edit similarity (ES) on RepoBench v1.1. All models are evaluated at temperature 0.2 and top-p 0.95.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[325,665,505,710],\\\"page\\\":12,\\\"span\\\":[0,137]}]},{\\\"name\\\":\\\"Table\\\",\\\"type\\\":\\\"table\\\",\\\"$ref\\\":\\\"#\\\\/tables\\\\/4\\\"},{\\\"text\\\":\\\"Table 7: Mean pass@1 accuracy averaged over 40 samples on DS-1000. All models are evaluated at temperature 0.2 and top-p 0.95.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,442,504,464],\\\"page\\\":12,\\\"span\\\":[0,126]}]},{\\\"name\\\":\\\"Table\\\",\\\"type\\\":\\\"table\\\",\\\"$ref\\\":\\\"#\\\\/tables\\\\/5\\\"},{\\\"text\\\":\\\"Table 6 shows the performance of different models on RepoBench v1.1. Granite-3B-CodeBase demonstrates notable performance among the smaller models, with StarCoderBase-3B achieving the leading performance metrics. Among the medium models, Granite-8B-CodeBase shows very strong performance on Java, while ranks second best one in Python, with CodeGemma-7B being the best performing on both metrics. Among larger models, Granite-20B-Code not only outperforms StarCoder2-15B but also CodeLlama-34B on all 4 metrics across both programming languages. This demonstrates strong repository-level\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"Text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,59,506,136],\\\"page\\\":12,\\\"span\\\":[0,587]}]},{\\\"name\\\":\\\"Table\\\",\\\"type\\\":\\\"table\\\",\\\"$ref\\\":\\\"#\\\\/tables\\\\/6\\\"},{\\\"text\\\":\\\"On RepoBench, we evaluate using level 2k across three settings: cross-file-first (12,000 data points), cross-file-random (5,000 data points), and in-file (7,000 data points). We report the average edit similarity and exact match across the settings. Following Liu et al. (2023b), we set generation temperature to 0.2 and the top-p sampling parameter to 0.95 for all models. We constrain the models to generate a maximum of 64 new tokens per prompt, and the first non-empty and non-comment line of the output was selected as the prediction.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,224,505,290],\\\"page\\\":12,\\\"span\\\":[0,539]}]},{\\\"text\\\":\\\"For CrossCodeEval, following Ding et al. (2024), we use a max sequence length of 2k using the retrieve-and-generate (RG) method with OpenAI's ada embedding. We set the maximum cross-file context to 512 tokens and the max generation token to 50 tokens for all the models. We use the uniform prompt formatting in the original implementation, with a temperature of 0.2 and top-p of 0.95 for all model generations. Max sequence length was set to 8,192 for all models, with the exception of Granite-3B-Code-Base (2,048) and Granite-8B-Code-Base (4,096), given their respective context lengths.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,142,505,218],\\\"page\\\":12,\\\"span\\\":[0,588]}]},{\\\"text\\\":\\\"12\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[301,32,311,40],\\\"page\\\":12,\\\"span\\\":[0,2]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,764],\\\"page\\\":13,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"Table 8: CrossCodeEval (Ding et al. , 2023) evaluation results. We report Code Match (Edit Similarity) and Identifier Match (F1) results for four languages.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,688,504,711],\\\"page\\\":13,\\\"span\\\":[0,156]}]},{\\\"text\\\":\\\"Table 9: Exact-match on FIM-task (Allal et al. , 2023). All models are evaluated using greedy decoding with maximum new tokens set to 512.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,451,504,472],\\\"page\\\":13,\\\"span\\\":[0,138]}]},{\\\"name\\\":\\\"Table\\\",\\\"type\\\":\\\"table\\\",\\\"$ref\\\":\\\"#\\\\/tables\\\\/7\\\"},{\\\"name\\\":\\\"Table\\\",\\\"type\\\":\\\"table\\\",\\\"$ref\\\":\\\"#\\\\/tables\\\\/8\\\"},{\\\"text\\\":\\\"code generation capabilities of the Granite Code models despite being not trained with repo-level file packing as in (Lozhkov et al. , 2024; CodeGemma Team et al. , 2024); we leave this as an interesting future work to further improve performance of our models.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,243,504,276],\\\"page\\\":13,\\\"span\\\":[0,261]}]},{\\\"text\\\":\\\"Results on CrossCodeEval are shown in Table 8. As can be seen from the table, among the similar sized models, CodeGemma-7B is best on Python and TypeScript, while StarCoder27B performs best on Java and C#. Likewise, Granite-20B-Code-Base outperforms CodeLlama13B on 3 programming languages (Python, Java, C#), while falls behind on TypeScript. Across all model sizes and programming languages, there is no single model that is best at all the metrics, similar to the findings in MultiPL-E. This indicates that achieving uniformly high performance across all programming languages remains challenging.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,160,505,237],\\\"page\\\":13,\\\"span\\\":[0,600]}]},{\\\"text\\\":\\\"6.1.6 FIM: Infilling Evaluations\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,134,256,145],\\\"page\\\":13,\\\"span\\\":[0,32]}]},{\\\"text\\\":\\\"Granite Code models are trained for code completion purposes using FIM objective, as described in Sec. 4.2. We use SantaCoder-FIM benchmark (Allal et al. , 2023), for infilling evaluations which tests the ability of models to fill in a single line of code in Python, JavaScript, and Java solutions to HumanEval. We use greedy decoding and report the mean exact match for all the models. Table 9 shows that Granite Code models significantly outperforms StarCoder and StarCoder2 across all model sizes, demonstrating it to be\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,59,505,124],\\\"page\\\":13,\\\"span\\\":[0,523]}]},{\\\"text\\\":\\\"13\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[301,32,311,40],\\\"page\\\":13,\\\"span\\\":[0,2]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,764],\\\"page\\\":14,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"Figure 3: Performance of Granite-8B-Code-Instruct, Mistral-7B-Instruct-v0.2, Gemma-7B-IT, and Llama-3-8B-Instruct on HumanEvalPack. Best viewed in color.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,518,505,539],\\\"page\\\":14,\\\"span\\\":[0,153]}]},{\\\"name\\\":\\\"Picture\\\",\\\"type\\\":\\\"figure\\\",\\\"$ref\\\":\\\"#\\\\/figures\\\\/2\\\"},{\\\"text\\\":\\\"excellent well-rounded models for code completion use cases. Moreover, we observe no performance improvement in scaling the model sizes from 8B to 34B, indicating that smaller models are often more suitable for FIM code completion tasks.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,463,505,495],\\\"page\\\":14,\\\"span\\\":[0,237]}]},{\\\"text\\\":\\\"6.2 Code Explanation and Fixing\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,438,263,448],\\\"page\\\":14,\\\"span\\\":[0,31]}]},{\\\"text\\\":\\\"While most of the prior code LLMs primarily focus on evaluating performance using code generation benchmarks, users may want to use these models in other challenging scenarios beyond synthesis like explaining and fixing codes. Thus, following (Muennighoff et al. , 2023), we test the performance of different code models on the code explanation and fixing variants of HumanEvalPack benchmark, spanning 6 different programming languages. For both HumanEvalExplain and HumanEvalFix, we evaluate all models in a zero-shot manner using greedy decoding with completion format for the base models, and instruction template for the instruction-tuned models.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,340,505,428],\\\"page\\\":14,\\\"span\\\":[0,650]}]},{\\\"text\\\":\\\"The results of the HumanEvalExplain benchmark are shown in Table 10. Granite Code Base models significantly outperform other SOTA base code LLMs, including StarCoder2 and CodeGemma, by a large margin. Interestingly, Granite-8B-Code-Base beats CodeLlama-34B by 9.3% on average, while being close to CodeLlama-70B. We attribute this performance to our data mixture and base model training decisions. After instruction tuning, performance of all the base models significantly improves across languages. Among code instruct models, Granite-34B-Code-Instruct performs the best reaching the average score of 41.9%, which is very close of 41.1% score of CodeLlama-70B-Instruct. Remarkably, CodeGemma-7B-IT gains the most improvement from instruction tuning but still falls behind Granite-8b-CodeInstruct by 2.5% on average. Mixtral-8x22B-Instruct-v0.1 performs best among all models benchmarked with a significant margin, indicating the potential advantage of bigger models and training on general natural language data could potentially help on this task.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,202,505,334],\\\"page\\\":14,\\\"span\\\":[0,1049]}]},{\\\"text\\\":\\\"Table 11 reports the results on HumanEvalFix. Like HumanEvalExplain, Granite Code models base models significantly outperform other base models. Notably, Granite-8B-Code-Base again shows impressive performance, making it close to CodeLlama-70B and Llama-3-70B. After instruction tuning, we observe a performance improvement on almost all models. Notably, our 8B and 20B instruct models achieve the best performance among models with less that has less than 34B parameters. However, we see a significant performance improvement (about 10 points) after moving to larger models with more than 34B parameters. Among large instruct models, Granite-34B-Code-Instruct performs similarly to other models with at least twice the parameters, thus having a better cost and performance balance.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,98,505,197],\\\"page\\\":14,\\\"span\\\":[0,782]}]},{\\\"text\\\":\\\"Figure 3 compares the performance of Granite-8B-Code-Instruct with state-of-the-art opensource instruction-tuned general LLMs. Granite-8B-Code-Instruct consistently outperforms the compared models, emphasizing the need for domain-specific code models. To summa-\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,59,506,92],\\\"page\\\":14,\\\"span\\\":[0,261]}]},{\\\"text\\\":\\\"14\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[301,32,311,39],\\\"page\\\":14,\\\"span\\\":[0,2]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,764],\\\"page\\\":15,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"Table 10: Pass@1 performance on HumanEvalExplain.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[186,700,425,710],\\\"page\\\":15,\\\"span\\\":[0,49]}]},{\\\"name\\\":\\\"Table\\\",\\\"type\\\":\\\"table\\\",\\\"$ref\\\":\\\"#\\\\/tables\\\\/9\\\"},{\\\"text\\\":\\\"rize, these results show that both our base and instruct models are capable of generating good code but also in code fixing and explanation, demonstrating their ability to solve diverse coding tasks in enterprise software development.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,178,504,210],\\\"page\\\":15,\\\"span\\\":[0,234]}]},{\\\"text\\\":\\\"6.3 Code Editing and Translation\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,148,264,158],\\\"page\\\":15,\\\"span\\\":[0,32]}]},{\\\"text\\\":\\\"CanItEdit is a recent benchmark designed to evaluate code LLMs on instructional code editing tasks. The benchmark contains 105 hand-crafted Python programs where each problem consists of a code snippet accompanied by an instruction of two types: descriptive or lazy. The goal is to modify the code according to the instruction; both lazy and descriptive instructions should lead to the same edit. Following Cassano et al. (2024), we compare different instruction-tuned models using their corresponding instruction format, by random sampling with a temperature of 0.2 and a top-p of 0.95, with 20 completions per problem.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,59,504,136],\\\"page\\\":15,\\\"span\\\":[0,620]}]},{\\\"text\\\":\\\"15\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[301,32,311,39],\\\"page\\\":15,\\\"span\\\":[0,2]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,764],\\\"page\\\":16,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"Table 11: Pass@1 performance on HumanEvalFix.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[196,698,414,709],\\\"page\\\":16,\\\"span\\\":[0,45]}]},{\\\"text\\\":\\\"Table 12: Pass@1 and ExcessCode performance on CanItEdit. Pass@1 assesses functional correctness, and ExcessCode assesses conciseness and precision of code edits.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,190,504,212],\\\"page\\\":16,\\\"span\\\":[0,162]}]},{\\\"name\\\":\\\"Table\\\",\\\"type\\\":\\\"table\\\",\\\"$ref\\\":\\\"#\\\\/tables\\\\/10\\\"},{\\\"name\\\":\\\"Table\\\",\\\"type\\\":\\\"table\\\",\\\"$ref\\\":\\\"#\\\\/tables\\\\/11\\\"},{\\\"text\\\":\\\"16\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[301,32,311,39],\\\"page\\\":16,\\\"span\\\":[0,2]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,764],\\\"page\\\":17,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"Table 13: Performance of different instruction-tuned models on CodeLingua (Pan et al. , 2024). We report Pass@1 for translation across five languages.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,689,506,711],\\\"page\\\":17,\\\"span\\\":[0,150]}]},{\\\"name\\\":\\\"Table\\\",\\\"type\\\":\\\"table\\\",\\\"$ref\\\":\\\"#\\\\/tables\\\\/12\\\"},{\\\"text\\\":\\\"Table 13 shows the results on the CodeLingua benchmark. For the source languages C, C++ and Go the results reported in the table are taken directly from the runs on Codenet, whereas for Java and Python the results are reported as the average of the runs on Avatar and CodeNet. We report the numbers of Octocoder and CodeLlama from the CodeLingua leaderboard 16. The Granite Code models performs comparably to CodeGemma. It is worth noticing that the correctness of the translation is not only due to the code generated by the model, but also by the extra metadata and explanation provided as part of the answer. We tested instruction tuned models, as we observed that base models often struggle to understand the request itself to translate code. Instruct models, on the other hand, tend to add additional information besides the translated code as part of the generations. The CodeLLama family seems to suffer especially from this issue, as post-processing the\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"Text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,80,505,202],\\\"page\\\":17,\\\"span\\\":[0,961]}]},{\\\"name\\\":\\\"Table\\\",\\\"type\\\":\\\"table\\\",\\\"$ref\\\":\\\"#\\\\/tables\\\\/13\\\"},{\\\"text\\\":\\\"From Table 12, we make the following observations on the performance of different models on CanItEdit benchmark. It shows that Granite Code models have better pass rates, as well as less presence of unnecessary code changes, compared to CodeGemma and CodeLlama. This result shows that Granite Code models can better understand users' intentions and make accurate changes to the existing code in practical situations.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,357,505,411],\\\"page\\\":17,\\\"span\\\":[0,416]}]},{\\\"text\\\":\\\"CodeLingua (Pan et al. , 2024) is a dataset designed to test model's capabilities in code translation. It contains two sets of programs: one containing 251 programs in Java and 251 prgrams in Python sampled from Avatar (Ahmad et al. , 2021), and one from CodeNet Puri et al. (2021) containing 250 programs for each of five languages: C, C++, Go, Java and Python. For each program a set of unit tests in the form of input and expected outputs is provided. The task consists in translating each program from the source language to five target languages (the ones sampled from CodeNet). Pass@1 is used as the metric to evaluate translation accuracy. For every generation, we used greedy decoding and the suggested prompt format for each instruction tuned model. For base models, or cases where the instruction format was not specified, we used the default prompt from the dataset. Basic post-processing is applied to each generation, to remove generation artifacts such as repetition of the input instruction, source language code, target language name and formatting tokens ( ''' , for example).\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,208,505,351],\\\"page\\\":17,\\\"span\\\":[0,1093]}]},{\\\"text\\\":\\\"16 https:\\\\/\\\\/codetlingua.github.io\\\\/leaderboard.html\\\",\\\"type\\\":\\\"footnote\\\",\\\"name\\\":\\\"Footnote\\\",\\\"prov\\\":[{\\\"bbox\\\":[117,60,318,70],\\\"page\\\":17,\\\"span\\\":[0,49]}]},{\\\"text\\\":\\\"17\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[301,32,311,39],\\\"page\\\":17,\\\"span\\\":[0,2]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,764],\\\"page\\\":18,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"generations to extract only the relevant code constitutes a non-trivial task. The CodeGemma and Granite Models on the other hand, produce a nicely formatted output that can be easily parsed. Interestingly, Go seems to be the hardest target language to translate to, while C is the source language with the highest translation success rate from, for the Granite models.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,664,504,708],\\\"page\\\":18,\\\"span\\\":[0,368]}]},{\\\"text\\\":\\\"6.4 Code Reasoning, Understanding and Execution\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,640,345,650],\\\"page\\\":18,\\\"span\\\":[0,47]}]},{\\\"text\\\":\\\"Table 14: Performance on the CRUXEval benchmark. We use temperature 0.2 for pass@1 and temperature 0.8 for pass@5, both using 10 samples, as in (Gu et al. , 2024).\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"Text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,605,504,627],\\\"page\\\":18,\\\"span\\\":[0,163]}]},{\\\"name\\\":\\\"Table\\\",\\\"type\\\":\\\"table\\\",\\\"$ref\\\":\\\"#\\\\/tables\\\\/14\\\"},{\\\"text\\\":\\\"CRUXEval (Gu et al. , 2024) is a benchmark of 800 Python functions and input-output pairs, consisting of two tasks: CRUXEval-I (input prediction) and CRUXEval-O (output prediction). We use temperature 0.2 to report pass@1 and temperature 0.8 to report pass@5, both using 10 samples, as in Lozhkov et al. (2024); Gu et al. (2024). Table 14 shows that Granite Code models perform competitively with other models. Granite-3B-Code-Base outperforms CodeGemma-2B on CRUXEval-I but lags behind on CRUXEval-O. Interestingly, there is not a single model which performs consistently best at 3B parameters. However, at 7B-8B parameters, CodeGemma-7B outperforms all the models on both tasks. For the large models, Granite-34B-Code-Base lags behind CodeLlama-34B on CRUXEval-I but outperforms on CRUXEval-O. Performance on both CRUXEval-I and CRUXEval-O increases as we scale the size of the Granite Code models from 3B to 34B parameters, demonstrating the advantage of larger models for code reasoning and execution tasks.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,242,505,374],\\\"page\\\":18,\\\"span\\\":[0,1011]}]},{\\\"text\\\":\\\"6.5 Math Reasoning\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,218,206,228],\\\"page\\\":18,\\\"span\\\":[0,18]}]},{\\\"text\\\":\\\"We use the following four widely used benchmarks to assess the mathematical reasoning capabilities of Granite-8B-Code-Base and various 7B-8B baseline models:\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,186,504,208],\\\"page\\\":18,\\\"span\\\":[0,157]}]},{\\\"text\\\":\\\"\\\\u00b7 MATH (Hendrycks et al. , 2021): a dataset from high-school math competitions; we use the 4-shot experiment setting from Gao et al. (2023);\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[133,156,504,178],\\\"page\\\":18,\\\"span\\\":[0,140]}]},{\\\"text\\\":\\\"\\\\u00b7 GSM8K (Cobbe et al. , 2021): a dataset of middle-school level math word problems; we use the 5-shot experiment setting from Gao et al. (2023);\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[133,131,505,153],\\\"page\\\":18,\\\"span\\\":[0,144]}]},{\\\"text\\\":\\\"\\\\u00b7 SAT (Azerbayev et al. , 2023): a dataset consisting of the 32 math questions with no figures from the May 2023 College Board SAT examination; we use the same experiment setting from Azerbayev et al. (2023)\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[133,95,504,128],\\\"page\\\":18,\\\"span\\\":[0,207]}]},{\\\"text\\\":\\\"\\\\u00b7 OCW (Lewkowycz et al. , 2022): a collection of undergraduate-level STEM problems harvested from MIT's OpenCourseWare; we use the 4-shot experiment setting from Azerbayev et al. (2023).\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[133,59,504,92],\\\"page\\\":18,\\\"span\\\":[0,186]}]},{\\\"text\\\":\\\"18\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[301,32,311,39],\\\"page\\\":18,\\\"span\\\":[0,2]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,764],\\\"page\\\":19,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"Table 15: Performance on 4 chain-of-thought math tasks and 2 tool-aided math tasks.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[118,700,492,711],\\\"page\\\":19,\\\"span\\\":[0,83]}]},{\\\"name\\\":\\\"Table\\\",\\\"type\\\":\\\"table\\\",\\\"$ref\\\":\\\"#\\\\/tables\\\\/15\\\"},{\\\"text\\\":\\\"\\\\u22c6 We noticed that Llama-3-8B-Base tends to generate invalid programs given the same prompts as the other model, resulting in very low scores on MATH+Py and GSM8K+Py tasks. Similar issues have been observed in our Python code generation experiment.\\\",\\\"type\\\":\\\"footnote\\\",\\\"name\\\":\\\"Footnote\\\",\\\"prov\\\":[{\\\"bbox\\\":[118,534,494,564],\\\"page\\\":19,\\\"span\\\":[0,247]}]},{\\\"text\\\":\\\"Figure 4: Performance of Granite Code models on Berkeley Function-Calling Leaderboard. Overall accuracy keeps increasing with increase in model sizes, indicating the advantage of large models for function calling abilities. Best viewed in color.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,306,505,339],\\\"page\\\":19,\\\"span\\\":[0,245]}]},{\\\"name\\\":\\\"Picture\\\",\\\"type\\\":\\\"figure\\\",\\\"$ref\\\":\\\"#\\\\/figures\\\\/3\\\"},{\\\"text\\\":\\\"Following Azerbayev et al. (2023), we also evaluate models on two tasks that involve solving problems with access to computational tools:\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,261,504,283],\\\"page\\\":19,\\\"span\\\":[0,137]}]},{\\\"text\\\":\\\"\\\\u00b7 MATH+Py solving MATH task by writing a Python program that uses built-in numeric operations, the math module, and SymPy; we use the 5-shot prompt and experiment setting from Azerbayev et al. (2023);\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[133,218,504,251],\\\"page\\\":19,\\\"span\\\":[0,200]}]},{\\\"text\\\":\\\"\\\\u00b7 GSM8K+Py solving GSM8K task by writing a Python program that executes to generate an integer answer; we use the 8-shot prompt and experiment setting from Azerbayev et al. (2023).\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[133,181,504,214],\\\"page\\\":19,\\\"span\\\":[0,180]}]},{\\\"text\\\":\\\"Table 15 summarizes the results. Despite not being specifically tuned for mathematical reasoning, Granite-8B-Code-Base shows impressive reasoning ability, outperforming most existing 7B to 8B models. While other models may be particularly strong on a few tasks, our model consistently achieves top-1 or top-2 performance on all tasks.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,128,504,171],\\\"page\\\":19,\\\"span\\\":[0,334]}]},{\\\"text\\\":\\\"6.6 Calling Functions and Tools\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,102,258,112],\\\"page\\\":19,\\\"span\\\":[0,31]}]},{\\\"text\\\":\\\"We adopt Berkeley Function-Calling Leaderboard (BFCL) (Yan et al. , 2024), to evaluate LLM's ability to call functions and tools. BFCL is a function-calling dataset with 1700 functions across 4 categories: simple, multiple, parallel, and parallel multiple function calls -\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,59,505,92],\\\"page\\\":19,\\\"span\\\":[0,272]}]},{\\\"text\\\":\\\"19\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[301,32,311,39],\\\"page\\\":19,\\\"span\\\":[0,2]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,764],\\\"page\\\":20,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"Figure 5: Granite-8B-Code vs CodeLlama-7B on Berkley Function-Calling Leaderboard. Granite-8B-Code (Base\\\\/Instruct) consistently outperforms CodeLlama-7B (Base\\\\/Instruct) on all three metrics. Best viewed in color.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,485,505,516],\\\"page\\\":20,\\\"span\\\":[0,212]}]},{\\\"name\\\":\\\"Picture\\\",\\\"type\\\":\\\"figure\\\",\\\"$ref\\\":\\\"#\\\\/figures\\\\/4\\\"},{\\\"text\\\":\\\"ch differing in the number of potential functions the model has access to and the number of output functions the model has to generate. We use two popular methods to evaluate the accuracy of the model-generated answers: AST evaluation based on Abstract Syntax Tree (AST) based metric for fuzzy evaluation of output, and Executable evaluation to match the outputs of model-generated and ground-truth functions.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,406,504,461],\\\"page\\\":20,\\\"span\\\":[0,409]}]},{\\\"text\\\":\\\"Figure 4 shows the results of different Granite Code models on BFCL benchmark. As can be seen from the figure, overall accuracy improves from 25.65% to 57.12% for Granite-3BCode-Base to Granite-34B-Code-Base, showing the effectiveness of model scaling in function (tool) calling capabilities. We also compare Granite-8B-Code with CodeLlama-7B in Figure 5 and find that Granite-8B-Code-Instruct beats CodeLlama-7B-Instruct by 22%, 14% and 12% on AST Summary, Execution Summary and Overall accuracy respectively. Additionally, Figure 5 shows that instruction tuning consistently improves performance of both base models, with more noticeable improvements in Granite Code models. E.g., +17.88% in overall accuracy from Granite-8B-Code-Base to Granite-8B-Code-Instruct, indicating the effectiveness of our well-curated data mixture in finetuning base models.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,290,505,400],\\\"page\\\":20,\\\"span\\\":[0,854]}]},{\\\"text\\\":\\\"6.7 Model Robustness\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,266,215,275],\\\"page\\\":20,\\\"span\\\":[0,20]}]},{\\\"text\\\":\\\"While the performance on canonical code generative tasks is essential, we argue that the evaluation of practical robustness is also necessary to characterize different models systematically. We therefore consider benchmarking the robustness of code synthesis, one of the most representative downstream tasks of source code. ReCode (Wang et al. , 2022) provides 30 different general perturbations on docstrings, function names, and codes to evaluate the robustness of code-generation models. We use the perturbed version of the HumanEval benchmark using greedy generation with 5 seeds, as recommended in (Wang et al. , 2022).\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,175,505,251],\\\"page\\\":20,\\\"span\\\":[0,624]}]},{\\\"text\\\":\\\"Table 16 shows the worst-case RP@1 of different models for each perturbation category. While Granite-3B-Code-Base consistently outperforms CodeGemma-2B, Granite-8B-CodeBase lags behind CodeGemma-7B on all categories. Granite Code models obtains much better performance compared to CodeLlama models, showing its generalization in a robust way at every sizes. Our largest model, Granite-34B-Code-Base consistently outperforms CodeLlama-34B on all four categories. This indicates that Granite-34B-Code-Base has more capacity to deal with unseen instances and perturbations. In general, we also observe higher RP@1 for larger models within the Granite Code family (e.g., improved from 40.1% to 52.0% for Granite-3B-Code-Base to Granite-34B-Code-Base on average across all perturbations), showing that larger model helps improve worst-case robustness.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,59,505,169],\\\"page\\\":20,\\\"span\\\":[0,846]}]},{\\\"text\\\":\\\"20\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[301,31,311,40],\\\"page\\\":20,\\\"span\\\":[0,2]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,764],\\\"page\\\":21,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"Table 16: RP@1 performance on the Recode benchmark. Following (Wang et al. , 2022), we use the perturbed version of the HumanEval benchmark with greedy sampling for all the models to eliminate randomness effect and enable fair comparison.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,678,504,710],\\\"page\\\":21,\\\"span\\\":[0,238]}]},{\\\"name\\\":\\\"Table\\\",\\\"type\\\":\\\"table\\\",\\\"$ref\\\":\\\"#\\\\/tables\\\\/16\\\"},{\\\"text\\\":\\\"7 Conclusion\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,442,188,452],\\\"page\\\":21,\\\"span\\\":[0,12]}]},{\\\"text\\\":\\\"We presented a family of decoder-only Granite Code models ranging in size from 3 to 34 billion parameters that are highly versatile in their ability to accomplish a wide range of tasks from code generation to fixing bugs, explaining and documenting code, maintaining repositories, and more. These models have proven to be suitable for applications ranging from complex application modernization tasks (IBM , 2023) to on-device memory-constrained use cases. Extensive evaluation demonstrates that Granite Code models consistently reach state-of-the-art performance among open-source code LLMs, matching or exceeding the performance of recently released CodeGemma, StarCoder2, and Llama3 models on average performance across various code-related tasks of code generation, explanation, and bug fixing in a variety of popular programming languages. Our experience and results demonstrate that Granite Code models have a proven ability to better handle different tasks in enterprise software development workflows. We release all our Granite Code models under an Apache 2.0 license for both research and commercial use. We plan to continuously release updates to these models to improve their performance, e.g. leveraging the CodeNet instruction dataset (Puri et al. , 2021), and in the near future we plan to release long-context as well as Python- and Java-specialized model variants.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,252,505,427],\\\"page\\\":21,\\\"span\\\":[0,1381]}]},{\\\"text\\\":\\\"Acknowledgments\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,222,211,234],\\\"page\\\":21,\\\"span\\\":[0,15]}]},{\\\"text\\\":\\\"We would like to acknowledge the efforts of numerous teams at IBM Research AI and Hybrid Cloud Platform, IBM AI Infrastructure team, IBM WatsonX Code Assistant and platform team. Special thanks to IBM Research leaders - Dario Gil, Sriram Raghavan, Mukesh Khare, Danny Barnett, Talia Gershon, Priya Nagpurkar, Nicholas Fuller for their support.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,166,505,209],\\\"page\\\":21,\\\"span\\\":[0,343]}]},{\\\"text\\\":\\\"Thanks and acknowledgement to Trent Gray-Donald, Keri Olson, Alvin Tan, Hillery Hunter, Dakshi Agrawal, Xuan Liu, Mudhakar Srivatsa, Raghu Kiran Ganti, Carlos Costa, Darrell Reimer, Maja Vukovic, Dinesh Garg, Akash Srivastava, Abhishek Bhandwaldar, Aldo Pareja, Shiv Sudalairaj, Atin Sood, Sandeep Gopisetty, Nick Hill, Ray Rose, Tulio Coppola, Allysson ' Oliveira, Aadarsh Sahoo, Apoorve Mohan, Yuan Chi Chang, Jitendra Singh, Yuya Ong, Eric Butler, David Brotherton, Rakesh Mohan, David Kung, Dinesh Khandelwal, Naigang Wang, Nelson Mimura Gonzalez, Olivier Tardieu, Tuan Hoang Trong, Luis Angel Bathen, Kevin O'Connor, Christopher Laibinis, Tatsuhiro Chiba, Sunyanan Choochotkaew, Robert Walkup, Antoni Viros i Martin, Adnan Hoque, Davis Wertheimer and Marquita Ellis.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"Text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,59,505,160],\\\"page\\\":21,\\\"span\\\":[0,771]}]},{\\\"text\\\":\\\"21\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[301,32,310,40],\\\"page\\\":21,\\\"span\\\":[0,2]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,765],\\\"page\\\":22,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"References\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,699,168,709],\\\"page\\\":22,\\\"span\\\":[0,10]}]},{\\\"text\\\":\\\"Wasi Uddin Ahmad, Md Golam Rahman Tushar, Saikat Chakraborty, and Kai-Wei Chang. Avatar: A parallel corpus for java-python program translation. arXiv preprint arXiv:2108.11590, 2021.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,658,504,690],\\\"page\\\":22,\\\"span\\\":[0,182]}]},{\\\"text\\\":\\\"AI@Meta. Llama 3 model card. 2024. URL https:\\\\/\\\\/github.com\\\\/meta-llama\\\\/llama3\\\\/blob\\\\/ main\\\\/MODEL CARD.md .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,628,505,649],\\\"page\\\":22,\\\"span\\\":[0,102]}]},{\\\"text\\\":\\\"Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, ' and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,585,505,618],\\\"page\\\":22,\\\"span\\\":[0,197]}]},{\\\"text\\\":\\\"Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. Santacoder: don't reach for the stars! arXiv preprint arXiv:2301.03988, 2023.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,543,506,576],\\\"page\\\":22,\\\"span\\\":[0,242]}]},{\\\"text\\\":\\\"Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,502,504,535],\\\"page\\\":22,\\\"span\\\":[0,211]}]},{\\\"text\\\":\\\"Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310.10631, 2023.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,460,505,492],\\\"page\\\":22,\\\"span\\\":[0,235]}]},{\\\"text\\\":\\\"Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,440,479,451],\\\"page\\\":22,\\\"span\\\":[0,82]}]},{\\\"text\\\":\\\"Kinjal Basu, Ibrahim Abdelaziz, Subhajit Chaudhury, Soham Dan, Maxwell Crouse, Asim Munawar, Sadhana Kumaravel, Vinod Muthusamy, Pavan Kapanipathi, and Luis A Lastras. Api-blend: A comprehensive corpora for training and benchmarking api llms. arXiv preprint arXiv:2402.15491, 2024.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,388,505,431],\\\"page\\\":22,\\\"span\\\":[0,281]}]},{\\\"text\\\":\\\"Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle, 2022.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,348,505,379],\\\"page\\\":22,\\\"span\\\":[0,176]}]},{\\\"text\\\":\\\"Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. Multipl-e: a scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering, 2023.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,294,506,337],\\\"page\\\":22,\\\"span\\\":[0,303]}]},{\\\"text\\\":\\\"Federico Cassano, Luisa Li, Akul Sethi, Noah Shinn, Abby Brennan-Jones, Jacob Ginesin, Edward Berman, George Chakhnashvili, Anton Lozhkov, Carolyn Jane Anderson, and Arjun Guha. Can it edit? evaluating the ability of large language models to follow code editing instructions, 2024.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,241,505,285],\\\"page\\\":22,\\\"span\\\":[0,281]}]},{\\\"text\\\":\\\"Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"List-item\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,103,506,232],\\\"page\\\":22,\\\"span\\\":[0,950]}]},{\\\"text\\\":\\\"Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,59,505,92],\\\"page\\\":22,\\\"span\\\":[0,244]}]},{\\\"text\\\":\\\"22\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[300,32,311,40],\\\"page\\\":22,\\\"span\\\":[0,2]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,765],\\\"page\\\":23,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"CodeGemma Team, Ale Jakse Hartman, Andrea Hu, Christopher A. Choquette-Choo, Heri Zhao, Jane Fine, Jeffrey Hui, Jingyue Shen, Joe Kelley, Joshua Howland, Kshitij Bansal, Luke Vilnis, Mateo Wirth, Nam Nguyen, Paul Michel, Peter Choy, Pratik Joshi, Ravin Kumar, Sarmad Hashmi, Shubham Agrawal, Siqi Zuo, Tris Warkentin, and Zhitao et al. Gong. Codegemma: Open code models based on gemma. 2024. URL https: \\\\/\\\\/goo.gle\\\\/codegemma .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,642,505,708],\\\"page\\\":23,\\\"span\\\":[0,424]}]},{\\\"text\\\":\\\"Cohere. Command r+. https:\\\\/\\\\/docs.cohere.com\\\\/docs\\\\/command-r-plus .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,623,430,634],\\\"page\\\":23,\\\"span\\\":[0,65]}]},{\\\"text\\\":\\\"Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"List-item\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,595,506,614],\\\"page\\\":23,\\\"span\\\":[0,96]}]},{\\\"text\\\":\\\"Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Re. Flashatten- ' ' tion: Fast and memory-efficient exact attention with io-awareness. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 16344-16359. Curran Associates, Inc., 2022. URL https:\\\\/\\\\/proceedings.neurips.cc\\\\/paper files\\\\/paper\\\\/2022\\\\/file\\\\/ 67d57c32e20fd0a7a302cb81d36e40d5-Paper-Conference.pdf .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,519,506,584],\\\"page\\\":23,\\\"span\\\":[0,453]}]},{\\\"text\\\":\\\"Databricks. Introducing dbrx: A new state-of-the-art open llm - databricks blog. https: \\\\/\\\\/www.databricks.com\\\\/blog\\\\/introducing-dbrx-new-state-art-open-llm .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"List-item\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,488,505,510],\\\"page\\\":23,\\\"span\\\":[0,155]}]},{\\\"text\\\":\\\"Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https:\\\\/\\\\/openreview.net\\\\/forum?id=wgDcbBMSfh .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,425,506,479],\\\"page\\\":23,\\\"span\\\":[0,411]}]},{\\\"text\\\":\\\"Yangruibo Ding, Zijian Wang, Wasi Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, et al. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. Advances in Neural Information Processing Systems, 36, 2024.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,372,506,416],\\\"page\\\":23,\\\"span\\\":[0,301]}]},{\\\"text\\\":\\\"Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL https:\\\\/\\\\/zenodo.org\\\\/records\\\\/10256836 .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,298,506,363],\\\"page\\\":23,\\\"span\\\":[0,448]}]},{\\\"text\\\":\\\"Gemma-Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Riviere, Mihir Sanjay Kale, Juliette Love, Pouya ' ' Tafti, Leonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, ' ' Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amelie H ' ' eliou, Andrea ' ' Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clement Crepy, Daniel Cer, Daphne Ippolito, David Reid, ' ' Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Miku\\\\u0142a, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross McIlroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clement Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis ' ' Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open models based on gemini research and technology, 2024.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"Text\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,59,506,290],\\\"page\\\":23,\\\"span\\\":[0,1760]}]},{\\\"text\\\":\\\"23\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[300,31,311,40],\\\"page\\\":23,\\\"span\\\":[0,2]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,764],\\\"page\\\":24,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"Alex Gu, Baptiste Roziere, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and ' ' Sida I. Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065, 2024.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,675,505,708],\\\"page\\\":24,\\\"span\\\":[0,212]}]},{\\\"text\\\":\\\"Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), 2023.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,657,444,668],\\\"page\\\":24,\\\"span\\\":[0,74]}]},{\\\"text\\\":\\\"Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,618,505,650],\\\"page\\\":24,\\\"span\\\":[0,193]}]},{\\\"text\\\":\\\"Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,577,504,610],\\\"page\\\":24,\\\"span\\\":[0,255]}]},{\\\"text\\\":\\\"IBM. watsonx code assistant, 2023. URL https:\\\\/\\\\/www.ibm.com\\\\/products\\\\/ watsonx-code-assistant .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,549,505,570],\\\"page\\\":24,\\\"span\\\":[0,93]}]},{\\\"text\\\":\\\"Neel Jain, Ping yeh Chiang, Yuxin Wen, John Kirchenbauer, Hong-Min Chu, Gowthami Somepalli, Brian R. Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Neftune: Noisy embeddings improve instruction finetuning, 2023.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,497,505,541],\\\"page\\\":24,\\\"span\\\":[0,279]}]},{\\\"text\\\":\\\"Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023a.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,457,504,489],\\\"page\\\":24,\\\"span\\\":[0,241]}]},{\\\"text\\\":\\\"Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lelio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut ' ' Lavril, Thomas Wang, Timothee Lacroix, and William El Sayed. Mistral 7b, 2023b. ' '\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"List-item\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,406,504,449],\\\"page\\\":24,\\\"span\\\":[0,349]}]},{\\\"text\\\":\\\"Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey. A study of bfloat16 for deep learning training, 2019.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"List-item\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,344,505,399],\\\"page\\\":24,\\\"span\\\":[0,384]}]},{\\\"text\\\":\\\"Dahyun Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, Changbae Ahn, Seonghoon Yang, Sukyung Lee, Hyunbyung Park, Gyoungjin Gim, Mikyoung Cha, Hwalsuk Lee, and Sunghun Kim. Solar 10.7b: Scaling large language models with simple yet effective depth up-scaling, 2024.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,282,506,337],\\\"page\\\":24,\\\"span\\\":[0,335]}]},{\\\"text\\\":\\\"Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,264,489,274],\\\"page\\\":24,\\\"span\\\":[0,82]}]},{\\\"text\\\":\\\"Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023. URL https:\\\\/\\\\/proceedings.mlsys.org\\\\/paper files\\\\/paper\\\\/2023\\\\/hash\\\\/ e851ca7b43815718fbbac8afb2246bf8-Abstract-mlsys2023.html .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,202,506,256],\\\"page\\\":24,\\\"span\\\":[0,364]}]},{\\\"text\\\":\\\"Andreas Kopf, Yannic Kilcher, Dimitri von R \\\\u00a8 utte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith \\\\u00a8 Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richard Nagyfi, Shahul ' ' ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. Openassistant conversations - democratizing large language model alignment, 2023.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"List-item\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,139,505,194],\\\"page\\\":24,\\\"span\\\":[0,385]}]},{\\\"text\\\":\\\"Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wentau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pp. 18319-18345. PMLR, 2023.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,90,506,132],\\\"page\\\":24,\\\"span\\\":[0,284]}]},{\\\"text\\\":\\\"Ariel N. Lee, Cole J. Hunter, and Nataniel Ruiz. Platypus: Quick, cheap, and powerful refinement of llms. 2023.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,61,504,81],\\\"page\\\":24,\\\"span\\\":[0,111]}]},{\\\"text\\\":\\\"24\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[300,32,311,40],\\\"page\\\":24,\\\"span\\\":[0,2]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,765],\\\"page\\\":25,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,664,506,708],\\\"page\\\":25,\\\"span\\\":[0,294]}]},{\\\"text\\\":\\\"Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, Joao Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, \\\\u02dc \\\\u02dc Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Munoz Ferrandis, Sean Hughes, \\\\u02dc \\\\u02dc Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you!, 2023a.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"List-item\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,504,506,657],\\\"page\\\":25,\\\"span\\\":[0,1101]}]},{\\\"text\\\":\\\"Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161, 2023b.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,464,505,497],\\\"page\\\":25,\\\"span\\\":[0,230]}]},{\\\"text\\\":\\\"Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a. URL https:\\\\/\\\\/openreview.net\\\\/forum?id=1qvx610Cu7 .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,413,505,457],\\\"page\\\":25,\\\"span\\\":[0,306]}]},{\\\"text\\\":\\\"Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint arXiv:2306.03091, 2023b.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,384,504,405],\\\"page\\\":25,\\\"span\\\":[0,155]}]},{\\\"text\\\":\\\"Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pp. 22631- 22648. PMLR, 2023.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,334,506,376],\\\"page\\\":25,\\\"span\\\":[0,285]}]},{\\\"text\\\":\\\"Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173, 2024.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,292,505,325],\\\"page\\\":25,\\\"span\\\":[0,243]}]},{\\\"text\\\":\\\"Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In International Conference on Learning Representations, 2018. URL https:\\\\/\\\\/openreview.net\\\\/forum?id=r1gs9JgRZ .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,242,505,285],\\\"page\\\":25,\\\"span\\\":[0,310]}]},{\\\"text\\\":\\\"MistralAI. Mixtral 8x22b. https:\\\\/\\\\/mistral.ai\\\\/news\\\\/mixtral-8x22b\\\\/ .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,224,408,234],\\\"page\\\":25,\\\"span\\\":[0,66]}]},{\\\"text\\\":\\\"Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models, 2023.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,183,506,216],\\\"page\\\":25,\\\"span\\\":[0,216]}]},{\\\"text\\\":\\\"Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '21, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450384421. doi: 10.1145\\\\/3458817.3476209. URL https:\\\\/\\\\/doi.org\\\\/10.1145\\\\/3458817.3476209 .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,99,506,176],\\\"page\\\":25,\\\"span\\\":[0,573]}]},{\\\"text\\\":\\\"Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis, 2023.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,59,505,92],\\\"page\\\":25,\\\"span\\\":[0,196]}]},{\\\"text\\\":\\\"25\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[300,31,311,40],\\\"page\\\":25,\\\"span\\\":[0,2]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,765],\\\"page\\\":26,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"Rangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar, Lambert Pouguem Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha, and Reyhaneh Jabbarvand. Lost in translation: A study of bugs introduced by large language models while translating code. In Proceedings of the IEEE\\\\/ACM 46th International Conference on Software Engineering , ICSE '24. Association for Computing Machinery, 2024.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,654,506,708],\\\"page\\\":26,\\\"span\\\":[0,409]}]},{\\\"text\\\":\\\"Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text. arXiv preprint arXiv:2310.06786, 2023.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,623,505,645],\\\"page\\\":26,\\\"span\\\":[0,172]}]},{\\\"text\\\":\\\"Nikhil Pinnaparaju, Reshinth Adithyan, Duy Phung, Jonathan Tow, James Baicoianu, Ashish Datta, Maksym Zhuravinskyi, Dakota Mahan, Marco Bellagente, Carlos Riquelme, et al. Stable code technical report. arXiv preprint arXiv:2404.01226, 2024.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,582,505,615],\\\"page\\\":26,\\\"span\\\":[0,240]}]},{\\\"text\\\":\\\"Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks. NeurIPS , 2021.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,522,505,574],\\\"page\\\":26,\\\"span\\\":[0,359]}]},{\\\"text\\\":\\\"Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions, 2017.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,500,505,510],\\\"page\\\":26,\\\"span\\\":[0,91]}]},{\\\"text\\\":\\\"Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jer ' emy Rapin, et al. Code llama: Open foundation ' ' models for code. arXiv preprint arXiv:2308.12950, 2023.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,459,505,492],\\\"page\\\":26,\\\"span\\\":[0,238]}]},{\\\"text\\\":\\\"Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, ' ' Yossi Adi, Jingyu Liu, Tal Remez, Jer ' emy Rapin, Artyom Kozhevnikov, Ivan Evtimov, ' ' Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Defossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas ' ' Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code, 2023.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"List-item\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,385,505,450],\\\"page\\\":26,\\\"span\\\":[0,455]}]},{\\\"text\\\":\\\"Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,355,504,376],\\\"page\\\":26,\\\"span\\\":[0,149]}]},{\\\"text\\\":\\\"Noam Shazeer. Fast transformer decoding: One write-head is all you need, 2019.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,336,462,347],\\\"page\\\":26,\\\"span\\\":[0,78]}]},{\\\"text\\\":\\\"Noam Shazeer. Glu variants improve transformer, 2020.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,317,353,327],\\\"page\\\":26,\\\"span\\\":[0,53]}]},{\\\"text\\\":\\\"Yikang Shen, Zhen Guo, Tianle Cai, and Zengyi Qin. Jetmoe: Reaching llama2 performance with 0.1 m dollars. arXiv preprint arXiv:2404.07413, 2024.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,286,504,308],\\\"page\\\":26,\\\"span\\\":[0,145]}]},{\\\"text\\\":\\\"Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,245,504,278],\\\"page\\\":26,\\\"span\\\":[0,225]}]},{\\\"text\\\":\\\"Snowflake. Snowflake arctic - llm for enterprise ai. https:\\\\/\\\\/www.snowflake.com\\\\/blog\\\\/ arctic-open-efficient-foundation-language-models-snowflake\\\\/ .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"List-item\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,215,505,237],\\\"page\\\":26,\\\"span\\\":[0,146]}]},{\\\"text\\\":\\\"Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,185,505,207],\\\"page\\\":26,\\\"span\\\":[0,143]}]},{\\\"text\\\":\\\"Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Rozi ' ' ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. ' ' Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,136,506,176],\\\"page\\\":26,\\\"span\\\":[0,269]}]},{\\\"text\\\":\\\"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \\\\u0141 ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https:\\\\/\\\\/proceedings.neurips.cc\\\\/paper files\\\\/paper\\\\/2017\\\\/file\\\\/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,59,506,125],\\\"page\\\":26,\\\"span\\\":[0,457]}]},{\\\"text\\\":\\\"26\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[300,31,311,40],\\\"page\\\":26,\\\"span\\\":[0,2]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,764],\\\"page\\\":27,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"Shiqi Wang, Li Zheng, Haifeng Qian, Chenghao Yang, Zijian Wang, Varun Kumar, Mingyue Shang, Samson Tan, Baishakhi Ray, Parminder Bhatia, Ramesh Nallapati, Murali Krishna Ramanathan, Dan Roth, and Bing Xiang. Recode: Robustness evaluation of code generation models. 2022. doi: 10.48550\\\\/arXiv.2212.10264. URL https:\\\\/\\\\/arxiv.org\\\\/abs\\\\/2212.10264 .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,654,506,708],\\\"page\\\":27,\\\"span\\\":[0,341]}]},{\\\"text\\\":\\\"Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev. Helpsteer: Multi-attribute helpfulness dataset for steerlm, 2023.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"List-item\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,612,504,645],\\\"page\\\":27,\\\"span\\\":[0,243]}]},{\\\"text\\\":\\\"Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In Hal Daume III and Aarti Singh (eds.), ' ' Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 10524-10533. PMLR, 13-18 Jul 2020. URL https:\\\\/\\\\/proceedings.mlr.press\\\\/ v119\\\\/xiong20b.html .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,539,505,604],\\\"page\\\":27,\\\"span\\\":[0,449]}]},{\\\"text\\\":\\\"Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Berkeley function calling leaderboard. https:\\\\/\\\\/gorilla.cs. berkeley.edu\\\\/blogs\\\\/8 berkeley function calling leaderboard.html, 2024.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,498,506,530],\\\"page\\\":27,\\\"span\\\":[0,245]}]},{\\\"text\\\":\\\"Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284, 2023.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,457,505,490],\\\"page\\\":27,\\\"span\\\":[0,248]}]},{\\\"text\\\":\\\"Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653, 2023.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,416,505,449],\\\"page\\\":27,\\\"span\\\":[0,202]}]},{\\\"text\\\":\\\"Biao Zhang and Rico Sennrich. Root mean square layer normalization, 2019.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,397,443,408],\\\"page\\\":27,\\\"span\\\":[0,73]}]},{\\\"text\\\":\\\"Yifan Zhang. Stackmathqa: A curated collection of 2 million mathematical questions and answers sourced from stack exchange, 2024.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,368,504,389],\\\"page\\\":27,\\\"span\\\":[0,129]}]},{\\\"text\\\":\\\"27\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[300,31,311,40],\\\"page\\\":27,\\\"span\\\":[0,2]}]},{\\\"text\\\":\\\"IBM Granite Code Models\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,756,504,764],\\\"page\\\":28,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"A Programming Languages\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,696,266,709],\\\"page\\\":28,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"ABAP, Ada, Agda, Alloy, ANTLR, AppleScript, Arduino, ASP, Assembly, Augeas, Awk, Batchfile, Bison, Bluespec, C, C-sharp, C++, Clojure, CMake, COBOL, CoffeeScript, Common-Lisp, CSS, Cucumber, Cuda, Cython, Dart, Dockerfile, Eagle, Elixir, Elm, EmacsLisp, Erlang, F-sharp, FORTRAN, GLSL, GO, Gradle, GraphQL, Groovy, Haskell, Haxe, HCL, HTML, Idris, Isabelle, Java, Java-Server-Pages, JavaScript, JSON, JSON5, JSONiq, JSONLD, JSX, Julia, Jupyter, Kotlin, Lean, Literate-Agda, Literate-CoffeeScript, LiterateHaskell, Lua, Makefile, Maple, Markdown, Mathematica, Matlab, Objective-C++, OCaml, OpenCL, Pascal, Perl, PHP, PowerShell, Prolog, Protocol-Buffer, Python, Python-traceback, R, Racket, RDoc, Restructuredtext, RHTML, RMarkdown, Ruby, Rust, SAS, Scala, Scheme, Shell, Smalltalk, Solidity, SPARQL, SQL, Stan, Standard-ML, Stata, Swift, SystemVerilog, Tcl, Tcsh, Tex, Thrift, Twig, TypeScript, Verilog, VHDL, Visual-Basic, Vue, Web-OntologyLanguage, WebAssembly, XML, XSLT, Yacc, YAML, Zig\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"Text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,553,506,684],\\\"page\\\":28,\\\"span\\\":[0,990]}]},{\\\"text\\\":\\\"28\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[300,31,311,40],\\\"page\\\":28,\\\"span\\\":[0,2]}]}],\\\"figures\\\":[{\\\"prov\\\":[{\\\"bbox\\\":[109,409,502,710],\\\"page\\\":2,\\\"span\\\":[0,268]}],\\\"text\\\":\\\"Figure 1: Comparison of Granite-8B-Code (Base\\\\/Instruct) with other open source (code) LLMs of similar size on HumanEvalPack (Muennighoff et al. , 2023), spanning 3 coding tasks and 6 programming languages. See Tables 3 , 10 , 11 for more details. Best viewed in color.\\\",\\\"type\\\":\\\"figure\\\"},{\\\"prov\\\":[{\\\"bbox\\\":[130,584,483,709],\\\"page\\\":6,\\\"span\\\":[0,274]}],\\\"text\\\":\\\"Figure 2: An overview of depth upscaling (Kim et al. , 2024) for efficient training of Granite34B-Code. We utilize the 20B model after 1.6T tokens to start training of 34B model with the same code pretraining data without any changes to the training and inference framework.\\\",\\\"type\\\":\\\"figure\\\"},{\\\"prov\\\":[{\\\"bbox\\\":[111,558,500,708],\\\"page\\\":14,\\\"span\\\":[0,153]}],\\\"text\\\":\\\"Figure 3: Performance of Granite-8B-Code-Instruct, Mistral-7B-Instruct-v0.2, Gemma-7B-IT, and Llama-3-8B-Instruct on HumanEvalPack. Best viewed in color.\\\",\\\"type\\\":\\\"figure\\\"},{\\\"prov\\\":[{\\\"bbox\\\":[159,356,454,521],\\\"page\\\":19,\\\"span\\\":[0,245]}],\\\"text\\\":\\\"Figure 4: Performance of Granite Code models on Berkeley Function-Calling Leaderboard. Overall accuracy keeps increasing with increase in model sizes, indicating the advantage of large models for function calling abilities. Best viewed in color.\\\",\\\"type\\\":\\\"figure\\\"},{\\\"prov\\\":[{\\\"bbox\\\":[128,537,485,710],\\\"page\\\":20,\\\"span\\\":[0,212]}],\\\"text\\\":\\\"Figure 5: Granite-8B-Code vs CodeLlama-7B on Berkley Function-Calling Leaderboard. Granite-8B-Code (Base\\\\/Instruct) consistently outperforms CodeLlama-7B (Base\\\\/Instruct) on all three metrics. Best viewed in color.\\\",\\\"type\\\":\\\"figure\\\"}],\\\"tables\\\":[{\\\"prov\\\":[{\\\"bbox\\\":[138,300,472,432],\\\"page\\\":5,\\\"span\\\":[0,0]}],\\\"text\\\":\\\"Table 1: Model configurations for Granite Code models.\\\",\\\"type\\\":\\\"table\\\",\\\"#-cols\\\":1,\\\"#-rows\\\":11,\\\"data\\\":[[{\\\"bbox\\\":[145,420,447,428],\\\"spans\\\":[[0,0]],\\\"text\\\":\\\"Model 3B 8B 20B 34B\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]}],[{\\\"bbox\\\":[145,404,447,412],\\\"spans\\\":[[1,0]],\\\"text\\\":\\\"Batch size 2048 1024 576 532\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]}],[{\\\"bbox\\\":[144,391,449,401],\\\"spans\\\":[[2,0]],\\\"text\\\":\\\"Context length 2048 4096 8192 8192\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]}],[{\\\"bbox\\\":[144,382,449,390],\\\"spans\\\":[[3,0]],\\\"text\\\":\\\"Hidden size 2560 4096 6144 6144\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]}],[{\\\"bbox\\\":[144,371,452,379],\\\"spans\\\":[[4,0]],\\\"text\\\":\\\"FFN hidden size 10240 14336 24576 24576\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]}],[{\\\"bbox\\\":[144,361,444,368],\\\"spans\\\":[[5,0]],\\\"text\\\":\\\"Attention heads 32 32 48 48\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]}],[{\\\"bbox\\\":[144,347,458,357],\\\"spans\\\":[[6,0]],\\\"text\\\":\\\"Key-Value heads 32 (MHA) 8 (GQA) 1 (MQA) 1 (MQA)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]}],[{\\\"bbox\\\":[144,336,444,346],\\\"spans\\\":[[7,0]],\\\"text\\\":\\\"Layers 32 36 52 88\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]}],[{\\\"bbox\\\":[144,325,465,335],\\\"spans\\\":[[8,0]],\\\"text\\\":\\\"Normalization RMSNorm RMSNorm LayerNorm LayerNorm\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]}],[{\\\"bbox\\\":[144,314,449,324],\\\"spans\\\":[[9,0]],\\\"text\\\":\\\"Activation swiglu swiglu gelu gelu\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]}],[{\\\"bbox\\\":[144,306,452,313],\\\"spans\\\":[[10,0]],\\\"text\\\":\\\"Vocab size 49152 49152 49152 49152\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]}]]},{\\\"prov\\\":[{\\\"bbox\\\":[108,459,509,690],\\\"page\\\":8,\\\"span\\\":[0,0]}],\\\"text\\\":\\\"Table 2: Summary of evaluation tasks.\\\",\\\"type\\\":\\\"table\\\",\\\"#-cols\\\":2,\\\"#-rows\\\":20,\\\"data\\\":[[{\\\"spans\\\":[[0,0]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[180,678,470,685],\\\"spans\\\":[[0,1]],\\\"text\\\":\\\"Task Benchmark Reference\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]}],[{\\\"bbox\\\":[127,659,380,669],\\\"spans\\\":[[1,0]],\\\"text\\\":\\\"Multilingual code generation HumanEvalSynthesize\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"bbox\\\":[392,659,502,669],\\\"spans\\\":[[1,1]],\\\"text\\\":\\\"Muennighoff et al. (2023)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]}],[{\\\"bbox\\\":[127,648,353,658],\\\"spans\\\":[[2,0]],\\\"text\\\":\\\"Multilingual code generation MultiPL-E\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"bbox\\\":[403,649,491,658],\\\"spans\\\":[[2,1]],\\\"text\\\":\\\"Cassano et al. (2023)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]}],[{\\\"bbox\\\":[139,637,343,647],\\\"spans\\\":[[3,0]],\\\"text\\\":\\\"Python code generation MBPP\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[406,638,488,647],\\\"spans\\\":[[3,1]],\\\"text\\\":\\\"Austin et al. (2021)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]}],[{\\\"bbox\\\":[139,626,346,636],\\\"spans\\\":[[4,0]],\\\"text\\\":\\\"Python code generation MBPP+\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"bbox\\\":[411,627,483,636],\\\"spans\\\":[[4,1]],\\\"text\\\":\\\"Liu et al. (2023a)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]}],[{\\\"bbox\\\":[127,615,346,625],\\\"spans\\\":[[5,0]],\\\"text\\\":\\\"Data science code generation DS1000\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[414,616,480,625],\\\"spans\\\":[[5,1]],\\\"text\\\":\\\"Lai et al. (2023)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]}],[{\\\"bbox\\\":[118,604,355,614],\\\"spans\\\":[[6,0]],\\\"text\\\":\\\"Repository-level code generation RepoBench\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"bbox\\\":[411,605,483,614],\\\"spans\\\":[[6,1]],\\\"text\\\":\\\"Liu et al. (2023b)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]}],[{\\\"bbox\\\":[118,593,364,603],\\\"spans\\\":[[7,0]],\\\"text\\\":\\\"Repository-level code generation CrossCodeEval\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"bbox\\\":[410,593,484,603],\\\"spans\\\":[[7,1]],\\\"text\\\":\\\"Ding et al. (2023)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]}],[{\\\"bbox\\\":[114,582,366,593],\\\"spans\\\":[[8,0]],\\\"text\\\":\\\"Fill-in-the-middle code completion SantaCoder-FIM\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"bbox\\\":[411,583,484,593],\\\"spans\\\":[[8,1]],\\\"text\\\":\\\"Allal et al. (2023)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]}],[{\\\"bbox\\\":[124,572,373,582],\\\"spans\\\":[[9,0]],\\\"text\\\":\\\"Multilingual code explanation HumanEvalExplain\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[392,572,502,582],\\\"spans\\\":[[9,1]],\\\"text\\\":\\\"Muennighoff et al. (2023)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]}],[{\\\"bbox\\\":[138,561,363,571],\\\"spans\\\":[[10,0]],\\\"text\\\":\\\"Multilingual code fixing HumanEvalFix\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[392,561,502,571],\\\"spans\\\":[[10,1]],\\\"text\\\":\\\"Muennighoff et al. (2023)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]}],[{\\\"bbox\\\":[162,550,351,560],\\\"spans\\\":[[11,0]],\\\"text\\\":\\\"Code editing CanItEdit\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"bbox\\\":[403,550,491,560],\\\"spans\\\":[[11,1]],\\\"text\\\":\\\"Cassano et al. (2024)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]}],[{\\\"bbox\\\":[154,539,357,549],\\\"spans\\\":[[12,0]],\\\"text\\\":\\\"Code translation CodeLingua\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"bbox\\\":[413,539,481,549],\\\"spans\\\":[[12,1]],\\\"text\\\":\\\"Pan et al. (2024)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]}],[{\\\"bbox\\\":[157,530,351,538],\\\"spans\\\":[[13,0]],\\\"text\\\":\\\"Code execution CruxEval\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"bbox\\\":[415,528,480,538],\\\"spans\\\":[[13,1]],\\\"text\\\":\\\"Gu et al. (2024)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]}],[{\\\"bbox\\\":[156,517,345,527],\\\"spans\\\":[[14,0]],\\\"text\\\":\\\"Math reasoning MATH\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"bbox\\\":[397,517,497,527],\\\"spans\\\":[[14,1]],\\\"text\\\":\\\"Hendrycks et al. (2021)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]}],[{\\\"bbox\\\":[156,506,347,516],\\\"spans\\\":[[15,0]],\\\"text\\\":\\\"Math reasoning GSM8K\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"bbox\\\":[407,506,487,516],\\\"spans\\\":[[15,1]],\\\"text\\\":\\\"Cobbe et al. (2021)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]}],[{\\\"bbox\\\":[156,495,339,505],\\\"spans\\\":[[16,0]],\\\"text\\\":\\\"Math reasoning SAT\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"bbox\\\":[397,495,497,505],\\\"spans\\\":[[16,1]],\\\"text\\\":\\\"Azerbayev et al. (2023)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]}],[{\\\"bbox\\\":[156,484,342,494],\\\"spans\\\":[[17,0]],\\\"text\\\":\\\"Math reasoning OCW\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"bbox\\\":[395,484,500,494],\\\"spans\\\":[[17,1]],\\\"text\\\":\\\"Lewkowycz et al. (2022)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]}],[{\\\"bbox\\\":[155,473,342,483],\\\"spans\\\":[[18,0]],\\\"text\\\":\\\"Function calling BFCL\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]},{\\\"bbox\\\":[413,474,481,483],\\\"spans\\\":[[18,1]],\\\"text\\\":\\\"Yan et al. (2024)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]}],[{\\\"bbox\\\":[152,465,347,472],\\\"spans\\\":[[19,0]],\\\"text\\\":\\\"Model robustness ReCode\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]},{\\\"bbox\\\":[409,462,486,472],\\\"spans\\\":[[19,1]],\\\"text\\\":\\\"Wang et al. (2022)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]}]]},{\\\"prov\\\":[{\\\"bbox\\\":[107,205,505,668],\\\"page\\\":10,\\\"span\\\":[0,0]}],\\\"text\\\":\\\"Table 3: Pass@1 performance on HumanEvalSynthesize benchmark (Muennighoff et al. , 2023). All models are evaluated using greedy decoding with completion format for the base models, and instruction template for the instruction-tuned models.\\\",\\\"type\\\":\\\"table\\\",\\\"#-cols\\\":4,\\\"#-rows\\\":44,\\\"data\\\":[[{\\\"bbox\\\":[152,655,263,663],\\\"spans\\\":[[0,0]],\\\"text\\\":\\\"Model Prompt\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[282,655,470,663],\\\"spans\\\":[[0,1]],\\\"text\\\":\\\"Python JavaScript Java Go C++ Rust\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"spans\\\":[[0,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[481,655,498,663],\\\"spans\\\":[[0,3]],\\\"text\\\":\\\"Avg.\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]}],[{\\\"bbox\\\":[283,643,329,649],\\\"spans\\\":[[1,0],[1,1],[1,2],[1,3]],\\\"text\\\":\\\"Base Models\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,4],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"bbox\\\":[283,643,329,649],\\\"spans\\\":[[1,0],[1,1],[1,2],[1,3]],\\\"text\\\":\\\"Base Models\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[0,4],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"bbox\\\":[283,643,329,649],\\\"spans\\\":[[1,0],[1,1],[1,2],[1,3]],\\\"text\\\":\\\"Base Models\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[0,4],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"bbox\\\":[283,643,329,649],\\\"spans\\\":[[1,0],[1,1],[1,2],[1,3]],\\\"text\\\":\\\"Base Models\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[0,4],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]}],[{\\\"bbox\\\":[131,627,271,635],\\\"spans\\\":[[2,0]],\\\"text\\\":\\\"StarCoderBase-3B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"bbox\\\":[289,629,468,635],\\\"spans\\\":[[2,2]],\\\"text\\\":\\\"25.6 22.6 24.4 18.3 23.2 16.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"bbox\\\":[483,629,497,635],\\\"spans\\\":[[2,3]],\\\"text\\\":\\\"21.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]}],[{\\\"bbox\\\":[138,618,271,626],\\\"spans\\\":[[3,0]],\\\"text\\\":\\\"StableCode-3B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"spans\\\":[[3,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[289,620,468,626],\\\"spans\\\":[[3,2]],\\\"text\\\":\\\"24.4 32.3 34.1 21.3 33.5 21.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[483,620,497,626],\\\"spans\\\":[[3,3]],\\\"text\\\":\\\"27.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]}],[{\\\"bbox\\\":[138,608,271,617],\\\"spans\\\":[[4,0]],\\\"text\\\":\\\"StarCoder2-3B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"bbox\\\":[289,611,347,617],\\\"spans\\\":[[4,1]],\\\"text\\\":\\\"27.4 36.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"bbox\\\":[372,611,468,616],\\\"spans\\\":[[4,2]],\\\"text\\\":\\\"42.1  23.8  36.6 24.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"bbox\\\":[482,611,497,617],\\\"spans\\\":[[4,3]],\\\"text\\\":\\\"31.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]}],[{\\\"bbox\\\":[134,599,271,607],\\\"spans\\\":[[5,0]],\\\"text\\\":\\\"CodeGemma-2B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[289,601,347,607],\\\"spans\\\":[[5,1]],\\\"text\\\":\\\"39.0 37.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[372,601,468,607],\\\"spans\\\":[[5,2]],\\\"text\\\":\\\"37.8 13.4 33.5 20.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[482,601,497,607],\\\"spans\\\":[[5,3]],\\\"text\\\":\\\"30.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]}],[{\\\"bbox\\\":[124,590,271,598],\\\"spans\\\":[[6,0]],\\\"text\\\":\\\"Granite-3B-Code-Base Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"bbox\\\":[289,592,386,598],\\\"spans\\\":[[6,1]],\\\"text\\\":\\\"36.6 37.2 40.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"bbox\\\":[399,592,468,598],\\\"spans\\\":[[6,2]],\\\"text\\\":\\\"26.2  35.4 22.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"bbox\\\":[483,592,497,598],\\\"spans\\\":[[6,3]],\\\"text\\\":\\\"33.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]}],[{\\\"bbox\\\":[131,576,271,584],\\\"spans\\\":[[7,0]],\\\"text\\\":\\\"StarCoderBase-7B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"bbox\\\":[289,578,468,584],\\\"spans\\\":[[7,2]],\\\"text\\\":\\\"26.8 28.7 31.7 22.6 28.0 22.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"bbox\\\":[483,578,497,584],\\\"spans\\\":[[7,3]],\\\"text\\\":\\\"26.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]}],[{\\\"bbox\\\":[137,566,271,575],\\\"spans\\\":[[8,0]],\\\"text\\\":\\\"CodeLlama-7B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"bbox\\\":[289,569,468,575],\\\"spans\\\":[[8,2]],\\\"text\\\":\\\"35.4 36.0 39.0 21.3 31.1 24.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"bbox\\\":[482,569,497,575],\\\"spans\\\":[[8,3]],\\\"text\\\":\\\"31.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]}],[{\\\"spans\\\":[[9,0]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[289,559,386,565],\\\"spans\\\":[[9,1]],\\\"text\\\":\\\"38.4 43.3 48.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[399,559,468,565],\\\"spans\\\":[[9,2]],\\\"text\\\":\\\"31.7  38.4 24.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"spans\\\":[[9,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]}],[{\\\"bbox\\\":[138,557,271,565],\\\"spans\\\":[[10,0]],\\\"text\\\":\\\"StarCoder2-7B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"spans\\\":[[10,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"spans\\\":[[10,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[482,559,497,565],\\\"spans\\\":[[10,3]],\\\"text\\\":\\\"37.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]}],[{\\\"bbox\\\":[134,548,271,556],\\\"spans\\\":[[11,0]],\\\"text\\\":\\\"CodeGemma-7B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"spans\\\":[[11,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"bbox\\\":[289,550,468,556],\\\"spans\\\":[[11,2]],\\\"text\\\":\\\"41.5 48.8 54.9 26.8  44.5  32.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"bbox\\\":[482,550,497,556],\\\"spans\\\":[[11,3]],\\\"text\\\":\\\"41.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]}],[{\\\"bbox\\\":[124,538,271,547],\\\"spans\\\":[[12,0]],\\\"text\\\":\\\"Granite-8B-Code-Base Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"spans\\\":[[12,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"bbox\\\":[289,541,468,547],\\\"spans\\\":[[12,2]],\\\"text\\\":\\\"43.9 52.4 56.1 31.7  43.9  32.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"bbox\\\":[482,541,497,546],\\\"spans\\\":[[12,3]],\\\"text\\\":\\\"43.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]}],[{\\\"bbox\\\":[129,524,271,533],\\\"spans\\\":[[13,0]],\\\"text\\\":\\\"StarCoderBase-15B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"spans\\\":[[13,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"bbox\\\":[289,526,468,532],\\\"spans\\\":[[13,2]],\\\"text\\\":\\\"32.3 36.6 40.2 25.6 31.1 25.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"bbox\\\":[482,526,497,532],\\\"spans\\\":[[13,3]],\\\"text\\\":\\\"31.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]}],[{\\\"bbox\\\":[135,515,271,523],\\\"spans\\\":[[14,0]],\\\"text\\\":\\\"CodeLlama-13B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"spans\\\":[[14,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"bbox\\\":[289,517,468,523],\\\"spans\\\":[[14,2]],\\\"text\\\":\\\"41.5 42.7 51.8 26.8 40.9 23.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"bbox\\\":[482,517,497,523],\\\"spans\\\":[[14,3]],\\\"text\\\":\\\"37.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]}],[{\\\"bbox\\\":[136,506,271,514],\\\"spans\\\":[[15,0]],\\\"text\\\":\\\"StarCoder2-15B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"bbox\\\":[289,508,386,514],\\\"spans\\\":[[15,1]],\\\"text\\\":\\\"44.5 47.0 51.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"bbox\\\":[399,508,468,514],\\\"spans\\\":[[15,2]],\\\"text\\\":\\\"33.5 50.0 39.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"bbox\\\":[482,508,497,514],\\\"spans\\\":[[15,3]],\\\"text\\\":\\\"44.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]}],[{\\\"bbox\\\":[121,496,271,505],\\\"spans\\\":[[16,0]],\\\"text\\\":\\\"Granite-20B-Code-Base Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"bbox\\\":[289,499,386,499],\\\"spans\\\":[[16,1]],\\\"text\\\":\\\"48.2 50.0 59.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"bbox\\\":[289,490,468,504],\\\"spans\\\":[[16,2]],\\\"text\\\":\\\"32.3 40.9 35.4  47.4 48.2 45.6 34.1 47.0 37.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"bbox\\\":[482,498,497,504],\\\"spans\\\":[[16,3]],\\\"text\\\":\\\"44.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]}],[{\\\"bbox\\\":[121,473,271,491],\\\"spans\\\":[[17,0]],\\\"text\\\":\\\"CodeLlama-34B Completion  Granite-34B-Code-Base Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"spans\\\":[[17,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"bbox\\\":[289,475,468,481],\\\"spans\\\":[[17,2]],\\\"text\\\":\\\"48.2 54.9 61.6 40.2 50.0 39.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"bbox\\\":[482,475,497,490],\\\"spans\\\":[[17,3]],\\\"text\\\":\\\"43.3 49.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]}],[{\\\"bbox\\\":[135,464,271,472],\\\"spans\\\":[[18,0]],\\\"text\\\":\\\"CodeLlama-70B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]},{\\\"spans\\\":[[18,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]},{\\\"bbox\\\":[289,466,468,472],\\\"spans\\\":[[18,2]],\\\"text\\\":\\\"55.5 55.5 65.2 40.9 55.5 43.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]},{\\\"bbox\\\":[482,466,497,472],\\\"spans\\\":[[18,3]],\\\"text\\\":\\\"52.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]}],[{\\\"bbox\\\":[144,450,271,458],\\\"spans\\\":[[19,0]],\\\"text\\\":\\\"Gemma-2B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]},{\\\"spans\\\":[[19,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]},{\\\"bbox\\\":[289,452,466,458],\\\"spans\\\":[[19,2]],\\\"text\\\":\\\"20.1 23.2 19.5 13.4 18.3 8.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]},{\\\"bbox\\\":[483,452,497,458],\\\"spans\\\":[[19,3]],\\\"text\\\":\\\"17.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]}],[{\\\"bbox\\\":[144,440,271,449],\\\"spans\\\":[[20,0]],\\\"text\\\":\\\"Gemma-7B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":20,\\\"row-header\\\":false,\\\"row-span\\\":[20,21]},{\\\"spans\\\":[[20,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":20,\\\"row-header\\\":false,\\\"row-span\\\":[20,21]},{\\\"bbox\\\":[289,442,468,448],\\\"spans\\\":[[20,2]],\\\"text\\\":\\\"33.5 41.5 43.9 26.2 35.4 26.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":20,\\\"row-header\\\":false,\\\"row-span\\\":[20,21]},{\\\"bbox\\\":[482,442,497,448],\\\"spans\\\":[[20,3]],\\\"text\\\":\\\"34.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":20,\\\"row-header\\\":false,\\\"row-span\\\":[20,21]}],[{\\\"bbox\\\":[136,431,271,439],\\\"spans\\\":[[21,0]],\\\"text\\\":\\\"Mistral-7B-v0.2 Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":21,\\\"row-header\\\":false,\\\"row-span\\\":[21,22]},{\\\"spans\\\":[[21,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":21,\\\"row-header\\\":false,\\\"row-span\\\":[21,22]},{\\\"bbox\\\":[289,433,468,439],\\\"spans\\\":[[21,2]],\\\"text\\\":\\\"32.9 34.1 36.6 22.6 30.5 18.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":21,\\\"row-header\\\":false,\\\"row-span\\\":[21,22]},{\\\"bbox\\\":[483,433,497,439],\\\"spans\\\":[[21,3]],\\\"text\\\":\\\"29.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":21,\\\"row-header\\\":false,\\\"row-span\\\":[21,22]}],[{\\\"bbox\\\":[132,422,271,430],\\\"spans\\\":[[22,0]],\\\"text\\\":\\\"Mixtral-8x7B-v0.1 Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":22,\\\"row-header\\\":false,\\\"row-span\\\":[22,23]},{\\\"spans\\\":[[22,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":22,\\\"row-header\\\":false,\\\"row-span\\\":[22,23]},{\\\"bbox\\\":[289,424,468,430],\\\"spans\\\":[[22,2]],\\\"text\\\":\\\"42.1 53.7 52.4 33.5 42.7 35.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":22,\\\"row-header\\\":false,\\\"row-span\\\":[22,23]},{\\\"bbox\\\":[482,424,497,430],\\\"spans\\\":[[22,3]],\\\"text\\\":\\\"43.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":22,\\\"row-header\\\":false,\\\"row-span\\\":[22,23]}],[{\\\"bbox\\\":[130,412,271,421],\\\"spans\\\":[[23,0]],\\\"text\\\":\\\"Mixtral-8x22B-v0.1 Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":23,\\\"row-header\\\":false,\\\"row-span\\\":[23,24]},{\\\"spans\\\":[[23,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":23,\\\"row-header\\\":false,\\\"row-span\\\":[23,24]},{\\\"bbox\\\":[289,414,468,420],\\\"spans\\\":[[23,2]],\\\"text\\\":\\\"51.2 57.9 64.6 40.9 57.3 32.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":23,\\\"row-header\\\":false,\\\"row-span\\\":[23,24]},{\\\"bbox\\\":[482,414,497,420],\\\"spans\\\":[[23,3]],\\\"text\\\":\\\"50.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":23,\\\"row-header\\\":false,\\\"row-span\\\":[23,24]}],[{\\\"bbox\\\":[143,403,271,411],\\\"spans\\\":[[24,0]],\\\"text\\\":\\\"Llama-3-8B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":24,\\\"row-header\\\":false,\\\"row-span\\\":[24,25]},{\\\"spans\\\":[[24,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":24,\\\"row-header\\\":false,\\\"row-span\\\":[24,25]},{\\\"bbox\\\":[289,405,468,411],\\\"spans\\\":[[24,2]],\\\"text\\\":\\\"26.2 37.8 40.2 11.0 37.2 21.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":24,\\\"row-header\\\":false,\\\"row-span\\\":[24,25]},{\\\"bbox\\\":[483,405,497,411],\\\"spans\\\":[[24,3]],\\\"text\\\":\\\"29.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":24,\\\"row-header\\\":false,\\\"row-span\\\":[24,25]}],[{\\\"bbox\\\":[141,394,271,402],\\\"spans\\\":[[25,0]],\\\"text\\\":\\\"Llama-3-70B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":25,\\\"row-header\\\":false,\\\"row-span\\\":[25,26]},{\\\"bbox\\\":[289,396,307,402],\\\"spans\\\":[[25,1]],\\\"text\\\":\\\"25.0 \\\\u22c6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":25,\\\"row-header\\\":false,\\\"row-span\\\":[25,26]},{\\\"bbox\\\":[333,396,468,402],\\\"spans\\\":[[25,2]],\\\"text\\\":\\\"51.2 62.2 21.3 53.7 37.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":25,\\\"row-header\\\":false,\\\"row-span\\\":[25,26]},{\\\"bbox\\\":[482,396,497,402],\\\"spans\\\":[[25,3]],\\\"text\\\":\\\"41.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":25,\\\"row-header\\\":false,\\\"row-span\\\":[25,26]}],[{\\\"bbox\\\":[277,382,335,388],\\\"spans\\\":[[26,0],[26,1],[26,2],[26,3]],\\\"text\\\":\\\"Instruct Models\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,4],\\\"row\\\":26,\\\"row-header\\\":false,\\\"row-span\\\":[26,27]},{\\\"bbox\\\":[277,382,335,388],\\\"spans\\\":[[26,0],[26,1],[26,2],[26,3]],\\\"text\\\":\\\"Instruct Models\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[0,4],\\\"row\\\":26,\\\"row-header\\\":false,\\\"row-span\\\":[26,27]},{\\\"bbox\\\":[277,382,335,388],\\\"spans\\\":[[26,0],[26,1],[26,2],[26,3]],\\\"text\\\":\\\"Instruct Models\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[0,4],\\\"row\\\":26,\\\"row-header\\\":false,\\\"row-span\\\":[26,27]},{\\\"bbox\\\":[277,382,335,388],\\\"spans\\\":[[26,0],[26,1],[26,2],[26,3]],\\\"text\\\":\\\"Instruct Models\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[0,4],\\\"row\\\":26,\\\"row-header\\\":false,\\\"row-span\\\":[26,27]}],[{\\\"bbox\\\":[129,368,269,374],\\\"spans\\\":[[27,0]],\\\"text\\\":\\\"CodeGemma-7B-IT Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":27,\\\"row-header\\\":false,\\\"row-span\\\":[27,28]},{\\\"spans\\\":[[27,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":27,\\\"row-header\\\":false,\\\"row-span\\\":[27,28]},{\\\"bbox\\\":[289,368,468,374],\\\"spans\\\":[[27,2]],\\\"text\\\":\\\"48.4 46.0 48.4 28.6 42.2 36.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":27,\\\"row-header\\\":false,\\\"row-span\\\":[27,28]},{\\\"bbox\\\":[482,368,497,374],\\\"spans\\\":[[27,3]],\\\"text\\\":\\\"41.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":27,\\\"row-header\\\":false,\\\"row-span\\\":[27,28]}],[{\\\"bbox\\\":[121,358,269,365],\\\"spans\\\":[[28,0]],\\\"text\\\":\\\"CodeLlama-7B-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":28,\\\"row-header\\\":false,\\\"row-span\\\":[28,29]},{\\\"spans\\\":[[28,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":28,\\\"row-header\\\":false,\\\"row-span\\\":[28,29]},{\\\"bbox\\\":[289,358,468,364],\\\"spans\\\":[[28,2]],\\\"text\\\":\\\"47.0 39.0 45.7 26.8 38.4 30.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":28,\\\"row-header\\\":false,\\\"row-span\\\":[28,29]},{\\\"bbox\\\":[482,358,497,364],\\\"spans\\\":[[28,3]],\\\"text\\\":\\\"37.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":28,\\\"row-header\\\":false,\\\"row-span\\\":[28,29]}],[{\\\"bbox\\\":[119,349,269,355],\\\"spans\\\":[[29,0]],\\\"text\\\":\\\"CodeLlama-13B-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":29,\\\"row-header\\\":false,\\\"row-span\\\":[29,30]},{\\\"spans\\\":[[29,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":29,\\\"row-header\\\":false,\\\"row-span\\\":[29,30]},{\\\"bbox\\\":[289,349,468,355],\\\"spans\\\":[[29,2]],\\\"text\\\":\\\"50.6 45.1 47.0 29.9 37.8 26.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":29,\\\"row-header\\\":false,\\\"row-span\\\":[29,30]},{\\\"bbox\\\":[482,349,497,355],\\\"spans\\\":[[29,3]],\\\"text\\\":\\\"39.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":29,\\\"row-header\\\":false,\\\"row-span\\\":[29,30]}],[{\\\"bbox\\\":[119,340,269,346],\\\"spans\\\":[[30,0]],\\\"text\\\":\\\"CodeLlama-34B-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":30,\\\"row-header\\\":false,\\\"row-span\\\":[30,31]},{\\\"spans\\\":[[30,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":30,\\\"row-header\\\":false,\\\"row-span\\\":[30,31]},{\\\"bbox\\\":[289,340,468,346],\\\"spans\\\":[[30,2]],\\\"text\\\":\\\"48.8 48.8 48.8 26.2 42.7 32.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":30,\\\"row-header\\\":false,\\\"row-span\\\":[30,31]},{\\\"bbox\\\":[482,340,497,346],\\\"spans\\\":[[30,3]],\\\"text\\\":\\\"41.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":30,\\\"row-header\\\":false,\\\"row-span\\\":[30,31]}],[{\\\"bbox\\\":[119,330,269,337],\\\"spans\\\":[[31,0]],\\\"text\\\":\\\"CodeLlama-70B-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":31,\\\"row-header\\\":false,\\\"row-span\\\":[31,32]},{\\\"spans\\\":[[31,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":31,\\\"row-header\\\":false,\\\"row-span\\\":[31,32]},{\\\"bbox\\\":[289,330,468,336],\\\"spans\\\":[[31,2]],\\\"text\\\":\\\"67.8 61.6 70.7 51.2 60.4 41.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":31,\\\"row-header\\\":false,\\\"row-span\\\":[31,32]},{\\\"bbox\\\":[482,330,497,336],\\\"spans\\\":[[31,3]],\\\"text\\\":\\\"58.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":31,\\\"row-header\\\":false,\\\"row-span\\\":[31,32]}],[{\\\"bbox\\\":[136,321,269,327],\\\"spans\\\":[[32,0]],\\\"text\\\":\\\"OctoCoder-15B Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":32,\\\"row-header\\\":false,\\\"row-span\\\":[32,33]},{\\\"spans\\\":[[32,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":32,\\\"row-header\\\":false,\\\"row-span\\\":[32,33]},{\\\"bbox\\\":[289,321,468,327],\\\"spans\\\":[[32,2]],\\\"text\\\":\\\"43.9 39.0 39.6 30.5 36.0 25.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":32,\\\"row-header\\\":false,\\\"row-span\\\":[32,33]},{\\\"bbox\\\":[482,321,497,327],\\\"spans\\\":[[32,3]],\\\"text\\\":\\\"35.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":32,\\\"row-header\\\":false,\\\"row-span\\\":[32,33]}],[{\\\"bbox\\\":[118,307,269,313],\\\"spans\\\":[[33,0]],\\\"text\\\":\\\"Granite-3b-Code-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":33,\\\"row-header\\\":false,\\\"row-span\\\":[33,34]},{\\\"spans\\\":[[33,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":33,\\\"row-header\\\":false,\\\"row-span\\\":[33,34]},{\\\"bbox\\\":[289,307,468,313],\\\"spans\\\":[[33,2]],\\\"text\\\":\\\"51.2 43.9 41.5 31.7 40.2 29.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":33,\\\"row-header\\\":false,\\\"row-span\\\":[33,34]},{\\\"bbox\\\":[482,307,497,313],\\\"spans\\\":[[33,3]],\\\"text\\\":\\\"39.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":33,\\\"row-header\\\":false,\\\"row-span\\\":[33,34]}],[{\\\"bbox\\\":[118,298,269,304],\\\"spans\\\":[[34,0]],\\\"text\\\":\\\"Granite-8b-Code-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":34,\\\"row-header\\\":false,\\\"row-span\\\":[34,35]},{\\\"spans\\\":[[34,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":34,\\\"row-header\\\":false,\\\"row-span\\\":[34,35]},{\\\"bbox\\\":[289,298,468,304],\\\"spans\\\":[[34,2]],\\\"text\\\":\\\"57.9 52.4 58.5 43.3 48.2 37.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":34,\\\"row-header\\\":false,\\\"row-span\\\":[34,35]},{\\\"bbox\\\":[482,298,497,304],\\\"spans\\\":[[34,3]],\\\"text\\\":\\\"49.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":34,\\\"row-header\\\":false,\\\"row-span\\\":[34,35]}],[{\\\"bbox\\\":[116,288,269,295],\\\"spans\\\":[[35,0]],\\\"text\\\":\\\"Granite-20B-Code-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":35,\\\"row-header\\\":false,\\\"row-span\\\":[35,36]},{\\\"spans\\\":[[35,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":35,\\\"row-header\\\":false,\\\"row-span\\\":[35,36]},{\\\"bbox\\\":[289,288,468,294],\\\"spans\\\":[[35,2]],\\\"text\\\":\\\"60.4 53.7 58.5 42.1 45.7 42.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":35,\\\"row-header\\\":false,\\\"row-span\\\":[35,36]},{\\\"bbox\\\":[482,288,497,294],\\\"spans\\\":[[35,3]],\\\"text\\\":\\\"50.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":35,\\\"row-header\\\":false,\\\"row-span\\\":[35,36]}],[{\\\"bbox\\\":[116,279,269,285],\\\"spans\\\":[[36,0]],\\\"text\\\":\\\"Granite-34B-Code-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":36,\\\"row-header\\\":false,\\\"row-span\\\":[36,37]},{\\\"spans\\\":[[36,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":36,\\\"row-header\\\":false,\\\"row-span\\\":[36,37]},{\\\"bbox\\\":[289,279,468,285],\\\"spans\\\":[[36,2]],\\\"text\\\":\\\"62.2 56.7 62.8 47.6 57.9 41.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":36,\\\"row-header\\\":false,\\\"row-span\\\":[36,37]},{\\\"bbox\\\":[482,279,497,285],\\\"spans\\\":[[36,3]],\\\"text\\\":\\\"54.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":36,\\\"row-header\\\":false,\\\"row-span\\\":[36,37]}],[{\\\"bbox\\\":[139,265,269,271],\\\"spans\\\":[[37,0]],\\\"text\\\":\\\"Gemma-2B-IT Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":37,\\\"row-header\\\":false,\\\"row-span\\\":[37,38]},{\\\"spans\\\":[[37,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":37,\\\"row-header\\\":false,\\\"row-span\\\":[37,38]},{\\\"bbox\\\":[289,265,466,271],\\\"spans\\\":[[37,2]],\\\"text\\\":\\\"17.7 13.4 10.4 7.3 17.7 2.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":37,\\\"row-header\\\":false,\\\"row-span\\\":[37,38]},{\\\"bbox\\\":[483,265,497,271],\\\"spans\\\":[[37,3]],\\\"text\\\":\\\"11.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":37,\\\"row-header\\\":false,\\\"row-span\\\":[37,38]}],[{\\\"bbox\\\":[139,256,269,262],\\\"spans\\\":[[38,0]],\\\"text\\\":\\\"Gemma-7B-IT Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":38,\\\"row-header\\\":false,\\\"row-span\\\":[38,39]},{\\\"spans\\\":[[38,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":38,\\\"row-header\\\":false,\\\"row-span\\\":[38,39]},{\\\"bbox\\\":[289,256,466,262],\\\"spans\\\":[[38,2]],\\\"text\\\":\\\"28.7 17.1 29.9 18.3 18.9 6.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":38,\\\"row-header\\\":false,\\\"row-span\\\":[38,39]},{\\\"bbox\\\":[483,256,497,262],\\\"spans\\\":[[38,3]],\\\"text\\\":\\\"19.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":38,\\\"row-header\\\":false,\\\"row-span\\\":[38,39]}],[{\\\"bbox\\\":[120,246,269,253],\\\"spans\\\":[[39,0]],\\\"text\\\":\\\"Mistral-7B-Instruct-v0.2 Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":39,\\\"row-header\\\":false,\\\"row-span\\\":[39,40]},{\\\"spans\\\":[[39,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":39,\\\"row-header\\\":false,\\\"row-span\\\":[39,40]},{\\\"bbox\\\":[289,246,468,252],\\\"spans\\\":[[39,2]],\\\"text\\\":\\\"39.6 32.9 36.6 22.0 33.5 19.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":39,\\\"row-header\\\":false,\\\"row-span\\\":[39,40]},{\\\"bbox\\\":[482,246,497,252],\\\"spans\\\":[[39,3]],\\\"text\\\":\\\"30.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":39,\\\"row-header\\\":false,\\\"row-span\\\":[39,40]}],[{\\\"bbox\\\":[116,237,269,243],\\\"spans\\\":[[40,0]],\\\"text\\\":\\\"Mixtral-8x7B-Instruct-v0.1 Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":40,\\\"row-header\\\":false,\\\"row-span\\\":[40,41]},{\\\"spans\\\":[[40,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":40,\\\"row-header\\\":false,\\\"row-span\\\":[40,41]},{\\\"bbox\\\":[289,237,468,243],\\\"spans\\\":[[40,2]],\\\"text\\\":\\\"52.4 53.0 56.1 38.4 54.9 35.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":40,\\\"row-header\\\":false,\\\"row-span\\\":[40,41]},{\\\"bbox\\\":[482,237,497,243],\\\"spans\\\":[[40,3]],\\\"text\\\":\\\"48.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":40,\\\"row-header\\\":false,\\\"row-span\\\":[40,41]}],[{\\\"bbox\\\":[114,228,269,234],\\\"spans\\\":[[41,0]],\\\"text\\\":\\\"Mixtral-8x22B-Instruct-v0.1 Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":41,\\\"row-header\\\":false,\\\"row-span\\\":[41,42]},{\\\"spans\\\":[[41,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":41,\\\"row-header\\\":false,\\\"row-span\\\":[41,42]},{\\\"bbox\\\":[289,228,468,234],\\\"spans\\\":[[41,2]],\\\"text\\\":\\\"70.7 69.5 75.6 55.5 69.5 48.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":41,\\\"row-header\\\":false,\\\"row-span\\\":[41,42]},{\\\"bbox\\\":[483,228,497,234],\\\"spans\\\":[[41,3]],\\\"text\\\":\\\"64.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":41,\\\"row-header\\\":false,\\\"row-span\\\":[41,42]}],[{\\\"bbox\\\":[128,218,269,225],\\\"spans\\\":[[42,0]],\\\"text\\\":\\\"Llama-3-8B-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":42,\\\"row-header\\\":false,\\\"row-span\\\":[42,43]},{\\\"spans\\\":[[42,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":42,\\\"row-header\\\":false,\\\"row-span\\\":[42,43]},{\\\"bbox\\\":[289,218,468,224],\\\"spans\\\":[[42,2]],\\\"text\\\":\\\"60.4 30.5 30.5 22.6 46.3 31.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":42,\\\"row-header\\\":false,\\\"row-span\\\":[42,43]},{\\\"bbox\\\":[482,218,497,224],\\\"spans\\\":[[42,3]],\\\"text\\\":\\\"36.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":42,\\\"row-header\\\":false,\\\"row-span\\\":[42,43]}],[{\\\"bbox\\\":[126,209,269,215],\\\"spans\\\":[[43,0]],\\\"text\\\":\\\"Llama-3-70B-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":43,\\\"row-header\\\":false,\\\"row-span\\\":[43,44]},{\\\"spans\\\":[[43,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":43,\\\"row-header\\\":false,\\\"row-span\\\":[43,44]},{\\\"bbox\\\":[289,209,468,215],\\\"spans\\\":[[43,2]],\\\"text\\\":\\\"76.2 69.5 76.2 51.8 65.2 54.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":43,\\\"row-header\\\":false,\\\"row-span\\\":[43,44]},{\\\"bbox\\\":[483,209,497,215],\\\"spans\\\":[[43,3]],\\\"text\\\":\\\"65.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":43,\\\"row-header\\\":false,\\\"row-span\\\":[43,44]}]]},{\\\"prov\\\":[{\\\"bbox\\\":[109,322,504,679],\\\"page\\\":11,\\\"span\\\":[0,0]}],\\\"text\\\":\\\"Table 4: Pass@1 results on MultiPL-E averaged over 50 samples for each problem. All models are evaluated at temperature 0.2 and top-p 0.95.\\\",\\\"type\\\":\\\"table\\\",\\\"#-cols\\\":4,\\\"#-rows\\\":33,\\\"data\\\":[[{\\\"bbox\\\":[146,669,170,675],\\\"spans\\\":[[0,0]],\\\"text\\\":\\\"Model\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[204,667,487,675],\\\"spans\\\":[[0,1]],\\\"text\\\":\\\"C++ C# D Go Java Julia JavaScript Lua PHP\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"spans\\\":[[0,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"spans\\\":[[0,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]}],[{\\\"bbox\\\":[125,655,485,662],\\\"spans\\\":[[1,0]],\\\"text\\\":\\\"StarCoderBase-3B 19.9 13.0 12.3 13.3 15.0 16.6 16.7 16.8 17.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]}],[{\\\"bbox\\\":[131,646,485,652],\\\"spans\\\":[[2,0]],\\\"text\\\":\\\"StableCode-3B 31.5 15.0 12.3 19.7 23.9 24.9 26.0 23.7 23.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]}],[{\\\"bbox\\\":[131,637,184,643],\\\"spans\\\":[[3,0]],\\\"text\\\":\\\"StarCoder2-3B\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[204,637,485,643],\\\"spans\\\":[[3,1]],\\\"text\\\":\\\"32.8 23.4 24.5 24.2 27.0 27.5 29.3 27.6 28.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"spans\\\":[[3,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"spans\\\":[[3,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]}],[{\\\"bbox\\\":[128,628,485,634],\\\"spans\\\":[[4,0]],\\\"text\\\":\\\"CodeGemma-2B 29.4 18.6 16.8 22.0 19.4 12.5 15.6 11.9 12.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]}],[{\\\"bbox\\\":[117,619,485,625],\\\"spans\\\":[[5,0]],\\\"text\\\":\\\"Granite-3B-Code-Base 31.6 21.5 22.8 22.7 25.7 27.0 27.2 26.8 27.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"spans\\\":[[5,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"spans\\\":[[5,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"spans\\\":[[5,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]}],[{\\\"bbox\\\":[125,605,485,612],\\\"spans\\\":[[6,0]],\\\"text\\\":\\\"StarCoderBase-7B 24.0 19.6 16.3 19.8 19.5 21.7 21.6 21.9 22.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]}],[{\\\"bbox\\\":[131,596,486,602],\\\"spans\\\":[[7,0]],\\\"text\\\":\\\"CodeLlama-7B 29.0 21.6 20.5 21.2 24.4 26.2 26.2 26.9 26.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]}],[{\\\"bbox\\\":[131,587,485,593],\\\"spans\\\":[[8,0]],\\\"text\\\":\\\"StarCoder2-7B 39.1 24.7 27.6 22.2 30.5 29.6 31.8 29.8 30.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]}],[{\\\"bbox\\\":[128,578,244,584],\\\"spans\\\":[[9,0]],\\\"text\\\":\\\"CodeGemma-7B 43.7  28.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[257,578,336,584],\\\"spans\\\":[[9,1]],\\\"text\\\":\\\"27.6 27.9 31.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[351,578,404,584],\\\"spans\\\":[[9,2]],\\\"text\\\":\\\"34.4  35.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[430,578,486,584],\\\"spans\\\":[[9,3]],\\\"text\\\":\\\"35.3 35.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]}],[{\\\"bbox\\\":[117,569,244,575],\\\"spans\\\":[[10,0]],\\\"text\\\":\\\"Granite-8B-Code-Base 44.3  21.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[257,569,336,574],\\\"spans\\\":[[10,1]],\\\"text\\\":\\\"30.2 28.0 33.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[351,569,404,575],\\\"spans\\\":[[10,2]],\\\"text\\\":\\\"33.7  35.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[430,569,485,575],\\\"spans\\\":[[10,3]],\\\"text\\\":\\\"33.4 33.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]}],[{\\\"bbox\\\":[123,555,485,562],\\\"spans\\\":[[11,0]],\\\"text\\\":\\\"StarCoderBase-15B 30.2 20.6 20.4 22.0 22.9 24.6 25.2 24.9 25.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"spans\\\":[[11,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"spans\\\":[[11,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"spans\\\":[[11,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]}],[{\\\"bbox\\\":[129,546,485,552],\\\"spans\\\":[[12,0]],\\\"text\\\":\\\"CodeLlama-13B 38.8 24.5 27.3 26.7 30.4 31.7 32.5 31.7 31.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"spans\\\":[[12,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"spans\\\":[[12,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"spans\\\":[[12,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]}],[{\\\"bbox\\\":[129,537,271,543],\\\"spans\\\":[[13,0]],\\\"text\\\":\\\"StarCoder2-15B  47.4 31.7 35.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"bbox\\\":[290,537,305,543],\\\"spans\\\":[[13,1]],\\\"text\\\":\\\"27.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"bbox\\\":[322,537,444,543],\\\"spans\\\":[[13,2]],\\\"text\\\":\\\"36.6 37.5  38.5  38.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"bbox\\\":[471,537,485,543],\\\"spans\\\":[[13,3]],\\\"text\\\":\\\"31.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]}],[{\\\"bbox\\\":[115,528,271,534],\\\"spans\\\":[[14,0]],\\\"text\\\":\\\"Granite-20B-Code-Base 43.3 30.6 16.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"bbox\\\":[291,528,305,534],\\\"spans\\\":[[14,1]],\\\"text\\\":\\\"29.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"bbox\\\":[322,528,444,534],\\\"spans\\\":[[14,2]],\\\"text\\\":\\\"35.9 31.8  38.9  30.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"bbox\\\":[471,528,485,534],\\\"spans\\\":[[14,3]],\\\"text\\\":\\\"38.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]}],[{\\\"bbox\\\":[129,515,271,521],\\\"spans\\\":[[15,0]],\\\"text\\\":\\\"CodeLlama-34B 45.9 31.0  30.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"bbox\\\":[290,515,305,521],\\\"spans\\\":[[15,1]],\\\"text\\\":\\\"28.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"bbox\\\":[322,515,365,520],\\\"spans\\\":[[15,2]],\\\"text\\\":\\\"35.2 36.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"bbox\\\":[390,515,485,521],\\\"spans\\\":[[15,3]],\\\"text\\\":\\\"37.0 36.4 37.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]}],[{\\\"bbox\\\":[115,505,272,512],\\\"spans\\\":[[16,0]],\\\"text\\\":\\\"Granite-34B-Code-Base 46.7 32.8  18.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"bbox\\\":[291,505,305,511],\\\"spans\\\":[[16,1]],\\\"text\\\":\\\"33.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"bbox\\\":[322,505,365,511],\\\"spans\\\":[[16,2]],\\\"text\\\":\\\"33.3 31.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"bbox\\\":[390,505,485,511],\\\"spans\\\":[[16,3]],\\\"text\\\":\\\"44.5 38.2 42.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]}],[{\\\"bbox\\\":[146,490,170,496],\\\"spans\\\":[[17,0]],\\\"text\\\":\\\"Model\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"spans\\\":[[17,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"spans\\\":[[17,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"bbox\\\":[204,488,499,496],\\\"spans\\\":[[17,3]],\\\"text\\\":\\\"Perl R Ruby Racket Rust Scala Bash Swift TypeScript\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]}],[{\\\"bbox\\\":[131,467,485,474],\\\"spans\\\":[[18,0]],\\\"text\\\":\\\"StableCode-3B 9.7 22.5 18.7 20.7 19.1 14.6 8.4 17.5 29.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]},{\\\"spans\\\":[[18,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]},{\\\"spans\\\":[[18,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]},{\\\"spans\\\":[[18,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]}],[{\\\"bbox\\\":[131,458,218,465],\\\"spans\\\":[[19,0]],\\\"text\\\":\\\"StarCoder2-3B 13.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]},{\\\"bbox\\\":[230,458,336,464],\\\"spans\\\":[[19,1]],\\\"text\\\":\\\"26.7 25.2 24.6 25.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]},{\\\"bbox\\\":[351,458,365,464],\\\"spans\\\":[[19,2]],\\\"text\\\":\\\"21.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]},{\\\"bbox\\\":[390,458,485,464],\\\"spans\\\":[[19,3]],\\\"text\\\":\\\"12.5 23.2 35.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]}],[{\\\"bbox\\\":[128,449,483,455],\\\"spans\\\":[[20,0]],\\\"text\\\":\\\"CodeGemma-2B 2.5 11.3 9.9 10.9 11.6 25.6 1.6 10.7 1.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":20,\\\"row-header\\\":false,\\\"row-span\\\":[20,21]},{\\\"spans\\\":[[20,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":20,\\\"row-header\\\":false,\\\"row-span\\\":[20,21]},{\\\"spans\\\":[[20,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":20,\\\"row-header\\\":false,\\\"row-span\\\":[20,21]},{\\\"spans\\\":[[20,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":20,\\\"row-header\\\":false,\\\"row-span\\\":[20,21]}],[{\\\"bbox\\\":[117,440,218,446],\\\"spans\\\":[[21,0]],\\\"text\\\":\\\"Granite-3B-Code-Base 18.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":21,\\\"row-header\\\":false,\\\"row-span\\\":[21,22]},{\\\"bbox\\\":[229,440,336,446],\\\"spans\\\":[[21,1]],\\\"text\\\":\\\"25.8 24.1 24.1 24.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":21,\\\"row-header\\\":false,\\\"row-span\\\":[21,22]},{\\\"bbox\\\":[351,440,365,446],\\\"spans\\\":[[21,2]],\\\"text\\\":\\\"26.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":21,\\\"row-header\\\":false,\\\"row-span\\\":[21,22]},{\\\"bbox\\\":[393,440,485,446],\\\"spans\\\":[[21,3]],\\\"text\\\":\\\"7.0 22.0 31.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":21,\\\"row-header\\\":false,\\\"row-span\\\":[21,22]}],[{\\\"bbox\\\":[125,427,485,433],\\\"spans\\\":[[22,0]],\\\"text\\\":\\\"StarCoderBase-7B 16.8 21.1 19.9 20.0 20.1 21.2 7.5 18.5 27.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":22,\\\"row-header\\\":false,\\\"row-span\\\":[22,23]},{\\\"spans\\\":[[22,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":22,\\\"row-header\\\":false,\\\"row-span\\\":[22,23]},{\\\"spans\\\":[[22,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":22,\\\"row-header\\\":false,\\\"row-span\\\":[22,23]},{\\\"spans\\\":[[22,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":22,\\\"row-header\\\":false,\\\"row-span\\\":[22,23]}],[{\\\"bbox\\\":[131,417,486,424],\\\"spans\\\":[[23,0]],\\\"text\\\":\\\"CodeLlama-7B 17.4 25.7 24.7 24.2 24.9 25.5 10.0 22.8 33.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":23,\\\"row-header\\\":false,\\\"row-span\\\":[23,24]},{\\\"spans\\\":[[23,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":23,\\\"row-header\\\":false,\\\"row-span\\\":[23,24]},{\\\"spans\\\":[[23,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":23,\\\"row-header\\\":false,\\\"row-span\\\":[23,24]},{\\\"spans\\\":[[23,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":23,\\\"row-header\\\":false,\\\"row-span\\\":[23,24]}],[{\\\"bbox\\\":[131,408,485,415],\\\"spans\\\":[[24,0]],\\\"text\\\":\\\"StarCoder2-7B 17.6 29.0 27.3 27.2 27.6 22.0 13.2 25.3 37.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":24,\\\"row-header\\\":false,\\\"row-span\\\":[24,25]},{\\\"spans\\\":[[24,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":24,\\\"row-header\\\":false,\\\"row-span\\\":[24,25]},{\\\"spans\\\":[[24,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":24,\\\"row-header\\\":false,\\\"row-span\\\":[24,25]},{\\\"spans\\\":[[24,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":24,\\\"row-header\\\":false,\\\"row-span\\\":[24,25]}],[{\\\"bbox\\\":[128,399,188,405],\\\"spans\\\":[[25,0]],\\\"text\\\":\\\"CodeGemma-7B\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":25,\\\"row-header\\\":false,\\\"row-span\\\":[25,26]},{\\\"bbox\\\":[204,399,365,405],\\\"spans\\\":[[25,1]],\\\"text\\\":\\\"31.2 34.0 33.1 32.0 33.6 39.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":25,\\\"row-header\\\":false,\\\"row-span\\\":[25,26]},{\\\"bbox\\\":[391,399,404,405],\\\"spans\\\":[[25,2]],\\\"text\\\":\\\"11.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":25,\\\"row-header\\\":false,\\\"row-span\\\":[25,26]},{\\\"bbox\\\":[430,399,485,405],\\\"spans\\\":[[25,3]],\\\"text\\\":\\\"30.8 45.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":25,\\\"row-header\\\":false,\\\"row-span\\\":[25,26]}],[{\\\"bbox\\\":[117,390,365,396],\\\"spans\\\":[[26,0]],\\\"text\\\":\\\"Granite-8B-Code-Base 18.8 32.2 30.4 30.4 30.4 26.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":26,\\\"row-header\\\":false,\\\"row-span\\\":[26,27]},{\\\"spans\\\":[[26,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":26,\\\"row-header\\\":false,\\\"row-span\\\":[26,27]},{\\\"bbox\\\":[390,390,404,396],\\\"spans\\\":[[26,2]],\\\"text\\\":\\\"13.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":26,\\\"row-header\\\":false,\\\"row-span\\\":[26,27]},{\\\"bbox\\\":[430,390,485,396],\\\"spans\\\":[[26,3]],\\\"text\\\":\\\"27.9 31.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":26,\\\"row-header\\\":false,\\\"row-span\\\":[26,27]}],[{\\\"bbox\\\":[123,377,485,383],\\\"spans\\\":[[27,0]],\\\"text\\\":\\\"StarCoderBase-15B 16.8 23.9 22.0 22.6 22.3 28.6 11.2 20.4 32.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":27,\\\"row-header\\\":false,\\\"row-span\\\":[27,28]},{\\\"spans\\\":[[27,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":27,\\\"row-header\\\":false,\\\"row-span\\\":[27,28]},{\\\"spans\\\":[[27,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":27,\\\"row-header\\\":false,\\\"row-span\\\":[27,28]},{\\\"spans\\\":[[27,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":27,\\\"row-header\\\":false,\\\"row-span\\\":[27,28]}],[{\\\"bbox\\\":[129,368,485,374],\\\"spans\\\":[[28,0]],\\\"text\\\":\\\"CodeLlama-13B 22.5 30.1 28.8 28.4 29.0 29.7 13.8 26.6 40.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":28,\\\"row-header\\\":false,\\\"row-span\\\":[28,29]},{\\\"spans\\\":[[28,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":28,\\\"row-header\\\":false,\\\"row-span\\\":[28,29]},{\\\"spans\\\":[[28,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":28,\\\"row-header\\\":false,\\\"row-span\\\":[28,29]},{\\\"spans\\\":[[28,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":28,\\\"row-header\\\":false,\\\"row-span\\\":[28,29]}],[{\\\"bbox\\\":[129,358,187,365],\\\"spans\\\":[[29,0]],\\\"text\\\":\\\"StarCoder2-15B\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":29,\\\"row-header\\\":false,\\\"row-span\\\":[29,30]},{\\\"bbox\\\":[204,358,336,364],\\\"spans\\\":[[29,1]],\\\"text\\\":\\\"36.6 37.2 36.4 35.9 36.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":29,\\\"row-header\\\":false,\\\"row-span\\\":[29,30]},{\\\"bbox\\\":[351,358,365,364],\\\"spans\\\":[[29,2]],\\\"text\\\":\\\"18.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":29,\\\"row-header\\\":false,\\\"row-span\\\":[29,30]},{\\\"bbox\\\":[390,358,485,364],\\\"spans\\\":[[29,3]],\\\"text\\\":\\\"18.7 33.5 43.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":29,\\\"row-header\\\":false,\\\"row-span\\\":[29,30]}],[{\\\"bbox\\\":[115,349,337,355],\\\"spans\\\":[[30,0]],\\\"text\\\":\\\"Granite-20B-Code-Base 26.5 15.5 28.1 17.9 35.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":30,\\\"row-header\\\":false,\\\"row-span\\\":[30,31]},{\\\"spans\\\":[[30,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":30,\\\"row-header\\\":false,\\\"row-span\\\":[30,31]},{\\\"bbox\\\":[351,349,365,355],\\\"spans\\\":[[30,2]],\\\"text\\\":\\\"37.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":30,\\\"row-header\\\":false,\\\"row-span\\\":[30,31]},{\\\"bbox\\\":[391,349,486,355],\\\"spans\\\":[[30,3]],\\\"text\\\":\\\"16.7 27.6 38.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":30,\\\"row-header\\\":false,\\\"row-span\\\":[30,31]}],[{\\\"bbox\\\":[129,336,271,342],\\\"spans\\\":[[31,0]],\\\"text\\\":\\\"CodeLlama-34B 28.9  35.4  33.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":31,\\\"row-header\\\":false,\\\"row-span\\\":[31,32]},{\\\"bbox\\\":[291,336,305,342],\\\"spans\\\":[[31,1]],\\\"text\\\":\\\"33.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":31,\\\"row-header\\\":false,\\\"row-span\\\":[31,32]},{\\\"spans\\\":[[31,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":31,\\\"row-header\\\":false,\\\"row-span\\\":[31,32]},{\\\"spans\\\":[[31,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":31,\\\"row-header\\\":false,\\\"row-span\\\":[31,32]}],[{\\\"bbox\\\":[115,327,271,333],\\\"spans\\\":[[32,0]],\\\"text\\\":\\\"Granite-34B-Code-Base 31.5  25.4  33.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":32,\\\"row-header\\\":false,\\\"row-span\\\":[32,33]},{\\\"bbox\\\":[291,327,305,333],\\\"spans\\\":[[32,1]],\\\"text\\\":\\\"18.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":32,\\\"row-header\\\":false,\\\"row-span\\\":[32,33]},{\\\"bbox\\\":[322,336,485,342],\\\"spans\\\":[[32,2]],\\\"text\\\":\\\"34.3 32.9 16.2 31.5 39.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":32,\\\"row-header\\\":false,\\\"row-span\\\":[32,33]},{\\\"bbox\\\":[322,327,485,332],\\\"spans\\\":[[32,3]],\\\"text\\\":\\\"38.9 41.7 19.3 36.5 41.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":32,\\\"row-header\\\":false,\\\"row-span\\\":[32,33]}]]},{\\\"prov\\\":[{\\\"bbox\\\":[107,480,287,668],\\\"page\\\":12,\\\"span\\\":[0,0]}],\\\"text\\\":\\\"Table 6: Average exact match (EM) and edit similarity (ES) on RepoBench v1.1. All models are evaluated at temperature 0.2 and top-p 0.95.\\\",\\\"type\\\":\\\"table\\\",\\\"#-cols\\\":1,\\\"#-rows\\\":17,\\\"data\\\":[[{\\\"bbox\\\":[146,657,280,664],\\\"spans\\\":[[0,0]],\\\"text\\\":\\\"Model MBPP MBPP+\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]}],[{\\\"bbox\\\":[124,643,273,650],\\\"spans\\\":[[1,0]],\\\"text\\\":\\\"StarCoderBase-3B 29.4 37.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]}],[{\\\"bbox\\\":[130,633,273,640],\\\"spans\\\":[[2,0]],\\\"text\\\":\\\"StableCode-3B 34.8 43.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]}],[{\\\"bbox\\\":[130,624,273,630],\\\"spans\\\":[[3,0]],\\\"text\\\":\\\"StarCoder2-3B  42.4 48.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]}],[{\\\"bbox\\\":[127,614,273,621],\\\"spans\\\":[[4,0]],\\\"text\\\":\\\"CodeGemma-2B 30.4 30.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]}],[{\\\"bbox\\\":[116,604,273,611],\\\"spans\\\":[[5,0]],\\\"text\\\":\\\"Granite-3B-Code-Base 36.0 45.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]}],[{\\\"bbox\\\":[124,590,273,597],\\\"spans\\\":[[6,0]],\\\"text\\\":\\\"StarCoderBase-7B 34.8 42.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]}],[{\\\"bbox\\\":[130,581,273,587],\\\"spans\\\":[[7,0]],\\\"text\\\":\\\"CodeLlama-7B 39.0 42.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]}],[{\\\"bbox\\\":[130,571,273,578],\\\"spans\\\":[[8,0]],\\\"text\\\":\\\"StarCoder2-7B 45.4 46.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]}],[{\\\"bbox\\\":[127,561,273,568],\\\"spans\\\":[[9,0]],\\\"text\\\":\\\"CodeGemma-7B  53.0 54.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]}],[{\\\"bbox\\\":[116,552,273,558],\\\"spans\\\":[[10,0]],\\\"text\\\":\\\"Granite-8B-Code-Base 42.2 49.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]}],[{\\\"bbox\\\":[122,538,273,544],\\\"spans\\\":[[11,0]],\\\"text\\\":\\\"StarCoderBase-15B 37.4 46.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]}],[{\\\"bbox\\\":[128,528,273,535],\\\"spans\\\":[[12,0]],\\\"text\\\":\\\"CodeLlama-13B 30.6 30.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]}],[{\\\"bbox\\\":[128,518,273,525],\\\"spans\\\":[[13,0]],\\\"text\\\":\\\"StarCoder2-15B  51.2 56.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]}],[{\\\"bbox\\\":[113,509,273,515],\\\"spans\\\":[[14,0]],\\\"text\\\":\\\"Granite-20B-Code-Base 43.8 51.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]}],[{\\\"bbox\\\":[128,495,273,501],\\\"spans\\\":[[15,0]],\\\"text\\\":\\\"CodeLlama-34B  48.6 53.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]}],[{\\\"bbox\\\":[113,485,273,492],\\\"spans\\\":[[16,0]],\\\"text\\\":\\\"Granite-34B-Code-Base 47.2 53.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]}]]},{\\\"prov\\\":[{\\\"bbox\\\":[325,481,505,656],\\\"page\\\":12,\\\"span\\\":[0,0]}],\\\"text\\\":\\\"Table 7: Mean pass@1 accuracy averaged over 40 samples on DS-1000. All models are evaluated at temperature 0.2 and top-p 0.95.\\\",\\\"type\\\":\\\"table\\\",\\\"#-cols\\\":2,\\\"#-rows\\\":17,\\\"data\\\":[[{\\\"bbox\\\":[359,643,381,648],\\\"spans\\\":[[0,0]],\\\"text\\\":\\\"Model\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[419,635,497,653],\\\"spans\\\":[[0,1]],\\\"text\\\":\\\"Python Java EM ES EM ES\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]}],[{\\\"bbox\\\":[340,623,400,628],\\\"spans\\\":[[1,0]],\\\"text\\\":\\\"StarCoderBase-3B\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"bbox\\\":[419,623,499,628],\\\"spans\\\":[[1,1]],\\\"text\\\":\\\"29.9 69.3 36.0 74.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]}],[{\\\"bbox\\\":[345,614,499,620],\\\"spans\\\":[[2,0]],\\\"text\\\":\\\"StableCode-3B 29.4 68.5 34.9 72.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]}],[{\\\"bbox\\\":[345,606,499,611],\\\"spans\\\":[[3,0]],\\\"text\\\":\\\"StarCoder2-3B 27.2 67.0 35.9 74.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"spans\\\":[[3,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]}],[{\\\"bbox\\\":[342,597,499,603],\\\"spans\\\":[[4,0]],\\\"text\\\":\\\"CodeGemma-2B 26.2 66.9 33.6 71.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]}],[{\\\"bbox\\\":[332,589,499,595],\\\"spans\\\":[[5,0]],\\\"text\\\":\\\"Granite-3B-Code-Base 27.1 66.8 34.9 73.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"spans\\\":[[5,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]}],[{\\\"bbox\\\":[340,577,499,582],\\\"spans\\\":[[6,0]],\\\"text\\\":\\\"StarCoderBase-7B 27.1 66.5 36.5 75.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]}],[{\\\"bbox\\\":[345,568,477,574],\\\"spans\\\":[[7,0]],\\\"text\\\":\\\"CodeLlama-7B 29.2 67.4 37.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"bbox\\\":[486,568,499,574],\\\"spans\\\":[[7,1]],\\\"text\\\":\\\"76.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]}],[{\\\"bbox\\\":[345,560,499,566],\\\"spans\\\":[[8,0]],\\\"text\\\":\\\"StarCoder2-7B 28.1 67.6 37.0 75.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]}],[{\\\"bbox\\\":[342,551,454,557],\\\"spans\\\":[[9,0]],\\\"text\\\":\\\"CodeGemma-7B  36.8 72.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[464,551,499,557],\\\"spans\\\":[[9,1]],\\\"text\\\":\\\"38.3 74.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]}],[{\\\"bbox\\\":[332,543,454,549],\\\"spans\\\":[[10,0]],\\\"text\\\":\\\"Granite-8B-Code-Base 31.8 69.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[464,543,499,549],\\\"spans\\\":[[10,1]],\\\"text\\\":\\\"38.4  76.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]}],[{\\\"bbox\\\":[338,531,499,537],\\\"spans\\\":[[11,0]],\\\"text\\\":\\\"StarCoderBase-15B 29.4 67.8 37.1 75.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"spans\\\":[[11,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]}],[{\\\"bbox\\\":[343,522,499,528],\\\"spans\\\":[[12,0]],\\\"text\\\":\\\"CodeLlama-13B 31.4 68.8 39.4 77.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"spans\\\":[[12,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]}],[{\\\"bbox\\\":[343,514,499,520],\\\"spans\\\":[[13,0]],\\\"text\\\":\\\"StarCoder2-15B 31.3 69.6 39.9 77.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"spans\\\":[[13,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]}],[{\\\"bbox\\\":[331,506,409,511],\\\"spans\\\":[[14,0]],\\\"text\\\":\\\"Granite-20B-Code-Base\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"bbox\\\":[419,506,499,511],\\\"spans\\\":[[14,1]],\\\"text\\\":\\\"38.0 72.2 42.3 78.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]}],[{\\\"bbox\\\":[343,493,477,499],\\\"spans\\\":[[15,0]],\\\"text\\\":\\\"CodeLlama-34B 34.4 70.2 40.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"bbox\\\":[486,493,499,499],\\\"spans\\\":[[15,1]],\\\"text\\\":\\\"78.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]}],[{\\\"bbox\\\":[331,485,409,491],\\\"spans\\\":[[16,0]],\\\"text\\\":\\\"Granite-34B-Code-Base\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"bbox\\\":[419,485,499,490],\\\"spans\\\":[[16,1]],\\\"text\\\":\\\"35.9 71.5 42.0  77.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]}]]},{\\\"prov\\\":[{\\\"bbox\\\":[109,317,502,432],\\\"page\\\":12,\\\"span\\\":[0,0]}],\\\"text\\\":\\\"Table 6 shows the performance of different models on RepoBench v1.1. Granite-3B-CodeBase demonstrates notable performance among the smaller models, with StarCoderBase-3B achieving the leading performance metrics. Among the medium models, Granite-8B-CodeBase shows very strong performance on Java, while ranks second best one in Python, with CodeGemma-7B being the best performing on both metrics. Among larger models, Granite-20B-Code not only outperforms StarCoder2-15B but also CodeLlama-34B on all 4 metrics across both programming languages. This demonstrates strong repository-level\\\",\\\"type\\\":\\\"table\\\",\\\"#-cols\\\":4,\\\"#-rows\\\":11,\\\"data\\\":[[{\\\"bbox\\\":[115,418,139,424],\\\"spans\\\":[[0,0]],\\\"text\\\":\\\"Model\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"spans\\\":[[0,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[204,417,473,424],\\\"spans\\\":[[0,2]],\\\"text\\\":\\\"Matplotlib NumPy Pandas PyTorch SciPy Scikit-Learn TensorFlow\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[483,416,497,424],\\\"spans\\\":[[0,3]],\\\"text\\\":\\\"Avg\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]}],[{\\\"bbox\\\":[115,405,497,411],\\\"spans\\\":[[1,0]],\\\"text\\\":\\\"StarCoderBase-3B 32.1 16.8 5.3 9.2 13.2 10.5 17.2 14.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]}],[{\\\"bbox\\\":[115,396,269,403],\\\"spans\\\":[[2,0]],\\\"text\\\":\\\"StableCode-3B 42.5 24.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"bbox\\\":[290,397,303,402],\\\"spans\\\":[[2,1]],\\\"text\\\":\\\"16.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"bbox\\\":[325,396,497,402],\\\"spans\\\":[[2,3]],\\\"text\\\":\\\"15.4 13.5 20.2 27.7 22.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]}],[{\\\"bbox\\\":[115,388,229,394],\\\"spans\\\":[[3,0]],\\\"text\\\":\\\"StarCoder2-3B 45.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[255,388,303,393],\\\"spans\\\":[[3,1]],\\\"text\\\":\\\"27.7 16.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[325,388,411,393],\\\"spans\\\":[[3,2]],\\\"text\\\":\\\"12.9 15.8  30.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[446,388,497,393],\\\"spans\\\":[[3,3]],\\\"text\\\":\\\"22.8  25.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]}],[{\\\"bbox\\\":[115,379,497,385],\\\"spans\\\":[[4,0]],\\\"text\\\":\\\"CodeGemma-2B 30.3 17.7 5.5 4.4 10.3 2.6 4.4 10.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]}],[{\\\"bbox\\\":[115,370,269,376],\\\"spans\\\":[[5,0]],\\\"text\\\":\\\"Granite-3B-Code-Base  43.3 27.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[290,370,303,376],\\\"spans\\\":[[5,1]],\\\"text\\\":\\\"11.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[325,370,411,376],\\\"spans\\\":[[5,2]],\\\"text\\\":\\\"19.1 21.7  16.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[446,370,497,376],\\\"spans\\\":[[5,3]],\\\"text\\\":\\\"24.4  23.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]}],[{\\\"bbox\\\":[115,357,496,363],\\\"spans\\\":[[6,0]],\\\"text\\\":\\\"StarCoderBase-7B 38.0 23.0 8.2 13.1 13.7 24.5 14.6 19.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]}],[{\\\"bbox\\\":[115,348,497,354],\\\"spans\\\":[[7,0]],\\\"text\\\":\\\"CodeLlama-7B 46.3 21.6 13.9 12.2 17.5 16.7 20.6 21.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]}],[{\\\"bbox\\\":[115,339,497,345],\\\"spans\\\":[[8,0]],\\\"text\\\":\\\"StarCoder2-7B 53.6 33.3 16.9 16.2 20.6 22.2 31.9 27.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]}],[{\\\"bbox\\\":[115,330,229,336],\\\"spans\\\":[[9,0]],\\\"text\\\":\\\"CodeGemma-7B  56.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[255,330,371,336],\\\"spans\\\":[[9,1]],\\\"text\\\":\\\"37.2 20.9 20.5 24.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[397,330,411,336],\\\"spans\\\":[[9,2]],\\\"text\\\":\\\"34.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[446,330,496,336],\\\"spans\\\":[[9,3]],\\\"text\\\":\\\"31.1 32.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]}],[{\\\"bbox\\\":[115,321,229,327],\\\"spans\\\":[[10,0]],\\\"text\\\":\\\"Granite-8B-Code-Base 51.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[255,321,371,327],\\\"spans\\\":[[10,1]],\\\"text\\\":\\\"40.0 23.4 32.4 22.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[397,321,411,327],\\\"spans\\\":[[10,2]],\\\"text\\\":\\\"29.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[446,321,497,327],\\\"spans\\\":[[10,3]],\\\"text\\\":\\\"42.2 34.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]}]]},{\\\"prov\\\":[{\\\"bbox\\\":[109,487,502,679],\\\"page\\\":13,\\\"span\\\":[0,0]}],\\\"text\\\":\\\"Table 9: Exact-match on FIM-task (Allal et al. , 2023). All models are evaluated using greedy decoding with maximum new tokens set to 512.\\\",\\\"type\\\":\\\"table\\\",\\\"#-cols\\\":5,\\\"#-rows\\\":18,\\\"data\\\":[[{\\\"spans\\\":[[0,0]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"spans\\\":[[0,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[230,667,470,675],\\\"spans\\\":[[0,2],[0,3]],\\\"text\\\":\\\"Python Java TypeScript C#\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,4],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[230,667,470,675],\\\"spans\\\":[[0,2],[0,3]],\\\"text\\\":\\\"Python Java TypeScript C#\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[2,4],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"spans\\\":[[0,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]}],[{\\\"bbox\\\":[146,659,171,666],\\\"spans\\\":[[1,0]],\\\"text\\\":\\\"Model\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"bbox\\\":[212,655,496,662],\\\"spans\\\":[[1,2]],\\\"text\\\":\\\"Code ES ID F1 Code ES ID F1 Code ES ID F1 Code ES ID F1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]}],[{\\\"bbox\\\":[125,642,493,648],\\\"spans\\\":[[2,0]],\\\"text\\\":\\\"StarCoderBase-3B 63.5 53.8 63.3 55.7 44.2 40.8 65.3 45.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]}],[{\\\"bbox\\\":[131,633,272,639],\\\"spans\\\":[[3,0]],\\\"text\\\":\\\"StableCode-3B 65.3 56.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"spans\\\":[[3,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[295,633,420,638],\\\"spans\\\":[[3,2]],\\\"text\\\":\\\"68.2 61.0 60.9 55.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[442,633,494,639],\\\"spans\\\":[[3,3]],\\\"text\\\":\\\"59.9 41.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"spans\\\":[[3,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]}],[{\\\"bbox\\\":[132,623,185,630],\\\"spans\\\":[[4,0]],\\\"text\\\":\\\"StarCoder2-3B\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"bbox\\\":[221,623,272,629],\\\"spans\\\":[[4,1]],\\\"text\\\":\\\"65.5 56.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"bbox\\\":[295,623,419,629],\\\"spans\\\":[[4,2]],\\\"text\\\":\\\"64.8 57.3 44.7 41.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"bbox\\\":[442,623,493,629],\\\"spans\\\":[[4,3]],\\\"text\\\":\\\"66.0 47.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]}],[{\\\"bbox\\\":[128,614,493,620],\\\"spans\\\":[[5,0]],\\\"text\\\":\\\"CodeGemma-2B 60.5 50.6 55.1 46.4 55.6 49.0 44.2 27.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"spans\\\":[[5,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"spans\\\":[[5,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"spans\\\":[[5,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"spans\\\":[[5,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]}],[{\\\"bbox\\\":[117,605,493,611],\\\"spans\\\":[[6,0]],\\\"text\\\":\\\"Granite-3B-Code-Base 65.1 56.0 64.1 56.6 43.2 39.4 65.9 46.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]}],[{\\\"bbox\\\":[125,591,493,598],\\\"spans\\\":[[7,0]],\\\"text\\\":\\\"StarCoderBase-7B 65.3 56.3 65.4 58.0 46.2 43.2 66.1 47.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]}],[{\\\"bbox\\\":[131,582,493,588],\\\"spans\\\":[[8,0]],\\\"text\\\":\\\"CodeLlama-7B 64.9 55.4 65.0 57.8 62.1 56.9 65.1 46.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]}],[{\\\"bbox\\\":[132,573,272,579],\\\"spans\\\":[[9,0]],\\\"text\\\":\\\"StarCoder2-7B 66.5 57.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"spans\\\":[[9,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[295,573,419,579],\\\"spans\\\":[[9,2]],\\\"text\\\":\\\"67.0 59.8  46.9 43.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[442,573,493,579],\\\"spans\\\":[[9,3]],\\\"text\\\":\\\"67.2  48.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"spans\\\":[[9,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]}],[{\\\"bbox\\\":[128,564,189,570],\\\"spans\\\":[[10,0]],\\\"text\\\":\\\"CodeGemma-7B\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[221,564,272,570],\\\"spans\\\":[[10,1]],\\\"text\\\":\\\"68.1 59.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[295,564,420,570],\\\"spans\\\":[[10,2]],\\\"text\\\":\\\"65.9 59.6  63.5 58.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[442,564,493,570],\\\"spans\\\":[[10,3]],\\\"text\\\":\\\"56.2 42.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"spans\\\":[[10,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]}],[{\\\"bbox\\\":[117,555,456,561],\\\"spans\\\":[[11,0]],\\\"text\\\":\\\"Granite-8B-Code-Base 66.3 57.8 66.5 59.0 45.1 42.0 66.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"spans\\\":[[11,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"spans\\\":[[11,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"bbox\\\":[479,555,494,560],\\\"spans\\\":[[11,3]],\\\"text\\\":\\\"48.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"spans\\\":[[11,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]}],[{\\\"bbox\\\":[123,541,494,547],\\\"spans\\\":[[12,0]],\\\"text\\\":\\\"StarCoderBase-15B 66.0 57.3 67.0 60.2 46.6 43.3 66.4 47.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"spans\\\":[[12,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"spans\\\":[[12,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"spans\\\":[[12,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"spans\\\":[[12,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]}],[{\\\"bbox\\\":[129,532,383,538],\\\"spans\\\":[[13,0]],\\\"text\\\":\\\"CodeLlama-13B 66.2 57.4 66.8 59.5 63.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"spans\\\":[[13,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"spans\\\":[[13,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"bbox\\\":[405,532,493,538],\\\"spans\\\":[[13,3]],\\\"text\\\":\\\"58.6  65.6 48.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"spans\\\":[[13,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]}],[{\\\"bbox\\\":[130,523,235,529],\\\"spans\\\":[[14,0]],\\\"text\\\":\\\"StarCoder2-15B 68.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"bbox\\\":[258,523,272,528],\\\"spans\\\":[[14,1]],\\\"text\\\":\\\"59.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"bbox\\\":[295,523,420,529],\\\"spans\\\":[[14,2]],\\\"text\\\":\\\"68.1  61.5  47.0 43.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"bbox\\\":[442,523,493,528],\\\"spans\\\":[[14,3]],\\\"text\\\":\\\"68.5 51.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"spans\\\":[[14,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]}],[{\\\"bbox\\\":[115,513,201,520],\\\"spans\\\":[[15,0]],\\\"text\\\":\\\"Granite-20B-Code-Base\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"bbox\\\":[221,513,272,519],\\\"spans\\\":[[15,1]],\\\"text\\\":\\\"68.2  58.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"bbox\\\":[295,513,383,519],\\\"spans\\\":[[15,2]],\\\"text\\\":\\\"68.4  60.4  48.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"bbox\\\":[405,513,493,519],\\\"spans\\\":[[15,3]],\\\"text\\\":\\\"43.1 67.5 48.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"spans\\\":[[15,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]}],[{\\\"bbox\\\":[129,500,188,506],\\\"spans\\\":[[16,0]],\\\"text\\\":\\\"CodeLlama-34B\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"bbox\\\":[221,500,272,506],\\\"spans\\\":[[16,1]],\\\"text\\\":\\\"69.3 59.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"bbox\\\":[295,500,419,506],\\\"spans\\\":[[16,2]],\\\"text\\\":\\\"68.2  61.1 64.4 56.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"bbox\\\":[442,500,493,506],\\\"spans\\\":[[16,3]],\\\"text\\\":\\\"67.2  49.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"spans\\\":[[16,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]}],[{\\\"bbox\\\":[115,491,272,497],\\\"spans\\\":[[17,0]],\\\"text\\\":\\\"Granite-34B-Code-Base 68.3 58.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"spans\\\":[[17,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"bbox\\\":[295,491,420,497],\\\"spans\\\":[[17,2]],\\\"text\\\":\\\"68.6  60.8 48.9 43.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"bbox\\\":[442,491,493,497],\\\"spans\\\":[[17,3]],\\\"text\\\":\\\"67.5  49.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"spans\\\":[[17,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]}]]},{\\\"prov\\\":[{\\\"bbox\\\":[176,300,437,441],\\\"page\\\":13,\\\"span\\\":[0,0]}],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"table\\\",\\\"#-cols\\\":3,\\\"#-rows\\\":12,\\\"data\\\":[[{\\\"spans\\\":[[0,0]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[215,427,399,436],\\\"spans\\\":[[0,1]],\\\"text\\\":\\\"Model Java JavaScript Python\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[411,427,429,436],\\\"spans\\\":[[0,2]],\\\"text\\\":\\\"Avg.\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]}],[{\\\"bbox\\\":[193,415,392,421],\\\"spans\\\":[[1,0]],\\\"text\\\":\\\"StarCoderBase-3B 76.0 68.5 53.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"bbox\\\":[413,415,428,421],\\\"spans\\\":[[1,2]],\\\"text\\\":\\\"66.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]}],[{\\\"bbox\\\":[200,405,303,411],\\\"spans\\\":[[2,0]],\\\"text\\\":\\\"StableCode-3B 64.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"bbox\\\":[330,405,392,411],\\\"spans\\\":[[2,1]],\\\"text\\\":\\\"74.5  59.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"bbox\\\":[413,405,428,411],\\\"spans\\\":[[2,2]],\\\"text\\\":\\\"66.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]}],[{\\\"bbox\\\":[200,395,392,401],\\\"spans\\\":[[3,0]],\\\"text\\\":\\\"StarCoder2-3B 76.0 73.5 59.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"spans\\\":[[3,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[413,395,428,401],\\\"spans\\\":[[3,2]],\\\"text\\\":\\\"69.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]}],[{\\\"bbox\\\":[185,385,303,391],\\\"spans\\\":[[4,0]],\\\"text\\\":\\\"Granite-3B-Code-Base  79.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"bbox\\\":[329,385,392,391],\\\"spans\\\":[[4,1]],\\\"text\\\":\\\"71.6  61.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"bbox\\\":[413,385,428,391],\\\"spans\\\":[[4,2]],\\\"text\\\":\\\"71.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]}],[{\\\"bbox\\\":[193,370,392,376],\\\"spans\\\":[[5,0]],\\\"text\\\":\\\"StarCoderBase-7B 81.1 74.5 62.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"spans\\\":[[5,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[413,370,428,376],\\\"spans\\\":[[5,2]],\\\"text\\\":\\\"72.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]}],[{\\\"bbox\\\":[200,360,392,366],\\\"spans\\\":[[6,0]],\\\"text\\\":\\\"StarCoder2-7B 82.1 78.4 61.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"bbox\\\":[413,360,428,366],\\\"spans\\\":[[6,2]],\\\"text\\\":\\\"74.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]}],[{\\\"bbox\\\":[185,350,272,356],\\\"spans\\\":[[7,0]],\\\"text\\\":\\\"Granite-8B-Code-Base\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"bbox\\\":[288,350,392,356],\\\"spans\\\":[[7,1]],\\\"text\\\":\\\"83.6 79.9 66.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"bbox\\\":[413,350,428,356],\\\"spans\\\":[[7,2]],\\\"text\\\":\\\"76.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]}],[{\\\"bbox\\\":[191,335,392,341],\\\"spans\\\":[[8,0]],\\\"text\\\":\\\"StarCoderBase-15B 74.6 74.6 63.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"bbox\\\":[413,335,428,341],\\\"spans\\\":[[8,2]],\\\"text\\\":\\\"70.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]}],[{\\\"bbox\\\":[198,325,392,332],\\\"spans\\\":[[9,0]],\\\"text\\\":\\\"StarCoder2-15B 61.1 54.8 48.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"spans\\\":[[9,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[413,325,428,331],\\\"spans\\\":[[9,2]],\\\"text\\\":\\\"54.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]}],[{\\\"bbox\\\":[182,315,303,322],\\\"spans\\\":[[10,0]],\\\"text\\\":\\\"Granite-20B-Code-Base 79.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[329,315,392,321],\\\"spans\\\":[[10,1]],\\\"text\\\":\\\"82.2 66.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[413,315,428,321],\\\"spans\\\":[[10,2]],\\\"text\\\":\\\"76.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]}],[{\\\"bbox\\\":[182,305,303,312],\\\"spans\\\":[[11,0]],\\\"text\\\":\\\"Granite-34B-Code-Base  80.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"bbox\\\":[329,305,392,311],\\\"spans\\\":[[11,1]],\\\"text\\\":\\\"79.4 67.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"bbox\\\":[413,305,428,311],\\\"spans\\\":[[11,2]],\\\"text\\\":\\\"76.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]}]]},{\\\"prov\\\":[{\\\"bbox\\\":[107,227,505,690],\\\"page\\\":15,\\\"span\\\":[0,0]}],\\\"text\\\":\\\"Table 10: Pass@1 performance on HumanEvalExplain.\\\",\\\"type\\\":\\\"table\\\",\\\"#-cols\\\":4,\\\"#-rows\\\":43,\\\"data\\\":[[{\\\"bbox\\\":[152,677,263,685],\\\"spans\\\":[[0,0]],\\\"text\\\":\\\"Model Prompt\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[282,677,470,685],\\\"spans\\\":[[0,1]],\\\"text\\\":\\\"Python JavaScript Java Go C++ Rust\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"spans\\\":[[0,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[481,677,498,685],\\\"spans\\\":[[0,3]],\\\"text\\\":\\\"Avg.\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]}],[{\\\"bbox\\\":[283,665,329,671],\\\"spans\\\":[[1,0],[1,1],[1,2],[1,3]],\\\"text\\\":\\\"Base Models\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,4],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"bbox\\\":[283,665,329,671],\\\"spans\\\":[[1,0],[1,1],[1,2],[1,3]],\\\"text\\\":\\\"Base Models\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[0,4],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"bbox\\\":[283,665,329,671],\\\"spans\\\":[[1,0],[1,1],[1,2],[1,3]],\\\"text\\\":\\\"Base Models\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[0,4],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"bbox\\\":[283,665,329,671],\\\"spans\\\":[[1,0],[1,1],[1,2],[1,3]],\\\"text\\\":\\\"Base Models\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[0,4],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]}],[{\\\"bbox\\\":[131,649,271,657],\\\"spans\\\":[[2,0]],\\\"text\\\":\\\"StarCoderBase-3B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"bbox\\\":[289,651,468,657],\\\"spans\\\":[[2,1]],\\\"text\\\":\\\"11.0 10.4 14.6 11.0 13.4 11.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"bbox\\\":[483,651,497,657],\\\"spans\\\":[[2,3]],\\\"text\\\":\\\"11.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]}],[{\\\"bbox\\\":[138,640,271,648],\\\"spans\\\":[[3,0]],\\\"text\\\":\\\"StableCode-3B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[289,642,468,648],\\\"spans\\\":[[3,1]],\\\"text\\\":\\\"11.0 7.9 22.0 4.3 14.6 14.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"spans\\\":[[3,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[483,642,497,648],\\\"spans\\\":[[3,3]],\\\"text\\\":\\\"12.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]}],[{\\\"bbox\\\":[138,630,271,639],\\\"spans\\\":[[4,0]],\\\"text\\\":\\\"StarCoder2-3B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"bbox\\\":[289,632,468,638],\\\"spans\\\":[[4,2]],\\\"text\\\":\\\"12.2 13.4 19.5 6.7 14.0 12.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"bbox\\\":[483,632,496,638],\\\"spans\\\":[[4,3]],\\\"text\\\":\\\"13.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]}],[{\\\"bbox\\\":[134,621,271,629],\\\"spans\\\":[[5,0]],\\\"text\\\":\\\"CodeGemma-2B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"spans\\\":[[5,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[289,623,468,629],\\\"spans\\\":[[5,2]],\\\"text\\\":\\\"20.7 15.9 20.7 12.8 17.7 15.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[483,623,497,629],\\\"spans\\\":[[5,3]],\\\"text\\\":\\\"17.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]}],[{\\\"bbox\\\":[124,612,271,620],\\\"spans\\\":[[6,0]],\\\"text\\\":\\\"Granite-3B-Code-Base Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"bbox\\\":[289,614,468,620],\\\"spans\\\":[[6,2]],\\\"text\\\":\\\"25.0 18.9 29.9 17.1 26.8 14.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"bbox\\\":[483,614,497,620],\\\"spans\\\":[[6,3]],\\\"text\\\":\\\"21.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]}],[{\\\"bbox\\\":[131,598,271,606],\\\"spans\\\":[[7,0]],\\\"text\\\":\\\"StarCoderBase-7B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"bbox\\\":[289,600,468,606],\\\"spans\\\":[[7,2]],\\\"text\\\":\\\"14.0 17.1 17.7 10.4 17.1 12.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"bbox\\\":[483,600,497,606],\\\"spans\\\":[[7,3]],\\\"text\\\":\\\"14.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]}],[{\\\"bbox\\\":[137,588,271,597],\\\"spans\\\":[[8,0]],\\\"text\\\":\\\"CodeLlama-7B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"bbox\\\":[289,590,468,596],\\\"spans\\\":[[8,2]],\\\"text\\\":\\\"11.0 14.0 16.5 9.8 17.7 14.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"bbox\\\":[483,590,497,596],\\\"spans\\\":[[8,3]],\\\"text\\\":\\\"13.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]}],[{\\\"bbox\\\":[138,579,271,587],\\\"spans\\\":[[9,0]],\\\"text\\\":\\\"StarCoder2-7B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"spans\\\":[[9,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[291,581,468,587],\\\"spans\\\":[[9,2]],\\\"text\\\":\\\"4.9 12.8 22.0 4.9 22.0 14.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[483,581,497,587],\\\"spans\\\":[[9,3]],\\\"text\\\":\\\"13.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]}],[{\\\"bbox\\\":[134,570,271,578],\\\"spans\\\":[[10,0]],\\\"text\\\":\\\"CodeGemma-7B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"spans\\\":[[10,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[289,572,468,578],\\\"spans\\\":[[10,2]],\\\"text\\\":\\\"13.1 13.8 2.0 8.0 18.6 18.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[483,572,497,578],\\\"spans\\\":[[10,3]],\\\"text\\\":\\\"12.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]}],[{\\\"bbox\\\":[124,560,271,569],\\\"spans\\\":[[11,0]],\\\"text\\\":\\\"Granite-8B-Code-Base Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"spans\\\":[[11,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"bbox\\\":[289,562,468,568],\\\"spans\\\":[[11,2]],\\\"text\\\":\\\"23.5 32.3 25.0 23.2 28.0 19.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"bbox\\\":[483,562,497,568],\\\"spans\\\":[[11,3]],\\\"text\\\":\\\"26.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]}],[{\\\"bbox\\\":[129,546,271,555],\\\"spans\\\":[[12,0]],\\\"text\\\":\\\"StarCoderBase-15B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"bbox\\\":[291,548,387,554],\\\"spans\\\":[[12,1]],\\\"text\\\":\\\"9.8 15.2  24.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"bbox\\\":[401,548,468,554],\\\"spans\\\":[[12,2]],\\\"text\\\":\\\"9.1 20.1 13.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"bbox\\\":[483,548,497,554],\\\"spans\\\":[[12,3]],\\\"text\\\":\\\"15.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]}],[{\\\"bbox\\\":[135,537,271,545],\\\"spans\\\":[[13,0]],\\\"text\\\":\\\"CodeLlama-13B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"spans\\\":[[13,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"bbox\\\":[289,539,468,545],\\\"spans\\\":[[13,2]],\\\"text\\\":\\\"13.4 14.0 23.2 9.8 15.9 13.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"bbox\\\":[483,539,497,545],\\\"spans\\\":[[13,3]],\\\"text\\\":\\\"15.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]}],[{\\\"bbox\\\":[136,528,271,536],\\\"spans\\\":[[14,0]],\\\"text\\\":\\\"StarCoder2-15B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"bbox\\\":[289,530,440,536],\\\"spans\\\":[[14,1]],\\\"text\\\":\\\"20.1 19.5  7.3 9.8 23.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"bbox\\\":[454,530,468,536],\\\"spans\\\":[[14,2]],\\\"text\\\":\\\"21.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"bbox\\\":[483,530,497,536],\\\"spans\\\":[[14,3]],\\\"text\\\":\\\"17.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]}],[{\\\"bbox\\\":[121,518,271,527],\\\"spans\\\":[[15,0]],\\\"text\\\":\\\"Granite-20B-Code-Base Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"bbox\\\":[289,506,468,520],\\\"spans\\\":[[15,1]],\\\"text\\\":\\\"17.1 18.3 23.2  11.6 18.3 22.0 9.8 20.1 20.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"bbox\\\":[399,526,468,526],\\\"spans\\\":[[15,2]],\\\"text\\\":\\\"10.4 25.6  18.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"bbox\\\":[483,520,497,526],\\\"spans\\\":[[15,3]],\\\"text\\\":\\\"18.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]}],[{\\\"bbox\\\":[121,495,271,513],\\\"spans\\\":[[16,0]],\\\"text\\\":\\\"CodeLlama-34B Completion  Granite-34B-Code-Base Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"spans\\\":[[16,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"bbox\\\":[289,497,468,503],\\\"spans\\\":[[16,2]],\\\"text\\\":\\\"42.7 26.2 47.0 26.8 36.6 25.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"bbox\\\":[483,497,497,512],\\\"spans\\\":[[16,3]],\\\"text\\\":\\\"17.1 34.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]}],[{\\\"bbox\\\":[135,486,271,494],\\\"spans\\\":[[17,0]],\\\"text\\\":\\\"CodeLlama-70B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"spans\\\":[[17,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"bbox\\\":[289,488,468,494],\\\"spans\\\":[[17,2]],\\\"text\\\":\\\"24.4 30.5 43.9 19.5 31.1 20.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"bbox\\\":[483,488,497,494],\\\"spans\\\":[[17,3]],\\\"text\\\":\\\"28.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]}],[{\\\"bbox\\\":[144,472,271,480],\\\"spans\\\":[[18,0]],\\\"text\\\":\\\"Gemma-2B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]},{\\\"bbox\\\":[291,474,466,480],\\\"spans\\\":[[18,1]],\\\"text\\\":\\\"9.8 9.8 14.6 7.9 14.0 9.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]},{\\\"spans\\\":[[18,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]},{\\\"bbox\\\":[483,474,497,480],\\\"spans\\\":[[18,3]],\\\"text\\\":\\\"10.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]}],[{\\\"bbox\\\":[144,462,271,471],\\\"spans\\\":[[19,0]],\\\"text\\\":\\\"Gemma-7B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]},{\\\"spans\\\":[[19,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]},{\\\"bbox\\\":[289,464,468,470],\\\"spans\\\":[[19,2]],\\\"text\\\":\\\"10.4 18.3 19.5 9.8 18.3 14.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]},{\\\"bbox\\\":[483,464,497,470],\\\"spans\\\":[[19,3]],\\\"text\\\":\\\"15.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]}],[{\\\"bbox\\\":[136,453,271,461],\\\"spans\\\":[[20,0]],\\\"text\\\":\\\"Mistral-7B-v0.2 Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":20,\\\"row-header\\\":false,\\\"row-span\\\":[20,21]},{\\\"spans\\\":[[20,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":20,\\\"row-header\\\":false,\\\"row-span\\\":[20,21]},{\\\"bbox\\\":[289,455,468,461],\\\"spans\\\":[[20,2]],\\\"text\\\":\\\"22.0 23.8 34.8 16.5 14.6 12.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":20,\\\"row-header\\\":false,\\\"row-span\\\":[20,21]},{\\\"bbox\\\":[483,455,497,461],\\\"spans\\\":[[20,3]],\\\"text\\\":\\\"20.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":20,\\\"row-header\\\":false,\\\"row-span\\\":[20,21]}],[{\\\"bbox\\\":[132,444,271,452],\\\"spans\\\":[[21,0]],\\\"text\\\":\\\"Mixtral-8x7B-v0.1 Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":21,\\\"row-header\\\":false,\\\"row-span\\\":[21,22]},{\\\"spans\\\":[[21,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":21,\\\"row-header\\\":false,\\\"row-span\\\":[21,22]},{\\\"bbox\\\":[289,446,468,452],\\\"spans\\\":[[21,2]],\\\"text\\\":\\\"17.1 18.3 35.4 19.5 18.9 15.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":21,\\\"row-header\\\":false,\\\"row-span\\\":[21,22]},{\\\"bbox\\\":[483,446,497,452],\\\"spans\\\":[[21,3]],\\\"text\\\":\\\"20.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":21,\\\"row-header\\\":false,\\\"row-span\\\":[21,22]}],[{\\\"bbox\\\":[130,434,271,443],\\\"spans\\\":[[22,0]],\\\"text\\\":\\\"Mixtral-8x22B-v0.1 Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":22,\\\"row-header\\\":false,\\\"row-span\\\":[22,23]},{\\\"spans\\\":[[22,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":22,\\\"row-header\\\":false,\\\"row-span\\\":[22,23]},{\\\"bbox\\\":[289,436,468,442],\\\"spans\\\":[[22,2]],\\\"text\\\":\\\"29.9 20.1 40.2 17.7 22.0 17.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":22,\\\"row-header\\\":false,\\\"row-span\\\":[22,23]},{\\\"bbox\\\":[483,436,497,442],\\\"spans\\\":[[22,3]],\\\"text\\\":\\\"24.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":22,\\\"row-header\\\":false,\\\"row-span\\\":[22,23]}],[{\\\"bbox\\\":[143,425,271,433],\\\"spans\\\":[[23,0]],\\\"text\\\":\\\"Llama-3-8B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":23,\\\"row-header\\\":false,\\\"row-span\\\":[23,24]},{\\\"bbox\\\":[289,427,466,433],\\\"spans\\\":[[23,1]],\\\"text\\\":\\\"15.2 14.0 18.9 5.5 18.3 8.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":23,\\\"row-header\\\":false,\\\"row-span\\\":[23,24]},{\\\"spans\\\":[[23,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":23,\\\"row-header\\\":false,\\\"row-span\\\":[23,24]},{\\\"bbox\\\":[483,427,497,433],\\\"spans\\\":[[23,3]],\\\"text\\\":\\\"13.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":23,\\\"row-header\\\":false,\\\"row-span\\\":[23,24]}],[{\\\"bbox\\\":[141,416,271,424],\\\"spans\\\":[[24,0]],\\\"text\\\":\\\"Llama-3-70B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":24,\\\"row-header\\\":false,\\\"row-span\\\":[24,25]},{\\\"spans\\\":[[24,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":24,\\\"row-header\\\":false,\\\"row-span\\\":[24,25]},{\\\"bbox\\\":[289,418,468,424],\\\"spans\\\":[[24,2]],\\\"text\\\":\\\"12.2 18.9 20.7 9.1 16.5 15.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":24,\\\"row-header\\\":false,\\\"row-span\\\":[24,25]},{\\\"bbox\\\":[483,418,497,424],\\\"spans\\\":[[24,3]],\\\"text\\\":\\\"15.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":24,\\\"row-header\\\":false,\\\"row-span\\\":[24,25]}],[{\\\"bbox\\\":[277,404,335,410],\\\"spans\\\":[[25,0],[25,1],[25,2],[25,3]],\\\"text\\\":\\\"Instruct Models\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,4],\\\"row\\\":25,\\\"row-header\\\":false,\\\"row-span\\\":[25,26]},{\\\"bbox\\\":[277,404,335,410],\\\"spans\\\":[[25,0],[25,1],[25,2],[25,3]],\\\"text\\\":\\\"Instruct Models\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[0,4],\\\"row\\\":25,\\\"row-header\\\":false,\\\"row-span\\\":[25,26]},{\\\"bbox\\\":[277,404,335,410],\\\"spans\\\":[[25,0],[25,1],[25,2],[25,3]],\\\"text\\\":\\\"Instruct Models\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[0,4],\\\"row\\\":25,\\\"row-header\\\":false,\\\"row-span\\\":[25,26]},{\\\"bbox\\\":[277,404,335,410],\\\"spans\\\":[[25,0],[25,1],[25,2],[25,3]],\\\"text\\\":\\\"Instruct Models\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[0,4],\\\"row\\\":25,\\\"row-header\\\":false,\\\"row-span\\\":[25,26]}],[{\\\"bbox\\\":[129,390,269,396],\\\"spans\\\":[[26,0]],\\\"text\\\":\\\"CodeGemma-7B-IT Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":26,\\\"row-header\\\":false,\\\"row-span\\\":[26,27]},{\\\"spans\\\":[[26,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":26,\\\"row-header\\\":false,\\\"row-span\\\":[26,27]},{\\\"bbox\\\":[289,390,468,396],\\\"spans\\\":[[26,2]],\\\"text\\\":\\\"48.2 40.9 51.8 31.1 33.5 25.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":26,\\\"row-header\\\":false,\\\"row-span\\\":[26,27]},{\\\"bbox\\\":[482,390,497,396],\\\"spans\\\":[[26,3]],\\\"text\\\":\\\"38.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":26,\\\"row-header\\\":false,\\\"row-span\\\":[26,27]}],[{\\\"bbox\\\":[121,380,269,387],\\\"spans\\\":[[27,0]],\\\"text\\\":\\\"CodeLlama-7B-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":27,\\\"row-header\\\":false,\\\"row-span\\\":[27,28]},{\\\"spans\\\":[[27,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":27,\\\"row-header\\\":false,\\\"row-span\\\":[27,28]},{\\\"bbox\\\":[289,380,468,386],\\\"spans\\\":[[27,2]],\\\"text\\\":\\\"29.9 29.9 32.9 19.5 25.0 13.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":27,\\\"row-header\\\":false,\\\"row-span\\\":[27,28]},{\\\"bbox\\\":[483,380,496,386],\\\"spans\\\":[[27,3]],\\\"text\\\":\\\"25.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":27,\\\"row-header\\\":false,\\\"row-span\\\":[27,28]}],[{\\\"bbox\\\":[119,371,269,377],\\\"spans\\\":[[28,0]],\\\"text\\\":\\\"CodeLlama-13B-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":28,\\\"row-header\\\":false,\\\"row-span\\\":[28,29]},{\\\"spans\\\":[[28,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":28,\\\"row-header\\\":false,\\\"row-span\\\":[28,29]},{\\\"bbox\\\":[289,371,468,377],\\\"spans\\\":[[28,2]],\\\"text\\\":\\\"38.4 28.0 36.0 22.6 26.8 14.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":28,\\\"row-header\\\":false,\\\"row-span\\\":[28,29]},{\\\"bbox\\\":[483,371,497,377],\\\"spans\\\":[[28,3]],\\\"text\\\":\\\"27.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":28,\\\"row-header\\\":false,\\\"row-span\\\":[28,29]}],[{\\\"bbox\\\":[119,362,269,368],\\\"spans\\\":[[29,0]],\\\"text\\\":\\\"CodeLlama-34B-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":29,\\\"row-header\\\":false,\\\"row-span\\\":[29,30]},{\\\"spans\\\":[[29,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":29,\\\"row-header\\\":false,\\\"row-span\\\":[29,30]},{\\\"bbox\\\":[289,362,468,368],\\\"spans\\\":[[29,2]],\\\"text\\\":\\\"42.1 31.1 39.6 20.1 31.7 17.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":29,\\\"row-header\\\":false,\\\"row-span\\\":[29,30]},{\\\"bbox\\\":[482,362,497,368],\\\"spans\\\":[[29,3]],\\\"text\\\":\\\"30.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":29,\\\"row-header\\\":false,\\\"row-span\\\":[29,30]}],[{\\\"bbox\\\":[119,352,269,359],\\\"spans\\\":[[30,0]],\\\"text\\\":\\\"CodeLlama-70B-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":30,\\\"row-header\\\":false,\\\"row-span\\\":[30,31]},{\\\"spans\\\":[[30,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":30,\\\"row-header\\\":false,\\\"row-span\\\":[30,31]},{\\\"bbox\\\":[289,352,468,358],\\\"spans\\\":[[30,2]],\\\"text\\\":\\\"47.0 40.2 54.9 34.1 46.3 23.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":30,\\\"row-header\\\":false,\\\"row-span\\\":[30,31]},{\\\"bbox\\\":[482,352,496,358],\\\"spans\\\":[[30,3]],\\\"text\\\":\\\"41.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":30,\\\"row-header\\\":false,\\\"row-span\\\":[30,31]}],[{\\\"bbox\\\":[136,343,269,349],\\\"spans\\\":[[31,0]],\\\"text\\\":\\\"OctoCoder-15B Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":31,\\\"row-header\\\":false,\\\"row-span\\\":[31,32]},{\\\"spans\\\":[[31,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":31,\\\"row-header\\\":false,\\\"row-span\\\":[31,32]},{\\\"bbox\\\":[289,343,468,349],\\\"spans\\\":[[31,2]],\\\"text\\\":\\\"37.8 26.8 31.1 18.3 22.6 14.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":31,\\\"row-header\\\":false,\\\"row-span\\\":[31,32]},{\\\"bbox\\\":[483,343,496,349],\\\"spans\\\":[[31,3]],\\\"text\\\":\\\"25.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":31,\\\"row-header\\\":false,\\\"row-span\\\":[31,32]}],[{\\\"bbox\\\":[118,329,269,335],\\\"spans\\\":[[32,0]],\\\"text\\\":\\\"Granite-3b-Code-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":32,\\\"row-header\\\":false,\\\"row-span\\\":[32,33]},{\\\"spans\\\":[[32,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":32,\\\"row-header\\\":false,\\\"row-span\\\":[32,33]},{\\\"bbox\\\":[289,329,468,335],\\\"spans\\\":[[32,2]],\\\"text\\\":\\\"39.6 26.8 39.0 14.0 23.8 12.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":32,\\\"row-header\\\":false,\\\"row-span\\\":[32,33]},{\\\"bbox\\\":[483,329,497,335],\\\"spans\\\":[[32,3]],\\\"text\\\":\\\"26.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":32,\\\"row-header\\\":false,\\\"row-span\\\":[32,33]}],[{\\\"bbox\\\":[118,320,269,326],\\\"spans\\\":[[33,0]],\\\"text\\\":\\\"Granite-8b-Code-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":33,\\\"row-header\\\":false,\\\"row-span\\\":[33,34]},{\\\"spans\\\":[[33,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":33,\\\"row-header\\\":false,\\\"row-span\\\":[33,34]},{\\\"bbox\\\":[289,320,468,326],\\\"spans\\\":[[33,2]],\\\"text\\\":\\\"53.0 42.7 52.4 36.6 43.9 16.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":33,\\\"row-header\\\":false,\\\"row-span\\\":[33,34]},{\\\"bbox\\\":[482,320,497,326],\\\"spans\\\":[[33,3]],\\\"text\\\":\\\"40.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":33,\\\"row-header\\\":false,\\\"row-span\\\":[33,34]}],[{\\\"bbox\\\":[116,310,269,317],\\\"spans\\\":[[34,0]],\\\"text\\\":\\\"Granite-20B-Code-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":34,\\\"row-header\\\":false,\\\"row-span\\\":[34,35]},{\\\"spans\\\":[[34,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":34,\\\"row-header\\\":false,\\\"row-span\\\":[34,35]},{\\\"bbox\\\":[289,310,468,316],\\\"spans\\\":[[34,2]],\\\"text\\\":\\\"44.5 42.7 49.4 32.3 42.1 18.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":34,\\\"row-header\\\":false,\\\"row-span\\\":[34,35]},{\\\"bbox\\\":[482,310,497,316],\\\"spans\\\":[[34,3]],\\\"text\\\":\\\"38.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":34,\\\"row-header\\\":false,\\\"row-span\\\":[34,35]}],[{\\\"bbox\\\":[116,301,269,307],\\\"spans\\\":[[35,0]],\\\"text\\\":\\\"Granite-34B-Code-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":35,\\\"row-header\\\":false,\\\"row-span\\\":[35,36]},{\\\"spans\\\":[[35,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":35,\\\"row-header\\\":false,\\\"row-span\\\":[35,36]},{\\\"bbox\\\":[289,301,468,307],\\\"spans\\\":[[35,2]],\\\"text\\\":\\\"53.0 45.1 50.6 36.0 42.7 23.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":35,\\\"row-header\\\":false,\\\"row-span\\\":[35,36]},{\\\"bbox\\\":[482,301,497,307],\\\"spans\\\":[[35,3]],\\\"text\\\":\\\"41.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":35,\\\"row-header\\\":false,\\\"row-span\\\":[35,36]}],[{\\\"bbox\\\":[139,287,269,293],\\\"spans\\\":[[36,0]],\\\"text\\\":\\\"Gemma-2B-IT Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":36,\\\"row-header\\\":false,\\\"row-span\\\":[36,37]},{\\\"bbox\\\":[291,287,466,293],\\\"spans\\\":[[36,1]],\\\"text\\\":\\\"9.8 12.8 13.4 8.5 13.4 3.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":36,\\\"row-header\\\":false,\\\"row-span\\\":[36,37]},{\\\"spans\\\":[[36,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":36,\\\"row-header\\\":false,\\\"row-span\\\":[36,37]},{\\\"bbox\\\":[483,287,497,293],\\\"spans\\\":[[36,3]],\\\"text\\\":\\\"10.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":36,\\\"row-header\\\":false,\\\"row-span\\\":[36,37]}],[{\\\"bbox\\\":[139,278,269,284],\\\"spans\\\":[[37,0]],\\\"text\\\":\\\"Gemma-7B-IT Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":37,\\\"row-header\\\":false,\\\"row-span\\\":[37,38]},{\\\"bbox\\\":[289,278,466,284],\\\"spans\\\":[[37,1]],\\\"text\\\":\\\"31.1 23.2 28.0 14.6 23.8 8.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":37,\\\"row-header\\\":false,\\\"row-span\\\":[37,38]},{\\\"spans\\\":[[37,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":37,\\\"row-header\\\":false,\\\"row-span\\\":[37,38]},{\\\"bbox\\\":[483,278,497,284],\\\"spans\\\":[[37,3]],\\\"text\\\":\\\"21.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":37,\\\"row-header\\\":false,\\\"row-span\\\":[37,38]}],[{\\\"bbox\\\":[120,268,269,275],\\\"spans\\\":[[38,0]],\\\"text\\\":\\\"Mistral-7B-Instruct-v0.2 Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":38,\\\"row-header\\\":false,\\\"row-span\\\":[38,39]},{\\\"bbox\\\":[289,268,468,274],\\\"spans\\\":[[38,1]],\\\"text\\\":\\\"24.4 26.8 31.1 22.6 25.6 14.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":38,\\\"row-header\\\":false,\\\"row-span\\\":[38,39]},{\\\"spans\\\":[[38,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":38,\\\"row-header\\\":false,\\\"row-span\\\":[38,39]},{\\\"bbox\\\":[483,268,496,274],\\\"spans\\\":[[38,3]],\\\"text\\\":\\\"24.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":38,\\\"row-header\\\":false,\\\"row-span\\\":[38,39]}],[{\\\"bbox\\\":[116,259,269,265],\\\"spans\\\":[[39,0]],\\\"text\\\":\\\"Mixtral-8x7B-Instruct-v0.1 Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":39,\\\"row-header\\\":false,\\\"row-span\\\":[39,40]},{\\\"spans\\\":[[39,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":39,\\\"row-header\\\":false,\\\"row-span\\\":[39,40]},{\\\"bbox\\\":[289,259,468,265],\\\"spans\\\":[[39,2]],\\\"text\\\":\\\"47.0 40.9 48.2 28.0 32.9 25.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":39,\\\"row-header\\\":false,\\\"row-span\\\":[39,40]},{\\\"bbox\\\":[482,259,497,265],\\\"spans\\\":[[39,3]],\\\"text\\\":\\\"37.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":39,\\\"row-header\\\":false,\\\"row-span\\\":[39,40]}],[{\\\"bbox\\\":[114,250,269,256],\\\"spans\\\":[[40,0]],\\\"text\\\":\\\"Mixtral-8x22B-Instruct-v0.1 Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":40,\\\"row-header\\\":false,\\\"row-span\\\":[40,41]},{\\\"spans\\\":[[40,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":40,\\\"row-header\\\":false,\\\"row-span\\\":[40,41]},{\\\"bbox\\\":[289,250,468,256],\\\"spans\\\":[[40,2]],\\\"text\\\":\\\"67.1 56.7 67.7 44.5 64.0 39.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":40,\\\"row-header\\\":false,\\\"row-span\\\":[40,41]},{\\\"bbox\\\":[482,250,497,256],\\\"spans\\\":[[40,3]],\\\"text\\\":\\\"56.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":40,\\\"row-header\\\":false,\\\"row-span\\\":[40,41]}],[{\\\"bbox\\\":[128,240,269,247],\\\"spans\\\":[[41,0]],\\\"text\\\":\\\"Llama-3-8B-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":41,\\\"row-header\\\":false,\\\"row-span\\\":[41,42]},{\\\"bbox\\\":[289,240,468,246],\\\"spans\\\":[[41,1]],\\\"text\\\":\\\"36.0 31.7 40.2 19.5 31.1 15.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":41,\\\"row-header\\\":false,\\\"row-span\\\":[41,42]},{\\\"spans\\\":[[41,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":41,\\\"row-header\\\":false,\\\"row-span\\\":[41,42]},{\\\"bbox\\\":[483,240,496,246],\\\"spans\\\":[[41,3]],\\\"text\\\":\\\"29.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":41,\\\"row-header\\\":false,\\\"row-span\\\":[41,42]}],[{\\\"bbox\\\":[126,231,269,237],\\\"spans\\\":[[42,0]],\\\"text\\\":\\\"Llama-3-70B-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":42,\\\"row-header\\\":false,\\\"row-span\\\":[42,43]},{\\\"spans\\\":[[42,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":42,\\\"row-header\\\":false,\\\"row-span\\\":[42,43]},{\\\"bbox\\\":[289,231,468,237],\\\"spans\\\":[[42,2]],\\\"text\\\":\\\"50.6 47.6 57.9 34.8 48.2 33.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":42,\\\"row-header\\\":false,\\\"row-span\\\":[42,43]},{\\\"bbox\\\":[482,231,497,237],\\\"spans\\\":[[42,3]],\\\"text\\\":\\\"45.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":42,\\\"row-header\\\":false,\\\"row-span\\\":[42,43]}]]},{\\\"prov\\\":[{\\\"bbox\\\":[107,225,505,687],\\\"page\\\":16,\\\"span\\\":[0,0]}],\\\"text\\\":\\\"Table 12: Pass@1 and ExcessCode performance on CanItEdit. Pass@1 assesses functional correctness, and ExcessCode assesses conciseness and precision of code edits.\\\",\\\"type\\\":\\\"table\\\",\\\"#-cols\\\":4,\\\"#-rows\\\":45,\\\"data\\\":[[{\\\"bbox\\\":[152,675,263,683],\\\"spans\\\":[[0,0]],\\\"text\\\":\\\"Model Prompt\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[282,675,470,683],\\\"spans\\\":[[0,1]],\\\"text\\\":\\\"Python JavaScript Java Go C++ Rust\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"spans\\\":[[0,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[481,675,498,683],\\\"spans\\\":[[0,3]],\\\"text\\\":\\\"Avg.\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]}],[{\\\"bbox\\\":[283,663,329,669],\\\"spans\\\":[[1,0],[1,1],[1,2],[1,3]],\\\"text\\\":\\\"Base Models\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,4],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"bbox\\\":[283,663,329,669],\\\"spans\\\":[[1,0],[1,1],[1,2],[1,3]],\\\"text\\\":\\\"Base Models\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[0,4],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"bbox\\\":[283,663,329,669],\\\"spans\\\":[[1,0],[1,1],[1,2],[1,3]],\\\"text\\\":\\\"Base Models\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[0,4],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"bbox\\\":[283,663,329,669],\\\"spans\\\":[[1,0],[1,1],[1,2],[1,3]],\\\"text\\\":\\\"Base Models\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[0,4],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]}],[{\\\"bbox\\\":[131,647,271,655],\\\"spans\\\":[[2,0]],\\\"text\\\":\\\"StarCoderBase-3B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"bbox\\\":[289,649,466,655],\\\"spans\\\":[[2,2]],\\\"text\\\":\\\"12.2 9.8 6.1 7.9 1.8 0.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"bbox\\\":[485,649,495,655],\\\"spans\\\":[[2,3]],\\\"text\\\":\\\"6.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]}],[{\\\"bbox\\\":[138,638,271,646],\\\"spans\\\":[[3,0]],\\\"text\\\":\\\"StableCode-3B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"spans\\\":[[3,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[289,640,466,646],\\\"spans\\\":[[3,2]],\\\"text\\\":\\\"11.0 7.3 20.1 10.4 12.8 1.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[483,640,497,646],\\\"spans\\\":[[3,3]],\\\"text\\\":\\\"10.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]}],[{\\\"bbox\\\":[138,628,271,637],\\\"spans\\\":[[4,0]],\\\"text\\\":\\\"StarCoder2-3B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"bbox\\\":[289,630,303,636],\\\"spans\\\":[[4,1]],\\\"text\\\":\\\"18.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"bbox\\\":[334,630,466,636],\\\"spans\\\":[[4,2]],\\\"text\\\":\\\"15.9 12.8 12.8 5.5 0.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"bbox\\\":[483,630,497,636],\\\"spans\\\":[[4,3]],\\\"text\\\":\\\"11.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]}],[{\\\"bbox\\\":[134,619,271,627],\\\"spans\\\":[[5,0]],\\\"text\\\":\\\"CodeGemma-2B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"spans\\\":[[5,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[291,621,466,627],\\\"spans\\\":[[5,2]],\\\"text\\\":\\\"4.3 7.3 3.0 9.8 1.8 0.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[484,621,495,627],\\\"spans\\\":[[5,3]],\\\"text\\\":\\\"4.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]}],[{\\\"bbox\\\":[124,609,271,618],\\\"spans\\\":[[6,0]],\\\"text\\\":\\\"Granite-3B-Code-Base Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"bbox\\\":[289,612,466,617],\\\"spans\\\":[[6,2]],\\\"text\\\":\\\"18.3 23.2 29.9 24.4 16.5 3.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"bbox\\\":[483,612,497,617],\\\"spans\\\":[[6,3]],\\\"text\\\":\\\"19.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]}],[{\\\"bbox\\\":[131,595,271,604],\\\"spans\\\":[[7,0]],\\\"text\\\":\\\"StarCoderBase-7B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"bbox\\\":[289,598,466,604],\\\"spans\\\":[[7,2]],\\\"text\\\":\\\"15.9 21.3 17.1 14.6 5.5 0.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"bbox\\\":[483,598,497,604],\\\"spans\\\":[[7,3]],\\\"text\\\":\\\"12.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]}],[{\\\"bbox\\\":[137,586,271,595],\\\"spans\\\":[[8,0]],\\\"text\\\":\\\"CodeLlama-7B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"bbox\\\":[289,588,468,594],\\\"spans\\\":[[8,2]],\\\"text\\\":\\\"15.9 14.0 23.8 15.2 5.5 11.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"bbox\\\":[483,588,497,594],\\\"spans\\\":[[8,3]],\\\"text\\\":\\\"14.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]}],[{\\\"spans\\\":[[9,0]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"spans\\\":[[9,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[291,579,466,585],\\\"spans\\\":[[9,2]],\\\"text\\\":\\\"5.5 13.4 15.9 11.0 7.3 0.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"spans\\\":[[9,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]}],[{\\\"bbox\\\":[138,577,271,585],\\\"spans\\\":[[10,0]],\\\"text\\\":\\\"StarCoder2-7B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"spans\\\":[[10,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"spans\\\":[[10,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[485,579,495,585],\\\"spans\\\":[[10,3]],\\\"text\\\":\\\"8.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]}],[{\\\"bbox\\\":[134,567,271,576],\\\"spans\\\":[[11,0]],\\\"text\\\":\\\"CodeGemma-7B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"spans\\\":[[11,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"bbox\\\":[291,570,466,576],\\\"spans\\\":[[11,2]],\\\"text\\\":\\\"8.5 5.5 20.1 14.6 7.9 3.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"bbox\\\":[483,570,496,576],\\\"spans\\\":[[11,3]],\\\"text\\\":\\\"10.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]}],[{\\\"bbox\\\":[124,558,271,567],\\\"spans\\\":[[12,0]],\\\"text\\\":\\\"Granite-8B-Code-Base Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"spans\\\":[[12,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"bbox\\\":[289,560,468,566],\\\"spans\\\":[[12,2]],\\\"text\\\":\\\"22.6 35.4 38.4 37.2 28.7 15.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"bbox\\\":[483,560,497,566],\\\"spans\\\":[[12,3]],\\\"text\\\":\\\"29.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]}],[{\\\"bbox\\\":[129,544,271,553],\\\"spans\\\":[[13,0]],\\\"text\\\":\\\"StarCoderBase-15B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"spans\\\":[[13,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"bbox\\\":[289,546,466,552],\\\"spans\\\":[[13,2]],\\\"text\\\":\\\"10.4 17.7 17.1 18.9 9.8 3.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"bbox\\\":[483,546,497,552],\\\"spans\\\":[[13,3]],\\\"text\\\":\\\"12.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]}],[{\\\"bbox\\\":[135,535,271,543],\\\"spans\\\":[[14,0]],\\\"text\\\":\\\"CodeLlama-13B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"spans\\\":[[14,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"bbox\\\":[291,537,468,543],\\\"spans\\\":[[14,2]],\\\"text\\\":\\\"6.1 9.1 17.1 9.8 6.1 10.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"bbox\\\":[485,537,495,543],\\\"spans\\\":[[14,3]],\\\"text\\\":\\\"9.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]}],[{\\\"bbox\\\":[136,525,271,534],\\\"spans\\\":[[15,0]],\\\"text\\\":\\\"StarCoder2-15B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"spans\\\":[[15,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"bbox\\\":[291,528,468,534],\\\"spans\\\":[[15,2]],\\\"text\\\":\\\"9.1 18.9 25.0 37.2 25.0 16.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"bbox\\\":[483,528,497,534],\\\"spans\\\":[[15,3]],\\\"text\\\":\\\"21.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]}],[{\\\"bbox\\\":[135,502,271,511],\\\"spans\\\":[[16,0]],\\\"text\\\":\\\"CodeLlama-34B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"spans\\\":[[16,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"bbox\\\":[289,504,466,510],\\\"spans\\\":[[16,2]],\\\"text\\\":\\\"14.0 20.7 20.1 26.2 31.1 6.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"bbox\\\":[483,504,497,510],\\\"spans\\\":[[16,3]],\\\"text\\\":\\\"19.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]}],[{\\\"bbox\\\":[121,493,271,501],\\\"spans\\\":[[17,0]],\\\"text\\\":\\\"Granite-34B-Code-Base Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"spans\\\":[[17,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"bbox\\\":[289,495,468,501],\\\"spans\\\":[[17,2]],\\\"text\\\":\\\"20.1 30.5 40.9 34.1 39.0 12.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"bbox\\\":[483,495,497,501],\\\"spans\\\":[[17,3]],\\\"text\\\":\\\"29.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]}],[{\\\"bbox\\\":[135,483,271,492],\\\"spans\\\":[[18,0]],\\\"text\\\":\\\"CodeLlama-70B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]},{\\\"spans\\\":[[18,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]},{\\\"bbox\\\":[289,486,468,492],\\\"spans\\\":[[18,2]],\\\"text\\\":\\\"12.8 31.1 41.5 42.1 38.4 31.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]},{\\\"bbox\\\":[482,486,497,492],\\\"spans\\\":[[18,3]],\\\"text\\\":\\\"32.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]}],[{\\\"bbox\\\":[144,469,271,478],\\\"spans\\\":[[19,0]],\\\"text\\\":\\\"Gemma-2B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]},{\\\"spans\\\":[[19,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]},{\\\"bbox\\\":[291,472,466,478],\\\"spans\\\":[[19,2]],\\\"text\\\":\\\"1.2 2.4 4.3 2.4 1.2 0.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]},{\\\"bbox\\\":[485,472,495,478],\\\"spans\\\":[[19,3]],\\\"text\\\":\\\"1.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]}],[{\\\"bbox\\\":[144,460,271,469],\\\"spans\\\":[[20,0]],\\\"text\\\":\\\"Gemma-7B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":20,\\\"row-header\\\":false,\\\"row-span\\\":[20,21]},{\\\"spans\\\":[[20,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":20,\\\"row-header\\\":false,\\\"row-span\\\":[20,21]},{\\\"bbox\\\":[291,462,466,468],\\\"spans\\\":[[20,2]],\\\"text\\\":\\\"1.8 1.2 26.2 7.3 6.7 1.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":20,\\\"row-header\\\":false,\\\"row-span\\\":[20,21]},{\\\"bbox\\\":[485,462,495,468],\\\"spans\\\":[[20,3]],\\\"text\\\":\\\"7.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":20,\\\"row-header\\\":false,\\\"row-span\\\":[20,21]}],[{\\\"bbox\\\":[136,451,271,459],\\\"spans\\\":[[21,0]],\\\"text\\\":\\\"Mistral-7B-v0.2 Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":21,\\\"row-header\\\":false,\\\"row-span\\\":[21,22]},{\\\"spans\\\":[[21,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":21,\\\"row-header\\\":false,\\\"row-span\\\":[21,22]},{\\\"bbox\\\":[291,453,466,459],\\\"spans\\\":[[21,2]],\\\"text\\\":\\\"3.0 2.4 6.1 9.1 5.5 0.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":21,\\\"row-header\\\":false,\\\"row-span\\\":[21,22]},{\\\"bbox\\\":[484,453,495,459],\\\"spans\\\":[[21,3]],\\\"text\\\":\\\"4.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":21,\\\"row-header\\\":false,\\\"row-span\\\":[21,22]}],[{\\\"bbox\\\":[132,441,271,450],\\\"spans\\\":[[22,0]],\\\"text\\\":\\\"Mixtral-8x7B-v0.1 Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":22,\\\"row-header\\\":false,\\\"row-span\\\":[22,23]},{\\\"spans\\\":[[22,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":22,\\\"row-header\\\":false,\\\"row-span\\\":[22,23]},{\\\"bbox\\\":[289,444,466,450],\\\"spans\\\":[[22,2]],\\\"text\\\":\\\"11.0 12.8 18.3 25.0 15.9 3.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":22,\\\"row-header\\\":false,\\\"row-span\\\":[22,23]},{\\\"bbox\\\":[483,444,497,450],\\\"spans\\\":[[22,3]],\\\"text\\\":\\\"14.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":22,\\\"row-header\\\":false,\\\"row-span\\\":[22,23]}],[{\\\"bbox\\\":[130,432,271,441],\\\"spans\\\":[[23,0]],\\\"text\\\":\\\"Mixtral-8x22B-v0.1 Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":23,\\\"row-header\\\":false,\\\"row-span\\\":[23,24]},{\\\"spans\\\":[[23,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":23,\\\"row-header\\\":false,\\\"row-span\\\":[23,24]},{\\\"bbox\\\":[289,434,468,440],\\\"spans\\\":[[23,2]],\\\"text\\\":\\\"17.1 30.5 23.8 30.5 32.3 11.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":23,\\\"row-header\\\":false,\\\"row-span\\\":[23,24]},{\\\"bbox\\\":[483,434,497,440],\\\"spans\\\":[[23,3]],\\\"text\\\":\\\"24.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":23,\\\"row-header\\\":false,\\\"row-span\\\":[23,24]}],[{\\\"bbox\\\":[143,423,271,431],\\\"spans\\\":[[24,0]],\\\"text\\\":\\\"Llama-3-8B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":24,\\\"row-header\\\":false,\\\"row-span\\\":[24,25]},{\\\"spans\\\":[[24,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":24,\\\"row-header\\\":false,\\\"row-span\\\":[24,25]},{\\\"bbox\\\":[291,425,466,431],\\\"spans\\\":[[24,2]],\\\"text\\\":\\\"2.6 3.7 5.5 3.7 1.3 0.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":24,\\\"row-header\\\":false,\\\"row-span\\\":[24,25]},{\\\"bbox\\\":[485,425,495,431],\\\"spans\\\":[[24,3]],\\\"text\\\":\\\"2.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":24,\\\"row-header\\\":false,\\\"row-span\\\":[24,25]}],[{\\\"bbox\\\":[141,413,271,422],\\\"spans\\\":[[25,0]],\\\"text\\\":\\\"Llama-3-70B Completion\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":25,\\\"row-header\\\":false,\\\"row-span\\\":[25,26]},{\\\"spans\\\":[[25,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":25,\\\"row-header\\\":false,\\\"row-span\\\":[25,26]},{\\\"bbox\\\":[289,416,468,422],\\\"spans\\\":[[25,2]],\\\"text\\\":\\\"31.7 33.5 39.0 28.0 28.7 12.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":25,\\\"row-header\\\":false,\\\"row-span\\\":[25,26]},{\\\"bbox\\\":[483,416,497,422],\\\"spans\\\":[[25,3]],\\\"text\\\":\\\"28.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":25,\\\"row-header\\\":false,\\\"row-span\\\":[25,26]}],[{\\\"bbox\\\":[277,402,335,408],\\\"spans\\\":[[26,0],[26,1],[26,2],[26,3]],\\\"text\\\":\\\"Instruct Models\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,4],\\\"row\\\":26,\\\"row-header\\\":false,\\\"row-span\\\":[26,27]},{\\\"bbox\\\":[277,402,335,408],\\\"spans\\\":[[26,0],[26,1],[26,2],[26,3]],\\\"text\\\":\\\"Instruct Models\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[0,4],\\\"row\\\":26,\\\"row-header\\\":false,\\\"row-span\\\":[26,27]},{\\\"bbox\\\":[277,402,335,408],\\\"spans\\\":[[26,0],[26,1],[26,2],[26,3]],\\\"text\\\":\\\"Instruct Models\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[0,4],\\\"row\\\":26,\\\"row-header\\\":false,\\\"row-span\\\":[26,27]},{\\\"bbox\\\":[277,402,335,408],\\\"spans\\\":[[26,0],[26,1],[26,2],[26,3]],\\\"text\\\":\\\"Instruct Models\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[0,4],\\\"row\\\":26,\\\"row-header\\\":false,\\\"row-span\\\":[26,27]}],[{\\\"bbox\\\":[129,388,269,394],\\\"spans\\\":[[27,0]],\\\"text\\\":\\\"CodeGemma-7B-IT Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":27,\\\"row-header\\\":false,\\\"row-span\\\":[27,28]},{\\\"spans\\\":[[27,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":27,\\\"row-header\\\":false,\\\"row-span\\\":[27,28]},{\\\"bbox\\\":[289,388,468,394],\\\"spans\\\":[[27,2]],\\\"text\\\":\\\"46.3 45.7 52.4 48.2 43.9 38.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":27,\\\"row-header\\\":false,\\\"row-span\\\":[27,28]},{\\\"bbox\\\":[482,388,497,394],\\\"spans\\\":[[27,3]],\\\"text\\\":\\\"45.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":27,\\\"row-header\\\":false,\\\"row-span\\\":[27,28]}],[{\\\"bbox\\\":[121,378,269,384],\\\"spans\\\":[[28,0]],\\\"text\\\":\\\"CodeLlama-7B-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":28,\\\"row-header\\\":false,\\\"row-span\\\":[28,29]},{\\\"spans\\\":[[28,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":28,\\\"row-header\\\":false,\\\"row-span\\\":[28,29]},{\\\"bbox\\\":[289,378,466,384],\\\"spans\\\":[[28,2]],\\\"text\\\":\\\"19.5 18.9 13.4 14.6 6.1 4.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":28,\\\"row-header\\\":false,\\\"row-span\\\":[28,29]},{\\\"bbox\\\":[483,378,497,384],\\\"spans\\\":[[28,3]],\\\"text\\\":\\\"12.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":28,\\\"row-header\\\":false,\\\"row-span\\\":[28,29]}],[{\\\"bbox\\\":[119,369,269,375],\\\"spans\\\":[[29,0]],\\\"text\\\":\\\"CodeLlama-13B-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":29,\\\"row-header\\\":false,\\\"row-span\\\":[29,30]},{\\\"spans\\\":[[29,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":29,\\\"row-header\\\":false,\\\"row-span\\\":[29,30]},{\\\"bbox\\\":[289,369,466,375],\\\"spans\\\":[[29,2]],\\\"text\\\":\\\"18.9 18.9 24.4 22.6 9.8 0.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":29,\\\"row-header\\\":false,\\\"row-span\\\":[29,30]},{\\\"bbox\\\":[483,369,497,375],\\\"spans\\\":[[29,3]],\\\"text\\\":\\\"15.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":29,\\\"row-header\\\":false,\\\"row-span\\\":[29,30]}],[{\\\"bbox\\\":[119,360,269,366],\\\"spans\\\":[[30,0]],\\\"text\\\":\\\"CodeLlama-34B-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":30,\\\"row-header\\\":false,\\\"row-span\\\":[30,31]},{\\\"spans\\\":[[30,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":30,\\\"row-header\\\":false,\\\"row-span\\\":[30,31]},{\\\"bbox\\\":[289,360,468,366],\\\"spans\\\":[[30,2]],\\\"text\\\":\\\"37.8 28.0 37.2 24.4 24.4 17.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":30,\\\"row-header\\\":false,\\\"row-span\\\":[30,31]},{\\\"bbox\\\":[483,360,497,366],\\\"spans\\\":[[30,3]],\\\"text\\\":\\\"28.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":30,\\\"row-header\\\":false,\\\"row-span\\\":[30,31]}],[{\\\"bbox\\\":[119,350,269,356],\\\"spans\\\":[[31,0]],\\\"text\\\":\\\"CodeLlama-70B-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":31,\\\"row-header\\\":false,\\\"row-span\\\":[31,32]},{\\\"spans\\\":[[31,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":31,\\\"row-header\\\":false,\\\"row-span\\\":[31,32]},{\\\"bbox\\\":[289,350,468,356],\\\"spans\\\":[[31,2]],\\\"text\\\":\\\"64.6 52.4 57.3 51.8 51.2 40.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":31,\\\"row-header\\\":false,\\\"row-span\\\":[31,32]},{\\\"bbox\\\":[482,350,497,356],\\\"spans\\\":[[31,3]],\\\"text\\\":\\\"53.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":31,\\\"row-header\\\":false,\\\"row-span\\\":[31,32]}],[{\\\"bbox\\\":[136,341,269,347],\\\"spans\\\":[[32,0]],\\\"text\\\":\\\"OctoCoder-15B Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":32,\\\"row-header\\\":false,\\\"row-span\\\":[32,33]},{\\\"spans\\\":[[32,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":32,\\\"row-header\\\":false,\\\"row-span\\\":[32,33]},{\\\"bbox\\\":[289,341,468,347],\\\"spans\\\":[[32,2]],\\\"text\\\":\\\"28.0 28.7 34.1 26.8 25.0 15.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":32,\\\"row-header\\\":false,\\\"row-span\\\":[32,33]},{\\\"bbox\\\":[483,341,497,347],\\\"spans\\\":[[32,3]],\\\"text\\\":\\\"26.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":32,\\\"row-header\\\":false,\\\"row-span\\\":[32,33]}],[{\\\"bbox\\\":[118,327,269,333],\\\"spans\\\":[[33,0]],\\\"text\\\":\\\"Granite-3b-Code-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":33,\\\"row-header\\\":false,\\\"row-span\\\":[33,34]},{\\\"spans\\\":[[33,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":33,\\\"row-header\\\":false,\\\"row-span\\\":[33,34]},{\\\"bbox\\\":[289,327,468,333],\\\"spans\\\":[[33,2]],\\\"text\\\":\\\"26.8 28.0 33.5 27.4 31.7 16.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":33,\\\"row-header\\\":false,\\\"row-span\\\":[33,34]},{\\\"bbox\\\":[483,327,497,333],\\\"spans\\\":[[33,3]],\\\"text\\\":\\\"27.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":33,\\\"row-header\\\":false,\\\"row-span\\\":[33,34]}],[{\\\"bbox\\\":[118,318,269,324],\\\"spans\\\":[[34,0]],\\\"text\\\":\\\"Granite-8b-Code-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":34,\\\"row-header\\\":false,\\\"row-span\\\":[34,35]},{\\\"spans\\\":[[34,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":34,\\\"row-header\\\":false,\\\"row-span\\\":[34,35]},{\\\"bbox\\\":[289,318,468,324],\\\"spans\\\":[[34,2]],\\\"text\\\":\\\"39.6 40.9 48.2 41.5 39.0 32.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":34,\\\"row-header\\\":false,\\\"row-span\\\":[34,35]},{\\\"bbox\\\":[482,318,497,324],\\\"spans\\\":[[34,3]],\\\"text\\\":\\\"40.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":34,\\\"row-header\\\":false,\\\"row-span\\\":[34,35]}],[{\\\"bbox\\\":[116,308,269,314],\\\"spans\\\":[[35,0]],\\\"text\\\":\\\"Granite-20B-Code-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":35,\\\"row-header\\\":false,\\\"row-span\\\":[35,36]},{\\\"spans\\\":[[35,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":35,\\\"row-header\\\":false,\\\"row-span\\\":[35,36]},{\\\"bbox\\\":[289,308,468,314],\\\"spans\\\":[[35,2]],\\\"text\\\":\\\"43.9 43.9 45.7 41.5 41.5 29.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":35,\\\"row-header\\\":false,\\\"row-span\\\":[35,36]},{\\\"bbox\\\":[482,308,496,314],\\\"spans\\\":[[35,3]],\\\"text\\\":\\\"41.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":35,\\\"row-header\\\":false,\\\"row-span\\\":[35,36]}],[{\\\"bbox\\\":[116,299,269,305],\\\"spans\\\":[[36,0]],\\\"text\\\":\\\"Granite-34B-Code-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":36,\\\"row-header\\\":false,\\\"row-span\\\":[36,37]},{\\\"spans\\\":[[36,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":36,\\\"row-header\\\":false,\\\"row-span\\\":[36,37]},{\\\"bbox\\\":[289,299,468,305],\\\"spans\\\":[[36,2]],\\\"text\\\":\\\"54.9 47.6 55.5 51.2 47.0 45.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":36,\\\"row-header\\\":false,\\\"row-span\\\":[36,37]},{\\\"bbox\\\":[482,299,497,305],\\\"spans\\\":[[36,3]],\\\"text\\\":\\\"50.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":36,\\\"row-header\\\":false,\\\"row-span\\\":[36,37]}],[{\\\"spans\\\":[[37,0]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":37,\\\"row-header\\\":false,\\\"row-span\\\":[37,38]},{\\\"spans\\\":[[37,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":37,\\\"row-header\\\":false,\\\"row-span\\\":[37,38]},{\\\"bbox\\\":[289,285,468,291],\\\"spans\\\":[[37,2]],\\\"text\\\":\\\"18.9 15.9 25.6 13.4 15.9 10.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":37,\\\"row-header\\\":false,\\\"row-span\\\":[37,38]},{\\\"bbox\\\":[483,285,497,291],\\\"spans\\\":[[37,3]],\\\"text\\\":\\\"16.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":37,\\\"row-header\\\":false,\\\"row-span\\\":[37,38]}],[{\\\"bbox\\\":[139,285,269,291],\\\"spans\\\":[[38,0]],\\\"text\\\":\\\"Gemma-2B-IT Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":38,\\\"row-header\\\":false,\\\"row-span\\\":[38,39]},{\\\"spans\\\":[[38,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":38,\\\"row-header\\\":false,\\\"row-span\\\":[38,39]},{\\\"bbox\\\":[289,276,466,282],\\\"spans\\\":[[38,2]],\\\"text\\\":\\\"10.3 9.8 22.0 14.3 16.2 9.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":38,\\\"row-header\\\":false,\\\"row-span\\\":[38,39]},{\\\"spans\\\":[[38,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":38,\\\"row-header\\\":false,\\\"row-span\\\":[38,39]}],[{\\\"bbox\\\":[139,276,269,282],\\\"spans\\\":[[39,0]],\\\"text\\\":\\\"Gemma-7B-IT Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":39,\\\"row-header\\\":false,\\\"row-span\\\":[39,40]},{\\\"spans\\\":[[39,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":39,\\\"row-header\\\":false,\\\"row-span\\\":[39,40]},{\\\"spans\\\":[[39,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":39,\\\"row-header\\\":false,\\\"row-span\\\":[39,40]},{\\\"bbox\\\":[483,276,497,282],\\\"spans\\\":[[39,3]],\\\"text\\\":\\\"13.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":39,\\\"row-header\\\":false,\\\"row-span\\\":[39,40]}],[{\\\"bbox\\\":[120,266,269,272],\\\"spans\\\":[[40,0]],\\\"text\\\":\\\"Mistral-7B-Instruct-v0.2 Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":40,\\\"row-header\\\":false,\\\"row-span\\\":[40,41]},{\\\"spans\\\":[[40,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":40,\\\"row-header\\\":false,\\\"row-span\\\":[40,41]},{\\\"bbox\\\":[289,266,468,272],\\\"spans\\\":[[40,2]],\\\"text\\\":\\\"33.5 22.6 27.4 27.4 26.2 23.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":40,\\\"row-header\\\":false,\\\"row-span\\\":[40,41]},{\\\"bbox\\\":[483,266,497,272],\\\"spans\\\":[[40,3]],\\\"text\\\":\\\"26.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":40,\\\"row-header\\\":false,\\\"row-span\\\":[40,41]}],[{\\\"bbox\\\":[116,257,269,263],\\\"spans\\\":[[41,0]],\\\"text\\\":\\\"Mixtral-8x7B-Instruct-v0.1 Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":41,\\\"row-header\\\":false,\\\"row-span\\\":[41,42]},{\\\"spans\\\":[[41,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":41,\\\"row-header\\\":false,\\\"row-span\\\":[41,42]},{\\\"bbox\\\":[289,257,468,263],\\\"spans\\\":[[41,2]],\\\"text\\\":\\\"44.5 37.8 47.6 38.4 39.0 28.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":41,\\\"row-header\\\":false,\\\"row-span\\\":[41,42]},{\\\"bbox\\\":[482,257,497,263],\\\"spans\\\":[[41,3]],\\\"text\\\":\\\"39.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":41,\\\"row-header\\\":false,\\\"row-span\\\":[41,42]}],[{\\\"bbox\\\":[114,238,269,254],\\\"spans\\\":[[42,0]],\\\"text\\\":\\\"Mixtral-8x22B-Instruct-v0.1 Instruction  Llama-3-8B-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":42,\\\"row-header\\\":false,\\\"row-span\\\":[42,43]},{\\\"spans\\\":[[42,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":42,\\\"row-header\\\":false,\\\"row-span\\\":[42,43]},{\\\"bbox\\\":[289,248,468,253],\\\"spans\\\":[[42,2]],\\\"text\\\":\\\"59.1 53.7 66.5 55.5 56.1 45.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":42,\\\"row-header\\\":false,\\\"row-span\\\":[42,43]},{\\\"bbox\\\":[482,238,497,253],\\\"spans\\\":[[42,3]],\\\"text\\\":\\\"56.1 29.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":42,\\\"row-header\\\":false,\\\"row-span\\\":[42,43]}],[{\\\"bbox\\\":[126,229,269,235],\\\"spans\\\":[[43,0]],\\\"text\\\":\\\"Llama-3-70B-Instruct Instruction\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":43,\\\"row-header\\\":false,\\\"row-span\\\":[43,44]},{\\\"spans\\\":[[43,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":43,\\\"row-header\\\":false,\\\"row-span\\\":[43,44]},{\\\"bbox\\\":[289,238,468,244],\\\"spans\\\":[[43,2]],\\\"text\\\":\\\"40.2 25.6 43.3 29.9 13.4 23.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":43,\\\"row-header\\\":false,\\\"row-span\\\":[43,44]},{\\\"bbox\\\":[482,229,497,235],\\\"spans\\\":[[43,3]],\\\"text\\\":\\\"52.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":43,\\\"row-header\\\":false,\\\"row-span\\\":[43,44]}],[{\\\"spans\\\":[[44,0]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":44,\\\"row-header\\\":false,\\\"row-span\\\":[44,45]},{\\\"spans\\\":[[44,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":44,\\\"row-header\\\":false,\\\"row-span\\\":[44,45]},{\\\"bbox\\\":[289,229,468,235],\\\"spans\\\":[[44,2]],\\\"text\\\":\\\"57.3 51.2 54.3 51.8 50.6 49.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":44,\\\"row-header\\\":false,\\\"row-span\\\":[44,45]},{\\\"spans\\\":[[44,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":44,\\\"row-header\\\":false,\\\"row-span\\\":[44,45]}]]},{\\\"prov\\\":[{\\\"bbox\\\":[120,64,492,181],\\\"page\\\":16,\\\"span\\\":[0,0]}],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"table\\\",\\\"#-cols\\\":3,\\\"#-rows\\\":9,\\\"data\\\":[[{\\\"bbox\\\":[166,167,438,176],\\\"spans\\\":[[0,0]],\\\"text\\\":\\\"Model Descriptive Lazy\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"spans\\\":[[0,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"spans\\\":[[0,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]}],[{\\\"spans\\\":[[1,0]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"bbox\\\":[244,153,485,161],\\\"spans\\\":[[1,2]],\\\"text\\\":\\\"Pass@1 (\\\\u2191) ExcessCode (\\\\u2193) Pass@1 (\\\\u2191) ExcessCode (\\\\u2193)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]}],[{\\\"bbox\\\":[141,139,463,146],\\\"spans\\\":[[2,0]],\\\"text\\\":\\\"CodeGemma-7B-IT 31.57 1.03 25.28 0.58\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]}],[{\\\"bbox\\\":[133,129,463,136],\\\"spans\\\":[[3,0]],\\\"text\\\":\\\"CodeLlama-7B-Instruct 33.23 0.80 24.80 0.45\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"spans\\\":[[3,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"spans\\\":[[3,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]}],[{\\\"bbox\\\":[129,119,229,126],\\\"spans\\\":[[4,0]],\\\"text\\\":\\\"Granite-8B-Code-Instruct\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"bbox\\\":[255,119,463,126],\\\"spans\\\":[[4,1]],\\\"text\\\":\\\"39.72 0.30 32.38 0.08\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]}],[{\\\"bbox\\\":[131,104,227,111],\\\"spans\\\":[[5,0]],\\\"text\\\":\\\"CodeLlama-13B-Instruct\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[255,104,401,111],\\\"spans\\\":[[5,1]],\\\"text\\\":\\\"41.09 0.20 29.85\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[448,104,463,111],\\\"spans\\\":[[5,2]],\\\"text\\\":\\\"0.70\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]}],[{\\\"bbox\\\":[127,94,401,101],\\\"spans\\\":[[6,0]],\\\"text\\\":\\\"Granite-20B-Code-Instruct 40.52 0.30 29.72\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"bbox\\\":[448,95,463,101],\\\"spans\\\":[[6,2]],\\\"text\\\":\\\"0.02\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]}],[{\\\"bbox\\\":[131,80,274,86],\\\"spans\\\":[[7,0]],\\\"text\\\":\\\"CodeLlama-34B-Instruct 46.28\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"bbox\\\":[321,80,401,86],\\\"spans\\\":[[7,1]],\\\"text\\\":\\\"0.05 37.90\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"bbox\\\":[448,80,463,86],\\\"spans\\\":[[7,2]],\\\"text\\\":\\\"0.43\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]}],[{\\\"bbox\\\":[127,70,274,76],\\\"spans\\\":[[8,0]],\\\"text\\\":\\\"Granite-34B-Code-Instruct  50.28\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"bbox\\\":[320,70,401,76],\\\"spans\\\":[[8,1]],\\\"text\\\":\\\"0.25 37.28\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"bbox\\\":[448,70,463,76],\\\"spans\\\":[[8,2]],\\\"text\\\":\\\"0.05\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]}]]},{\\\"prov\\\":[{\\\"bbox\\\":[114,561,498,679],\\\"page\\\":17,\\\"span\\\":[0,0]}],\\\"text\\\":\\\"Table 13: Performance of different instruction-tuned models on CodeLingua (Pan et al. , 2024). We report Pass@1 for translation across five languages.\\\",\\\"type\\\":\\\"table\\\",\\\"#-cols\\\":5,\\\"#-rows\\\":11,\\\"data\\\":[[{\\\"bbox\\\":[135,667,455,675],\\\"spans\\\":[[0,0]],\\\"text\\\":\\\"Source Language C C++ Go\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"spans\\\":[[0,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"spans\\\":[[0,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"spans\\\":[[0,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"spans\\\":[[0,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]}],[{\\\"bbox\\\":[136,655,493,663],\\\"spans\\\":[[1,0]],\\\"text\\\":\\\"Target Language C++ Go Java Py C Go Java Py C C++ Java Py\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]}],[{\\\"bbox\\\":[121,645,492,650],\\\"spans\\\":[[2,0]],\\\"text\\\":\\\"Granite-3B-Code-Instruct 78.5 43.5 64.5 38.5 43.5 25.5 68 46.5 46.5 62.5 2.5 20.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]}],[{\\\"bbox\\\":[132,632,198,638],\\\"spans\\\":[[3,0]],\\\"text\\\":\\\"CodeGemma-7B-IT\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[221,632,257,638],\\\"spans\\\":[[3,1]],\\\"text\\\":\\\"79.5 48.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[269,632,328,638],\\\"spans\\\":[[3,2]],\\\"text\\\":\\\"60.5 44.0 29.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[337,632,351,637],\\\"spans\\\":[[3,3]],\\\"text\\\":\\\"32.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[362,632,492,638],\\\"spans\\\":[[3,4]],\\\"text\\\":\\\"58.0 39.0 31.0 49.5 55.5 33.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]}],[{\\\"bbox\\\":[125,624,492,629],\\\"spans\\\":[[4,0]],\\\"text\\\":\\\"CodeLlama-7B-Instruct 18.0 3.0 59.5 31.0 9.0 13.0 54.5 28.0 5.5 31.5 0.0 5.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]}],[{\\\"bbox\\\":[121,615,257,621],\\\"spans\\\":[[5,0]],\\\"text\\\":\\\"Granite-8B-Code-Instruct 78.5 18.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"spans\\\":[[5,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[269,615,328,620],\\\"spans\\\":[[5,2]],\\\"text\\\":\\\"72.0 57.0 54.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[337,615,350,621],\\\"spans\\\":[[5,3]],\\\"text\\\":\\\"24.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[362,615,493,620],\\\"spans\\\":[[5,4]],\\\"text\\\":\\\"74.0 56.5 50.0 65.5 58.0 52.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]}],[{\\\"bbox\\\":[123,603,492,608],\\\"spans\\\":[[6,0]],\\\"text\\\":\\\"CodeLlama-13B-Instruct 2.5 41.5 60.0 2.5 0.0 23.0 59.0 11.5 1.5 2.5 21.0 27.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]}],[{\\\"bbox\\\":[139,594,492,600],\\\"spans\\\":[[7,0]],\\\"text\\\":\\\"OctoCoder-15B 1.5 39.0 1.0 1.0 0.0 29.0 4.0 0.5 0.5 6.5 0.0 37.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]}],[{\\\"bbox\\\":[119,586,210,591],\\\"spans\\\":[[8,0]],\\\"text\\\":\\\"Granite-20B-Code-Instruct\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"bbox\\\":[221,586,493,591],\\\"spans\\\":[[8,3]],\\\"text\\\":\\\"81.5 50.0 71.5 39.0 55.5 31.5 70.5 40.5 55.0 63.0 58.0 55.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]}],[{\\\"bbox\\\":[123,573,234,579],\\\"spans\\\":[[9,0]],\\\"text\\\":\\\"CodeLlama-34B-Instruct 2.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[244,573,257,578],\\\"spans\\\":[[9,1]],\\\"text\\\":\\\"49.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[269,573,328,579],\\\"spans\\\":[[9,2]],\\\"text\\\":\\\"60.0 28.5 2.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[337,573,351,578],\\\"spans\\\":[[9,3]],\\\"text\\\":\\\"26.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[362,573,492,579],\\\"spans\\\":[[9,4]],\\\"text\\\":\\\"60.5 33.0 0.5 3.0 38.5 18.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]}],[{\\\"bbox\\\":[119,565,210,570],\\\"spans\\\":[[10,0]],\\\"text\\\":\\\"Granite-34B-Code-Instruct\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[221,565,257,570],\\\"spans\\\":[[10,1]],\\\"text\\\":\\\"83.0  15.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[269,565,328,570],\\\"spans\\\":[[10,2]],\\\"text\\\":\\\"74.5 54.5 61.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[337,565,350,570],\\\"spans\\\":[[10,3]],\\\"text\\\":\\\"25.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[362,565,492,570],\\\"spans\\\":[[10,4]],\\\"text\\\":\\\"74.0 42.5 58.0 62.0 68.0 66.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]}]]},{\\\"prov\\\":[{\\\"bbox\\\":[161,434,451,553],\\\"page\\\":17,\\\"span\\\":[0,0]}],\\\"text\\\":\\\"Table 13 shows the results on the CodeLingua benchmark. For the source languages C, C++ and Go the results reported in the table are taken directly from the runs on Codenet, whereas for Java and Python the results are reported as the average of the runs on Avatar and CodeNet. We report the numbers of Octocoder and CodeLlama from the CodeLingua leaderboard 16. The Granite Code models performs comparably to CodeGemma. It is worth noticing that the correctness of the translation is not only due to the code generated by the model, but also by the extra metadata and explanation provided as part of the answer. We tested instruction tuned models, as we observed that base models often struggle to understand the request itself to translate code. Instruct models, on the other hand, tend to add additional information besides the translated code as part of the generations. The CodeLLama family seems to suffer especially from this issue, as post-processing the\\\",\\\"type\\\":\\\"table\\\",\\\"#-cols\\\":3,\\\"#-rows\\\":11,\\\"data\\\":[[{\\\"bbox\\\":[182,541,415,548],\\\"spans\\\":[[0,0]],\\\"text\\\":\\\"Source Language Java Python\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"spans\\\":[[0,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"spans\\\":[[0,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]}],[{\\\"bbox\\\":[183,528,445,536],\\\"spans\\\":[[1,0]],\\\"text\\\":\\\"Target Language C C++ Go Py C C++ Go Java\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]}],[{\\\"bbox\\\":[178,518,445,524],\\\"spans\\\":[[2,0]],\\\"text\\\":\\\"Granite-3B-Code-IT 34.2 63.2 20.1 44.1 18.8 45.6 1.2 15.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]}],[{\\\"bbox\\\":[179,505,421,511],\\\"spans\\\":[[3,0]],\\\"text\\\":\\\"CodeGemma-7B-IT 33.7 31.3 10.3 31.3 22.9 33.4 4.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"spans\\\":[[3,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[432,506,445,511],\\\"spans\\\":[[3,2]],\\\"text\\\":\\\"44.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]}],[{\\\"bbox\\\":[172,497,398,503],\\\"spans\\\":[[4,0]],\\\"text\\\":\\\"CodeLlama-7B-Instruct 5.3 17.8 15.8 11.6 2.55 18.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"bbox\\\":[408,497,445,502],\\\"spans\\\":[[4,2]],\\\"text\\\":\\\"14.7  28.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]}],[{\\\"bbox\\\":[169,488,256,494],\\\"spans\\\":[[5,0]],\\\"text\\\":\\\"Granite-8B-Code-Instruct\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[267,488,398,494],\\\"spans\\\":[[5,1]],\\\"text\\\":\\\"49.6 57.3 17.1 57.9 41.3 55.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[408,488,445,494],\\\"spans\\\":[[5,2]],\\\"text\\\":\\\"10.5 39.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]}],[{\\\"bbox\\\":[170,476,445,482],\\\"spans\\\":[[6,0]],\\\"text\\\":\\\"CodeLlama-13B-Instruct 0.0 1.25 7.6 34.2 0.0 0.9 9.6 13.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]}],[{\\\"bbox\\\":[186,467,445,473],\\\"spans\\\":[[7,0]],\\\"text\\\":\\\"OctoCoder-15B 0.3 3.2 18.8 22.9 0.0 2.65 16.8 32.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]}],[{\\\"bbox\\\":[167,459,258,465],\\\"spans\\\":[[8,0]],\\\"text\\\":\\\"Granite-20B-Code-Instruct\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"bbox\\\":[267,459,445,464],\\\"spans\\\":[[8,1]],\\\"text\\\":\\\"44.8 52.8 33.4 34.3 41.8 48.4 26.1 53.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]}],[{\\\"bbox\\\":[170,446,445,452],\\\"spans\\\":[[9,0]],\\\"text\\\":\\\"CodeLlama-34B-Instruct 0.0 1.8 9.8 20.6 0.5 1.0 14.1 10.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"spans\\\":[[9,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"spans\\\":[[9,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]}],[{\\\"bbox\\\":[167,438,258,444],\\\"spans\\\":[[10,0]],\\\"text\\\":\\\"Granite-34B-Code-Instruct\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[267,438,445,443],\\\"spans\\\":[[10,1]],\\\"text\\\":\\\"55.0 65.9 28.5 56.3 45.7 53.7 34.1 61.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"spans\\\":[[10,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]}]]},{\\\"prov\\\":[{\\\"bbox\\\":[184,384,427,596],\\\"page\\\":18,\\\"span\\\":[0,0]}],\\\"text\\\":\\\"Table 14: Performance on the CRUXEval benchmark. We use temperature 0.2 for pass@1 and temperature 0.8 for pass@5, both using 10 samples, as in (Gu et al. , 2024).\\\",\\\"type\\\":\\\"table\\\",\\\"#-cols\\\":3,\\\"#-rows\\\":18,\\\"data\\\":[[{\\\"bbox\\\":[224,579,250,585],\\\"spans\\\":[[0,0]],\\\"text\\\":\\\"Model\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[299,584,419,590],\\\"spans\\\":[[0,1],[0,2]],\\\"text\\\":\\\"CRUXEval-I CRUXEval-O\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,3],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[299,584,419,590],\\\"spans\\\":[[0,1],[0,2]],\\\"text\\\":\\\"CRUXEval-I CRUXEval-O\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[1,3],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]}],[{\\\"spans\\\":[[1,0]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"bbox\\\":[296,569,421,575],\\\"spans\\\":[[1,1]],\\\"text\\\":\\\"Pass1 Pass5 Pass1 Pass5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]}],[{\\\"bbox\\\":[202,554,417,561],\\\"spans\\\":[[2,0]],\\\"text\\\":\\\"StarCoderBase-3B 27.5 44.9 27.0 41.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]}],[{\\\"bbox\\\":[209,544,266,551],\\\"spans\\\":[[3,0]],\\\"text\\\":\\\"StableCode-3B\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[299,544,417,550],\\\"spans\\\":[[3,1]],\\\"text\\\":\\\"33.5 54.2 32.1 44.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"spans\\\":[[3,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]}],[{\\\"bbox\\\":[209,534,417,541],\\\"spans\\\":[[4,0]],\\\"text\\\":\\\"StarCoder2-3B 32.1 50.3 34.0 48.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]}],[{\\\"bbox\\\":[205,524,417,531],\\\"spans\\\":[[5,0]],\\\"text\\\":\\\"CodeGemma-2B 29.6 45.5 32.1 45.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"spans\\\":[[5,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"spans\\\":[[5,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]}],[{\\\"bbox\\\":[194,514,417,521],\\\"spans\\\":[[6,0]],\\\"text\\\":\\\"Granite-3B-Code-Base 30.6 50.9 31.4 35.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]}],[{\\\"bbox\\\":[202,499,417,506],\\\"spans\\\":[[7,0]],\\\"text\\\":\\\"StarCoderBase-7B 29.8 47.5 32.1 44.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]}],[{\\\"bbox\\\":[208,489,417,496],\\\"spans\\\":[[8,0]],\\\"text\\\":\\\"CodeLlama-7B 36.2 53.7 34.0 48.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]}],[{\\\"bbox\\\":[209,479,418,486],\\\"spans\\\":[[9,0]],\\\"text\\\":\\\"StarCoder2-7B 34.2 53.8 35.8 49.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"spans\\\":[[9,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"spans\\\":[[9,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]}],[{\\\"bbox\\\":[205,469,270,476],\\\"spans\\\":[[10,0]],\\\"text\\\":\\\"CodeGemma-7B\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[299,469,418,475],\\\"spans\\\":[[10,1]],\\\"text\\\":\\\"42.6 60.9 43.9 56.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"spans\\\":[[10,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]}],[{\\\"bbox\\\":[194,459,417,466],\\\"spans\\\":[[11,0]],\\\"text\\\":\\\"Granite-8B-Code-Base 36.0 55.8 36.1 50.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"spans\\\":[[11,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"spans\\\":[[11,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]}],[{\\\"bbox\\\":[200,444,417,451],\\\"spans\\\":[[12,0]],\\\"text\\\":\\\"StarCoderBase-15B 31.0 49.2 34.4 47.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"spans\\\":[[12,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"spans\\\":[[12,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]}],[{\\\"bbox\\\":[206,434,417,441],\\\"spans\\\":[[13,0]],\\\"text\\\":\\\"CodeLlama-13B 42.2 61.8 39.9 55.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"spans\\\":[[13,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"spans\\\":[[13,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]}],[{\\\"bbox\\\":[206,424,268,431],\\\"spans\\\":[[14,0]],\\\"text\\\":\\\"StarCoder2-15B\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"bbox\\\":[299,424,417,431],\\\"spans\\\":[[14,1]],\\\"text\\\":\\\"47.4 68.3 46.7 59.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"spans\\\":[[14,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]}],[{\\\"bbox\\\":[191,414,418,421],\\\"spans\\\":[[15,0]],\\\"text\\\":\\\"Granite-20B-Code-Base 39.1 59.0 37.5 51.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"spans\\\":[[15,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"spans\\\":[[15,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]}],[{\\\"bbox\\\":[206,399,349,406],\\\"spans\\\":[[16,0]],\\\"text\\\":\\\"CodeLlama-34B  47.8 65.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"bbox\\\":[368,399,417,406],\\\"spans\\\":[[16,1]],\\\"text\\\":\\\"42.5  56.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"spans\\\":[[16,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]}],[{\\\"bbox\\\":[191,390,349,396],\\\"spans\\\":[[17,0]],\\\"text\\\":\\\"Granite-34B-Code-Base 43.3 61.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"bbox\\\":[368,390,417,396],\\\"spans\\\":[[17,1]],\\\"text\\\":\\\"44.8  55.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"spans\\\":[[17,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]}]]},{\\\"prov\\\":[{\\\"bbox\\\":[112,574,500,690],\\\"page\\\":19,\\\"span\\\":[0,0]}],\\\"text\\\":\\\"Table 15: Performance on 4 chain-of-thought math tasks and 2 tool-aided math tasks.\\\",\\\"type\\\":\\\"table\\\",\\\"#-cols\\\":3,\\\"#-rows\\\":10,\\\"data\\\":[[{\\\"spans\\\":[[0,0]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"spans\\\":[[0,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[149,676,484,685],\\\"spans\\\":[[0,2]],\\\"text\\\":\\\"Model MATH GSM8K SAT OCW MATH+Py GSM8K+Py\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]}],[{\\\"bbox\\\":[127,664,467,670],\\\"spans\\\":[[1,0]],\\\"text\\\":\\\"StarCoderBase-7B 2.4 3.8 18.7 2.2 18.2 15.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]}],[{\\\"bbox\\\":[133,654,467,660],\\\"spans\\\":[[2,0]],\\\"text\\\":\\\"CodeLlama-7B 4.1 11.9 12.5 2.9 20.8 26.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]}],[{\\\"bbox\\\":[133,644,467,650],\\\"spans\\\":[[3,0]],\\\"text\\\":\\\"StarCoder2-7B 10.4 27.2 37.5 4.8 28.7 39.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"spans\\\":[[3,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"spans\\\":[[3,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]}],[{\\\"bbox\\\":[129,634,467,640],\\\"spans\\\":[[4,0]],\\\"text\\\":\\\"CodeGemma-7B 21.8 49.0 53.1 6.9 31.1 60.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]}],[{\\\"bbox\\\":[118,624,240,630],\\\"spans\\\":[[5,0]],\\\"text\\\":\\\"Granite-8B-Code-Base 21.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[268,624,283,630],\\\"spans\\\":[[5,1]],\\\"text\\\":\\\"61.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[305,624,467,630],\\\"spans\\\":[[5,2]],\\\"text\\\":\\\"62.5 8.8  35.4 63.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]}],[{\\\"bbox\\\":[140,609,240,615],\\\"spans\\\":[[6,0]],\\\"text\\\":\\\"Gemma-7B  24.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"bbox\\\":[267,609,283,615],\\\"spans\\\":[[6,1]],\\\"text\\\":\\\"53.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"bbox\\\":[305,609,467,615],\\\"spans\\\":[[6,2]],\\\"text\\\":\\\"75.0  7.3 27.4 52.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]}],[{\\\"bbox\\\":[132,599,467,605],\\\"spans\\\":[[7,0]],\\\"text\\\":\\\"Mistral-7B-v0.2 12.8 37.2 53.1 5.8 25.7 45.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]}],[{\\\"bbox\\\":[139,589,320,595],\\\"spans\\\":[[8,0]],\\\"text\\\":\\\"Llama-3-8B 15.6 49.8 34.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"bbox\\\":[339,589,470,596],\\\"spans\\\":[[8,2]],\\\"text\\\":\\\"9.9  0.0 \\\\u22c6  2.4 \\\\u22c6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]}],[{\\\"bbox\\\":[139,579,467,585],\\\"spans\\\":[[9,0]],\\\"text\\\":\\\"Llemma-7B 17.3 33.7 59.4 7.0 25.6 40.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"spans\\\":[[9,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"spans\\\":[[9,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]}]]},{\\\"prov\\\":[{\\\"bbox\\\":[162,472,449,669],\\\"page\\\":21,\\\"span\\\":[0,0]}],\\\"text\\\":\\\"Table 16: RP@1 performance on the Recode benchmark. Following (Wang et al. , 2022), we use the perturbed version of the HumanEval benchmark with greedy sampling for all the models to eliminate randomness effect and enable fair comparison.\\\",\\\"type\\\":\\\"table\\\",\\\"#-cols\\\":3,\\\"#-rows\\\":17,\\\"data\\\":[[{\\\"spans\\\":[[0,0]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[202,654,443,663],\\\"spans\\\":[[0,1]],\\\"text\\\":\\\"Model Docstring Function Syntax Format\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"spans\\\":[[0,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]}],[{\\\"bbox\\\":[180,642,436,648],\\\"spans\\\":[[1,0]],\\\"text\\\":\\\"StarCoderBase-3B 12.3 11.4 17.2 24.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]}],[{\\\"bbox\\\":[187,632,436,638],\\\"spans\\\":[[2,0]],\\\"text\\\":\\\"StableCode-3B 22.8 25.8 37.1 46.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]}],[{\\\"bbox\\\":[187,622,301,628],\\\"spans\\\":[[3,0]],\\\"text\\\":\\\"StarCoder2-3B  28.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[336,622,352,628],\\\"spans\\\":[[3,1]],\\\"text\\\":\\\"29.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[380,622,436,628],\\\"spans\\\":[[3,2]],\\\"text\\\":\\\"49.6 57.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]}],[{\\\"bbox\\\":[183,612,436,618],\\\"spans\\\":[[4,0]],\\\"text\\\":\\\"CodeGemma-2B 12.3 11.4 17.2 24.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]}],[{\\\"bbox\\\":[172,602,301,608],\\\"spans\\\":[[5,0]],\\\"text\\\":\\\"Granite-3B-Code-Base 28.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[336,602,352,608],\\\"spans\\\":[[5,1]],\\\"text\\\":\\\"30.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[380,602,436,608],\\\"spans\\\":[[5,2]],\\\"text\\\":\\\"45.8 56.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]}],[{\\\"bbox\\\":[180,587,436,593],\\\"spans\\\":[[6,0]],\\\"text\\\":\\\"StarCoderBase-7B 23.7 25.3 38.2 47.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]}],[{\\\"bbox\\\":[186,577,436,584],\\\"spans\\\":[[7,0]],\\\"text\\\":\\\"CodeLlama-7B 24.7 27.6 43.0 53.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]}],[{\\\"bbox\\\":[187,567,436,574],\\\"spans\\\":[[8,0]],\\\"text\\\":\\\"StarCoder2-7B 27.6 30.4 45.8 57.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]}],[{\\\"bbox\\\":[183,557,248,564],\\\"spans\\\":[[9,0]],\\\"text\\\":\\\"CodeGemma-7B\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[286,557,436,563],\\\"spans\\\":[[9,1]],\\\"text\\\":\\\"32.3 37.8 55.3 64.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"spans\\\":[[9,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]}],[{\\\"bbox\\\":[172,547,436,554],\\\"spans\\\":[[10,0]],\\\"text\\\":\\\"Granite-8B-Code-Base 25.5 30.9 49.9 60.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"spans\\\":[[10,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"spans\\\":[[10,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]}],[{\\\"bbox\\\":[178,532,436,539],\\\"spans\\\":[[11,0]],\\\"text\\\":\\\"StarCoderBase-15B 26.6 30.7 44.3 52.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"spans\\\":[[11,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"spans\\\":[[11,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]}],[{\\\"bbox\\\":[184,522,436,529],\\\"spans\\\":[[12,0]],\\\"text\\\":\\\"CodeLlama-13B 25.8 29.7 50.6 60.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"spans\\\":[[12,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"spans\\\":[[12,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]}],[{\\\"bbox\\\":[184,512,246,519],\\\"spans\\\":[[13,0]],\\\"text\\\":\\\"StarCoder2-15B\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"bbox\\\":[286,512,436,518],\\\"spans\\\":[[13,1]],\\\"text\\\":\\\"36.9 43.9 60.4 70.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"spans\\\":[[13,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]}],[{\\\"bbox\\\":[169,502,436,509],\\\"spans\\\":[[14,0]],\\\"text\\\":\\\"Granite-20B-Code-Base 35.2 43.0 55.1 63.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"spans\\\":[[14,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"spans\\\":[[14,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]}],[{\\\"bbox\\\":[184,487,436,494],\\\"spans\\\":[[15,0]],\\\"text\\\":\\\"CodeLlama-34B 33.1 38.0 54.7 64.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"spans\\\":[[15,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"spans\\\":[[15,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]}],[{\\\"bbox\\\":[169,477,261,484],\\\"spans\\\":[[16,0]],\\\"text\\\":\\\"Granite-34B-Code-Base\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"spans\\\":[[16,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"bbox\\\":[286,477,436,483],\\\"spans\\\":[[16,2]],\\\"text\\\":\\\"36.3 44.4 59.2 68.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]}]]}],\\\"equations\\\":[],\\\"footnotes\\\":[],\\\"page-dimensions\\\":[{\\\"height\\\":792,\\\"page\\\":1,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":2,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":3,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":4,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":5,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":6,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":7,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":8,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":9,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":10,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":11,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":12,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":13,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":14,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":15,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":16,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":17,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":18,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":19,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":20,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":21,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":22,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":23,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":24,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":25,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":26,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":27,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":28,\\\"width\\\":612}],\\\"page-footers\\\":[],\\\"page-headers\\\":[]}\",\n          \"{\\\"_name\\\":\\\"\\\",\\\"type\\\":\\\"pdf-document\\\",\\\"description\\\":{\\\"logs\\\":[]},\\\"file-info\\\":{\\\"filename\\\":\\\"attention_is_all_you_need.pdf\\\",\\\"document-hash\\\":\\\"bdfaa68d8984f0dc02beaca527b76f207d99b666d31d1da728ee0728182df697\\\",\\\"#-pages\\\":15,\\\"page-hashes\\\":[{\\\"hash\\\":\\\"8834a09ad99e9297886c9f8ad786c2784b7dc66dc6e6adfeff6bf2c1f07926d6\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":1},{\\\"hash\\\":\\\"72ded7022ad3cbfa9b5c4377a9c9b44511251f9489973956c23d2f3321e6307e\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":2},{\\\"hash\\\":\\\"38733274891513257d051950018621d95f73d05d5c70bfd7331def2f1194973d\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":3},{\\\"hash\\\":\\\"699ed16bf81021d0f86374d05c7b4b2b1049e63a28d2951ec1fb930747d755b9\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":4},{\\\"hash\\\":\\\"a17e6b313bdd51eff07a824253eff394d78ae1d6ebc985de3580bdfece38d2e1\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":5},{\\\"hash\\\":\\\"b3e9b63f2e8728fa83a5b7d911df2827585cf6040d2a4734cb3b44be264da6b6\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":6},{\\\"hash\\\":\\\"7b23bd1c80383b757a39456a4fd95ed2e9aaefd6a04512f181279c27a66c54a4\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":7},{\\\"hash\\\":\\\"c1dbbbf5b2ad441bf20149c26fc440a95f714987edf1f39690d703e67699dbc4\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":8},{\\\"hash\\\":\\\"ae2f68db548ba7a95d11ba1a1f0e36ca46c7d71d40a27c73dc56cf32932a9638\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":9},{\\\"hash\\\":\\\"58c67da2ef05339a90ab5c62b29eece0f60a7d8bb2f9eb390ba45a6d49e042f5\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":10},{\\\"hash\\\":\\\"b39d3fefe795d9d05aad7432498b5baeee7b255ed2da5bc88f60c051b8fae865\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":11},{\\\"hash\\\":\\\"1e5bf23dc7cd6799ee19684b7152560e0bda459b639ac3ea3f6f3fabf96362a4\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":12},{\\\"hash\\\":\\\"0ea258bf68e3835a534da65a943f679a19d02a19ae9b8afe9cd34af2cd29e9c4\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":13},{\\\"hash\\\":\\\"34c515052914fa661b7cedf6a25d7d4bd22a4d329971f74b9d1e75b3f6f02d16\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":14},{\\\"hash\\\":\\\"6ab332fddb6e68c5979d7481a53c9b75d91ac31564cad8972e3fa12cd7362769\\\",\\\"model\\\":\\\"default\\\",\\\"page\\\":15}]},\\\"main-text\\\":[{\\\"text\\\":\\\"arXiv:1706.03762v7 [cs.CL] 2 Aug 2023\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[17,237,36,577],\\\"page\\\":1,\\\"span\\\":[0,37]}]},{\\\"text\\\":\\\"Provided proper attribution is provided, Google hereby grants permission to reproduce the tables and figures in this paper solely for use in journalistic or scholarly works.\\\",\\\"type\\\":\\\"page-header\\\",\\\"name\\\":\\\"Page-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[124,679,488,719],\\\"page\\\":1,\\\"span\\\":[0,173]}]},{\\\"text\\\":\\\"Attention Is All You Need\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[211,630,400,643],\\\"page\\\":1,\\\"span\\\":[0,25]}]},{\\\"text\\\":\\\"Ashish Vaswani \\\\u2217 Google Brain avaswani@google.com\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"meta-data\\\",\\\"prov\\\":[{\\\"bbox\\\":[117,524,216,557],\\\"page\\\":1,\\\"span\\\":[0,49]}]},{\\\"text\\\":\\\"Noam Shazeer \\\\u2217 Google Brain noam@google.com\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"meta-data\\\",\\\"prov\\\":[{\\\"bbox\\\":[231,525,309,557],\\\"page\\\":1,\\\"span\\\":[0,43]}]},{\\\"text\\\":\\\"Niki Parmar \\\\u2217 Google Research nikip@google.com\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"meta-data\\\",\\\"prov\\\":[{\\\"bbox\\\":[323,525,408,557],\\\"page\\\":1,\\\"span\\\":[0,46]}]},{\\\"text\\\":\\\"Jakob Uszkoreit \\\\u2217 Google Research usz@google.com\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"Text\\\",\\\"prov\\\":[{\\\"bbox\\\":[422,525,497,557],\\\"page\\\":1,\\\"span\\\":[0,48]}]},{\\\"text\\\":\\\"Llion Jones \\\\u2217 Google Research llion@google.com\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"meta-data\\\",\\\"prov\\\":[{\\\"bbox\\\":[126,475,210,507],\\\"page\\\":1,\\\"span\\\":[0,46]}]},{\\\"text\\\":\\\"Aidan N. Gomez\\\\u2217 \\\\u2020 University of Toronto aidan@cs.toronto.edu\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"meta-data\\\",\\\"prov\\\":[{\\\"bbox\\\":[235,476,340,508],\\\"page\\\":1,\\\"span\\\":[0,60]}]},{\\\"text\\\":\\\"\\\\u0141ukasz Kaiser \\\\u2217 Google Brain lukaszkaiser@google.com\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"meta-data\\\",\\\"prov\\\":[{\\\"bbox\\\":[364,474,485,507],\\\"page\\\":1,\\\"span\\\":[0,52]}]},{\\\"text\\\":\\\"Illia Polosukhin\\\\u2217 \\\\u2021 illia.polosukhin@gmail.com\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"meta-data\\\",\\\"prov\\\":[{\\\"bbox\\\":[238,436,374,458],\\\"page\\\":1,\\\"span\\\":[0,46]}]},{\\\"text\\\":\\\"Abstract\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[283,397,329,406],\\\"page\\\":1,\\\"span\\\":[0,8]}]},{\\\"text\\\":\\\"The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 Englishto-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[143,215,469,379],\\\"page\\\":1,\\\"span\\\":[0,1138]}]},{\\\"text\\\":\\\"\\\\u2217 Equal contribution. Listing order is random. Jakob proposed replacing RNNs with self-attention and started the effort to evaluate this idea. Ashish, with Illia, designed and implemented the first Transformer models and has been crucially involved in every aspect of this work. Noam proposed scaled dot-product attention, multi-head attention and the parameter-free position representation and became the other person involved in nearly every detail. Niki designed, implemented, tuned and evaluated countless model variants in our original codebase and tensor2tensor. Llion also experimented with novel model variants, was responsible for our initial codebase, and efficient inference and visualizations. Lukasz and Aidan spent countless long days designing various parts of and implementing tensor2tensor, replacing our earlier codebase, greatly improving results and massively accelerating our research.\\\",\\\"type\\\":\\\"footnote\\\",\\\"name\\\":\\\"Footnote\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,105,504,193],\\\"page\\\":1,\\\"span\\\":[0,906]}]},{\\\"text\\\":\\\"\\\\u2020 Work performed while at Google Brain.\\\",\\\"type\\\":\\\"footnote\\\",\\\"name\\\":\\\"Footnote\\\",\\\"prov\\\":[{\\\"bbox\\\":[120,93,267,104],\\\"page\\\":1,\\\"span\\\":[0,39]}]},{\\\"text\\\":\\\"\\\\u2021 Work performed while at Google Research.\\\",\\\"type\\\":\\\"footnote\\\",\\\"name\\\":\\\"Footnote\\\",\\\"prov\\\":[{\\\"bbox\\\":[120,82,280,93],\\\"page\\\":1,\\\"span\\\":[0,42]}]},{\\\"text\\\":\\\"31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,50,460,59],\\\"page\\\":1,\\\"span\\\":[0,90]}]},{\\\"text\\\":\\\"1 Introduction\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,710,191,719],\\\"page\\\":2,\\\"span\\\":[0,14]}]},{\\\"text\\\":\\\"Recurrent neural networks, long short-term memory [13] and gated recurrent [7] neural networks in particular, have been firmly established as state of the art approaches in sequence modeling and transduction problems such as language modeling and machine translation [35 , 2 , 5]. Numerous efforts have since continued to push the boundaries of recurrent language models and encoder-decoder architectures [38, 24, 15].\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,640,504,693],\\\"page\\\":2,\\\"span\\\":[0,418]}]},{\\\"text\\\":\\\"Recurrent models typically factor computation along the symbol positions of the input and output sequences. Aligning the positions to steps in computation time, they generate a sequence of hidden states h t , as a function of the previous hidden state ht - 1 and the input for position t. This inherently sequential nature precludes parallelization within training examples, which becomes critical at longer sequence lengths, as memory constraints limit batching across examples. Recent work has achieved significant improvements in computational efficiency through factorization tricks [21] and conditional computation [32], while also improving model performance in case of the latter. The fundamental constraint of sequential computation, however, remains.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,546,504,633],\\\"page\\\":2,\\\"span\\\":[0,759]}]},{\\\"text\\\":\\\"Attention mechanisms have become an integral part of compelling sequence modeling and transduction models in various tasks, allowing modeling of dependencies without regard to their distance in the input or output sequences [2 , 19]. In all but a few cases [27], however, such attention mechanisms are used in conjunction with a recurrent network.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,498,505,540],\\\"page\\\":2,\\\"span\\\":[0,347]}]},{\\\"text\\\":\\\"In this work we propose the Transformer, a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output. The Transformer allows for significantly more parallelization and can reach a new state of the art in translation quality after being trained for as little as twelve hours on eight P100 GPUs.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,448,505,491],\\\"page\\\":2,\\\"span\\\":[0,383]}]},{\\\"text\\\":\\\"2 Background\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,417,189,429],\\\"page\\\":2,\\\"span\\\":[0,12]}]},{\\\"text\\\":\\\"The goal of reducing sequential computation also forms the foundation of the Extended Neural GPU [16], ByteNet [18] and ConvS2S [9], all of which use convolutional neural networks as basic building block, computing hidden representations in parallel for all input and output positions. In these models, the number of operations required to relate signals from two arbitrary input or output positions grows in the distance between positions, linearly for ConvS2S and logarithmically for ByteNet. This makes it more difficult to learn dependencies between distant positions [12]. In the Transformer this is reduced to a constant number of operations, albeit at the cost of reduced effective resolution due to averaging attention-weighted positions, an effect we counteract with Multi-Head Attention as described in section 3.2.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,307,505,403],\\\"page\\\":2,\\\"span\\\":[0,825]}]},{\\\"text\\\":\\\"Self-attention, sometimes called intra-attention is an attention mechanism relating different positions of a single sequence in order to compute a representation of the sequence. Self-attention has been used successfully in a variety of tasks including reading comprehension, abstractive summarization, textual entailment and learning task-independent sentence representations [4, 27, 28, 22].\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,256,505,299],\\\"page\\\":2,\\\"span\\\":[0,393]}]},{\\\"text\\\":\\\"End-to-end memory networks are based on a recurrent attention mechanism instead of sequencealigned recurrence and have been shown to perform well on simple-language question answering and language modeling tasks [34].\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,218,505,249],\\\"page\\\":2,\\\"span\\\":[0,217]}]},{\\\"text\\\":\\\"To the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequencealigned RNNs or convolution. In the following sections, we will describe the Transformer, motivate self-attention and discuss its advantages over models such as [17, 18] and [9].\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,169,505,211],\\\"page\\\":2,\\\"span\\\":[0,373]}]},{\\\"text\\\":\\\"3 Model Architecture\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,140,226,149],\\\"page\\\":2,\\\"span\\\":[0,20]}]},{\\\"text\\\":\\\"Most competitive neural sequence transduction models have an encoder-decoder structure [5 , 2 , 35]. Here, the encoder maps an input sequence of symbol representations (x1, ..., x n ) to a sequence of continuous representations z = (z1, ..., z n ). Given z, the decoder then generates an output sequence (y1, ..., y m ) of symbols one element at a time. At each step the model is auto-regressive [10], consuming the previously generated symbols as additional input when generating the next.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,70,505,123],\\\"page\\\":2,\\\"span\\\":[0,490]}]},{\\\"text\\\":\\\"2\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[303,42,308,49],\\\"page\\\":2,\\\"span\\\":[0,1]}]},{\\\"text\\\":\\\"Figure 1: The Transformer - model architecture.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[210,378,401,387],\\\"page\\\":3,\\\"span\\\":[0,47]}]},{\\\"name\\\":\\\"Picture\\\",\\\"type\\\":\\\"figure\\\",\\\"$ref\\\":\\\"#\\\\/figures\\\\/0\\\"},{\\\"text\\\":\\\"The Transformer follows this overall architecture using stacked self-attention and point-wise, fully connected layers for both the encoder and decoder, shown in the left and right halves of Figure 1, respectively.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,324,505,355],\\\"page\\\":3,\\\"span\\\":[0,213]}]},{\\\"text\\\":\\\"3.1 Encoder and Decoder Stacks\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,301,253,309],\\\"page\\\":3,\\\"span\\\":[0,30]}]},{\\\"text\\\":\\\"Encoder: The encoder is composed of a stack of N = 6 identical layers. Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, positionwise fully connected feed-forward network. We employ a residual connection [11] around each of the two sub-layers, followed by layer normalization [1]. That is, the output of each sub-layer is LayerNorm(x + Sublayer(x)), where Sublayer(x) is the function implemented by the sub-layer itself. To facilitate these residual connections, all sub-layers in the model, as well as the embedding layers, produce outputs of dimension dmodel = 512 .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,214,505,289],\\\"page\\\":3,\\\"span\\\":[0,629]}]},{\\\"text\\\":\\\"Decoder: The decoder is also composed of a stack of N = 6 identical layers. In addition to the two sub-layers in each encoder layer, the decoder inserts a third sub-layer, which performs multi-head attention over the output of the encoder stack. Similar to the encoder, we employ residual connections around each of the sub-layers, followed by layer normalization. We also modify the self-attention sub-layer in the decoder stack to prevent positions from attending to subsequent positions. This masking, combined with fact that the output embeddings are offset by one position, ensures that the predictions for position i can depend only on the known outputs at positions less than i .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,125,504,200],\\\"page\\\":3,\\\"span\\\":[0,686]}]},{\\\"text\\\":\\\"3.2 Attention\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,102,171,110],\\\"page\\\":3,\\\"span\\\":[0,13]}]},{\\\"text\\\":\\\"An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is computed as a weighted sum\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,69,505,90],\\\"page\\\":3,\\\"span\\\":[0,200]}]},{\\\"text\\\":\\\"3\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[303,42,308,49],\\\"page\\\":3,\\\"span\\\":[0,1]}]},{\\\"text\\\":\\\"Scaled Dot-Product Attention\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"Text\\\",\\\"prov\\\":[{\\\"bbox\\\":[147,713,266,721],\\\"page\\\":4,\\\"span\\\":[0,28]}]},{\\\"name\\\":\\\"Picture\\\",\\\"type\\\":\\\"figure\\\",\\\"$ref\\\":\\\"#\\\\/figures\\\\/1\\\"},{\\\"text\\\":\\\"Figure 2: (left) Scaled Dot-Product Attention. (right) Multi-Head Attention consists of several attention layers running in parallel.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,497,504,518],\\\"page\\\":4,\\\"span\\\":[0,133]}]},{\\\"name\\\":\\\"Picture\\\",\\\"type\\\":\\\"figure\\\",\\\"$ref\\\":\\\"#\\\\/figures\\\\/2\\\"},{\\\"text\\\":\\\"of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,454,504,475],\\\"page\\\":4,\\\"span\\\":[0,135]}]},{\\\"text\\\":\\\"3.2.1 Scaled Dot-Product Attention\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,433,264,441],\\\"page\\\":4,\\\"span\\\":[0,34]}]},{\\\"text\\\":\\\"We call our particular attention \\\\\\\"Scaled Dot-Product Attention\\\\\\\" (Figure 2). The input consists of queries and keys of dimension dk, and values of dimension d v . We compute the dot products of the query with all keys, divide each by \\\\u221adk, and apply a softmax function to obtain the weights on the values.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,381,505,422],\\\"page\\\":4,\\\"span\\\":[0,303]}]},{\\\"text\\\":\\\"In practice, we compute the attention function on a set of queries simultaneously, packed together into a matrix Q. The keys and values are also packed together into matrices K and V . We compute the matrix of outputs as:\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,341,504,373],\\\"page\\\":4,\\\"span\\\":[0,221]}]},{\\\"text\\\":\\\"Attention(Q, K, V ) = softmax( QK T \\\\u221adk )V (1)\\\",\\\"type\\\":\\\"equation\\\",\\\"name\\\":\\\"Formula\\\",\\\"prov\\\":[{\\\"bbox\\\":[220,302,504,327],\\\"page\\\":4,\\\"span\\\":[0,46]}]},{\\\"text\\\":\\\"The two most commonly used attention functions are additive attention [2], and dot-product (multiplicative) attention. Dot-product attention is identical to our algorithm, except for the scaling factor of \\\\u221a 1 dk . Additive attention computes the compatibility function using a feed-forward network with a single hidden layer. While the two are similar in theoretical complexity, dot-product attention is much faster and more space-efficient in practice, since it can be implemented using highly optimized matrix multiplication code.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,225,505,292],\\\"page\\\":4,\\\"span\\\":[0,532]}]},{\\\"text\\\":\\\"While for small values of d k the two mechanisms perform similarly, additive attention outperforms dot product attention without scaling for larger values of dk [3]. We suspect that for large values of d k , the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients 4 . To counteract this effect, we scale the dot products by \\\\u221a 1 dk .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,174,505,220],\\\"page\\\":4,\\\"span\\\":[0,399]}]},{\\\"text\\\":\\\"3.2.2 Multi-Head Attention\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,153,230,161],\\\"page\\\":4,\\\"span\\\":[0,26]}]},{\\\"text\\\":\\\"Instead of performing a single attention function with dmodel-dimensional keys, values and queries, we found it beneficial to linearly project the queries, keys and values h times with different, learned linear projections to dk , d k and d v dimensions, respectively. On each of these projected versions of queries, keys and values we then perform the attention function in parallel, yielding d v -dimensional\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,99,505,142],\\\"page\\\":4,\\\"span\\\":[0,410]}]},{\\\"text\\\":\\\"4 To illustrate why the dot products get large, assume that the components of q and k are independent random variables with mean 0 and variance 1. Then their dot product, q \\\\u00b7 k =P dk i=1 qiki, has mean 0 and variance dk .\\\",\\\"type\\\":\\\"footnote\\\",\\\"name\\\":\\\"Footnote\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,69,504,91],\\\"page\\\":4,\\\"span\\\":[0,221]}]},{\\\"text\\\":\\\"4\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[303,42,308,49],\\\"page\\\":4,\\\"span\\\":[0,1]}]},{\\\"text\\\":\\\"output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,697,504,718],\\\"page\\\":5,\\\"span\\\":[0,119]}]},{\\\"text\\\":\\\"Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,669,504,690],\\\"page\\\":5,\\\"span\\\":[0,189]}]},{\\\"text\\\":\\\"MultiHead(Q, K, V ) = Concat(head1 , ..., head h )W O where head i = Attention(QW  i Q W  i , KW   i K W   i , V W  i V W  i )\\\",\\\"type\\\":\\\"equation\\\",\\\"name\\\":\\\"Formula\\\",\\\"prov\\\":[{\\\"bbox\\\":[186,617,425,646],\\\"page\\\":5,\\\"span\\\":[0,126]}]},{\\\"text\\\":\\\"Where the projections are parameter matrices W  i Q W  i \\\\u2208 R dmodel\\\\u00d7dk , W   i K W   i \\\\u2208 R dmodel\\\\u00d7dk , W  i V W  i \\\\u2208 R dmodel\\\\u00d7d v and W O \\\\u2208 R hd v \\\\u00d7dmodel .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,566,503,588],\\\"page\\\":5,\\\"span\\\":[0,156]}]},{\\\"text\\\":\\\"In this work we employ h = 8 parallel attention layers, or heads. For each of these we use d k = d v = d model \\\\/h = 64. Due to the reduced dimension of each head, the total computational cost is similar to that of single-head attention with full dimensionality.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,526,504,558],\\\"page\\\":5,\\\"span\\\":[0,261]}]},{\\\"text\\\":\\\"3.2.3 Applications of Attention in our Model\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,501,303,512],\\\"page\\\":5,\\\"span\\\":[0,44]}]},{\\\"text\\\":\\\"The Transformer uses multi-head attention in three different ways:\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,483,372,493],\\\"page\\\":5,\\\"span\\\":[0,66]}]},{\\\"text\\\":\\\"\\\\u00b7 In \\\\\\\"encoder-decoder attention\\\\\\\" layers, the queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder. This allows every position in the decoder to attend over all positions in the input sequence. This mimics the typical encoder-decoder attention mechanisms in sequence-to-sequence models such as [38, 2, 9].\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[135,419,505,472],\\\"page\\\":5,\\\"span\\\":[0,364]}]},{\\\"text\\\":\\\"\\\\u00b7 The encoder contains self-attention layers. In a self-attention layer all of the keys, values and queries come from the same place, in this case, the output of the previous layer in the encoder. Each position in the encoder can attend to all positions in the previous layer of the encoder.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[135,372,504,412],\\\"page\\\":5,\\\"span\\\":[0,291]}]},{\\\"text\\\":\\\"\\\\u00b7 Similarly, self-attention layers in the decoder allow each position in the decoder to attend to all positions in the decoder up to and including that position. We need to prevent leftward information flow in the decoder to preserve the auto-regressive property. We implement this inside of scaled dot-product attention by masking out (setting to -\\\\u221e) all values in the input of the softmax which correspond to illegal connections. See Figure 2.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[135,310,504,364],\\\"page\\\":5,\\\"span\\\":[0,445]}]},{\\\"text\\\":\\\"3.3 Position-wise Feed-Forward Networks\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,286,293,294],\\\"page\\\":5,\\\"span\\\":[0,39]}]},{\\\"text\\\":\\\"In addition to attention sub-layers, each of the layers in our encoder and decoder contains a fully connected feed-forward network, which is applied to each position separately and identically. This consists of two linear transformations with a ReLU activation in between.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,243,504,274],\\\"page\\\":5,\\\"span\\\":[0,272]}]},{\\\"text\\\":\\\"FFN(x) = max(0, xW1 W1 + b1)W2 W2 + b2 (2)\\\",\\\"type\\\":\\\"equation\\\",\\\"name\\\":\\\"Formula\\\",\\\"prov\\\":[{\\\"bbox\\\":[227,214,505,224],\\\"page\\\":5,\\\"span\\\":[0,42]}]},{\\\"text\\\":\\\"While the linear transformations are the same across different positions, they use different parameters from layer to layer. Another way of describing this is as two convolutions with kernel size 1. The dimensionality of input and output is dmodel = 512, and the inner-layer has dimensionality df f = 2048 .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,158,505,202],\\\"page\\\":5,\\\"span\\\":[0,307]}]},{\\\"text\\\":\\\"3.4 Embeddings and Softmax\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,134,240,143],\\\"page\\\":5,\\\"span\\\":[0,26]}]},{\\\"text\\\":\\\"Similarly to other sequence transduction models, we use learned embeddings to convert the input tokens and output tokens to vectors of dimension dmodel. We also use the usual learned linear transformation and softmax function to convert the decoder output to predicted next-token probabilities. In our model, we share the same weight matrix between the two embedding layers and the pre-softmax linear transformation, similar to [30]. In the embedding layers, we multiply those weights by \\\\u221admodel .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,70,505,123],\\\"page\\\":5,\\\"span\\\":[0,497]}]},{\\\"text\\\":\\\"5\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[303,41,308,49],\\\"page\\\":5,\\\"span\\\":[0,1]}]},{\\\"text\\\":\\\"Table 1: Maximum path lengths, per-layer complexity and minimum number of sequential operations for different layer types. n is the sequence length, d is the representation dimension, k is the kernel size of convolutions and r the size of the neighborhood in restricted self-attention.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,689,504,721],\\\"page\\\":6,\\\"span\\\":[0,285]}]},{\\\"name\\\":\\\"Table\\\",\\\"type\\\":\\\"table\\\",\\\"$ref\\\":\\\"#\\\\/tables\\\\/0\\\"},{\\\"text\\\":\\\"3.5 Positional Encoding\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,568,215,578],\\\"page\\\":6,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"Since our model contains no recurrence and no convolution, in order for the model to make use of the order of the sequence, we must inject some information about the relative or absolute position of the tokens in the sequence. To this end, we add \\\\\\\"positional encodings\\\\\\\" to the input embeddings at the bottoms of the encoder and decoder stacks. The positional encodings have the same dimension dmodel as the embeddings, so that the two can be summed. There are many choices of positional encodings, learned and fixed [9].\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,494,505,558],\\\"page\\\":6,\\\"span\\\":[0,520]}]},{\\\"text\\\":\\\"In this work, we use sine and cosine functions of different frequencies:\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,477,390,487],\\\"page\\\":6,\\\"span\\\":[0,72]}]},{\\\"text\\\":\\\"P E (pos,2i) = sin(pos\\\\/10000 2i\\\\/dmodel ) P E (pos,2i+1) = cos (pos\\\\/10000 2i\\\\/dmodel )\\\",\\\"type\\\":\\\"equation\\\",\\\"name\\\":\\\"Formula\\\",\\\"prov\\\":[{\\\"bbox\\\":[226,425,386,456],\\\"page\\\":6,\\\"span\\\":[0,84]}]},{\\\"text\\\":\\\"where pos is the position and i is the dimension. That is, each dimension of the positional encoding corresponds to a sinusoid. The wavelengths form a geometric progression from 2\\\\u03c0 to 10000 \\\\u00b7 2\\\\u03c0. We chose this function because we hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset k , P Ep Epos+k can be represented as a linear function of P Ep Epos .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,360,505,414],\\\"page\\\":6,\\\"span\\\":[0,414]}]},{\\\"text\\\":\\\"We also experimented with using learned positional embeddings [9] instead, and found that the two versions produced nearly identical results (see Table 3 row (E)). We chose the sinusoidal version because it may allow the model to extrapolate to sequence lengths longer than the ones encountered during training.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,312,504,354],\\\"page\\\":6,\\\"span\\\":[0,311]}]},{\\\"text\\\":\\\"4 Why Self-Attention\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,283,225,295],\\\"page\\\":6,\\\"span\\\":[0,20]}]},{\\\"text\\\":\\\"In this section we compare various aspects of self-attention layers to the recurrent and convolutional layers commonly used for mapping one variable-length sequence of symbol representations (x1, ..., x n ) to another sequence of equal length (z1, ..., z n ), with xi, zi \\\\u2208 R d , such as a hidden layer in a typical sequence transduction encoder or decoder. Motivating our use of self-attention we consider three desiderata.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,218,505,270],\\\"page\\\":6,\\\"span\\\":[0,424]}]},{\\\"text\\\":\\\"One is the total computational complexity per layer. Another is the amount of computation that can be parallelized, as measured by the minimum number of sequential operations required.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,189,504,210],\\\"page\\\":6,\\\"span\\\":[0,184]}]},{\\\"text\\\":\\\"The third is the path length between long-range dependencies in the network. Learning long-range dependencies is a key challenge in many sequence transduction tasks. One key factor affecting the ability to learn such dependencies is the length of the paths forward and backward signals have to traverse in the network. The shorter these paths between any combination of positions in the input and output sequences, the easier it is to learn long-range dependencies [12]. Hence we also compare the maximum path length between any two input and output positions in networks composed of the different layer types.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,107,504,183],\\\"page\\\":6,\\\"span\\\":[0,610]}]},{\\\"text\\\":\\\"As noted in Table 1, a self-attention layer connects all positions with a constant number of sequentially executed operations, whereas a recurrent layer requires O(n) sequential operations. In terms of computational complexity, self-attention layers are faster than recurrent layers when the sequence\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,70,505,102],\\\"page\\\":6,\\\"span\\\":[0,300]}]},{\\\"text\\\":\\\"6\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[303,42,308,49],\\\"page\\\":6,\\\"span\\\":[0,1]}]},{\\\"text\\\":\\\"length n is smaller than the representation dimensionality d, which is most often the case with sentence representations used by state-of-the-art models in machine translations, such as word-piece [38] and byte-pair [31] representations. To improve computational performance for tasks involving very long sequences, self-attention could be restricted to considering only a neighborhood of size r in the input sequence centered around the respective output position. This would increase the maximum path length to O(n\\\\/r). We plan to investigate this approach further in future work.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,653,504,718],\\\"page\\\":7,\\\"span\\\":[0,581]}]},{\\\"text\\\":\\\"A single convolutional layer with kernel width k < n does not connect all pairs of input and output positions. Doing so requires a stack of O(n\\\\/k) convolutional layers in the case of contiguous kernels, or O(logk(n)) in the case of dilated convolutions [18], increasing the length of the longest paths between any two positions in the network. Convolutional layers are generally more expensive than recurrent layers, by a factor of k. Separable convolutions [6], however, decrease the complexity considerably, to O(k \\\\u00b7 n \\\\u00b7 d + n \\\\u00b7 d 2 ). Even with k = n, however, the complexity of a separable convolution is equal to the combination of a self-attention layer and a point-wise feed-forward layer, the approach we take in our model.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,560,505,647],\\\"page\\\":7,\\\"span\\\":[0,731]}]},{\\\"text\\\":\\\"As side benefit, self-attention could yield more interpretable models. We inspect attention distributions from our models and present and discuss examples in the appendix. Not only do individual attention heads clearly learn to perform different tasks, many appear to exhibit behavior related to the syntactic and semantic structure of the sentences.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,512,504,554],\\\"page\\\":7,\\\"span\\\":[0,350]}]},{\\\"text\\\":\\\"5 Training\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,480,170,492],\\\"page\\\":7,\\\"span\\\":[0,10]}]},{\\\"text\\\":\\\"This section describes the training regime for our models.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,456,337,466],\\\"page\\\":7,\\\"span\\\":[0,58]}]},{\\\"text\\\":\\\"5.1 Training Data and Batching\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,429,250,439],\\\"page\\\":7,\\\"span\\\":[0,30]}]},{\\\"text\\\":\\\"We trained on the standard WMT 2014 English-German dataset consisting of about 4.5 million sentence pairs. Sentences were encoded using byte-pair encoding [3], which has a shared sourcetarget vocabulary of about 37000 tokens. For English-French, we used the significantly larger WMT 2014 English-French dataset consisting of 36M sentences and split tokens into a 32000 word-piece vocabulary [38]. Sentence pairs were batched together by approximate sequence length. Each training batch contained a set of sentence pairs containing approximately 25000 source tokens and 25000 target tokens.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,343,505,418],\\\"page\\\":7,\\\"span\\\":[0,589]}]},{\\\"text\\\":\\\"5.2 Hardware and Schedule\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,318,233,326],\\\"page\\\":7,\\\"span\\\":[0,25]}]},{\\\"text\\\":\\\"We trained our models on one machine with 8 NVIDIA P100 GPUs. For our base models using the hyperparameters described throughout the paper, each training step took about 0.4 seconds. We trained the base models for a total of 100,000 steps or 12 hours. For our big models,(described on the bottom line of table 3), step time was 1.0 seconds. The big models were trained for 300,000 steps (3.5 days).\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,252,504,305],\\\"page\\\":7,\\\"span\\\":[0,398]}]},{\\\"text\\\":\\\"5.3 Optimizer\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,225,174,235],\\\"page\\\":7,\\\"span\\\":[0,13]}]},{\\\"text\\\":\\\"We used the Adam optimizer [20] with \\\\u03b21 = 0 . 9 , \\\\u03b22 = 0 . 98 and \\\\u03f5 = 10 - 9 . We varied the learning rate over the course of training, according to the formula:\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,193,504,215],\\\"page\\\":7,\\\"span\\\":[0,161]}]},{\\\"text\\\":\\\"lrate = d - 0 . 5 model \\\\u00b7 min(step _ num - 0 . 5 , step _ num \\\\u00b7 warmup _ steps - 1 . 5 ) (3)\\\",\\\"type\\\":\\\"equation\\\",\\\"name\\\":\\\"Formula\\\",\\\"prov\\\":[{\\\"bbox\\\":[162,162,505,175],\\\"page\\\":7,\\\"span\\\":[0,92]}]},{\\\"text\\\":\\\"This corresponds to increasing the learning rate linearly for the first warmup _ steps training steps, and decreasing it thereafter proportionally to the inverse square root of the step number. We used warmup _ steps = 4000 .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,118,505,149],\\\"page\\\":7,\\\"span\\\":[0,225]}]},{\\\"text\\\":\\\"5.4 Regularization\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,90,193,100],\\\"page\\\":7,\\\"span\\\":[0,18]}]},{\\\"text\\\":\\\"We employ three types of regularization during training:\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,69,331,79],\\\"page\\\":7,\\\"span\\\":[0,56]}]},{\\\"text\\\":\\\"7\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[303,42,308,49],\\\"page\\\":7,\\\"span\\\":[0,1]}]},{\\\"text\\\":\\\"Table 2: The Transformer achieves better BLEU scores than previous state-of-the-art models on the English-to-German and English-to-French newstest2014 tests at a fraction of the training cost.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,700,504,721],\\\"page\\\":8,\\\"span\\\":[0,192]}]},{\\\"name\\\":\\\"Table\\\",\\\"type\\\":\\\"table\\\",\\\"$ref\\\":\\\"#\\\\/tables\\\\/1\\\"},{\\\"text\\\":\\\"Residual Dropout We apply dropout [33] to the output of each sub-layer, before it is added to the sub-layer input and normalized. In addition, we apply dropout to the sums of the embeddings and the positional encodings in both the encoder and decoder stacks. For the base model, we use a rate of Pd Pdrop = 0 . 1 .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,476,505,519],\\\"page\\\":8,\\\"span\\\":[0,314]}]},{\\\"text\\\":\\\"Label Smoothing During training, we employed label smoothing of value \\\\u03f5ls = 0 . 1 [36]. This hurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,439,504,461],\\\"page\\\":8,\\\"span\\\":[0,187]}]},{\\\"text\\\":\\\"6 Results\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,411,163,420],\\\"page\\\":8,\\\"span\\\":[0,9]}]},{\\\"text\\\":\\\"6.1 Machine Translation\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,387,219,395],\\\"page\\\":8,\\\"span\\\":[0,23]}]},{\\\"text\\\":\\\"On the WMT 2014 English-to-German translation task, the big transformer model (Transformer (big) in Table 2) outperforms the best previously reported models (including ensembles) by more than 2 . 0 BLEU, establishing a new state-of-the-art BLEU score of 28 . 4. The configuration of this model is listed in the bottom line of Table 3. Training took 3 . 5 days on 8 P100 GPUs. Even our base model surpasses all previously published models and ensembles, at a fraction of the training cost of any of the competitive models.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,310,505,374],\\\"page\\\":8,\\\"span\\\":[0,521]}]},{\\\"text\\\":\\\"On the WMT 2014 English-to-French translation task, our big model achieves a BLEU score of 41 . 0 , outperforming all of the previously published single models, at less than 1\\\\/4 the training cost of the previous state-of-the-art model. The Transformer (big) model trained for English-to-French used dropout rate PdPdrop = 0 . 1, instead of 0 . 3 .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,260,505,303],\\\"page\\\":8,\\\"span\\\":[0,347]}]},{\\\"text\\\":\\\"For the base models, we used a single model obtained by averaging the last 5 checkpoints, which were written at 10-minute intervals. For the big models, we averaged the last 20 checkpoints. We used beam search with a beam size of 4 and length penalty \\\\u03b1 = 0 . 6 [38]. These hyperparameters were chosen after experimentation on the development set. We set the maximum output length during inference to input length + 50, but terminate early when possible [38].\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,200,504,253],\\\"page\\\":8,\\\"span\\\":[0,458]}]},{\\\"text\\\":\\\"Table 2 summarizes our results and compares our translation quality and training costs to other model architectures from the literature. We estimate the number of floating point operations used to train a model by multiplying the training time, the number of GPUs used, and an estimate of the sustained single-precision floating-point capacity of each GPU 5 .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,151,504,193],\\\"page\\\":8,\\\"span\\\":[0,359]}]},{\\\"text\\\":\\\"6.2 Model Variations\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,126,204,134],\\\"page\\\":8,\\\"span\\\":[0,20]}]},{\\\"text\\\":\\\"To evaluate the importance of different components of the Transformer, we varied our base model in different ways, measuring the change in performance on English-to-German translation on the\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,92,504,114],\\\"page\\\":8,\\\"span\\\":[0,190]}]},{\\\"text\\\":\\\"5 We used values of 2.8, 3.7, 6.0 and 9.5 TFLOPS for K80, K40, M40 and P100, respectively.\\\",\\\"type\\\":\\\"footnote\\\",\\\"name\\\":\\\"Footnote\\\",\\\"prov\\\":[{\\\"bbox\\\":[120,70,453,80],\\\"page\\\":8,\\\"span\\\":[0,90]}]},{\\\"text\\\":\\\"8\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[303,42,308,49],\\\"page\\\":8,\\\"span\\\":[0,1]}]},{\\\"text\\\":\\\"Table 3: Variations on the Transformer architecture. Unlisted values are identical to those of the base model. All metrics are on the English-to-German translation development set, newstest2013. Listed perplexities are per-wordpiece, according to our byte-pair encoding, and should not be compared to per-word perplexities.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,678,504,721],\\\"page\\\":9,\\\"span\\\":[0,323]}]},{\\\"name\\\":\\\"Table\\\",\\\"type\\\":\\\"table\\\",\\\"$ref\\\":\\\"#\\\\/tables\\\\/2\\\"},{\\\"text\\\":\\\"development set, newstest2013. We used beam search as described in the previous section, but no checkpoint averaging. We present these results in Table 3.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,358,504,378],\\\"page\\\":9,\\\"span\\\":[0,154]}]},{\\\"text\\\":\\\"In Table 3 rows (A), we vary the number of attention heads and the attention key and value dimensions, keeping the amount of computation constant, as described in Section 3.2.2. While single-head attention is 0.9 BLEU worse than the best setting, quality also drops off with too many heads.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,319,505,351],\\\"page\\\":9,\\\"span\\\":[0,290]}]},{\\\"text\\\":\\\"In Table 3 rows (B), we observe that reducing the attention key size dk hurts model quality. This suggests that determining compatibility is not easy and that a more sophisticated compatibility function than dot product may be beneficial. We further observe in rows (C) and (D) that, as expected, bigger models are better, and dropout is very helpful in avoiding over-fitting. In row (E) we replace our sinusoidal positional encoding with learned positional embeddings [9], and observe nearly identical results to the base model.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,250,505,313],\\\"page\\\":9,\\\"span\\\":[0,529]}]},{\\\"text\\\":\\\"6.3 English Constituency Parsing\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,222,256,232],\\\"page\\\":9,\\\"span\\\":[0,32]}]},{\\\"text\\\":\\\"To evaluate if the Transformer can generalize to other tasks we performed experiments on English constituency parsing. This task presents specific challenges: the output is subject to strong structural constraints and is significantly longer than the input. Furthermore, RNN sequence-to-sequence models have not been able to attain state-of-the-art results in small-data regimes [37].\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,168,504,210],\\\"page\\\":9,\\\"span\\\":[0,384]}]},{\\\"text\\\":\\\"We trained a 4-layer transformer with dmodel = 1024 on the Wall Street Journal (WSJ) portion of the Penn Treebank [25], about 40K training sentences. We also trained it in a semi-supervised setting, using the larger high-confidence and BerkleyParser corpora from with approximately 17M sentences [37]. We used a vocabulary of 16K tokens for the WSJ only setting and a vocabulary of 32K tokens for the semi-supervised setting.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,107,505,162],\\\"page\\\":9,\\\"span\\\":[0,425]}]},{\\\"text\\\":\\\"We performed only a small number of experiments to select the dropout, both attention and residual (section 5.4), learning rates and beam size on the Section 22 development set, all other parameters remained unchanged from the English-to-German base translation model. During inference, we\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,70,504,102],\\\"page\\\":9,\\\"span\\\":[0,289]}]},{\\\"text\\\":\\\"9\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[303,42,308,49],\\\"page\\\":9,\\\"span\\\":[0,1]}]},{\\\"text\\\":\\\"Table 4: The Transformer generalizes well to English constituency parsing (Results are on Section 23 of WSJ)\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,701,504,721],\\\"page\\\":10,\\\"span\\\":[0,108]}]},{\\\"name\\\":\\\"Table\\\",\\\"type\\\":\\\"table\\\",\\\"$ref\\\":\\\"#\\\\/tables\\\\/3\\\"},{\\\"text\\\":\\\"increased the maximum output length to input length + 300. We used a beam size of 21 and \\\\u03b1 = 0 . 3 for both WSJ only and the semi-supervised setting.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,509,504,530],\\\"page\\\":10,\\\"span\\\":[0,149]}]},{\\\"text\\\":\\\"Our results in Table 4 show that despite the lack of task-specific tuning our model performs surprisingly well, yielding better results than all previously reported models with the exception of the Recurrent Neural Network Grammar [8].\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,471,505,503],\\\"page\\\":10,\\\"span\\\":[0,235]}]},{\\\"text\\\":\\\"In contrast to RNN sequence-to-sequence models [37], the Transformer outperforms the BerkeleyParser [29] even when training only on the WSJ training set of 40K sentences.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,443,506,464],\\\"page\\\":10,\\\"span\\\":[0,170]}]},{\\\"text\\\":\\\"7 Conclusion\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,418,183,427],\\\"page\\\":10,\\\"span\\\":[0,12]}]},{\\\"text\\\":\\\"In this work, we presented the Transformer, the first sequence transduction model based entirely on attention, replacing the recurrent layers most commonly used in encoder-decoder architectures with multi-headed self-attention.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,373,504,403],\\\"page\\\":10,\\\"span\\\":[0,227]}]},{\\\"text\\\":\\\"For translation tasks, the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers. On both WMT 2014 English-to-German and WMT 2014 English-to-French translation tasks, we achieve a new state of the art. In the former task our best model outperforms even all previously reported ensembles.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,322,504,364],\\\"page\\\":10,\\\"span\\\":[0,343]}]},{\\\"text\\\":\\\"We are excited about the future of attention-based models and plan to apply them to other tasks. We plan to extend the Transformer to problems involving input and output modalities other than text and to investigate local, restricted attention mechanisms to efficiently handle large inputs and outputs such as images, audio and video. Making generation less sequential is another research goals of ours.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,273,505,316],\\\"page\\\":10,\\\"span\\\":[0,403]}]},{\\\"text\\\":\\\"The code we used to train and evaluate our models is available at https:\\\\/\\\\/github.com\\\\/ tensorflow\\\\/tensor2tensor .\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,247,505,266],\\\"page\\\":10,\\\"span\\\":[0,112]}]},{\\\"text\\\":\\\"Acknowledgements We are grateful to Nal Kalchbrenner and Stephan Gouws for their fruitful comments, corrections and inspiration.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"text\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,212,504,233],\\\"page\\\":10,\\\"span\\\":[0,128]}]},{\\\"text\\\":\\\"References\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,186,164,195],\\\"page\\\":10,\\\"span\\\":[0,10]}]},{\\\"text\\\":\\\"[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[113,157,504,177],\\\"page\\\":10,\\\"span\\\":[0,118]}]},{\\\"text\\\":\\\"[2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. CoRR, abs\\\\/1409.0473, 2014.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[113,127,504,148],\\\"page\\\":10,\\\"span\\\":[0,153]}]},{\\\"text\\\":\\\"[3] Denny Britz, Anna Goldie, Minh-Thang Luong, and Quoc V. Le. Massive exploration of neural machine translation architectures. CoRR, abs\\\\/1703.03906, 2017.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[113,99,504,119],\\\"page\\\":10,\\\"span\\\":[0,156]}]},{\\\"text\\\":\\\"[4] Jianpeng Cheng, Li Dong, and Mirella Lapata. Long short-term memory-networks for machine reading. arXiv preprint arXiv:1601.06733, 2016.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[113,70,504,90],\\\"page\\\":10,\\\"span\\\":[0,140]}]},{\\\"text\\\":\\\"10\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[301,42,311,49],\\\"page\\\":10,\\\"span\\\":[0,2]}]},{\\\"text\\\":\\\"[5] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. CoRR, abs\\\\/1406.1078, 2014.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[113,686,505,718],\\\"page\\\":11,\\\"span\\\":[0,230]}]},{\\\"text\\\":\\\"[6] Francois Chollet. Xception: Deep learning with depthwise separable convolutions. arXiv preprint arXiv:1610.02357, 2016.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[113,655,504,676],\\\"page\\\":11,\\\"span\\\":[0,123]}]},{\\\"text\\\":\\\"[7] Junyoung Chung, \\\\u00c7aglar G\\\\u00fcl\\\\u00e7ehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, abs\\\\/1412.3555, 2014.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[113,624,504,645],\\\"page\\\":11,\\\"span\\\":[0,175]}]},{\\\"text\\\":\\\"[8] Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. Recurrent neural network grammars. In Proc. of NAACL, 2016.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[113,594,504,614],\\\"page\\\":11,\\\"span\\\":[0,132]}]},{\\\"text\\\":\\\"[9] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolutional sequence to sequence learning. arXiv preprint arXiv:1705.03122v2, 2017.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[113,563,506,583],\\\"page\\\":11,\\\"span\\\":[0,169]}]},{\\\"text\\\":\\\"[10] Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,533,505,553],\\\"page\\\":11,\\\"span\\\":[0,108]}]},{\\\"text\\\":\\\"[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,491,505,522],\\\"page\\\":11,\\\"span\\\":[0,208]}]},{\\\"text\\\":\\\"[12] Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, and J\\\\u00fcrgen Schmidhuber. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, 2001.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,460,504,481],\\\"page\\\":11,\\\"span\\\":[0,166]}]},{\\\"text\\\":\\\"[13] Sepp Hochreiter and J\\\\u00fcrgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,430,505,450],\\\"page\\\":11,\\\"span\\\":[0,111]}]},{\\\"text\\\":\\\"[14] Zhongqiang Huang and Mary Harper. Self-training PCFG grammars with latent annotations across languages. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 832-841. ACL, August 2009.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,387,505,420],\\\"page\\\":11,\\\"span\\\":[0,232]}]},{\\\"text\\\":\\\"[15] Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,357,504,378],\\\"page\\\":11,\\\"span\\\":[0,164]}]},{\\\"text\\\":\\\"[16] \\\\u0141ukasz Kaiser and Samy Bengio. Can active memory replace attention? In Advances in Neural Information Processing Systems, (NIPS), 2016.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,326,505,347],\\\"page\\\":11,\\\"span\\\":[0,140]}]},{\\\"text\\\":\\\"[17] \\\\u0141ukasz Kaiser and Ilya Sutskever. Neural GPUs learn algorithms. In International Conference on Learning Representations (ICLR), 2016.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,296,504,316],\\\"page\\\":11,\\\"span\\\":[0,138]}]},{\\\"text\\\":\\\"[18] Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, and Koray Kavukcuoglu. Neural machine translation in linear time. arXiv preprint arXiv:1610.10099v2 , 2017.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,256,506,286],\\\"page\\\":11,\\\"span\\\":[0,195]}]},{\\\"text\\\":\\\"[19] Yoon Kim, Carl Denton, Luong Hoang, and Alexander M. Rush. Structured attention networks. In International Conference on Learning Representations, 2017.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,223,506,244],\\\"page\\\":11,\\\"span\\\":[0,157]}]},{\\\"text\\\":\\\"[20] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,203,505,213],\\\"page\\\":11,\\\"span\\\":[0,93]}]},{\\\"text\\\":\\\"[21] Oleksii Kuchaiev and Boris Ginsburg. Factorization tricks for LSTM networks. arXiv preprint arXiv:1703.10722, 2017.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,174,504,193],\\\"page\\\":11,\\\"span\\\":[0,120]}]},{\\\"text\\\":\\\"[22] Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. arXiv preprint arXiv:1703.03130, 2017.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,132,504,163],\\\"page\\\":11,\\\"span\\\":[0,195]}]},{\\\"text\\\":\\\"[23] Minh-Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. Multi-task sequence to sequence learning. arXiv preprint arXiv:1511.06114, 2015.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,100,505,121],\\\"page\\\":11,\\\"span\\\":[0,165]}]},{\\\"text\\\":\\\"[24] Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attentionbased neural machine translation. arXiv preprint arXiv:1508.04025, 2015.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,70,506,90],\\\"page\\\":11,\\\"span\\\":[0,166]}]},{\\\"text\\\":\\\"11\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[301,42,310,49],\\\"page\\\":11,\\\"span\\\":[0,2]}]},{\\\"text\\\":\\\"[25] Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313-330, 1993.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,697,504,718],\\\"page\\\":12,\\\"span\\\":[0,184]}]},{\\\"text\\\":\\\"[26] David McClosky, Eugene Charniak, and Mark Johnson. Effective self-training for parsing. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference , pages 152-159. ACL, June 2006.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,652,505,683],\\\"page\\\":12,\\\"span\\\":[0,214]}]},{\\\"text\\\":\\\"[27] Ankur Parikh, Oscar T\\\\u00e4ckstr\\\\u00f6m, Dipanjan Das, and Jakob Uszkoreit. A decomposable attention model. In Empirical Methods in Natural Language Processing, 2016.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,617,504,638],\\\"page\\\":12,\\\"span\\\":[0,161]}]},{\\\"text\\\":\\\"[28] Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304, 2017.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,583,504,603],\\\"page\\\":12,\\\"span\\\":[0,148]}]},{\\\"text\\\":\\\"[29] Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein. Learning accurate, compact, and interpretable tree annotation. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL, pages 433-440. ACL, July 2006.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,528,505,569],\\\"page\\\":12,\\\"span\\\":[0,273]}]},{\\\"text\\\":\\\"[30] Ofir Press and Lior Wolf. Using the output embedding to improve language models. arXiv preprint arXiv:1608.05859, 2016.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,492,504,513],\\\"page\\\":12,\\\"span\\\":[0,124]}]},{\\\"text\\\":\\\"[31] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,457,504,478],\\\"page\\\":12,\\\"span\\\":[0,154]}]},{\\\"text\\\":\\\"[32] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,412,505,444],\\\"page\\\":12,\\\"span\\\":[0,229]}]},{\\\"text\\\":\\\"[33] Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-1958, 2014.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,367,505,399],\\\"page\\\":12,\\\"span\\\":[0,229]}]},{\\\"text\\\":\\\"[34] Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-to-end memory networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2440-2448. Curran Associates, Inc., 2015.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,312,505,353],\\\"page\\\":12,\\\"span\\\":[0,279]}]},{\\\"text\\\":\\\"[35] Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104-3112, 2014.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,276,504,297],\\\"page\\\":12,\\\"span\\\":[0,180]}]},{\\\"text\\\":\\\"[36] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. CoRR, abs\\\\/1512.00567, 2015.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,242,506,263],\\\"page\\\":12,\\\"span\\\":[0,180]}]},{\\\"text\\\":\\\"[37] Vinyals & Kaiser, Koo, Petrov, Sutskever, and Hinton. Grammar as a foreign language. In Advances in Neural Information Processing Systems, 2015.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,208,504,228],\\\"page\\\":12,\\\"span\\\":[0,149]}]},{\\\"text\\\":\\\"[38] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,152,504,194],\\\"page\\\":12,\\\"span\\\":[0,288]}]},{\\\"text\\\":\\\"[39] Jie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with fast-forward connections for neural machine translation. CoRR, abs\\\\/1606.04199, 2016.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,117,504,138],\\\"page\\\":12,\\\"span\\\":[0,171]}]},{\\\"text\\\":\\\"[40] Muhua Zhu, Yue Zhang, Wenliang Chen, Min Zhang, and Jingbo Zhu. Fast and accurate shift-reduce constituent parsing. In Proceedings of the 51st Annual Meeting of the ACL (Volume 1: Long Papers), pages 434-443. ACL, August 2013.\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"reference\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,72,504,104],\\\"page\\\":12,\\\"span\\\":[0,231]}]},{\\\"text\\\":\\\"12\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[301,42,311,49],\\\"page\\\":12,\\\"span\\\":[0,2]}]},{\\\"text\\\":\\\"Attention Visualizations  Input-Input Laye Attention Visualizations  Input-Input Layer5\\\",\\\"type\\\":\\\"subtitle-level-1\\\",\\\"name\\\":\\\"Section-header\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,710,250,726],\\\"page\\\":13,\\\"span\\\":[0,87]}]},{\\\"text\\\":\\\"Figure 3: An example of the attention mechanism following long-distance dependencies in the encoder self-attention in layer 5 of 6. Many of the attention heads attend to a distant dependency of the verb 'making', completing the phrase 'making...more difficult'. Attentions here shown only for the word 'making'. Different colors represent different heads. Best viewed in color.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,437,504,480],\\\"page\\\":13,\\\"span\\\":[0,377]}]},{\\\"name\\\":\\\"Picture\\\",\\\"type\\\":\\\"figure\\\",\\\"$ref\\\":\\\"#\\\\/figures\\\\/3\\\"},{\\\"text\\\":\\\"13\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[301,41,311,49],\\\"page\\\":13,\\\"span\\\":[0,2]}]},{\\\"text\\\":\\\"Input-Input Layer5\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"Text\\\",\\\"prov\\\":[{\\\"bbox\\\":[109,672,278,690],\\\"page\\\":14,\\\"span\\\":[0,18]}]},{\\\"text\\\":\\\"Figure 4: Two attention heads, also in layer 5 of 6, apparently involved in anaphora resolution. Top: Full attentions for head 5. Bottom: Isolated attentions from just the word 'its' for attention heads 5 and 6. Note that the attentions are very sharp for this word.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,146,505,178],\\\"page\\\":14,\\\"span\\\":[0,266]}]},{\\\"name\\\":\\\"Picture\\\",\\\"type\\\":\\\"figure\\\",\\\"$ref\\\":\\\"#\\\\/figures\\\\/4\\\"},{\\\"text\\\":\\\"14\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[301,42,311,49],\\\"page\\\":14,\\\"span\\\":[0,2]}]},{\\\"text\\\":\\\"Input-Input Layer5\\\",\\\"type\\\":\\\"paragraph\\\",\\\"name\\\":\\\"Text\\\",\\\"prov\\\":[{\\\"bbox\\\":[108,658,278,676],\\\"page\\\":15,\\\"span\\\":[0,18]}]},{\\\"name\\\":\\\"Picture\\\",\\\"type\\\":\\\"figure\\\",\\\"$ref\\\":\\\"#\\\\/figures\\\\/5\\\"},{\\\"text\\\":\\\"Figure 5: Many of the attention heads exhibit behaviour that seems related to the structure of the sentence. We give two such examples above, from two different heads from the encoder self-attention at layer 5 of 6. The heads clearly learned to perform different tasks.\\\",\\\"type\\\":\\\"caption\\\",\\\"name\\\":\\\"Caption\\\",\\\"prov\\\":[{\\\"bbox\\\":[107,157,504,190],\\\"page\\\":15,\\\"span\\\":[0,269]}]},{\\\"name\\\":\\\"Picture\\\",\\\"type\\\":\\\"figure\\\",\\\"$ref\\\":\\\"#\\\\/figures\\\\/6\\\"},{\\\"text\\\":\\\"15\\\",\\\"type\\\":\\\"page-footer\\\",\\\"name\\\":\\\"Page-footer\\\",\\\"prov\\\":[{\\\"bbox\\\":[301,41,311,49],\\\"page\\\":15,\\\"span\\\":[0,2]}]}],\\\"figures\\\":[{\\\"prov\\\":[{\\\"bbox\\\":[196,398,414,720],\\\"page\\\":3,\\\"span\\\":[0,47]}],\\\"text\\\":\\\"Figure 1: The Transformer - model architecture.\\\",\\\"type\\\":\\\"figure\\\"},{\\\"prov\\\":[{\\\"bbox\\\":[174,573,240,698],\\\"page\\\":4,\\\"span\\\":[0,0]}],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"figure\\\"},{\\\"prov\\\":[{\\\"bbox\\\":[346,554,467,720],\\\"page\\\":4,\\\"span\\\":[0,133]}],\\\"text\\\":\\\"Figure 2: (left) Scaled Dot-Product Attention. (right) Multi-Head Attention consists of several attention layers running in parallel.\\\",\\\"type\\\":\\\"figure\\\"},{\\\"prov\\\":[{\\\"bbox\\\":[118,490,503,692],\\\"page\\\":13,\\\"span\\\":[0,377]}],\\\"text\\\":\\\"Figure 3: An example of the attention mechanism following long-distance dependencies in the encoder self-attention in layer 5 of 6. Many of the attention heads attend to a distant dependency of the verb 'making', completing the phrase 'making...more difficult'. Attentions here shown only for the word 'making'. Different colors represent different heads. Best viewed in color.\\\",\\\"type\\\":\\\"figure\\\"},{\\\"prov\\\":[{\\\"bbox\\\":[109,187,501,621],\\\"page\\\":14,\\\"span\\\":[0,266]}],\\\"text\\\":\\\"Figure 4: Two attention heads, also in layer 5 of 6, apparently involved in anaphora resolution. Top: Full attentions for head 5. Bottom: Isolated attentions from just the word 'its' for attention heads 5 and 6. Note that the attentions are very sharp for this word.\\\",\\\"type\\\":\\\"figure\\\"},{\\\"prov\\\":[{\\\"bbox\\\":[108,421,498,608],\\\"page\\\":15,\\\"span\\\":[0,0]}],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"figure\\\"},{\\\"prov\\\":[{\\\"bbox\\\":[122,202,497,389],\\\"page\\\":15,\\\"span\\\":[0,269]}],\\\"text\\\":\\\"Figure 5: Many of the attention heads exhibit behaviour that seems related to the structure of the sentence. We give two such examples above, from two different heads from the encoder self-attention at layer 5 of 6. The heads clearly learned to perform different tasks.\\\",\\\"type\\\":\\\"figure\\\"}],\\\"tables\\\":[{\\\"prov\\\":[{\\\"bbox\\\":[118,605,494,680],\\\"page\\\":6,\\\"span\\\":[0,0]}],\\\"text\\\":\\\"Table 1: Maximum path lengths, per-layer complexity and minimum number of sequential operations for different layer types. n is the sequence length, d is the representation dimension, k is the kernel size of convolutions and r the size of the neighborhood in restricted self-attention.\\\",\\\"type\\\":\\\"table\\\",\\\"#-cols\\\":4,\\\"#-rows\\\":5,\\\"data\\\":[[{\\\"spans\\\":[[0,0]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"spans\\\":[[0,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[340,655,383,664],\\\"spans\\\":[[0,2]],\\\"text\\\":\\\"Operations\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[125,666,487,675],\\\"spans\\\":[[0,3]],\\\"text\\\":\\\"Layer Type Complexity per Layer Sequential Maximum Path Length\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]}],[{\\\"bbox\\\":[125,644,181,651],\\\"spans\\\":[[1,0]],\\\"text\\\":\\\"Self-Attention\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"bbox\\\":[265,642,302,653],\\\"spans\\\":[[1,1]],\\\"text\\\":\\\"O(n 2 \\\\u00b7  d)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"bbox\\\":[352,642,371,652],\\\"spans\\\":[[1,2]],\\\"text\\\":\\\"O(1)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"bbox\\\":[431,642,451,652],\\\"spans\\\":[[1,3]],\\\"text\\\":\\\"O(1)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]}],[{\\\"bbox\\\":[125,633,164,639],\\\"spans\\\":[[2,0]],\\\"text\\\":\\\"Recurrent\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"bbox\\\":[265,630,302,641],\\\"spans\\\":[[2,1]],\\\"text\\\":\\\"O(n  \\\\u00b7 d 2 )\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"bbox\\\":[351,630,371,640],\\\"spans\\\":[[2,2]],\\\"text\\\":\\\"O(n)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"bbox\\\":[431,630,451,640],\\\"spans\\\":[[2,3]],\\\"text\\\":\\\"O(n)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]}],[{\\\"bbox\\\":[125,621,181,628],\\\"spans\\\":[[3,0]],\\\"text\\\":\\\"Convolutional\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[259,619,308,630],\\\"spans\\\":[[3,1]],\\\"text\\\":\\\"O(k \\\\u00b7  n  \\\\u00b7 d 2 )\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[352,619,371,629],\\\"spans\\\":[[3,2]],\\\"text\\\":\\\"O(1)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[418,619,464,629],\\\"spans\\\":[[3,3]],\\\"text\\\":\\\"O(logk(n))\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]}],[{\\\"bbox\\\":[125,609,227,617],\\\"spans\\\":[[4,0]],\\\"text\\\":\\\"Self-Attention (restricted)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"bbox\\\":[261,608,306,618],\\\"spans\\\":[[4,1]],\\\"text\\\":\\\"O(r  \\\\u00b7  n  \\\\u00b7 d)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"bbox\\\":[352,608,371,618],\\\"spans\\\":[[4,2]],\\\"text\\\":\\\"O(1)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"bbox\\\":[426,608,456,618],\\\"spans\\\":[[4,3]],\\\"text\\\":\\\"O(n\\\\/r)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]}]]},{\\\"prov\\\":[{\\\"bbox\\\":[130,548,482,697],\\\"page\\\":8,\\\"span\\\":[0,0]}],\\\"text\\\":\\\"Table 2: The Transformer achieves better BLEU scores than previous state-of-the-art models on the English-to-German and English-to-French newstest2014 tests at a fraction of the training cost.\\\",\\\"type\\\":\\\"table\\\",\\\"#-cols\\\":4,\\\"#-rows\\\":12,\\\"data\\\":[[{\\\"bbox\\\":[137,678,162,685],\\\"spans\\\":[[0,0]],\\\"text\\\":\\\"Model\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"spans\\\":[[0,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[311,684,475,693],\\\"spans\\\":[[0,2],[0,3]],\\\"text\\\":\\\"BLEU Training Cost (FLOPs)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,4],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[311,684,475,693],\\\"spans\\\":[[0,2],[0,3]],\\\"text\\\":\\\"BLEU Training Cost (FLOPs)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[2,4],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]}],[{\\\"spans\\\":[[1,0]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"bbox\\\":[289,671,469,677],\\\"spans\\\":[[1,1]],\\\"text\\\":\\\"EN-DE EN-FR EN-DE EN-FR\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]}],[{\\\"bbox\\\":[137,657,314,666],\\\"spans\\\":[[2,0]],\\\"text\\\":\\\"ByteNet [18] 23.75\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]}],[{\\\"bbox\\\":[137,646,353,655],\\\"spans\\\":[[3,0]],\\\"text\\\":\\\"Deep-Att + PosUnk [39] 39.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"spans\\\":[[3,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"spans\\\":[[3,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[436,648,473,656],\\\"spans\\\":[[3,3]],\\\"text\\\":\\\"1 . 0  \\\\u00b7  10 20\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]}],[{\\\"bbox\\\":[137,635,356,643],\\\"spans\\\":[[4,0]],\\\"text\\\":\\\"GNMT + RL [38] 24.6 39.92\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"bbox\\\":[384,636,421,645],\\\"spans\\\":[[4,2]],\\\"text\\\":\\\"2 . 3  \\\\u00b7  10 19\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"bbox\\\":[436,636,473,645],\\\"spans\\\":[[4,3]],\\\"text\\\":\\\"1 . 4  \\\\u00b7  10 20\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]}],[{\\\"bbox\\\":[137,624,356,632],\\\"spans\\\":[[5,0]],\\\"text\\\":\\\"ConvS2S [9] 25.16 40.46\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"spans\\\":[[5,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[384,625,421,633],\\\"spans\\\":[[5,2]],\\\"text\\\":\\\"9 . 6  \\\\u00b7  10 18\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[436,625,473,633],\\\"spans\\\":[[5,3]],\\\"text\\\":\\\"1 . 5  \\\\u00b7  10 20\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]}],[{\\\"bbox\\\":[137,612,356,621],\\\"spans\\\":[[6,0]],\\\"text\\\":\\\"MoE [32] 26.03 40.56\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"bbox\\\":[384,614,421,622],\\\"spans\\\":[[6,2]],\\\"text\\\":\\\"2 . 0  \\\\u00b7  10 19\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"bbox\\\":[436,614,473,622],\\\"spans\\\":[[6,3]],\\\"text\\\":\\\"1 . 2  \\\\u00b7  10 20\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]}],[{\\\"bbox\\\":[137,599,353,608],\\\"spans\\\":[[7,0]],\\\"text\\\":\\\"Deep-Att + PosUnk Ensemble [39] 40.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"bbox\\\":[436,601,473,609],\\\"spans\\\":[[7,3]],\\\"text\\\":\\\"8 . 0  \\\\u00b7  10 20\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]}],[{\\\"bbox\\\":[137,588,356,597],\\\"spans\\\":[[8,0]],\\\"text\\\":\\\"GNMT + RL Ensemble [38] 26.30 41.16\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"bbox\\\":[384,590,421,598],\\\"spans\\\":[[8,2]],\\\"text\\\":\\\"1 . 8  \\\\u00b7  10 20\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"bbox\\\":[436,590,472,598],\\\"spans\\\":[[8,3]],\\\"text\\\":\\\"1 . 1  \\\\u00b7  10 21\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]}],[{\\\"bbox\\\":[137,577,315,585],\\\"spans\\\":[[9,0]],\\\"text\\\":\\\"ConvS2S Ensemble [9] 26.36\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[334,578,356,585],\\\"spans\\\":[[9,1]],\\\"text\\\":\\\"41.29\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[384,578,421,587],\\\"spans\\\":[[9,2]],\\\"text\\\":\\\"7 . 7  \\\\u00b7  10 19\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[436,578,472,587],\\\"spans\\\":[[9,3]],\\\"text\\\":\\\"1 . 2  \\\\u00b7  10 21\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]}],[{\\\"bbox\\\":[137,563,353,572],\\\"spans\\\":[[10,0]],\\\"text\\\":\\\"Transformer (base model) 27.3 38.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"spans\\\":[[10,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[408,565,427,572],\\\"spans\\\":[[10,2]],\\\"text\\\":\\\"3 . 3 \\\\u00b7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[431,565,450,573],\\\"spans\\\":[[10,3]],\\\"text\\\":\\\"10 18\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]}],[{\\\"bbox\\\":[137,552,207,561],\\\"spans\\\":[[11,0]],\\\"text\\\":\\\"Transformer (big)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"bbox\\\":[295,554,353,561],\\\"spans\\\":[[11,1]],\\\"text\\\":\\\"28.4 41.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"bbox\\\":[411,554,440,560],\\\"spans\\\":[[11,2]],\\\"text\\\":\\\"2 . 3  \\\\u00b7  10\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"bbox\\\":[441,557,448,562],\\\"spans\\\":[[11,3]],\\\"text\\\":\\\"19\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]}]]},{\\\"prov\\\":[{\\\"bbox\\\":[107,407,509,664],\\\"page\\\":9,\\\"span\\\":[0,0]}],\\\"text\\\":\\\"Table 3: Variations on the Transformer architecture. Unlisted values are identical to those of the base model. All metrics are on the English-to-German translation development set, newstest2013. Listed perplexities are per-wordpiece, according to our byte-pair encoding, and should not be compared to per-word perplexities.\\\",\\\"type\\\":\\\"table\\\",\\\"#-cols\\\":10,\\\"#-rows\\\":20,\\\"data\\\":[[{\\\"spans\\\":[[0,0]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[147,647,172,654],\\\"spans\\\":[[0,1]],\\\"text\\\":\\\"N d\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[172,646,294,646],\\\"spans\\\":[[0,2]],\\\"text\\\":\\\"model  d ff  d v\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[237,654,268,654],\\\"spans\\\":[[0,3]],\\\"text\\\":\\\"h d k\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[310,645,332,654],\\\"spans\\\":[[0,4]],\\\"text\\\":\\\"Pd Pdrop\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[346,646,355,652],\\\"spans\\\":[[0,5]],\\\"text\\\":\\\"\\\\u03f5ls\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":5,\\\"col-header\\\":false,\\\"col-span\\\":[5,6],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[371,639,390,660],\\\"spans\\\":[[0,6]],\\\"text\\\":\\\"train  steps\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":6,\\\"col-header\\\":false,\\\"col-span\\\":[6,7],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[404,640,459,641],\\\"spans\\\":[[0,7]],\\\"text\\\":\\\"(dev) (dev)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":7,\\\"col-header\\\":false,\\\"col-span\\\":[7,8],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[405,648,502,660],\\\"spans\\\":[[0,8]],\\\"text\\\":\\\"PPL BLEU params \\\\u00d710 6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":8,\\\"col-header\\\":false,\\\"col-span\\\":[8,9],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"spans\\\":[[0,9]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":9,\\\"col-header\\\":false,\\\"col-span\\\":[9,10],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]}],[{\\\"bbox\\\":[116,629,134,636],\\\"spans\\\":[[1,0]],\\\"text\\\":\\\"base\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"bbox\\\":[149,629,391,636],\\\"spans\\\":[[1,2]],\\\"text\\\":\\\"6 512 2048 8 64 64 0.1 0.1 100K\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,5]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":5,\\\"col-header\\\":false,\\\"col-span\\\":[5,6],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,6]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":6,\\\"col-header\\\":false,\\\"col-span\\\":[6,7],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"bbox\\\":[405,629,493,636],\\\"spans\\\":[[1,7]],\\\"text\\\":\\\"4.92 25.8 65\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":7,\\\"col-header\\\":false,\\\"col-span\\\":[7,8],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,8]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":8,\\\"col-header\\\":false,\\\"col-span\\\":[8,9],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"spans\\\":[[1,9]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":9,\\\"col-header\\\":false,\\\"col-span\\\":[9,10],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]}],[{\\\"spans\\\":[[2,0]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"bbox\\\":[238,616,298,623],\\\"spans\\\":[[2,3]],\\\"text\\\":\\\"1 512 512\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,5]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":5,\\\"col-header\\\":false,\\\"col-span\\\":[5,6],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,6]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":6,\\\"col-header\\\":false,\\\"col-span\\\":[6,7],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"bbox\\\":[405,616,457,623],\\\"spans\\\":[[2,7]],\\\"text\\\":\\\"5.29 24.9\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":7,\\\"col-header\\\":false,\\\"col-span\\\":[7,8],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,8]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":8,\\\"col-header\\\":false,\\\"col-span\\\":[8,9],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"spans\\\":[[2,9]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":9,\\\"col-header\\\":false,\\\"col-span\\\":[9,10],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]}],[{\\\"spans\\\":[[3,0]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"spans\\\":[[3,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"spans\\\":[[3,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[237,605,297,612],\\\"spans\\\":[[3,3]],\\\"text\\\":\\\"4 128 128\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"spans\\\":[[3,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"spans\\\":[[3,5]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":5,\\\"col-header\\\":false,\\\"col-span\\\":[5,6],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"spans\\\":[[3,6]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":6,\\\"col-header\\\":false,\\\"col-span\\\":[6,7],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[405,605,457,612],\\\"spans\\\":[[3,7]],\\\"text\\\":\\\"5.00 25.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":7,\\\"col-header\\\":false,\\\"col-span\\\":[7,8],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"spans\\\":[[3,8]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":8,\\\"col-header\\\":false,\\\"col-span\\\":[8,9],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"spans\\\":[[3,9]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":9,\\\"col-header\\\":false,\\\"col-span\\\":[9,10],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]}],[{\\\"bbox\\\":[119,598,132,607],\\\"spans\\\":[[4,0],[5,0]],\\\"text\\\":\\\"(A)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,6]},{\\\"spans\\\":[[4,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"bbox\\\":[235,594,295,601],\\\"spans\\\":[[4,3]],\\\"text\\\":\\\"16 32 32\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,5]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":5,\\\"col-header\\\":false,\\\"col-span\\\":[5,6],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,6]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":6,\\\"col-header\\\":false,\\\"col-span\\\":[6,7],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"bbox\\\":[405,594,457,601],\\\"spans\\\":[[4,7]],\\\"text\\\":\\\"4.91 25.8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":7,\\\"col-header\\\":false,\\\"col-span\\\":[7,8],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,8]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":8,\\\"col-header\\\":false,\\\"col-span\\\":[8,9],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"spans\\\":[[4,9]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":9,\\\"col-header\\\":false,\\\"col-span\\\":[9,10],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]}],[{\\\"bbox\\\":[119,598,132,607],\\\"spans\\\":[[4,0],[5,0]],\\\"text\\\":\\\"(A)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[4,6]},{\\\"spans\\\":[[5,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[235,583,295,590],\\\"spans\\\":[[5,2]],\\\"text\\\":\\\"32 16 16\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"spans\\\":[[5,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"spans\\\":[[5,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"spans\\\":[[5,5]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":5,\\\"col-header\\\":false,\\\"col-span\\\":[5,6],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"spans\\\":[[5,6]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":6,\\\"col-header\\\":false,\\\"col-span\\\":[6,7],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[405,583,457,590],\\\"spans\\\":[[5,7]],\\\"text\\\":\\\"5.01 25.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":7,\\\"col-header\\\":false,\\\"col-span\\\":[7,8],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"spans\\\":[[5,8]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":8,\\\"col-header\\\":false,\\\"col-span\\\":[8,9],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"spans\\\":[[5,9]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":9,\\\"col-header\\\":false,\\\"col-span\\\":[9,10],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]}],[{\\\"spans\\\":[[6,0]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"bbox\\\":[260,571,268,578],\\\"spans\\\":[[6,3]],\\\"text\\\":\\\"16\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,5]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":5,\\\"col-header\\\":false,\\\"col-span\\\":[5,6],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,6]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":6,\\\"col-header\\\":false,\\\"col-span\\\":[6,7],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"bbox\\\":[405,571,493,578],\\\"spans\\\":[[6,7]],\\\"text\\\":\\\"5.16 25.1 58\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":7,\\\"col-header\\\":false,\\\"col-span\\\":[7,8],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,8]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":8,\\\"col-header\\\":false,\\\"col-span\\\":[8,9],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"spans\\\":[[6,9]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":9,\\\"col-header\\\":false,\\\"col-span\\\":[9,10],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]}],[{\\\"bbox\\\":[119,564,131,572],\\\"spans\\\":[[7,0]],\\\"text\\\":\\\"(B)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"bbox\\\":[259,560,268,567],\\\"spans\\\":[[7,3]],\\\"text\\\":\\\"32\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,5]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":5,\\\"col-header\\\":false,\\\"col-span\\\":[5,6],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,6]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":6,\\\"col-header\\\":false,\\\"col-span\\\":[6,7],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"bbox\\\":[405,560,493,567],\\\"spans\\\":[[7,7]],\\\"text\\\":\\\"5.01 25.4 60\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":7,\\\"col-header\\\":false,\\\"col-span\\\":[7,8],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,8]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":8,\\\"col-header\\\":false,\\\"col-span\\\":[8,9],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"spans\\\":[[7,9]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":9,\\\"col-header\\\":false,\\\"col-span\\\":[9,10],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]}],[{\\\"bbox\\\":[119,513,131,521],\\\"spans\\\":[[8,0],[9,0],[10,0]],\\\"text\\\":\\\"(C)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,11]},{\\\"bbox\\\":[148,547,153,554],\\\"spans\\\":[[8,1]],\\\"text\\\":\\\"2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,5]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":5,\\\"col-header\\\":false,\\\"col-span\\\":[5,6],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,6]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":6,\\\"col-header\\\":false,\\\"col-span\\\":[6,7],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"bbox\\\":[405,547,493,554],\\\"spans\\\":[[8,7]],\\\"text\\\":\\\"6.11 23.7 36\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":7,\\\"col-header\\\":false,\\\"col-span\\\":[7,8],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,8]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":8,\\\"col-header\\\":false,\\\"col-span\\\":[8,9],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"spans\\\":[[8,9]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":9,\\\"col-header\\\":false,\\\"col-span\\\":[9,10],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]}],[{\\\"bbox\\\":[119,513,131,521],\\\"spans\\\":[[8,0],[9,0],[10,0]],\\\"text\\\":\\\"(C)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[8,11]},{\\\"bbox\\\":[148,536,153,543],\\\"spans\\\":[[9,1]],\\\"text\\\":\\\"4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"spans\\\":[[9,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"spans\\\":[[9,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"spans\\\":[[9,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"spans\\\":[[9,5]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":5,\\\"col-header\\\":false,\\\"col-span\\\":[5,6],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"spans\\\":[[9,6]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":6,\\\"col-header\\\":false,\\\"col-span\\\":[6,7],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[405,536,493,543],\\\"spans\\\":[[9,7]],\\\"text\\\":\\\"5.19 25.3 50\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":7,\\\"col-header\\\":false,\\\"col-span\\\":[7,8],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"spans\\\":[[9,8]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":8,\\\"col-header\\\":false,\\\"col-span\\\":[8,9],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"spans\\\":[[9,9]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":9,\\\"col-header\\\":false,\\\"col-span\\\":[9,10],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]}],[{\\\"bbox\\\":[119,513,131,521],\\\"spans\\\":[[8,0],[9,0],[10,0]],\\\"text\\\":\\\"(C)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[8,11]},{\\\"bbox\\\":[149,525,153,532],\\\"spans\\\":[[10,1]],\\\"text\\\":\\\"8\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"spans\\\":[[10,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"spans\\\":[[10,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"spans\\\":[[10,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"spans\\\":[[10,5]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":5,\\\"col-header\\\":false,\\\"col-span\\\":[5,6],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"spans\\\":[[10,6]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":6,\\\"col-header\\\":false,\\\"col-span\\\":[6,7],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[405,525,493,532],\\\"spans\\\":[[10,7]],\\\"text\\\":\\\"4.88 25.5 80\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":7,\\\"col-header\\\":false,\\\"col-span\\\":[7,8],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"spans\\\":[[10,8]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":8,\\\"col-header\\\":false,\\\"col-span\\\":[8,9],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"spans\\\":[[10,9]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":9,\\\"col-header\\\":false,\\\"col-span\\\":[9,10],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]}],[{\\\"spans\\\":[[11,0]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"spans\\\":[[11,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"bbox\\\":[172,515,295,522],\\\"spans\\\":[[11,2]],\\\"text\\\":\\\"256 32 32\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"spans\\\":[[11,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"spans\\\":[[11,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"spans\\\":[[11,5]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":5,\\\"col-header\\\":false,\\\"col-span\\\":[5,6],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"spans\\\":[[11,6]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":6,\\\"col-header\\\":false,\\\"col-span\\\":[6,7],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"bbox\\\":[405,515,493,522],\\\"spans\\\":[[11,7]],\\\"text\\\":\\\"5.75 24.5 28\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":7,\\\"col-header\\\":false,\\\"col-span\\\":[7,8],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"spans\\\":[[11,8]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":8,\\\"col-header\\\":false,\\\"col-span\\\":[8,9],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"spans\\\":[[11,9]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":9,\\\"col-header\\\":false,\\\"col-span\\\":[9,10],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]}],[{\\\"spans\\\":[[12,0]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"spans\\\":[[12,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"bbox\\\":[170,504,297,510],\\\"spans\\\":[[12,2]],\\\"text\\\":\\\"1024 128 128\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"spans\\\":[[12,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"spans\\\":[[12,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"spans\\\":[[12,5]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":5,\\\"col-header\\\":false,\\\"col-span\\\":[5,6],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"spans\\\":[[12,6]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":6,\\\"col-header\\\":false,\\\"col-span\\\":[6,7],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"bbox\\\":[405,504,495,511],\\\"spans\\\":[[12,7]],\\\"text\\\":\\\"4.66 26.0 168\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":7,\\\"col-header\\\":false,\\\"col-span\\\":[7,8],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"spans\\\":[[12,8]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":8,\\\"col-header\\\":false,\\\"col-span\\\":[8,9],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"spans\\\":[[12,9]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":9,\\\"col-header\\\":false,\\\"col-span\\\":[9,10],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]}],[{\\\"spans\\\":[[13,0]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"spans\\\":[[13,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"bbox\\\":[203,493,222,500],\\\"spans\\\":[[13,2]],\\\"text\\\":\\\"1024\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"spans\\\":[[13,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"spans\\\":[[13,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"spans\\\":[[13,5]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":5,\\\"col-header\\\":false,\\\"col-span\\\":[5,6],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"spans\\\":[[13,6]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":6,\\\"col-header\\\":false,\\\"col-span\\\":[6,7],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"bbox\\\":[405,493,493,500],\\\"spans\\\":[[13,7]],\\\"text\\\":\\\"5.12 25.4 53\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":7,\\\"col-header\\\":false,\\\"col-span\\\":[7,8],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"spans\\\":[[13,8]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":8,\\\"col-header\\\":false,\\\"col-span\\\":[8,9],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]},{\\\"spans\\\":[[13,9]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":9,\\\"col-header\\\":false,\\\"col-span\\\":[9,10],\\\"row\\\":13,\\\"row-header\\\":false,\\\"row-span\\\":[13,14]}],[{\\\"spans\\\":[[14,0]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"bbox\\\":[202,482,222,489],\\\"spans\\\":[[14,1],[14,2]],\\\"text\\\":\\\"4096\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,3],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"bbox\\\":[202,482,222,489],\\\"spans\\\":[[14,1],[14,2]],\\\"text\\\":\\\"4096\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[1,3],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"spans\\\":[[14,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"spans\\\":[[14,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"spans\\\":[[14,5]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":5,\\\"col-header\\\":false,\\\"col-span\\\":[5,6],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"spans\\\":[[14,6]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":6,\\\"col-header\\\":false,\\\"col-span\\\":[6,7],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"bbox\\\":[405,482,493,489],\\\"spans\\\":[[14,7]],\\\"text\\\":\\\"4.75 26.2 90\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":7,\\\"col-header\\\":false,\\\"col-span\\\":[7,8],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"spans\\\":[[14,8]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":8,\\\"col-header\\\":false,\\\"col-span\\\":[8,9],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]},{\\\"spans\\\":[[14,9]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":9,\\\"col-header\\\":false,\\\"col-span\\\":[9,10],\\\"row\\\":14,\\\"row-header\\\":false,\\\"row-span\\\":[14,15]}],[{\\\"spans\\\":[[15,0]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"spans\\\":[[15,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"spans\\\":[[15,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"spans\\\":[[15,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"bbox\\\":[315,469,327,476],\\\"spans\\\":[[15,4]],\\\"text\\\":\\\"0.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"spans\\\":[[15,5]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":5,\\\"col-header\\\":false,\\\"col-span\\\":[5,6],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"spans\\\":[[15,6]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":6,\\\"col-header\\\":false,\\\"col-span\\\":[6,7],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"bbox\\\":[405,469,457,476],\\\"spans\\\":[[15,7]],\\\"text\\\":\\\"5.77 24.6\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":7,\\\"col-header\\\":false,\\\"col-span\\\":[7,8],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"spans\\\":[[15,8]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":8,\\\"col-header\\\":false,\\\"col-span\\\":[8,9],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]},{\\\"spans\\\":[[15,9]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":9,\\\"col-header\\\":false,\\\"col-span\\\":[9,10],\\\"row\\\":15,\\\"row-header\\\":false,\\\"row-span\\\":[15,16]}],[{\\\"spans\\\":[[16,0]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"spans\\\":[[16,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"spans\\\":[[16,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"spans\\\":[[16,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"bbox\\\":[315,458,327,465],\\\"spans\\\":[[16,4]],\\\"text\\\":\\\"0.2\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"spans\\\":[[16,5]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":5,\\\"col-header\\\":false,\\\"col-span\\\":[5,6],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"spans\\\":[[16,6]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":6,\\\"col-header\\\":false,\\\"col-span\\\":[6,7],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"bbox\\\":[405,458,457,465],\\\"spans\\\":[[16,7]],\\\"text\\\":\\\"4.95 25.5\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":7,\\\"col-header\\\":false,\\\"col-span\\\":[7,8],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"spans\\\":[[16,8]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":8,\\\"col-header\\\":false,\\\"col-span\\\":[8,9],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]},{\\\"spans\\\":[[16,9]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":9,\\\"col-header\\\":false,\\\"col-span\\\":[9,10],\\\"row\\\":16,\\\"row-header\\\":false,\\\"row-span\\\":[16,17]}],[{\\\"bbox\\\":[119,451,132,460],\\\"spans\\\":[[17,0]],\\\"text\\\":\\\"(D)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"spans\\\":[[17,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"spans\\\":[[17,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"spans\\\":[[17,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"spans\\\":[[17,4]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"bbox\\\":[345,447,357,454],\\\"spans\\\":[[17,5]],\\\"text\\\":\\\"0.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":5,\\\"col-header\\\":false,\\\"col-span\\\":[5,6],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"spans\\\":[[17,6]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":6,\\\"col-header\\\":false,\\\"col-span\\\":[6,7],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"bbox\\\":[405,447,457,454],\\\"spans\\\":[[17,7]],\\\"text\\\":\\\"4.67 25.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":7,\\\"col-header\\\":false,\\\"col-span\\\":[7,8],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"spans\\\":[[17,8]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":8,\\\"col-header\\\":false,\\\"col-span\\\":[8,9],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]},{\\\"spans\\\":[[17,9]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":9,\\\"col-header\\\":false,\\\"col-span\\\":[9,10],\\\"row\\\":17,\\\"row-header\\\":false,\\\"row-span\\\":[17,18]}],[{\\\"bbox\\\":[119,422,131,431],\\\"spans\\\":[[18,0]],\\\"text\\\":\\\"(E)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]},{\\\"bbox\\\":[179,422,345,431],\\\"spans\\\":[[18,1],[18,2],[18,3],[18,4],[18,5],[18,6],[18,7],[18,8],[18,9]],\\\"text\\\":\\\"positional embedding instead of sinusoids\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,10],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]},{\\\"bbox\\\":[179,422,345,431],\\\"spans\\\":[[18,1],[18,2],[18,3],[18,4],[18,5],[18,6],[18,7],[18,8],[18,9]],\\\"text\\\":\\\"positional embedding instead of sinusoids\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[1,10],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]},{\\\"bbox\\\":[179,422,345,431],\\\"spans\\\":[[18,1],[18,2],[18,3],[18,4],[18,5],[18,6],[18,7],[18,8],[18,9]],\\\"text\\\":\\\"positional embedding instead of sinusoids\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[1,10],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]},{\\\"bbox\\\":[179,422,345,431],\\\"spans\\\":[[18,1],[18,2],[18,3],[18,4],[18,5],[18,6],[18,7],[18,8],[18,9]],\\\"text\\\":\\\"positional embedding instead of sinusoids\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[1,10],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]},{\\\"bbox\\\":[179,422,345,431],\\\"spans\\\":[[18,1],[18,2],[18,3],[18,4],[18,5],[18,6],[18,7],[18,8],[18,9]],\\\"text\\\":\\\"positional embedding instead of sinusoids\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":5,\\\"col-header\\\":false,\\\"col-span\\\":[1,10],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]},{\\\"bbox\\\":[179,422,345,431],\\\"spans\\\":[[18,1],[18,2],[18,3],[18,4],[18,5],[18,6],[18,7],[18,8],[18,9]],\\\"text\\\":\\\"positional embedding instead of sinusoids\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":6,\\\"col-header\\\":false,\\\"col-span\\\":[1,10],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]},{\\\"bbox\\\":[405,424,457,431],\\\"spans\\\":[[18,7]],\\\"text\\\":\\\"4.92 25.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":7,\\\"col-header\\\":false,\\\"col-span\\\":[7,8],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]},{\\\"bbox\\\":[179,422,345,431],\\\"spans\\\":[[18,1],[18,2],[18,3],[18,4],[18,5],[18,6],[18,7],[18,8],[18,9]],\\\"text\\\":\\\"positional embedding instead of sinusoids\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":8,\\\"col-header\\\":false,\\\"col-span\\\":[1,10],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]},{\\\"bbox\\\":[179,422,345,431],\\\"spans\\\":[[18,1],[18,2],[18,3],[18,4],[18,5],[18,6],[18,7],[18,8],[18,9]],\\\"text\\\":\\\"positional embedding instead of sinusoids\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":9,\\\"col-header\\\":false,\\\"col-span\\\":[1,10],\\\"row\\\":18,\\\"row-header\\\":false,\\\"row-span\\\":[18,19]}],[{\\\"bbox\\\":[119,409,131,418],\\\"spans\\\":[[19,0]],\\\"text\\\":\\\"big\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]},{\\\"spans\\\":[[19,1]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]},{\\\"spans\\\":[[19,2]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]},{\\\"spans\\\":[[19,3]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":3,\\\"col-header\\\":false,\\\"col-span\\\":[3,4],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]},{\\\"bbox\\\":[149,411,391,418],\\\"spans\\\":[[19,4]],\\\"text\\\":\\\"6 1024 4096 16 0.3 300K\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":4,\\\"col-header\\\":false,\\\"col-span\\\":[4,5],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]},{\\\"spans\\\":[[19,5]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":5,\\\"col-header\\\":false,\\\"col-span\\\":[5,6],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]},{\\\"spans\\\":[[19,6]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":6,\\\"col-header\\\":false,\\\"col-span\\\":[6,7],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]},{\\\"bbox\\\":[405,411,457,418],\\\"spans\\\":[[19,7]],\\\"text\\\":\\\"4.33 26.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":7,\\\"col-header\\\":false,\\\"col-span\\\":[7,8],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]},{\\\"bbox\\\":[481,411,495,418],\\\"spans\\\":[[19,8]],\\\"text\\\":\\\"213\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":8,\\\"col-header\\\":false,\\\"col-span\\\":[8,9],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]},{\\\"spans\\\":[[19,9]],\\\"text\\\":\\\"\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":9,\\\"col-header\\\":false,\\\"col-span\\\":[9,10],\\\"row\\\":19,\\\"row-header\\\":false,\\\"row-span\\\":[19,20]}]]},{\\\"prov\\\":[{\\\"bbox\\\":[144,555,467,699],\\\"page\\\":10,\\\"span\\\":[0,0]}],\\\"text\\\":\\\"Table 4: The Transformer generalizes well to English constituency parsing (Results are on Section 23 of WSJ)\\\",\\\"type\\\":\\\"table\\\",\\\"#-cols\\\":3,\\\"#-rows\\\":13,\\\"data\\\":[[{\\\"bbox\\\":[207,690,235,697],\\\"spans\\\":[[0,0]],\\\"text\\\":\\\"Parser\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[334,688,371,697],\\\"spans\\\":[[0,1]],\\\"text\\\":\\\"Training\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]},{\\\"bbox\\\":[415,689,460,697],\\\"spans\\\":[[0,2]],\\\"text\\\":\\\"WSJ 23 F1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":0,\\\"row-header\\\":false,\\\"row-span\\\":[0,1]}],[{\\\"bbox\\\":[151,677,290,686],\\\"spans\\\":[[1,0]],\\\"text\\\":\\\"Vinyals & Kaiser el al. (2014) [37]\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"bbox\\\":[303,677,402,686],\\\"spans\\\":[[1,1]],\\\"text\\\":\\\"WSJ only, discriminative\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]},{\\\"bbox\\\":[430,679,446,686],\\\"spans\\\":[[1,2]],\\\"text\\\":\\\"88.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":1,\\\"row-header\\\":false,\\\"row-span\\\":[1,2]}],[{\\\"bbox\\\":[173,666,268,675],\\\"spans\\\":[[2,0]],\\\"text\\\":\\\"Petrov et al. (2006) [29]\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"bbox\\\":[303,666,402,675],\\\"spans\\\":[[2,1]],\\\"text\\\":\\\"WSJ only, discriminative\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]},{\\\"bbox\\\":[429,668,446,675],\\\"spans\\\":[[2,2]],\\\"text\\\":\\\"90.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":2,\\\"row-header\\\":false,\\\"row-span\\\":[2,3]}],[{\\\"bbox\\\":[178,655,263,664],\\\"spans\\\":[[3,0]],\\\"text\\\":\\\"Zhu et al. (2013) [40]\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[303,655,402,664],\\\"spans\\\":[[3,1]],\\\"text\\\":\\\"WSJ only, discriminative\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]},{\\\"bbox\\\":[429,657,446,664],\\\"spans\\\":[[3,2]],\\\"text\\\":\\\"90.4\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":3,\\\"row-header\\\":false,\\\"row-span\\\":[3,4]}],[{\\\"bbox\\\":[178,644,263,653],\\\"spans\\\":[[4,0]],\\\"text\\\":\\\"Dyer et al. (2016) [8]\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"bbox\\\":[303,644,402,653],\\\"spans\\\":[[4,1]],\\\"text\\\":\\\"WSJ only, discriminative\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]},{\\\"bbox\\\":[429,646,446,653],\\\"spans\\\":[[4,2]],\\\"text\\\":\\\"91.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":4,\\\"row-header\\\":false,\\\"row-span\\\":[4,5]}],[{\\\"bbox\\\":[176,633,265,642],\\\"spans\\\":[[5,0]],\\\"text\\\":\\\"Transformer (4 layers)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[303,633,402,642],\\\"spans\\\":[[5,1]],\\\"text\\\":\\\"WSJ only, discriminative\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]},{\\\"bbox\\\":[429,635,446,642],\\\"spans\\\":[[5,2]],\\\"text\\\":\\\"91.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":5,\\\"row-header\\\":false,\\\"row-span\\\":[5,6]}],[{\\\"bbox\\\":[178,623,263,631],\\\"spans\\\":[[6,0]],\\\"text\\\":\\\"Zhu et al. (2013) [40]\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"bbox\\\":[321,622,385,631],\\\"spans\\\":[[6,1]],\\\"text\\\":\\\"semi-supervised\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]},{\\\"bbox\\\":[429,624,446,631],\\\"spans\\\":[[6,2]],\\\"text\\\":\\\"91.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":6,\\\"row-header\\\":false,\\\"row-span\\\":[6,7]}],[{\\\"bbox\\\":[163,611,277,620],\\\"spans\\\":[[7,0]],\\\"text\\\":\\\"Huang & Harper (2009) [14]\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"bbox\\\":[321,611,385,620],\\\"spans\\\":[[7,1]],\\\"text\\\":\\\"semi-supervised\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]},{\\\"bbox\\\":[429,613,446,620],\\\"spans\\\":[[7,2]],\\\"text\\\":\\\"91.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":7,\\\"row-header\\\":false,\\\"row-span\\\":[7,8]}],[{\\\"bbox\\\":[165,600,276,609],\\\"spans\\\":[[8,0]],\\\"text\\\":\\\"McClosky et al. (2006) [26]\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"bbox\\\":[321,600,385,609],\\\"spans\\\":[[8,1]],\\\"text\\\":\\\"semi-supervised\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]},{\\\"bbox\\\":[429,602,445,609],\\\"spans\\\":[[8,2]],\\\"text\\\":\\\"92.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":8,\\\"row-header\\\":false,\\\"row-span\\\":[8,9]}],[{\\\"bbox\\\":[151,589,290,598],\\\"spans\\\":[[9,0]],\\\"text\\\":\\\"Vinyals & Kaiser el al. (2014) [37]\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[321,589,385,598],\\\"spans\\\":[[9,1]],\\\"text\\\":\\\"semi-supervised\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]},{\\\"bbox\\\":[429,591,445,598],\\\"spans\\\":[[9,2]],\\\"text\\\":\\\"92.1\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":9,\\\"row-header\\\":false,\\\"row-span\\\":[9,10]}],[{\\\"bbox\\\":[176,579,265,588],\\\"spans\\\":[[10,0]],\\\"text\\\":\\\"Transformer (4 layers)\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[321,579,385,588],\\\"spans\\\":[[10,1]],\\\"text\\\":\\\"semi-supervised\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]},{\\\"bbox\\\":[429,581,446,587],\\\"spans\\\":[[10,2]],\\\"text\\\":\\\"92.7\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":10,\\\"row-header\\\":false,\\\"row-span\\\":[10,11]}],[{\\\"bbox\\\":[173,568,268,577],\\\"spans\\\":[[11,0]],\\\"text\\\":\\\"Luong et al. (2015) [23]\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"bbox\\\":[332,570,373,577],\\\"spans\\\":[[11,1]],\\\"text\\\":\\\"multi-task\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]},{\\\"bbox\\\":[429,570,446,577],\\\"spans\\\":[[11,2]],\\\"text\\\":\\\"93.0\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":11,\\\"row-header\\\":false,\\\"row-span\\\":[11,12]}],[{\\\"bbox\\\":[178,557,263,566],\\\"spans\\\":[[12,0]],\\\"text\\\":\\\"Dyer et al. (2016) [8]\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":0,\\\"col-header\\\":false,\\\"col-span\\\":[0,1],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"bbox\\\":[332,557,373,566],\\\"spans\\\":[[12,1]],\\\"text\\\":\\\"generative\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":1,\\\"col-header\\\":false,\\\"col-span\\\":[1,2],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]},{\\\"bbox\\\":[429,559,446,566],\\\"spans\\\":[[12,2]],\\\"text\\\":\\\"93.3\\\",\\\"type\\\":\\\"\\\",\\\"col\\\":2,\\\"col-header\\\":false,\\\"col-span\\\":[2,3],\\\"row\\\":12,\\\"row-header\\\":false,\\\"row-span\\\":[12,13]}]]}],\\\"equations\\\":[],\\\"footnotes\\\":[],\\\"page-dimensions\\\":[{\\\"height\\\":792,\\\"page\\\":1,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":2,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":3,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":4,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":5,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":6,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":7,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":8,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":9,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":10,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":11,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":12,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":13,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":14,\\\"width\\\":612},{\\\"height\\\":792,\\\"page\\\":15,\\\"width\\\":612}],\\\"page-footers\\\":[],\\\"page-headers\\\":[]}\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"num_pages\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 9,\n        \"min\": 15,\n        \"max\": 28,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          28,\n          15\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"num_tables\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 9,\n        \"min\": 4,\n        \"max\": 17,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          17,\n          4\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"num_doc_elements\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 89,\n        \"min\": 193,\n        \"max\": 320,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          320,\n          193\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"document_id\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"5ec63530-a806-4779-a4ca-24d92e5ee47a\",\n          \"1371115a-5421-4289-bbae-43882017b531\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"ext\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"pdf\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"hash\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"66a506489ca7abed61a40dd7e3fc73302e33f2169fab73f9679c7e8819983399\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"size\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 142877,\n        \"min\": 108568,\n        \"max\": 310628,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          310628\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"date_acquired\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"2024-09-04T20:46:16.208916\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"pdf_convert_time\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 165.6320917549954,\n        \"min\": 91.00510263442993,\n        \"max\": 325.24425315856934,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          325.24425315856934\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"source_filename\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"Granite_code_models.pdf\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "output_df"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-0d0c989e-b6fd-4227-b9b2-44a47972f207\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>filename</th>\n",
              "      <th>contents</th>\n",
              "      <th>num_pages</th>\n",
              "      <th>num_tables</th>\n",
              "      <th>num_doc_elements</th>\n",
              "      <th>document_id</th>\n",
              "      <th>ext</th>\n",
              "      <th>hash</th>\n",
              "      <th>size</th>\n",
              "      <th>date_acquired</th>\n",
              "      <th>pdf_convert_time</th>\n",
              "      <th>source_filename</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>attention_is_all_you_need.pdf</td>\n",
              "      <td>{\"_name\":\"\",\"type\":\"pdf-document\",\"description...</td>\n",
              "      <td>15</td>\n",
              "      <td>4</td>\n",
              "      <td>193</td>\n",
              "      <td>1371115a-5421-4289-bbae-43882017b531</td>\n",
              "      <td>pdf</td>\n",
              "      <td>69c4e62b4bffc298b2231e31e392cb51a514bcb03507b8...</td>\n",
              "      <td>108568</td>\n",
              "      <td>2024-09-04T20:47:47.345013</td>\n",
              "      <td>91.005103</td>\n",
              "      <td>attention_is_all_you_need.pdf</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Granite_code_models.pdf</td>\n",
              "      <td>{\"_name\":\"\",\"type\":\"pdf-document\",\"description...</td>\n",
              "      <td>28</td>\n",
              "      <td>17</td>\n",
              "      <td>320</td>\n",
              "      <td>5ec63530-a806-4779-a4ca-24d92e5ee47a</td>\n",
              "      <td>pdf</td>\n",
              "      <td>66a506489ca7abed61a40dd7e3fc73302e33f2169fab73...</td>\n",
              "      <td>310628</td>\n",
              "      <td>2024-09-04T20:46:16.208916</td>\n",
              "      <td>325.244253</td>\n",
              "      <td>Granite_code_models.pdf</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-0d0c989e-b6fd-4227-b9b2-44a47972f207')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-0d0c989e-b6fd-4227-b9b2-44a47972f207 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-0d0c989e-b6fd-4227-b9b2-44a47972f207');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-2bfa0acb-85e3-4fa9-a7a2-f905a9b99813\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-2bfa0acb-85e3-4fa9-a7a2-f905a9b99813')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-2bfa0acb-85e3-4fa9-a7a2-f905a9b99813 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_ccf50082-8d7f-4a82-ae3c-b0ebc730cb1d\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('output_df')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_ccf50082-8d7f-4a82-ae3c-b0ebc730cb1d button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('output_df');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                        filename  \\\n",
              "0  attention_is_all_you_need.pdf   \n",
              "1        Granite_code_models.pdf   \n",
              "\n",
              "                                            contents  num_pages  num_tables  \\\n",
              "0  {\"_name\":\"\",\"type\":\"pdf-document\",\"description...         15           4   \n",
              "1  {\"_name\":\"\",\"type\":\"pdf-document\",\"description...         28          17   \n",
              "\n",
              "   num_doc_elements                           document_id  ext  \\\n",
              "0               193  1371115a-5421-4289-bbae-43882017b531  pdf   \n",
              "1               320  5ec63530-a806-4779-a4ca-24d92e5ee47a  pdf   \n",
              "\n",
              "                                                hash    size  \\\n",
              "0  69c4e62b4bffc298b2231e31e392cb51a514bcb03507b8...  108568   \n",
              "1  66a506489ca7abed61a40dd7e3fc73302e33f2169fab73...  310628   \n",
              "\n",
              "                date_acquired  pdf_convert_time                source_filename  \n",
              "0  2024-09-04T20:47:47.345013         91.005103  attention_is_all_you_need.pdf  \n",
              "1  2024-09-04T20:46:16.208916        325.244253        Granite_code_models.pdf  "
            ]
          },
          "execution_count": 6,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "output_df = read_parquet_files_as_df(OUTPUT_DIR)\n",
        "output_df"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NheyE8u4vgJL",
        "outputId": "40ae9eb0-f877-4e5f-a8f9-f475f3df48f1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "('[\\n'\n",
            " '    '\n",
            " '\"{\\\\\"_name\\\\\":\\\\\"\\\\\",\\\\\"type\\\\\":\\\\\"pdf-document\\\\\",\\\\\"description\\\\\":{\\\\\"logs\\\\\":[]},\\\\\"file-info\\\\\":{\\\\\"filename\\\\\":\\\\\"attention_is_all_you_need.pdf\\\\\",\\\\\"document-hash\\\\\":\\\\\"bdfaa68d8984f0dc02beaca527b76f207d99b666d31d1da728ee0728182df697\\\\\",\\\\\"#-pages\\\\\":15,\\\\\"page-hashes\\\\\":[{\\\\\"hash\\\\\":\\\\\"8834a09ad99e9297886c9f8ad786c2784b7dc66dc6e6adfeff6bf2c1f07926d6\\\\\",\\\\\"model\\\\\":\\\\\"default\\\\\",\\\\\"page\\\\\":1},{\\\\\"hash\\\\\":\\\\\"72ded7022ad3cbfa9b5c4377a9c9b44511251f9489973956c23d2f3321e6307e\\\\\",\\\\\"model\\\\\":\\\\\"default\\\\\",\\\\\"page\\\\\":2},{\\\\\"hash\\\\\":\\\\\"38733274891513257d051950018621d95f73d05d5c70bfd7331def2f1194973d\\\\\",\\\\\"model\\\\\":\\\\\"default\\\\\",\\\\\"page\\\\\":3},{\\\\\"hash\\\\\":\\\\\"699ed16bf81021d0f86374d05c7b4b2b1049e63a28d2951ec1fb930747d755b9\\\\\",\\\\\"model\\\\\":\\\\\"default\\\\\",\\\\\"page\\\\\":4},{\\\\\"hash\\\\\":\\\\\"a17e6b313bdd51eff07a824253eff394d78ae1d6ebc985de3580bdfece38d2e1\\\\\",\\\\\"model\\\\\":\\\\\"default\\\\\",\\\\\"page\\\\\":5},{\\\\\"hash\\\\\":\\\\\"b3e9b63f2e8728fa83a5b7d911df2827585cf6040d2a4734cb3b44be264da6b6\\\\\",\\\\\"model\\\\\":\\\\\"default\\\\\",\\\\\"page\\\\\":6},{\\\\\"hash\\\\\":\\\\\"7b23bd1c80383b757a39456a4fd95ed2e9aaefd6a04512f181279c27a66c54a4\\\\\",\\\\\"model\\\\\":\\\\\"default\\\\\",\\\\\"page\\\\\":7},{\\\\\"hash\\\\\":\\\\\"c1dbbbf5b2ad441bf20149c26fc440a95f714987edf1f39690d703e67699dbc4\\\\\",\\\\\"model\\\\\":\\\\\"default\\\\\",\\\\\"page\\\\\":8},{\\\\\"hash\\\\\":\\\\\"ae2f68db548ba7a95d11ba1a1f0e36ca46c7d71d40a27c73dc56cf32932a9638\\\\\",\\\\\"model\\\\\":\\\\\"default\\\\\",\\\\\"page\\\\\":9},{\\\\\"hash\\\\\":\\\\\"58c67da2ef05339a90ab5c62b29eece0f60a7d8bb2f9eb390ba45a6d49e042f5\\\\\",\\\\\"model\\\\\":\\\\\"default\\\\\",\\\\\"page\\\\\":10},{\\\\\"hash\\\\\":\\\\\"b39d3fefe795d9d05aad7432498b5baeee7b255ed2da5bc88f60c051b8fae865\\\\\",\\\\\"model\\\\\":\\\\\"default\\\\\",\\\\\"page\\\\\":11},{\\\\\"hash\\\\\":\\\\\"1e5bf23dc7cd6799ee19684b7152560e0bda459b639ac3ea3f6f3fabf96362a4\\\\\",\\\\\"model\\\\\":\\\\\"default\\\\\",\\\\\"page\\\\\":12},{\\\\\"hash\\\\\":\\\\\"0ea258bf68e3835a534da65a943f679a19d02a19ae9b8afe9cd34af2cd29e9c4\\\\\",\\\\\"model\\\\\":\\\\\"default\\\\\",\\\\\"page\\\\\":13},{\\\\\"hash\\\\\":\\\\\"34c515052914fa661b7cedf6a25d7d4bd22a4d329971f74b9d1e75b3f6f02d16\\\\\",\\\\\"model\\\\\":\\\\\"default\\\\\",\\\\\"page\\\\\":14},{\\\\\"hash\\\\\":\\\\\"6ab332fddb6e68c5979d7481a53c9b75d91ac31564cad8972e3fa12cd7362769\\\\\",\\\\\"model\\\\\":\\\\\"default\\\\\",\\\\\"page\\\\\":15}]},\\\\\"main-text\\\\\":[{\\\\\"text\\\\\":\\\\\"arXiv:1706')\n"
          ]
        }
      ],
      "source": [
        "# Inspect contents\n",
        "\n",
        "import json\n",
        "import pprint\n",
        "\n",
        "column_list = output_df['contents'].tolist()\n",
        "column_json = json.dumps(column_list, indent=4)\n",
        "pprint.pprint(column_json[:2000]) # display first few lines"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yMcE4Io6vgJL"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "data-prep-kit-3-py311",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.9"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "02192a51069f40a2a0f831ecc2809085": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7c4a475bdb434c91a051d3408cda94d9",
            "placeholder": "​",
            "style": "IPY_MODEL_bfe2e357e7384315a3614797e9d530ae",
            "value": " 41.0/41.0 [00:00&lt;00:00, 1.14kB/s]"
          }
        },
        "02ce2efff8214690a2d0cea6832492a4": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f3e425eac5734ff19306ae71f627e079",
            "placeholder": "​",
            "style": "IPY_MODEL_691f09e257d54b499faaf90f72e44a3b",
            "value": "Fetching 7 files: 100%"
          }
        },
        "0453e6e227e6466b8fb2f6f80be09b8a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "0756cd16dc96426ba54f20c75686e6e3": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "0873f21424a3404dbcb995ec862a9f44": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "08be842c2e5e46589fb13deeb9c036bc": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "091493736776456392e4493174f9fa0f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "0e851e377db1415fbe9c62bc8bcc7d1a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "1196c066c5784a1bbcadc9a0202bfe02": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ebe75f475b9c43aca827b1f5cf68e511",
            "max": 169182553,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_30cacbb0c6674d4fa198381a502ab44e",
            "value": 169182553
          }
        },
        "136199f932974fc7a1b07d24dbf527bb": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "15263bd25c9741c3b34b073010642944": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1a02de7a971448fdb47d6b22c93476b3": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1a1ce455b56c4c5abc61a67b04fa8983": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_295ffb18518d4f55bed187eae1bb4a91",
              "IPY_MODEL_7d1f4804da354359ba4ffa44db911ef2",
              "IPY_MODEL_463d76d334c9438882ac033baa6a2a71"
            ],
            "layout": "IPY_MODEL_98e88ee67f8940b0b7c1ab0f4e5ed7fe"
          }
        },
        "1f048b048640473bb0b215d7da9e1a92": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "1fc82f82701c4c918f39b85dab253723": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "266a3a063fe34782a6546beac0eecb62": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2c22a39735d3452593db594323ed27dc",
            "max": 41,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_402b3286fec044b7a8bd12c9e46e3286",
            "value": 41
          }
        },
        "295ffb18518d4f55bed187eae1bb4a91": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_cac6ab1306d4430f91be43b5f4fc4585",
            "placeholder": "​",
            "style": "IPY_MODEL_a4797fa4499941318dbbbdc06bd59912",
            "value": "README.md: 100%"
          }
        },
        "2c22a39735d3452593db594323ed27dc": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2eeb6f5268484ef196f0ad26c77307cb": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "30cacbb0c6674d4fa198381a502ab44e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "30dc5f48be2d4b4da8716d2e0d23e372": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3e0010c2a38243d2ae4d4f27518b503b": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "402b3286fec044b7a8bd12c9e46e3286": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "463d76d334c9438882ac033baa6a2a71": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_68f49fdff2154a72b9875756c8469dba",
            "placeholder": "​",
            "style": "IPY_MODEL_ac44b83e481e4a4482f323b6328e39ff",
            "value": " 40.0/40.0 [00:00&lt;00:00, 757B/s]"
          }
        },
        "4912fdf719054575a6f5695fb106e74d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "4b2609707adc42e980fe7bc8171c0d1d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "4b8d0c62fda9415ca77acd3b135d3fa0": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_93232da807aa4bdfaa6df7af867ca183",
            "placeholder": "​",
            "style": "IPY_MODEL_4912fdf719054575a6f5695fb106e74d",
            "value": " 7.09k/7.09k [00:00&lt;00:00, 282kB/s]"
          }
        },
        "4d5ceb4c8b51480cb82caa36d573ca0b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "4da9634138834af6b53830e77822bbee": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_02ce2efff8214690a2d0cea6832492a4",
              "IPY_MODEL_86df69d190354ddaa33adce96936a6df",
              "IPY_MODEL_e93c215aa5304de39220f244ebc4b911"
            ],
            "layout": "IPY_MODEL_86c48f1756cd4b0fa73125e39dcb77ba"
          }
        },
        "4fff11b9bfad46e98b7c933f2994a725": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8010d3af49824ca5a6f7f9dd302494ba",
            "placeholder": "​",
            "style": "IPY_MODEL_0e851e377db1415fbe9c62bc8bcc7d1a",
            "value": " 5.18k/5.18k [00:00&lt;00:00, 141kB/s]"
          }
        },
        "5797c0a5b8824a3f84b72e4ce2e6de16": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a425f8aa6b9c4b8bb95cdc59f20734cf",
            "placeholder": "​",
            "style": "IPY_MODEL_0756cd16dc96426ba54f20c75686e6e3",
            "value": ".gitattributes: 100%"
          }
        },
        "58207c5fd9594c988308dcd6e32775b5": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5dbede0d03044ce3b9a6f08f7a924c62": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_15263bd25c9741c3b34b073010642944",
            "max": 5180,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_91ba93a7c927481b9e5ee9793988e9de",
            "value": 5180
          }
        },
        "5dc4d60351e54257907737d5deaea73a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8916ed32df05413ebc9fa43076f03f77",
            "placeholder": "​",
            "style": "IPY_MODEL_ee33612510194eee92a956bad872838c",
            "value": " 1.60k/1.60k [00:00&lt;00:00, 48.6kB/s]"
          }
        },
        "657fccdce16d4e128d51b6cb06f91056": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e4efea17c0fe49409a9c970dbcba9a93",
            "placeholder": "​",
            "style": "IPY_MODEL_0873f21424a3404dbcb995ec862a9f44",
            "value": "config.json: 100%"
          }
        },
        "677a6dc64b124372bb44e6df8066aa38": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a87905e174e64986b59575a4b0a1e799",
            "placeholder": "​",
            "style": "IPY_MODEL_c451b2d71a7d4291baabc7b59ed47b96",
            "value": " 169M/169M [00:04&lt;00:00, 31.8MB/s]"
          }
        },
        "68f49fdff2154a72b9875756c8469dba": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "691f09e257d54b499faaf90f72e44a3b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "6e04886fc4e6485a9941d67b4577bfda": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3e0010c2a38243d2ae4d4f27518b503b",
            "placeholder": "​",
            "style": "IPY_MODEL_091493736776456392e4493174f9fa0f",
            "value": "model.pt: 100%"
          }
        },
        "70834764cd4242708965f339e9f1f08e": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "71b4cb950be7449c9a63fc552fccf647": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1a02de7a971448fdb47d6b22c93476b3",
            "placeholder": "​",
            "style": "IPY_MODEL_4d5ceb4c8b51480cb82caa36d573ca0b",
            "value": "(…)del_artifacts/tableformer/tm_config.json: 100%"
          }
        },
        "76c2058596f04b1c9adbd456a5799f36": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_8409eca6c9034f3f94fce36a2933e3f3",
              "IPY_MODEL_d69525c97a264fea8cf50f1243aab2db",
              "IPY_MODEL_fda7c65808ed440b917d7299b500b485"
            ],
            "layout": "IPY_MODEL_2eeb6f5268484ef196f0ad26c77307cb"
          }
        },
        "7c4a475bdb434c91a051d3408cda94d9": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7d1f4804da354359ba4ffa44db911ef2": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9d8ea99d01c940e9a10889d1639d74e8",
            "max": 40,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_c3922d4ff33641f388ab101d7d4a981f",
            "value": 40
          }
        },
        "7e426116c9ac4a228ba1f9d94c6519bf": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "8010d3af49824ca5a6f7f9dd302494ba": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8409eca6c9034f3f94fce36a2933e3f3": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_dc5a5200d9f94dcf8c47c14590ff3952",
            "placeholder": "​",
            "style": "IPY_MODEL_7e426116c9ac4a228ba1f9d94c6519bf",
            "value": "otslp_all_fast.check: 100%"
          }
        },
        "85ff85b8cc074a9caa85f9db2f669ac0": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "86c48f1756cd4b0fa73125e39dcb77ba": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "86df69d190354ddaa33adce96936a6df": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_30dc5f48be2d4b4da8716d2e0d23e372",
            "max": 7,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_08be842c2e5e46589fb13deeb9c036bc",
            "value": 7
          }
        },
        "88a9c683544b416e8b58c54627320901": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_71b4cb950be7449c9a63fc552fccf647",
              "IPY_MODEL_f0a4d1f74e0947c780e64be7e2d33f97",
              "IPY_MODEL_4b8d0c62fda9415ca77acd3b135d3fa0"
            ],
            "layout": "IPY_MODEL_b7eb047535a245a2b1e46c68aa9c86d8"
          }
        },
        "8916ed32df05413ebc9fa43076f03f77": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8beac29537a4472ba369ccc91674dbf1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_5797c0a5b8824a3f84b72e4ce2e6de16",
              "IPY_MODEL_b483da44bc634a0c9b7a6fbc9f8443db",
              "IPY_MODEL_5dc4d60351e54257907737d5deaea73a"
            ],
            "layout": "IPY_MODEL_904f7974a8d24a60a9b6fe65fd42f4a4"
          }
        },
        "904f7974a8d24a60a9b6fe65fd42f4a4": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "91ba93a7c927481b9e5ee9793988e9de": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "93232da807aa4bdfaa6df7af867ca183": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "93342e0027934b22b8e5967fb59af428": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_58207c5fd9594c988308dcd6e32775b5",
            "placeholder": "​",
            "style": "IPY_MODEL_1f048b048640473bb0b215d7da9e1a92",
            "value": ".gitignore: 100%"
          }
        },
        "950921b442fc4f739fd186f051c76d81": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_93342e0027934b22b8e5967fb59af428",
              "IPY_MODEL_5dbede0d03044ce3b9a6f08f7a924c62",
              "IPY_MODEL_4fff11b9bfad46e98b7c933f2994a725"
            ],
            "layout": "IPY_MODEL_ae8e349ca5a741878cef03f248061f2e"
          }
        },
        "98e88ee67f8940b0b7c1ab0f4e5ed7fe": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9959454762d64b199ab209eef345e8e3": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_657fccdce16d4e128d51b6cb06f91056",
              "IPY_MODEL_266a3a063fe34782a6546beac0eecb62",
              "IPY_MODEL_02192a51069f40a2a0f831ecc2809085"
            ],
            "layout": "IPY_MODEL_70834764cd4242708965f339e9f1f08e"
          }
        },
        "9d8ea99d01c940e9a10889d1639d74e8": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a1939affb9954aca86cdcd9430ab569b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "a425f8aa6b9c4b8bb95cdc59f20734cf": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a4797fa4499941318dbbbdc06bd59912": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a87905e174e64986b59575a4b0a1e799": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ac44b83e481e4a4482f323b6328e39ff": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ae837f1a69a74a3b863a9c19202da0e7": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ae8e349ca5a741878cef03f248061f2e": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b483da44bc634a0c9b7a6fbc9f8443db": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ef3251cf7d884620a30719e0ae5b49a0",
            "max": 1604,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_0453e6e227e6466b8fb2f6f80be09b8a",
            "value": 1604
          }
        },
        "b7eb047535a245a2b1e46c68aa9c86d8": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bfe2e357e7384315a3614797e9d530ae": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c3922d4ff33641f388ab101d7d4a981f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "c3c45d72c4e74a1faef0632f2c768608": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c451b2d71a7d4291baabc7b59ed47b96": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c92ce42d31784202b1b3b210ae77aae7": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "cac6ab1306d4430f91be43b5f4fc4585": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d69525c97a264fea8cf50f1243aab2db": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_136199f932974fc7a1b07d24dbf527bb",
            "max": 145516093,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_4b2609707adc42e980fe7bc8171c0d1d",
            "value": 145516093
          }
        },
        "dc5a5200d9f94dcf8c47c14590ff3952": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e0ce578a163c40f684053b3d6f3db5ec": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_6e04886fc4e6485a9941d67b4577bfda",
              "IPY_MODEL_1196c066c5784a1bbcadc9a0202bfe02",
              "IPY_MODEL_677a6dc64b124372bb44e6df8066aa38"
            ],
            "layout": "IPY_MODEL_85ff85b8cc074a9caa85f9db2f669ac0"
          }
        },
        "e4efea17c0fe49409a9c970dbcba9a93": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e93c215aa5304de39220f244ebc4b911": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ae837f1a69a74a3b863a9c19202da0e7",
            "placeholder": "​",
            "style": "IPY_MODEL_c92ce42d31784202b1b3b210ae77aae7",
            "value": " 7/7 [00:04&lt;00:00,  1.11it/s]"
          }
        },
        "ebe75f475b9c43aca827b1f5cf68e511": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ee33612510194eee92a956bad872838c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ef3251cf7d884620a30719e0ae5b49a0": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f0a4d1f74e0947c780e64be7e2d33f97": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f5cde5c28a3e4788bbcf193dc64ebe57",
            "max": 7087,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_a1939affb9954aca86cdcd9430ab569b",
            "value": 7087
          }
        },
        "f3e425eac5734ff19306ae71f627e079": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f5cde5c28a3e4788bbcf193dc64ebe57": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "fda7c65808ed440b917d7299b500b485": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1fc82f82701c4c918f39b85dab253723",
            "placeholder": "​",
            "style": "IPY_MODEL_c3c45d72c4e74a1faef0632f2c768608",
            "value": " 146M/146M [00:02&lt;00:00, 96.8MB/s]"
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}