02-colossalai-sft-colab.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "collapsed_sections": [
        "NY7XsonIG8Ev",
        "bYe-_gnDHEEa"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# 语言模型练习场02：ColossalAI（SFT部分）Google Colab版本"
      ],
      "metadata": {
        "id": "AamIytcNnxL1"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "- [https://github.com/hpcaitech/ColossalAI](https://github.com/hpcaitech/ColossalAI)"
      ],
      "metadata": {
        "id": "yAw_VzT6pVwW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**注意：**\n",
        "- 此notebook**只演示在google colab下如何跑通ColossalAI的SFT部分**，并不会包含超参数的调整、对结果的分析等\n",
        "- 如果想在**Kaggle Notebook**中跑，可以看这篇文章的Kaggle版：[穷穷穷孩子如何体验ColossalAI SFT（Kaggle篇）](https://mp.weixin.qq.com/s/Q29uSNxvPMy0rC-QxHiGZA)\n",
        "- **如果你有自己的机器**，则此notebook对你的帮助可能不大（因为你不需要在notebook上进行训练）\n",
        "- 此notebook的受众是手里没有GPU资源，但是又想熟悉和浅浅尝试ColossalAI的小伙伴\n",
        "- Google Colab目前可以获得的免费运算资源是**T4（大概有15GB的显存）**\n",
        "- **运行太长时间会自动终止**。所以建议重要的运行结果保存路径要指向自己的google drive目录中，而不那么重要的文件可以直接安装到默认目录下（终止后会被自动删除）"
      ],
      "metadata": {
        "id": "hmoMi8BspejU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**数据的准备：**\n",
        "- 根据[官方文档的提示](https://github.com/hpcaitech/ColossalAI/tree/main/applications/Chat/examples)，在运行前需要准备好数据（https://github.com/hpcaitech/ColossalAI/tree/main/applications/Chat/examples）\n",
        "- 数据可以在[这里下载](https://github.com/XueFuzhao/InstructionWild/tree/main/data)。注意不要下载seed文件（因为seed文件只有instruction，而没有response），要下载README里面提到的json文件，例如instinwild_ch.json（https://github.com/XueFuzhao/InstructionWild/tree/main/data）"
      ],
      "metadata": {
        "id": "lvubgzEuGch2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 安装环境"
      ],
      "metadata": {
        "id": "zN_47P5fG3c_"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 1 安装ColossalAI"
      ],
      "metadata": {
        "id": "NY7XsonIG8Ev"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PhpvqFt06DSm",
        "outputId": "62102879-51e9-4014-a248-7e2020d83ded"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'ColossalAI'...\n",
            "remote: Enumerating objects: 26116, done.\u001b[K\n",
            "remote: Counting objects: 100% (5/5), done.\u001b[K\n",
            "remote: Compressing objects: 100% (5/5), done.\u001b[K\n",
            "remote: Total 26116 (delta 0), reused 0 (delta 0), pack-reused 26111\u001b[K\n",
            "Receiving objects: 100% (26116/26116), 23.36 MiB | 34.12 MiB/s, done.\n",
            "Resolving deltas: 100% (17448/17448), done.\n"
          ]
        }
      ],
      "source": [
        "!git clone https://github.com/hpcaitech/ColossalAI.git"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "os.chdir('./ColossalAI')\n",
        "!pip install ."
      ],
      "metadata": {
        "id": "kWmQqVbu6Ncc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7554b306-038f-4b06-a5e0-a7734c8e8945"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Processing /content/ColossalAI\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from colossalai==0.2.8) (1.22.4)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from colossalai==0.2.8) (4.65.0)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from colossalai==0.2.8) (5.9.5)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from colossalai==0.2.8) (23.1)\n",
            "Collecting pre-commit (from colossalai==0.2.8)\n",
            "  Downloading pre_commit-3.3.2-py2.py3-none-any.whl (202 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m202.8/202.8 kB\u001b[0m \u001b[31m6.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from colossalai==0.2.8) (13.3.4)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from colossalai==0.2.8) (8.1.3)\n",
            "Collecting fabric (from colossalai==0.2.8)\n",
            "  Downloading fabric-3.0.1-py3-none-any.whl (53 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m53.3/53.3 kB\u001b[0m \u001b[31m7.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting contexttimer (from colossalai==0.2.8)\n",
            "  Downloading contexttimer-0.3.3.tar.gz (4.9 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting ninja (from colossalai==0.2.8)\n",
            "  Downloading ninja-1.11.1-py2.py3-none-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (145 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m146.0/146.0 kB\u001b[0m \u001b[31m21.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: torch>=1.11 in /usr/local/lib/python3.10/dist-packages (from colossalai==0.2.8) (2.0.1+cu118)\n",
            "Collecting safetensors (from colossalai==0.2.8)\n",
            "  Downloading safetensors-0.3.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m39.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch>=1.11->colossalai==0.2.8) (3.12.0)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch>=1.11->colossalai==0.2.8) (4.5.0)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.11->colossalai==0.2.8) (1.11.1)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.11->colossalai==0.2.8) (3.1)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11->colossalai==0.2.8) (3.1.2)\n",
            "Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11->colossalai==0.2.8) (2.0.0)\n",
            "Requirement already satisfied: cmake in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.11->colossalai==0.2.8) (3.25.2)\n",
            "Requirement already satisfied: lit in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.11->colossalai==0.2.8) (16.0.5)\n",
            "Collecting invoke>=2.0 (from fabric->colossalai==0.2.8)\n",
            "  Downloading invoke-2.1.2-py3-none-any.whl (160 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m160.1/160.1 kB\u001b[0m \u001b[31m19.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting paramiko>=2.4 (from fabric->colossalai==0.2.8)\n",
            "  Downloading paramiko-3.1.0-py3-none-any.whl (211 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m211.2/211.2 kB\u001b[0m \u001b[31m26.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting cfgv>=2.0.0 (from pre-commit->colossalai==0.2.8)\n",
            "  Downloading cfgv-3.3.1-py2.py3-none-any.whl (7.3 kB)\n",
            "Collecting identify>=1.0.0 (from pre-commit->colossalai==0.2.8)\n",
            "  Downloading identify-2.5.24-py2.py3-none-any.whl (98 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m98.8/98.8 kB\u001b[0m \u001b[31m13.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nodeenv>=0.11.1 (from pre-commit->colossalai==0.2.8)\n",
            "  Downloading nodeenv-1.8.0-py2.py3-none-any.whl (22 kB)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from pre-commit->colossalai==0.2.8) (6.0)\n",
            "Collecting virtualenv>=20.10.0 (from pre-commit->colossalai==0.2.8)\n",
            "  Downloading virtualenv-20.23.0-py3-none-any.whl (3.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.3/3.3 MB\u001b[0m \u001b[31m81.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: markdown-it-py<3.0.0,>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->colossalai==0.2.8) (2.2.0)\n",
            "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich->colossalai==0.2.8) (2.14.0)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py<3.0.0,>=2.2.0->rich->colossalai==0.2.8) (0.1.2)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from nodeenv>=0.11.1->pre-commit->colossalai==0.2.8) (67.7.2)\n",
            "Collecting bcrypt>=3.2 (from paramiko>=2.4->fabric->colossalai==0.2.8)\n",
            "  Downloading bcrypt-4.0.1-cp36-abi3-manylinux_2_28_x86_64.whl (593 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m593.7/593.7 kB\u001b[0m \u001b[31m54.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: cryptography>=3.3 in /usr/local/lib/python3.10/dist-packages (from paramiko>=2.4->fabric->colossalai==0.2.8) (40.0.2)\n",
            "Collecting pynacl>=1.5 (from paramiko>=2.4->fabric->colossalai==0.2.8)\n",
            "  Downloading PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (856 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m856.7/856.7 kB\u001b[0m \u001b[31m67.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting distlib<1,>=0.3.6 (from virtualenv>=20.10.0->pre-commit->colossalai==0.2.8)\n",
            "  Downloading distlib-0.3.6-py2.py3-none-any.whl (468 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m468.5/468.5 kB\u001b[0m \u001b[31m50.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: platformdirs<4,>=3.2 in /usr/local/lib/python3.10/dist-packages (from virtualenv>=20.10.0->pre-commit->colossalai==0.2.8) (3.3.0)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.11->colossalai==0.2.8) (2.1.2)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.11->colossalai==0.2.8) (1.3.0)\n",
            "Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.10/dist-packages (from cryptography>=3.3->paramiko>=2.4->fabric->colossalai==0.2.8) (1.15.1)\n",
            "Requirement already satisfied: pycparser in /usr/local/lib/python3.10/dist-packages (from cffi>=1.12->cryptography>=3.3->paramiko>=2.4->fabric->colossalai==0.2.8) (2.21)\n",
            "Building wheels for collected packages: colossalai, contexttimer\n",
            "  Building wheel for colossalai (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for colossalai: filename=colossalai-0.2.8-py3-none-any.whl size=1106778 sha256=c48863e1d61e5a17a6d7d3fbd80022a7990e07e012015591dd40042ddb3b2760\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-bobgacrn/wheels/b1/fb/16/e46aa3127ee272b8cac710c8f76aa02445d96aaeed9da956ea\n",
            "  Building wheel for contexttimer (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for contexttimer: filename=contexttimer-0.3.3-py3-none-any.whl size=5803 sha256=21ac5cf4b74bf53b9a96c4f8705106a7107d03a3c3f118b2515e3f7419170e06\n",
            "  Stored in directory: /root/.cache/pip/wheels/72/1c/da/cfd97201d88ccce214427fa84a5caeb91fef7c5a1b4c4312b4\n",
            "Successfully built colossalai contexttimer\n",
            "Installing collected packages: safetensors, ninja, distlib, contexttimer, virtualenv, nodeenv, invoke, identify, cfgv, bcrypt, pynacl, pre-commit, paramiko, fabric, colossalai\n",
            "Successfully installed bcrypt-4.0.1 cfgv-3.3.1 colossalai-0.2.8 contexttimer-0.3.3 distlib-0.3.6 fabric-3.0.1 identify-2.5.24 invoke-2.1.2 ninja-1.11.1 nodeenv-1.8.0 paramiko-3.1.0 pre-commit-3.3.2 pynacl-1.5.0 safetensors-0.3.1 virtualenv-20.23.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 2 安装transformers\n",
        "这里我们安装的是hpcaitech下的transformers，如果直接pip install transformers是否可行并没有测试。"
      ],
      "metadata": {
        "id": "bYe-_gnDHEEa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!git clone https://github.com/hpcaitech/transformers\n",
        "os.chdir('./transformers')\n",
        "!pip install ."
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PJTkDrHUhFpP",
        "outputId": "11383ade-42af-4f76-ab3e-f47f955a863d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'transformers'...\n",
            "remote: Enumerating objects: 124468, done.\u001b[K\n",
            "remote: Total 124468 (delta 0), reused 0 (delta 0), pack-reused 124468\u001b[K\n",
            "Receiving objects: 100% (124468/124468), 127.28 MiB | 27.99 MiB/s, done.\n",
            "Resolving deltas: 100% (93320/93320), done.\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Processing /content/ColossalAI/transformers\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers==4.28.0.dev0) (3.12.0)\n",
            "Collecting huggingface-hub<1.0,>=0.11.0 (from transformers==4.28.0.dev0)\n",
            "  Downloading huggingface_hub-0.14.1-py3-none-any.whl (224 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m224.5/224.5 kB\u001b[0m \u001b[31m6.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.28.0.dev0) (1.22.4)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers==4.28.0.dev0) (23.1)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers==4.28.0.dev0) (6.0)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.28.0.dev0) (2022.10.31)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers==4.28.0.dev0) (2.27.1)\n",
            "Collecting tokenizers!=0.11.3,<0.14,>=0.11.1 (from transformers==4.28.0.dev0)\n",
            "  Downloading tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m64.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers==4.28.0.dev0) (4.65.0)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.11.0->transformers==4.28.0.dev0) (2023.4.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.11.0->transformers==4.28.0.dev0) (4.5.0)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.28.0.dev0) (1.26.15)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.28.0.dev0) (2022.12.7)\n",
            "Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.28.0.dev0) (2.0.12)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.28.0.dev0) (3.4)\n",
            "Building wheels for collected packages: transformers\n",
            "  Building wheel for transformers (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for transformers: filename=transformers-4.28.0.dev0-py3-none-any.whl size=6790611 sha256=92968d77f2b7dc7aa1557f1ab5447bb5ddbf00b2936e257790df7e00d1659f10\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-pyix5uo5/wheels/87/f3/6f/b220a07b1eb427c5c698eed3338325ec784fe66427d0989fa6\n",
            "Successfully built transformers\n",
            "Installing collected packages: tokenizers, huggingface-hub, transformers\n",
            "Successfully installed huggingface-hub-0.14.1 tokenizers-0.13.3 transformers-4.28.0.dev0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 3 安装Chat需要的库"
      ],
      "metadata": {
        "id": "Y_KsSNVYHM_I"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "os.chdir('../applications/Chat/')\n",
        "!pip install ."
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "F73wlOCahZRz",
        "outputId": "e6c92639-0202-47d4-d917-637dc8ba064e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Processing /content/ColossalAI/applications/Chat\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: transformers>=4.20.1 in /usr/local/lib/python3.10/dist-packages (from coati==1.0.0) (4.28.0.dev0)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from coati==1.0.0) (4.65.0)\n",
            "Collecting datasets (from coati==1.0.0)\n",
            "  Downloading datasets-2.12.0-py3-none-any.whl (474 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m474.6/474.6 kB\u001b[0m \u001b[31m10.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting loralib (from coati==1.0.0)\n",
            "  Downloading loralib-0.1.1-py3-none-any.whl (8.8 kB)\n",
            "Requirement already satisfied: colossalai>=0.2.4 in /usr/local/lib/python3.10/dist-packages (from coati==1.0.0) (0.2.8)\n",
            "Collecting torch<2.0.0,>=1.12.1 (from coati==1.0.0)\n",
            "  Downloading torch-1.13.1-cp310-cp310-manylinux1_x86_64.whl (887.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m887.5/887.5 MB\u001b[0m \u001b[31m1.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting langchain (from coati==1.0.0)\n",
            "  Downloading langchain-0.0.178-py3-none-any.whl (892 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m892.2/892.2 kB\u001b[0m \u001b[31m15.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: tokenizers in /usr/local/lib/python3.10/dist-packages (from coati==1.0.0) (0.13.3)\n",
            "Collecting fastapi (from coati==1.0.0)\n",
            "  Downloading fastapi-0.95.2-py3-none-any.whl (56 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m57.0/57.0 kB\u001b[0m \u001b[31m7.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting sse_starlette (from coati==1.0.0)\n",
            "  Downloading sse_starlette-1.6.1-py3-none-any.whl (9.6 kB)\n",
            "Collecting wandb (from coati==1.0.0)\n",
            "  Downloading wandb-0.15.3-py3-none-any.whl (2.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m17.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting sentencepiece (from coati==1.0.0)\n",
            "  Downloading sentencepiece-0.1.99-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m16.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting gpustat (from coati==1.0.0)\n",
            "  Downloading gpustat-1.1.tar.gz (97 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m97.9/97.9 kB\u001b[0m \u001b[31m13.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from colossalai>=0.2.4->coati==1.0.0) (1.22.4)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from colossalai>=0.2.4->coati==1.0.0) (5.9.5)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from colossalai>=0.2.4->coati==1.0.0) (23.1)\n",
            "Requirement already satisfied: pre-commit in /usr/local/lib/python3.10/dist-packages (from colossalai>=0.2.4->coati==1.0.0) (3.3.2)\n",
            "Requirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from colossalai>=0.2.4->coati==1.0.0) (13.3.4)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from colossalai>=0.2.4->coati==1.0.0) (8.1.3)\n",
            "Requirement already satisfied: fabric in /usr/local/lib/python3.10/dist-packages (from colossalai>=0.2.4->coati==1.0.0) (3.0.1)\n",
            "Requirement already satisfied: contexttimer in /usr/local/lib/python3.10/dist-packages (from colossalai>=0.2.4->coati==1.0.0) (0.3.3)\n",
            "Requirement already satisfied: ninja in /usr/local/lib/python3.10/dist-packages (from colossalai>=0.2.4->coati==1.0.0) (1.11.1)\n",
            "Requirement already satisfied: safetensors in /usr/local/lib/python3.10/dist-packages (from colossalai>=0.2.4->coati==1.0.0) (0.3.1)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch<2.0.0,>=1.12.1->coati==1.0.0) (4.5.0)\n",
            "Collecting nvidia-cuda-runtime-cu11==11.7.99 (from torch<2.0.0,>=1.12.1->coati==1.0.0)\n",
            "  Downloading nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (849 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m849.3/849.3 kB\u001b[0m \u001b[31m16.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-cudnn-cu11==8.5.0.96 (from torch<2.0.0,>=1.12.1->coati==1.0.0)\n",
            "  Downloading nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m557.1/557.1 MB\u001b[0m \u001b[31m2.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-cublas-cu11==11.10.3.66 (from torch<2.0.0,>=1.12.1->coati==1.0.0)\n",
            "  Downloading nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl (317.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m317.1/317.1 MB\u001b[0m \u001b[31m4.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-cuda-nvrtc-cu11==11.7.99 (from torch<2.0.0,>=1.12.1->coati==1.0.0)\n",
            "  Downloading nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl (21.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m21.0/21.0 MB\u001b[0m \u001b[31m26.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from nvidia-cublas-cu11==11.10.3.66->torch<2.0.0,>=1.12.1->coati==1.0.0) (67.7.2)\n",
            "Requirement already satisfied: wheel in /usr/local/lib/python3.10/dist-packages (from nvidia-cublas-cu11==11.10.3.66->torch<2.0.0,>=1.12.1->coati==1.0.0) (0.40.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers>=4.20.1->coati==1.0.0) (3.12.0)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.11.0 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.20.1->coati==1.0.0) (0.14.1)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.20.1->coati==1.0.0) (6.0)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.20.1->coati==1.0.0) (2022.10.31)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers>=4.20.1->coati==1.0.0) (2.27.1)\n",
            "Requirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets->coati==1.0.0) (9.0.0)\n",
            "Collecting dill<0.3.7,>=0.3.0 (from datasets->coati==1.0.0)\n",
            "  Downloading dill-0.3.6-py3-none-any.whl (110 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m110.5/110.5 kB\u001b[0m \u001b[31m13.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets->coati==1.0.0) (1.5.3)\n",
            "Collecting xxhash (from datasets->coati==1.0.0)\n",
            "  Downloading xxhash-3.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (212 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m212.5/212.5 kB\u001b[0m \u001b[31m20.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting multiprocess (from datasets->coati==1.0.0)\n",
            "  Downloading multiprocess-0.70.14-py310-none-any.whl (134 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.3/134.3 kB\u001b[0m \u001b[31m16.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: fsspec[http]>=2021.11.1 in /usr/local/lib/python3.10/dist-packages (from datasets->coati==1.0.0) (2023.4.0)\n",
            "Collecting aiohttp (from datasets->coati==1.0.0)\n",
            "  Downloading aiohttp-3.8.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.0/1.0 MB\u001b[0m \u001b[31m28.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting responses<0.19 (from datasets->coati==1.0.0)\n",
            "  Downloading responses-0.18.0-py3-none-any.whl (38 kB)\n",
            "Requirement already satisfied: pydantic!=1.7,!=1.7.1,!=1.7.2,!=1.7.3,!=1.8,!=1.8.1,<2.0.0,>=1.6.2 in /usr/local/lib/python3.10/dist-packages (from fastapi->coati==1.0.0) (1.10.7)\n",
            "Collecting starlette<0.28.0,>=0.27.0 (from fastapi->coati==1.0.0)\n",
            "  Downloading starlette-0.27.0-py3-none-any.whl (66 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m67.0/67.0 kB\u001b[0m \u001b[31m8.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-ml-py>=11.450.129 (from gpustat->coati==1.0.0)\n",
            "  Downloading nvidia_ml_py-11.525.112-py3-none-any.whl (35 kB)\n",
            "Collecting blessed>=1.17.1 (from gpustat->coati==1.0.0)\n",
            "  Downloading blessed-1.20.0-py2.py3-none-any.whl (58 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.4/58.4 kB\u001b[0m \u001b[31m7.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: SQLAlchemy<3,>=1.4 in /usr/local/lib/python3.10/dist-packages (from langchain->coati==1.0.0) (2.0.10)\n",
            "Collecting async-timeout<5.0.0,>=4.0.0 (from langchain->coati==1.0.0)\n",
            "  Downloading async_timeout-4.0.2-py3-none-any.whl (5.8 kB)\n",
            "Collecting dataclasses-json<0.6.0,>=0.5.7 (from langchain->coati==1.0.0)\n",
            "  Downloading dataclasses_json-0.5.7-py3-none-any.whl (25 kB)\n",
            "Requirement already satisfied: numexpr<3.0.0,>=2.8.4 in /usr/local/lib/python3.10/dist-packages (from langchain->coati==1.0.0) (2.8.4)\n",
            "Collecting openapi-schema-pydantic<2.0,>=1.2 (from langchain->coati==1.0.0)\n",
            "  Downloading openapi_schema_pydantic-1.2.4-py3-none-any.whl (90 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m90.0/90.0 kB\u001b[0m \u001b[31m12.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: tenacity<9.0.0,>=8.1.0 in /usr/local/lib/python3.10/dist-packages (from langchain->coati==1.0.0) (8.2.2)\n",
            "Collecting GitPython!=3.1.29,>=1.0.0 (from wandb->coati==1.0.0)\n",
            "  Downloading GitPython-3.1.31-py3-none-any.whl (184 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m184.3/184.3 kB\u001b[0m \u001b[31m21.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting sentry-sdk>=1.0.0 (from wandb->coati==1.0.0)\n",
            "  Downloading sentry_sdk-1.24.0-py2.py3-none-any.whl (206 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m206.5/206.5 kB\u001b[0m \u001b[31m18.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting docker-pycreds>=0.4.0 (from wandb->coati==1.0.0)\n",
            "  Downloading docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB)\n",
            "Collecting pathtools (from wandb->coati==1.0.0)\n",
            "  Downloading pathtools-0.1.2.tar.gz (11 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting setproctitle (from wandb->coati==1.0.0)\n",
            "  Downloading setproctitle-1.3.2-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (30 kB)\n",
            "Requirement already satisfied: appdirs>=1.4.3 in /usr/local/lib/python3.10/dist-packages (from wandb->coati==1.0.0) (1.4.4)\n",
            "Requirement already satisfied: protobuf!=4.21.0,<5,>=3.19.0 in /usr/local/lib/python3.10/dist-packages (from wandb->coati==1.0.0) (3.20.3)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->coati==1.0.0) (23.1.0)\n",
            "Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->coati==1.0.0) (2.0.12)\n",
            "Collecting multidict<7.0,>=4.5 (from aiohttp->datasets->coati==1.0.0)\n",
            "  Downloading multidict-6.0.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (114 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.5/114.5 kB\u001b[0m \u001b[31m16.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting yarl<2.0,>=1.0 (from aiohttp->datasets->coati==1.0.0)\n",
            "  Downloading yarl-1.9.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (268 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m268.8/268.8 kB\u001b[0m \u001b[31m26.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting frozenlist>=1.1.1 (from aiohttp->datasets->coati==1.0.0)\n",
            "  Downloading frozenlist-1.3.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (149 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m149.6/149.6 kB\u001b[0m \u001b[31m19.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting aiosignal>=1.1.2 (from aiohttp->datasets->coati==1.0.0)\n",
            "  Downloading aiosignal-1.3.1-py3-none-any.whl (7.6 kB)\n",
            "Requirement already satisfied: wcwidth>=0.1.4 in /usr/local/lib/python3.10/dist-packages (from blessed>=1.17.1->gpustat->coati==1.0.0) (0.2.6)\n",
            "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.10/dist-packages (from blessed>=1.17.1->gpustat->coati==1.0.0) (1.16.0)\n",
            "Collecting marshmallow<4.0.0,>=3.3.0 (from dataclasses-json<0.6.0,>=0.5.7->langchain->coati==1.0.0)\n",
            "  Downloading marshmallow-3.19.0-py3-none-any.whl (49 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.1/49.1 kB\u001b[0m \u001b[31m7.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting marshmallow-enum<2.0.0,>=1.5.1 (from dataclasses-json<0.6.0,>=0.5.7->langchain->coati==1.0.0)\n",
            "  Downloading marshmallow_enum-1.5.1-py2.py3-none-any.whl (4.2 kB)\n",
            "Collecting typing-inspect>=0.4.0 (from dataclasses-json<0.6.0,>=0.5.7->langchain->coati==1.0.0)\n",
            "  Downloading typing_inspect-0.8.0-py3-none-any.whl (8.7 kB)\n",
            "Collecting gitdb<5,>=4.0.1 (from GitPython!=3.1.29,>=1.0.0->wandb->coati==1.0.0)\n",
            "  Downloading gitdb-4.0.10-py3-none-any.whl (62 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.7/62.7 kB\u001b[0m \u001b[31m8.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.20.1->coati==1.0.0) (1.26.15)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.20.1->coati==1.0.0) (2022.12.7)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.20.1->coati==1.0.0) (3.4)\n",
            "Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/dist-packages (from SQLAlchemy<3,>=1.4->langchain->coati==1.0.0) (2.0.2)\n",
            "Requirement already satisfied: anyio<5,>=3.4.0 in /usr/local/lib/python3.10/dist-packages (from starlette<0.28.0,>=0.27.0->fastapi->coati==1.0.0) (3.6.2)\n",
            "Requirement already satisfied: invoke>=2.0 in /usr/local/lib/python3.10/dist-packages (from fabric->colossalai>=0.2.4->coati==1.0.0) (2.1.2)\n",
            "Requirement already satisfied: paramiko>=2.4 in /usr/local/lib/python3.10/dist-packages (from fabric->colossalai>=0.2.4->coati==1.0.0) (3.1.0)\n",
            "Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->coati==1.0.0) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->coati==1.0.0) (2022.7.1)\n",
            "Requirement already satisfied: cfgv>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from pre-commit->colossalai>=0.2.4->coati==1.0.0) (3.3.1)\n",
            "Requirement already satisfied: identify>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from pre-commit->colossalai>=0.2.4->coati==1.0.0) (2.5.24)\n",
            "Requirement already satisfied: nodeenv>=0.11.1 in /usr/local/lib/python3.10/dist-packages (from pre-commit->colossalai>=0.2.4->coati==1.0.0) (1.8.0)\n",
            "Requirement already satisfied: virtualenv>=20.10.0 in /usr/local/lib/python3.10/dist-packages (from pre-commit->colossalai>=0.2.4->coati==1.0.0) (20.23.0)\n",
            "Requirement already satisfied: markdown-it-py<3.0.0,>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->colossalai>=0.2.4->coati==1.0.0) (2.2.0)\n",
            "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich->colossalai>=0.2.4->coati==1.0.0) (2.14.0)\n",
            "Requirement already satisfied: sniffio>=1.1 in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi->coati==1.0.0) (1.3.0)\n",
            "Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->GitPython!=3.1.29,>=1.0.0->wandb->coati==1.0.0)\n",
            "  Downloading smmap-5.0.0-py3-none-any.whl (24 kB)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py<3.0.0,>=2.2.0->rich->colossalai>=0.2.4->coati==1.0.0) (0.1.2)\n",
            "Requirement already satisfied: bcrypt>=3.2 in /usr/local/lib/python3.10/dist-packages (from paramiko>=2.4->fabric->colossalai>=0.2.4->coati==1.0.0) (4.0.1)\n",
            "Requirement already satisfied: cryptography>=3.3 in /usr/local/lib/python3.10/dist-packages (from paramiko>=2.4->fabric->colossalai>=0.2.4->coati==1.0.0) (40.0.2)\n",
            "Requirement already satisfied: pynacl>=1.5 in /usr/local/lib/python3.10/dist-packages (from paramiko>=2.4->fabric->colossalai>=0.2.4->coati==1.0.0) (1.5.0)\n",
            "Collecting mypy-extensions>=0.3.0 (from typing-inspect>=0.4.0->dataclasses-json<0.6.0,>=0.5.7->langchain->coati==1.0.0)\n",
            "  Downloading mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)\n",
            "Requirement already satisfied: distlib<1,>=0.3.6 in /usr/local/lib/python3.10/dist-packages (from virtualenv>=20.10.0->pre-commit->colossalai>=0.2.4->coati==1.0.0) (0.3.6)\n",
            "Requirement already satisfied: platformdirs<4,>=3.2 in /usr/local/lib/python3.10/dist-packages (from virtualenv>=20.10.0->pre-commit->colossalai>=0.2.4->coati==1.0.0) (3.3.0)\n",
            "Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.10/dist-packages (from cryptography>=3.3->paramiko>=2.4->fabric->colossalai>=0.2.4->coati==1.0.0) (1.15.1)\n",
            "Requirement already satisfied: pycparser in /usr/local/lib/python3.10/dist-packages (from cffi>=1.12->cryptography>=3.3->paramiko>=2.4->fabric->colossalai>=0.2.4->coati==1.0.0) (2.21)\n",
            "Building wheels for collected packages: coati, gpustat, pathtools\n",
            "  Building wheel for coati (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for coati: filename=coati-1.0.0-py3-none-any.whl size=75334 sha256=13d5d09bdbb4b2b47045312ca337e68b88dfd2044087fbaf0c05bd593c1b4a35\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-tez1zypt/wheels/49/ba/eb/98b39707d3bcca1d3ecf646b531cdb25f480bd44ec5c0edafb\n",
            "  Building wheel for gpustat (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for gpustat: filename=gpustat-1.1-py3-none-any.whl size=26280 sha256=a748ad4da7967293e3504c25de991a95a408f0d9a9caf2a91b7d4e09b0ee58fd\n",
            "  Stored in directory: /root/.cache/pip/wheels/ee/d0/2c/1e02440645c2318ba03aea99993a44a9108dc8f74de0bd370b\n",
            "  Building wheel for pathtools (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pathtools: filename=pathtools-0.1.2-py3-none-any.whl size=8791 sha256=48ee994acbcb9e407204168f303fd9c252c0f7f64f4948b689f062e60891e7e3\n",
            "  Stored in directory: /root/.cache/pip/wheels/e7/f3/22/152153d6eb222ee7a56ff8617d80ee5207207a8c00a7aab794\n",
            "Successfully built coati gpustat pathtools\n",
            "Installing collected packages: sentencepiece, pathtools, nvidia-ml-py, xxhash, smmap, setproctitle, sentry-sdk, nvidia-cuda-runtime-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cublas-cu11, mypy-extensions, multidict, marshmallow, loralib, frozenlist, docker-pycreds, dill, blessed, async-timeout, yarl, typing-inspect, starlette, responses, openapi-schema-pydantic, nvidia-cudnn-cu11, multiprocess, marshmallow-enum, gpustat, gitdb, aiosignal, torch, sse_starlette, GitPython, fastapi, dataclasses-json, aiohttp, wandb, langchain, datasets, coati\n",
            "  Attempting uninstall: torch\n",
            "    Found existing installation: torch 2.0.1+cu118\n",
            "    Uninstalling torch-2.0.1+cu118:\n",
            "      Successfully uninstalled torch-2.0.1+cu118\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "torchaudio 2.0.2+cu118 requires torch==2.0.1, but you have torch 1.13.1 which is incompatible.\n",
            "torchdata 0.6.1 requires torch==2.0.1, but you have torch 1.13.1 which is incompatible.\n",
            "torchtext 0.15.2 requires torch==2.0.1, but you have torch 1.13.1 which is incompatible.\n",
            "torchvision 0.15.2+cu118 requires torch==2.0.1, but you have torch 1.13.1 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0mSuccessfully installed GitPython-3.1.31 aiohttp-3.8.4 aiosignal-1.3.1 async-timeout-4.0.2 blessed-1.20.0 coati-1.0.0 dataclasses-json-0.5.7 datasets-2.12.0 dill-0.3.6 docker-pycreds-0.4.0 fastapi-0.95.2 frozenlist-1.3.3 gitdb-4.0.10 gpustat-1.1 langchain-0.0.178 loralib-0.1.1 marshmallow-3.19.0 marshmallow-enum-1.5.1 multidict-6.0.4 multiprocess-0.70.14 mypy-extensions-1.0.0 nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-8.5.0.96 nvidia-ml-py-11.525.112 openapi-schema-pydantic-1.2.4 pathtools-0.1.2 responses-0.18.0 sentencepiece-0.1.99 sentry-sdk-1.24.0 setproctitle-1.3.2 smmap-5.0.0 sse_starlette-1.6.1 starlette-0.27.0 torch-1.13.1 typing-inspect-0.8.0 wandb-0.15.3 xxhash-3.2.0 yarl-1.9.2\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 预训练模型的下载（以bloom为例）"
      ],
      "metadata": {
        "id": "qbtHtV4pHXQ4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "os.chdir('/content/ColossalAI/applications/Chat/examples')"
      ],
      "metadata": {
        "id": "WxRdvkAvhlJW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "如果不安装下面的命令会出现什么情况呢？\n",
        "- 从huggingface中git clone下来的模型看似下载下来了，但是其实下载下来的并不是实质的模型文件（如果你检查文件的大小，只有几B）\n",
        "- 一旦下载下来的文件并不是实质的模型，则在运行SFT代码的时候会报错：**safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge**"
      ],
      "metadata": {
        "id": "GdlFgQzmHo2t"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!sudo apt-get install git-lfs\n",
        "!git lfs install"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "C4Pu4RmJiEK6",
        "outputId": "e49226b3-fd13-4f48-9f41-43f23fa017d8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Reading package lists... Done\n",
            "Building dependency tree       \n",
            "Reading state information... Done\n",
            "git-lfs is already the newest version (2.9.2-1).\n",
            "0 upgraded, 0 newly installed, 0 to remove and 24 not upgraded.\n",
            "Updated git hooks.\n",
            "Git LFS initialized.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**下载ColossalAI支持的系列模型**，我们以bloomz-560m为例。在下面，我们将模型放在了当前的目录下（ColossalAI/applications/Chat/examples），但是这里并不是强制的，可以根据自己的喜欢变换位置。"
      ],
      "metadata": {
        "id": "9JeE6R7QH4dJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!git clone https://huggingface.co/bigscience/bloomz-560m"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3xUK88K7iydi",
        "outputId": "90626fc0-dd6c-46d9-d68b-08289ae034f6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'bloomz-560m'...\n",
            "remote: Enumerating objects: 1332, done.\u001b[K\n",
            "remote: Counting objects: 100% (10/10), done.\u001b[K\n",
            "remote: Compressing objects: 100% (7/7), done.\u001b[K\n",
            "remote: Total 1332 (delta 3), reused 10 (delta 3), pack-reused 1322\u001b[K\n",
            "Receiving objects: 100% (1332/1332), 7.18 MiB | 22.90 MiB/s, done.\n",
            "Resolving deltas: 100% (616/616), done.\n",
            "Filtering content: 100% (8/8), 2.11 GiB | 57.58 MiB/s, done.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 运行SFT"
      ],
      "metadata": {
        "id": "wnZ6xjQfIEpL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "我们这里是直接运行的py文件，如果你按照[文档的说明](https://github.com/hpcaitech/ColossalAI/tree/main/applications/Chat/examples)，运行sh脚本文件（!bash train_sft.sh）也是可以的。其本质上，都是运行这个train_sft.py文件。\n",
        "\n",
        "**训练完成的模型存放地址（save_path）**这里写的示例路径是临时路径，一旦notebook停止运行，这个路径就失效了。所以要么在notebook停止前先下载训练好的模型，要么这里可以考虑填写到google drive的目录下。\n",
        "\n",
        "在下面的命令中，为了演示\n",
        "- 我们只是用一个非常非常非常小的数据集（--dataset）去跑程序（小到就只有5条数据）\n",
        "- model：改为了“bloom”\n",
        "- pretrain：改成了我们自己下载的模型地址\n",
        "- save_path: 改成了我们想放的目录地址\n",
        "- 其他参数可能需要你自己去多多探索：比如lora、gradient checkingpoint等。更多参数的说明见[官方文档](https://github.com/hpcaitech/ColossalAI/tree/main/applications/Chat/examples)"
      ],
      "metadata": {
        "id": "9w6QPTmIIr8J"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!torchrun --standalone --nproc_per_node=1 train_sft.py \\\n",
        "    --pretrain \"./bloomz-560m\" \\\n",
        "    --model 'bloom' \\\n",
        "    --strategy colossalai_zero2 \\\n",
        "    --log_interval 50 \\\n",
        "    --save_path  \"./bloomz-560m-finetuned\" \\\n",
        "    --dataset \"./instinwild_ch_small.json\" \\\n",
        "    --batch_size 4 \\\n",
        "    --accumulation_steps 8 \\\n",
        "    --lr 2e-5 \\\n",
        "    --max_datasets_size 512 \\\n",
        "    --max_epochs 1"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WVu629ZLjf6G",
        "outputId": "6fd1ef2f-6483-42a4-b43a-65f9181bac2c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2023-05-24 11:46:01.994320: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
            "\u001b[2;36m[05/24/23 11:46:05]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m colossalai - colossalai - INFO:                    \n",
            "\u001b[2;36m                    \u001b[0m         \u001b[35m/usr/local/lib/python3.10/dist-packages/colossalai/\u001b[0m\n",
            "\u001b[2;36m                    \u001b[0m         \u001b[35mcontext/\u001b[0m\u001b[95mparallel_context.py\u001b[0m:\u001b[1;36m522\u001b[0m set_device         \n",
            "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m colossalai - colossalai - INFO: process rank \u001b[1;36m0\u001b[0m is  \n",
            "\u001b[2;36m                    \u001b[0m         bound to device \u001b[1;36m0\u001b[0m                                  \n",
            "\u001b[2;36m[05/24/23 11:46:13]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m colossalai - colossalai - INFO:                    \n",
            "\u001b[2;36m                    \u001b[0m         \u001b[35m/usr/local/lib/python3.10/dist-packages/colossalai/\u001b[0m\n",
            "\u001b[2;36m                    \u001b[0m         \u001b[35mcontext/\u001b[0m\u001b[95mparallel_context.py\u001b[0m:\u001b[1;36m558\u001b[0m set_seed           \n",
            "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m colossalai - colossalai - INFO: initialized seed on\n",
            "\u001b[2;36m                    \u001b[0m         rank \u001b[1;36m0\u001b[0m, numpy: \u001b[1;36m42\u001b[0m, python random: \u001b[1;36m42\u001b[0m,              \n",
            "\u001b[2;36m                    \u001b[0m         ParallelMode.DATA: \u001b[1;36m42\u001b[0m, ParallelMode.TENSOR: \u001b[1;36m42\u001b[0m,the \n",
            "\u001b[2;36m                    \u001b[0m         default parallel seed is ParallelMode.DATA.        \n",
            "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m colossalai - colossalai - INFO:                    \n",
            "\u001b[2;36m                    \u001b[0m         \u001b[35m/usr/local/lib/python3.10/dist-packages/colossalai/\u001b[0m\n",
            "\u001b[2;36m                    \u001b[0m         \u001b[95minitialize.py\u001b[0m:\u001b[1;36m115\u001b[0m launch                           \n",
            "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m colossalai - colossalai - INFO: Distributed        \n",
            "\u001b[2;36m                    \u001b[0m         environment is initialized, data parallel size: \u001b[1;36m1\u001b[0m, \n",
            "\u001b[2;36m                    \u001b[0m         pipeline parallel size: \u001b[1;36m1\u001b[0m, tensor parallel size: \u001b[1;36m1\u001b[0m \n",
            "/usr/local/lib/python3.10/dist-packages/colossalai/kernel/op_builder/utils.py:94: UserWarning: [extension] The CUDA version on the system (11.8) does not match with the version (11.7) torch was compiled with. The mismatch is found in the minor version. As the APIs are compatible, we will allow compilation to proceed. If you encounter any issue when using the built kernel, please try to build it again with fully matched CUDA versions\n",
            "  warnings.warn(\n",
            "/usr/local/lib/python3.10/dist-packages/colossalai/kernel/op_builder/utils.py:94: UserWarning: [extension] The CUDA version on the system (11.8) does not match with the version (11.7) torch was compiled with. The mismatch is found in the minor version. As the APIs are compatible, we will allow compilation to proceed. If you encounter any issue when using the built kernel, please try to build it again with fully matched CUDA versions\n",
            "  warnings.warn(\n",
            "\u001b[2;36m[05/24/23 11:52:04]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m colossalai - colossalai - INFO:                    \n",
            "\u001b[2;36m                    \u001b[0m         \u001b[35m/usr/local/lib/python3.10/dist-packages/coati/datas\u001b[0m\n",
            "\u001b[2;36m                    \u001b[0m         \u001b[35met/\u001b[0m\u001b[95msft_dataset.py\u001b[0m:\u001b[1;36m121\u001b[0m __init__                     \n",
            "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m colossalai - colossalai - INFO: Loading data\u001b[33m...\u001b[0m    \n",
            "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m colossalai - colossalai - INFO:                    \n",
            "\u001b[2;36m                    \u001b[0m         \u001b[35m/usr/local/lib/python3.10/dist-packages/coati/datas\u001b[0m\n",
            "\u001b[2;36m                    \u001b[0m         \u001b[35met/\u001b[0m\u001b[95msft_dataset.py\u001b[0m:\u001b[1;36m123\u001b[0m __init__                     \n",
            "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m colossalai - colossalai - INFO: Loaded \u001b[1;36m6\u001b[0m examples. \n",
            "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m colossalai - colossalai - INFO:                    \n",
            "\u001b[2;36m                    \u001b[0m         \u001b[35m/usr/local/lib/python3.10/dist-packages/coati/datas\u001b[0m\n",
            "\u001b[2;36m                    \u001b[0m         \u001b[35met/\u001b[0m\u001b[95msft_dataset.py\u001b[0m:\u001b[1;36m126\u001b[0m __init__                     \n",
            "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m colossalai - colossalai - INFO: Limiting dataset to\n",
            "\u001b[2;36m                    \u001b[0m         \u001b[1;36m512\u001b[0m examples.                                      \n",
            "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m colossalai - colossalai - INFO:                    \n",
            "\u001b[2;36m                    \u001b[0m         \u001b[35m/usr/local/lib/python3.10/dist-packages/coati/datas\u001b[0m\n",
            "\u001b[2;36m                    \u001b[0m         \u001b[35met/\u001b[0m\u001b[95msft_dataset.py\u001b[0m:\u001b[1;36m129\u001b[0m __init__                     \n",
            "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m colossalai - colossalai - INFO: Formatting         \n",
            "\u001b[2;36m                    \u001b[0m         inputs\u001b[33m...\u001b[0m                                          \n",
            "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m colossalai - colossalai - INFO:                    \n",
            "\u001b[2;36m                    \u001b[0m         \u001b[35m/usr/local/lib/python3.10/dist-packages/coati/datas\u001b[0m\n",
            "\u001b[2;36m                    \u001b[0m         \u001b[35met/\u001b[0m\u001b[95msft_dataset.py\u001b[0m:\u001b[1;36m137\u001b[0m __init__                     \n",
            "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m colossalai - colossalai - INFO: Tokenizing         \n",
            "\u001b[2;36m                    \u001b[0m         inputs\u001b[33m...\u001b[0m This may take some time\u001b[33m...\u001b[0m               \n",
            "steps: 0it [00:00, ?it/s]\u001b[2;36m[05/24/23 11:52:09]\u001b[0m\u001b[2;36m \u001b[0m\u001b[31mWARNING \u001b[0m colossalai - colossalai - WARNING:                 \n",
            "\u001b[2;36m                    \u001b[0m         \u001b[35m/usr/local/lib/python3.10/dist-packages/coati/train\u001b[0m\n",
            "\u001b[2;36m                    \u001b[0m         \u001b[35mer/\u001b[0m\u001b[95msft.py\u001b[0m:\u001b[1;36m86\u001b[0m fit                                   \n",
            "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[31mWARNING \u001b[0m colossalai - colossalai - WARNING: batch_i\u001b[1;92md:0\u001b[0m,     \n",
            "\u001b[2;36m                    \u001b[0m         abnormal loss: \u001b[1;36m3.6484375\u001b[0m                           \n",
            "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[31mWARNING \u001b[0m colossalai - colossalai - WARNING:                 \n",
            "\u001b[2;36m                    \u001b[0m         \u001b[35m/usr/local/lib/python3.10/dist-packages/coati/train\u001b[0m\n",
            "\u001b[2;36m                    \u001b[0m         \u001b[35mer/\u001b[0m\u001b[95msft.py\u001b[0m:\u001b[1;36m86\u001b[0m fit                                   \n",
            "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[31mWARNING \u001b[0m colossalai - colossalai - WARNING: batch_i\u001b[1;92md:1\u001b[0m,     \n",
            "\u001b[2;36m                    \u001b[0m         abnormal loss: \u001b[1;36m4.109375\u001b[0m                            \n",
            "steps: 0it [00:03, ?it/s]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 小结\n",
        "仅仅跑起来只是一个开始。祝愿每个小伙伴最终都会获得更好的GPU资源、更棒的数据和更出色的属于自己的语言模型！"
      ],
      "metadata": {
        "id": "RQKDNZogLUNs"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "H7HM6E5VLW2u"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}