Skip to content

DockerCompose.md

Latest commit

 

History

History
51 lines (36 loc) · 1.69 KB

DockerCompose.md

File metadata and controls

51 lines (36 loc) · 1.69 KB

Run with Docker Compose

You can choose to run with Docker or Docker Compose. Both are not needed.

Docker Compose Prerequisites

Environment Setup (Optional)

Assumptions will be made on all of these values if you choose to skip this step. Create a .env file if one does not exist and modify it to your needs. Here is an example .env file:

LOCAL_LLM_API_KEY=
THREADS=10
GPU_LAYERS=0
MAIN_GPU=0
  • LOCAL_LLM_API_KEY - The API key to use for the server. If not set, the server will not require an API key.
  • THREADS - The number of threads to use. Default is your CPU core count minus 1.

The following are only applicable to NVIDIA GPUs:

  • GPU_LAYERS - The number of layers to use on the GPU. Default is 0.
  • MAIN_GPU - The GPU to use for the main model. Default is 0.

Make sure to move your .env file to the Local-LLM directory if you set one up.

Run with Docker Compose (Without NVIDIA GPU)

git clone https://github.com/Josh-XT/Local-LLM
cd Local-LLM
docker-compose pull
docker-compose up

Run with Docker Compose (With NVIDIA GPU support)

You must have the NVIDIA Container Toolkit installed if using NVIDIA GPU.

git clone https://github.com/Josh-XT/Local-LLM
cd Local-LLM
docker-compose -f docker-compose-cuda.yml pull
docker-compose -f docker-compose-cuda.yml up