|
| 1 | +# TAGLINE |
| 2 | + |
| 3 | +deploy open-source LLMs into your own AWS or GCP account |
| 4 | + |
| 5 | +# TLDR |
| 6 | + |
| 7 | +**Check** that cloud credentials and compute quota are usable |
| 8 | + |
| 9 | +```veloxml check``` |
| 10 | + |
| 11 | +**Scaffold** a FastAPI model service in the current directory |
| 12 | + |
| 13 | +```veloxml init [my-model]``` |
| 14 | + |
| 15 | +Same, requesting a **GPU** accelerator |
| 16 | + |
| 17 | +```veloxml init [llm-service] --gpu [T4:1]``` |
| 18 | + |
| 19 | +**Deploy** the current project (reads `veloxml.yaml` and `app.py`) |
| 20 | + |
| 21 | +```veloxml deploy``` |
| 22 | + |
| 23 | +Validate the generated SkyPilot spec **without launching** instances |
| 24 | + |
| 25 | +```veloxml deploy --dry-run``` |
| 26 | + |
| 27 | +Deploy to **GCP** instead of the default AWS `us-east-1` |
| 28 | + |
| 29 | +```veloxml deploy --cloud [gcp] --region [us-central1]``` |
| 30 | + |
| 31 | +Override the **GPU** for this deploy only |
| 32 | + |
| 33 | +```veloxml deploy --gpu [A10G:1] --verbose``` |
| 34 | + |
| 35 | +List **active services** in the cloud account |
| 36 | + |
| 37 | +```veloxml status``` |
| 38 | + |
| 39 | +**Tail** logs for the service named in `veloxml.yaml` |
| 40 | + |
| 41 | +```veloxml logs --follow``` |
| 42 | + |
| 43 | +Show the last **N** lines for a named service |
| 44 | + |
| 45 | +```veloxml logs [my-model] --tail [50]``` |
| 46 | + |
| 47 | +Tear down **this** service |
| 48 | + |
| 49 | +```veloxml down``` |
| 50 | + |
| 51 | +Tear down the service **and** every leftover SkyPilot cluster |
| 52 | + |
| 53 | +```veloxml down --all``` |
| 54 | + |
| 55 | +Print the **CLI version** |
| 56 | + |
| 57 | +```veloxml version``` |
| 58 | + |
| 59 | +# SYNOPSIS |
| 60 | + |
| 61 | +**veloxml** **init** [_name_] [**-g** | **--gpu** _ACCEL_] |
| 62 | + |
| 63 | +**veloxml** **check** |
| 64 | + |
| 65 | +**veloxml** **deploy** [**--dry-run**] [**--cloud** _aws_|_gcp_] [**--region** _REGION_] [**-g** | **--gpu** _ACCEL_] [**-v** | **--verbose**] |
| 66 | + |
| 67 | +**veloxml** **status** |
| 68 | + |
| 69 | +**veloxml** **logs** [_name_] [**-t** | **--tail** _N_] [**-f** | **--follow**] |
| 70 | + |
| 71 | +**veloxml** **down** [**-n** | **--name** _name_] [**-a** | **--all**] |
| 72 | + |
| 73 | +**veloxml** **version** |
| 74 | + |
| 75 | +# COMMANDS |
| 76 | + |
| 77 | +**init** [_name_] |
| 78 | +> Write a starter FastAPI `app.py` (health + echo `/predict`), `requirements.txt`, and `veloxml.yaml` in the current directory. _name_ defaults to **my-model-service**. Existing `app.py` / `requirements.txt` are left untouched. |
| 79 | +
|
| 80 | +**check** |
| 81 | +> Run SkyPilot's cloud probe (`sky check`) and report whether AWS credentials and EC2 quota look usable. Fails with a hint to run `aws configure` when nothing is found. |
| 82 | +
|
| 83 | +**deploy** |
| 84 | +> Compile `.veloxml/<name>.sky.yaml`, then `sky serve up` (or `sky serve update` if the service already exists). Polls until the replica `/health` probe returns HTTP 200 (up to 10 minutes) and prints a sample `curl` against `/predict`. Requires `veloxml.yaml` or `app.py` in the current directory. |
| 85 | +
|
| 86 | +**status** |
| 87 | +> Parse `sky serve status` and print a table of VeloxML services (name, version, uptime, status, replicas, endpoint). |
| 88 | +
|
| 89 | +**logs** [_name_] |
| 90 | +> Print application logs via `sky serve logs`. _name_ defaults to the `name` field in `veloxml.yaml`. |
| 91 | +
|
| 92 | +**down** |
| 93 | +> Stop the named service with `sky serve down`. With **--all**, also run `sky down -a` to destroy leftover controller nodes and clusters. |
| 94 | +
|
| 95 | +**version** |
| 96 | +> Print `VeloxML CLI version <x.y.z>`. |
| 97 | +
|
| 98 | +# PARAMETERS |
| 99 | + |
| 100 | +**-g**, **--gpu** _ACCEL_ |
| 101 | +> GPU accelerator string understood by SkyPilot, e.g. **T4:1**, **A10G:1**, **L4:1**. On **init** it is stored in `veloxml.yaml`; on **deploy** it overrides the file for that run. |
| 102 | +
|
| 103 | +**--dry-run** |
| 104 | +> **deploy** only. Write the SkyPilot spec and exit without provisioning. Prints a dummy `http://127.0.0.1:8000` endpoint. |
| 105 | +
|
| 106 | +**--cloud** _aws_|_gcp_ |
| 107 | +> **deploy** only. Target cloud. Default **aws**. |
| 108 | +
|
| 109 | +**--region** _REGION_ |
| 110 | +> **deploy** only. Cloud region. Default **us-east-1**. |
| 111 | +
|
| 112 | +**-v**, **--verbose** |
| 113 | +> **deploy** only. Stream SkyPilot's full output and re-raise exceptions instead of collapsing them into a short error box. |
| 114 | +
|
| 115 | +**-t**, **--tail** _N_ |
| 116 | +> **logs** only. Number of trailing lines to print. Default **100**. Ignored with **--follow**. |
| 117 | +
|
| 118 | +**-f**, **--follow** |
| 119 | +> **logs** only. Stream live logs (passes through to `sky serve logs`). |
| 120 | +
|
| 121 | +**-n**, **--name** _name_ |
| 122 | +> **down** only. Service to tear down instead of the name in `veloxml.yaml`. |
| 123 | +
|
| 124 | +**-a**, **--all** |
| 125 | +> **down** only. After stopping the service, purge every SkyPilot cluster (`sky down -a -y`) so leftover controllers stop billing. |
| 126 | +
|
| 127 | +**--help** |
| 128 | +> Typer help for the CLI or a subcommand. Running **veloxml** with no arguments also prints help. |
| 129 | +
|
| 130 | +# DESCRIPTION |
| 131 | + |
| 132 | +**veloxml** is the command-line interface for **VeloxML**, a thin Python wrapper around **SkyPilot Serve**. It deploys a FastAPI (or similar) model service into **your** AWS or GCP account rather than onto a third-party inference host. |
| 133 | + |
| 134 | +The usual loop is **init** (or hand-write `app.py` + `veloxml.yaml`), **check**, **deploy**, then **down** when finished. **deploy** generates a SkyPilot service spec under `.veloxml/`, installs `requirements.txt` plus an optional `runtime.setup` command on the VM, starts `uvicorn app:app --host 0.0.0.0 --port 8000` by default, and waits for `/health`. Re-running **deploy** against an already-running name performs a rolling `sky serve update`. |
| 135 | + |
| 136 | +The PyPI package is **veloxml-deploy**; the console script it installs is **veloxml**. It depends on **skypilot** (the `sky` binary), **truss**, and **typer**. Python **3.10+** is required. The project is Apache-2.0 licensed. |
| 137 | + |
| 138 | +# CONFIGURATION |
| 139 | + |
| 140 | +**veloxml.yaml** |
| 141 | +> Per-project spec in the working directory. Loaded by **deploy**, **logs**, and **down**; written by **init**. Missing keys take the defaults below. |
| 142 | +
|
| 143 | +**name** |
| 144 | +> Service name passed to SkyPilot (default **my-model-service**). |
| 145 | +
|
| 146 | +**compute.cloud** / **compute.region** |
| 147 | +> Cloud and region (defaults **aws**, **us-east-1**). Overridden by **--cloud** / **--region** on **deploy**. |
| 148 | +
|
| 149 | +**compute.accelerator** |
| 150 | +> Optional GPU, e.g. `T4:1`. Set by **--gpu**. |
| 151 | +
|
| 152 | +**compute.cpus** / **compute.memory** |
| 153 | +> Size floors with a trailing `+` (defaults **2+** / **4+**). Values without `+` get it appended when the spec is compiled. |
| 154 | +
|
| 155 | +**compute.use_spot** |
| 156 | +> Request spot instances. Default **false** in the schema; README examples set **true**. |
| 157 | +
|
| 158 | +**service.port** |
| 159 | +> Published port (default **8000**). |
| 160 | +
|
| 161 | +**service.readiness_probe** / **service.predict_path** |
| 162 | +> HTTP paths (defaults **/health** and **/predict**). The readiness probe is given a 240 s initial delay. |
| 163 | +
|
| 164 | +**service.min_replicas** / **service.max_replicas** |
| 165 | +> Replica policy (both default **1**). Scale-to-zero is not the compiled default; set **min_replicas** to **0** if SkyPilot should idle the service. |
| 166 | +
|
| 167 | +**runtime.app_entrypoint** / **runtime.command** |
| 168 | +> ASGI target and the remote start command (defaults **app:app** and `uvicorn app:app --host 0.0.0.0 --port 8000`). |
| 169 | +
|
| 170 | +**runtime.setup** |
| 171 | +> Extra shell run on the VM after `pip install -r requirements.txt` (and a best-effort CPU PyTorch install). |
| 172 | +
|
| 173 | +**.veloxml/<name>.sky.yaml** |
| 174 | +> Generated SkyPilot Serve spec. Do not treat it as the source of truth; it is rewritten on every **deploy**. |
| 175 | +
|
| 176 | +Cloud credentials are **not** stored by VeloxML. Use `aws configure` or `gcloud auth application-default login` so SkyPilot can see them. |
| 177 | + |
| 178 | +# CAVEATS |
| 179 | + |
| 180 | +The repository marks itself **alpha** and warns of breaking changes. Version **0.1.1** is the package version at documentation time. |
| 181 | + |
| 182 | +Deployments run in **your** cloud account and **cost money**. Spot instances can be reclaimed. Always **veloxml down --all** when finished; leftover SkyPilot controllers keep billing after a single-service **down**. |
| 183 | + |
| 184 | +**check** looks for AWS in `sky check` output even though **--cloud gcp** is accepted. GCP still needs application-default credentials for **deploy** to work. |
| 185 | + |
| 186 | +**deploy** must be run from the project directory. Without `veloxml.yaml` or `app.py` it exits 1. |
| 187 | + |
| 188 | +The marketing site talks about scale-to-zero; the compiled replica policy defaults to **min_replicas: 1**. Setting `use_spot: true` is independent of replica count. |
| 189 | + |
| 190 | +**logs --tail** is ignored when **--follow** is set, because follow mode execs `sky serve logs` without capturing output. |
| 191 | + |
| 192 | +The `sky` binary must be on **PATH** (or next to the same Python that installed **veloxml-deploy**). A missing or unconfigured SkyPilot install makes every cloud subcommand fail. |
| 193 | + |
| 194 | +# HISTORY |
| 195 | + |
| 196 | +**VeloxML** is an Apache-2.0 Python CLI by **Pedro Águas Marques** (**paguasmar**). The public GitHub repository **paguasmar/veloxml-deploy** was created on **29 August 2026**. It wraps **SkyPilot** (UC Berkeley) for provisioning and borrows packaging conventions from **Truss**. The advertised goal is one-command LLM endpoints inside the caller's own VPC, without Dockerfiles, Kubernetes YAML, or a hosted inference vendor. |
| 197 | + |
| 198 | +# SEE ALSO |
| 199 | + |
| 200 | +[aws](/man/aws)(1), [gcloud](/man/gcloud)(1), [uvicorn](/man/uvicorn)(1), [pip](/man/pip)(1), [curl](/man/curl)(1), [docker](/man/docker)(1), [flyctl](/man/flyctl)(1) |
| 201 | + |
| 202 | +# RESOURCES |
| 203 | + |
| 204 | +```[Source code](https://github.com/paguasmar/veloxml-deploy)``` |
| 205 | + |
| 206 | +```[Homepage](https://www.veloxml.com/)``` |
| 207 | + |
| 208 | +```[Documentation](https://github.com/paguasmar/veloxml-deploy#getting-started)``` |
| 209 | + |
| 210 | +<!-- verified: 2026-09-09 --> |
0 commit comments