Adding vllm speculative decoding example#317
Conversation
| base_image: | ||
| image: nvcr.io/nvidia/pytorch:23.11-py3 | ||
| python_executable_path: /usr/bin/python3 |
There was a problem hiding this comment.
just cuious why we need this base image? Can you add a cooment?
There was a problem hiding this comment.
Without this base image the build does not succeed. The baseten base image does not have nvcc, which is required for the developer build of vLLM.
| tensor_parallel: 1 | ||
| max_num_seqs: 16 | ||
| model_name: vLLM Speculative Decoding | ||
| python_version: py310 |
There was a problem hiding this comment.
drop python version if using base image
| python_executable_path: /usr/bin/python3 | ||
| build_commands: [] | ||
| environment_variables: | ||
| HF_TOKEN: "" |
There was a problem hiding this comment.
vLLM reads only this specific environment variable for the access token. It doesn't work with secrets
|
|
3 participants
No description provided.