CUDA/Windows instructions do not work; there is no setup.py

[andymartin]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [X ] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• [ X] I carefully followed the README.md.
• [ X] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• [ X] I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

I expect CUDA support to work, or not to be claimed to work.

Current Behavior

CUDA does not work.

Environment and Context

Win11 x64, followed all instructions. Still runs 100% CPU.

• Physical (or virtual) hardware you are using, e.g. for Linux:

$ lscpu

• Operating System, e.g. for Linux:

$ uname -a

• SDK version, e.g. for Linux:

$ python3 --version
$ make --version
$ g++ --version

Failure Information (for bugs)

Your instructions are wrong.

Steps to Reproduce

There is no setup.py.

Try the following:

1. git clone https://github.com/abetlen/llama-cpp-python
2. cd llama-cpp-python
3. rm -rf _skbuild/ # delete any old builds
4. python setup.py develop
5. cd ./vendor/llama.cpp
6. Follow llama.cpp's instructions to cmake llama.cpp
7. Run llama.cpp's ./main with the same arguments you previously passed to llama-cpp-python and see if you can reproduce the issue. If you can, log an issue with llama.cpp

Failure Logs

There is no setup.py. Nothing else matters because your instructions don't even match the reality of which files exist.

[tk-master]

ok first, have you actually tried building the project with commands suggested in llama-cpp-python readme? because it seems you are trying to build straight to vendor llama.cpp.

Try the following commands on windows (cmd):

git clone --recurse-submodules https://github.com/abetlen/llama-cpp-python.git llama-cpp-python
cd lama-cpp-python

set FORCE_CMAKE=1 && set CMAKE_ARGS=-DLLAMA_CUBLAS=on
python -m pip install -e . --force-reinstall --no-cache-dir

Note: this requires cuda installed and vs studio with cmake

Secondly, what does your script look like? you need to set gpu_layers param to use the gpu.