Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

本地部署疑问 #81

Open
1NormalGuy opened this issue Feb 4, 2025 · 3 comments
Open

本地部署疑问 #81

1NormalGuy opened this issue Feb 4, 2025 · 3 comments
Labels
model

Comments

@1NormalGuy
Copy link

根据教程在window11 RTX 3060和macos M1下均无法实现本地部署，请问有更详细的本地部署教程吗？
@LindiaC
Copy link

LindiaC commented Feb 9, 2025

+1, vllm在windows系统下会报错，还有没有别的替代方案？

@Hwwwww-dev
Copy link

Hwwwww-dev commented Feb 12, 2025 •
edited
Loading

根据教程在window11 RTX 3060和macos M1下均无法实现本地部署，请问有更详细的本地部署教程吗？

我直接 clone vllm 项目，然后起 vllm server 是运行起来了，无奈 M1 Pro 推理太慢了...

#64

@ycjcl868 ycjcl868 added the model label Feb 12, 2025
@AHEADer
Copy link

AHEADer commented Feb 13, 2025

我们实际发现ollama运行qwen2-vl模型会有明显的模型性能损失，而且ollama对bf16的支持也有限；vLLM的话可能更推荐使用CUDA+Linux的环境，如果有对应的推理问题，也可以去ollama和vLLM的仓库提issues。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
model
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@AHEADer @ycjcl868 @Hwwwww-dev @LindiaC @1NormalGuy