-
Notifications
You must be signed in to change notification settings - Fork 177
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Roadmap #77
Comments
This sounds an interesting project, I like the idea of decentralized computing a bit like cryptocurrencies does, but not for currency, rather for general computing, because computing can be quite expensive. About the roadmap, I see exciting technical details, but I don't see how people will see and find themselves sharing their resources for a common goal? Example: Bob and Alice want to train a GPT3 200B parameters, but Bob can only afford half the training price, same for Alice, but with this awesome UI, they could see that they match into a common goal. |
Hi @louis030195!
Yeah, there are a couple of projects related to this idea: vast.ai provides a service for users to lease/rent each other's GPUs, and BitTensor (cc @unconst) is built around a cryptocurrency serving as an incentive for people who help train models with their GPUs. Currently, hivemind doesn't involve any financial incentives: we assume that volunteers are motivated by having access to the training outcome and recognition in the leaderboard. However, if time shows that the financial motivation is crucial, hivemind may serve as a backend for BitTensor nodes :)
For now, we assume this happens like this:
An example of such a page is our demo where we train a DALL-E-like model. However, I definitely agree that our project will benefit from a centralized UI where a new user can see all planned/ongoing training runs and join the ones they consider interesting :) |
This is a global project roadmap that states our priorities for the nearest future. These priorities can and should be disputed here or elsewhere, after which we will update the roadmap.
v0.7 "It runs something" (released)
v0.8 "It runs at scale" (released)
v0.9 "It trains something" (released)
v0.10 "You can train with us" (released)
tutorial on defining and training custom experts(postponed)Elastic scaling of moshpit averaging with the number of active trainers(found workaround for now)v1.0 "most of the code makes sense without reading the source" (nov-dec)
v1.1 "You can set up collaborative training easily"
Target scenario: 100 volunteers training 2xl-like over the internet
1.2 Decentralized Model-parallelism
Target scenario: 500 peers training 1B+ over the internet
Important, but not urgent
The text was updated successfully, but these errors were encountered: