solana-developer-data

Goal: Collecting Solana Developer Data from Github

This repository is an example crawler that searches for Solana related activity on Github and collects them into a postgres db.

The crawler uses the following to find Solana repositories:

Library	Query	Description
@solana/web3.js	solana/web3.js filename:package.json	Solana JS/TS SDK
@solana/web3.js	solana/web3.js filename:package-lock.json	Solana JS/TS SDK
@solana/web3.js	solana/web3.js filename:yarn.lock	Solana JS/TS SDK
serum/anchor	serum/anchor filename:package.json	Anchor JS/TS SDK
solana-program	solana-program filename:Cargo.toml	Solana Rust Program SDK
anchor-lang	anchor-lang filename:Cargo.toml	Anchor Framework Program SDK
Solnet	Solnet.Rpc filename:*.csproj	Solana C# SDK
solana-go	gagliardetto/solana filename:mod.go	Solana Go SDK
solana	"from solana rpc import" language:python	Solana Python SDK

How to Run Locally

Go into the /local folder and run docker compose up to start a docker container with a postgres db

cd local
docker compose up

Get a Github access token and place in the .env file under GITHUB_ACCESS_TOKEN
Run npm install
Run npm run start

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
crawled_repos		crawled_repos
local		local
src		src
.env		.env
LICENSE		LICENSE
README.md		README.md
nodemon.json		nodemon.json
package.json		package.json
server.js		server.js