A live version of my personal knowledge graph is available online.
Twice a day, a dedicated github workflow extracts:
-
GitHub stars
-
Twitter likes
-
HackerNews upvotes
-
Zotero records
database/database.json, while the knowledge graph of topics is saved in the file database/triples.json. Additionally, the workflow generates a search engine and saves it as database/retriever.pkl. Finally, the updated state of the application is pushed to the cloud provider Fly.io and the dedicated GitHub page is updated accordingly. The cost of hosting the application is under 8$ per month. It may increase if a large number of users query your bookmarks 24/7. Costs can be bounded via Fly.io and OpenAI dashboards.
FLY_API_TOKEN and OPENAI_API_KEY. If you do not plan to use ChatGPT, you can leave OPENAI_API_KEY empty. It is important to set secrets as repository secrets and not as environment secrets.
To extract likes, we will need a Twitter API TOKEN, available on the Developer Portal after having creater an account.
TWITTER_TOKENWe will need a "group library" to index content from Zotero. The ZOTERO_API_KEY is available at https://www.zotero.org/settings/keys.
ZOTERO_API_KEYThe ZOTERO_LIBRARY_ID can be found by opening the group's page https://www.zotero.org/groups/groupname, and hovering over the group settings link. The ID is the integer after /groups/.
ZOTERO_LIBRARY_IDWe will need to create secrets for both Hackernews username and password.
HACKERNEWS_USERNAMEHACKERNEWS_PASSWORDOpenAI API is used to call ChatGPT when pressing the button "ask" to re-rank documents based on our query. If we do not plan to use ChatGPT, we will need to set the secret OPENAI_API_KEY with an empty value. We can get our OpenAI key here.
OPENAI_API_KEYThe prompt to ChatGPT is stored in the api/api.py file.
We will need to install the flyctl client available here to set the FLY_API_TOKEN. The fly.io api token enables the github action to automatically push the updated state of the api. We can get the token using the command line:
flyctl auth signupfly auth loginflyctl auth tokenFLY_API_TOKENAfter finalizing the secrets, we can specify the Github and Twitter users whose liked content we wish to extract. To achieve this, we'll need to modify the sources.yml file located at the root of the repository. We'll be able to handpick the Github stars we want to index and set the Twitter ID and handle of the users whose content we want to include. To obtain the Twitter ID, we can use a tool like tweeterid.com."
github:
- "raphaelsty"
- "gbolmier"
- "MaxHalford"
- "AdilZouitine"
twitter:
- [1262679654239961088, "raphaelsrty"]Once secrets and sources are set. We will deploy the API following the Fly.io documentation. You won't need any database. fly client should generate a fly.toml file that looks like the toml file below where app_name is the name of our api.
app = "app_name"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []
[env]
[experimental]
auto_rollback = true
[[services]]
http_checks = []
internal_port = 8080
processes = ["app"]
protocol = "tcp"
script_checks = []
[services.concurrency]
hard_limit = 6
soft_limit = 3
type = "connections"
[[services.ports]]
force_https = true
handlers = ["http"]
port = 80
[[services.ports]]
handlers = ["tls", "http"]
port = 443
[[services.tcp_checks]]
grace_period = "1s"
interval = "15s"
restart_limit = 0
timeout = "2s"
⚠️ After having created our API, we will need to update the urls called by the web app in the filedocs/index.html. There are 3 urls to replace:https://knowledge.fly.devperhttps://app_name.fly.devwhereapp_nameis your API name.
We will need to set the Github Page from the repository configurations (fork).
⚠️ After creating your github page, you will have to modify theoriginsfield of theapi/api.pyfile:
origins = [
"https://raphaelsty.github.io", # Put your own github page name here.
]
⚠️ To avoid any financial incident, remember to define ahard_limitand asoft_limitwhich will bounder the number of instance Fly.io will deploy to answer to peak demands and therefore limit the costs. Those parameters are available in thefly.tomlfile.
[services.concurrency]
hard_limit = 6
soft_limit = 3
type = "connections"
⚠️ Setting a 2GB memory VM with a single shared cpu on FLy.io will do the job for the app.
⚠️ Don't forget to define the limit amount you want to spend on OpenAI platform (10$ here).
To run the API locally using Docker, we can export the OPENAI_API_KEY to our environment variables using:
export OPENAI_API_KEY="sk-..."
Then, we can run make launch at the root of the repository.
make launch
We can also deploy the API manually using:
fly deploy \
--build-secret OPENAI_API_KEY=$OPENAI_API_KEY
My personal Knowledge Base is inspired and extract resources from the Knowledge Base of François-Paul Servant namely Semanlink.
GNU GENERAL PUBLIC LICENSE Knowledge Copyright (C) 2023 Raphaël Sourty