[ENH] add example of forking using github repos #4413
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1 task
This stack of pull requests is managed by Graphite. Learn more about stacking. |
Reviewer ChecklistPlease leverage this checklist to ensure your code review is thorough before approving Testing, Bugs, Errors, Logs, Documentation
System Compatibility
Quality
|
jairad26
force-pushed
the
jai/fork-notebook
branch
3 times, most recently
from
4142605 to
1ab410d
Compare
jairad26
force-pushed
the
graphite-base/4413
branch
from
a5eb712 to
dd6d176
Compare
jairad26
force-pushed
the
jai/fork-notebook
branch
from
1ab410d to
b0e30ac
Compare
jairad26
force-pushed
the
jai/fork-notebook
branch
2 times, most recently
from
be5bb4d to
64bbb43
Compare
jairad26
force-pushed
the
jai/fork-notebook
branch
from
64bbb43 to
c54e0ca
Compare
HammadB
approved these changes
May 6, 2025
HammadB
reviewed
May 6, 2025
| "metadata": {}, | ||
| "source": [ | ||
| "## Forking\n", | ||
| "ChromaDB now supports forking. Below is an example using forking to chunk and embed a github repo, fork off of the collection for a new github branch, and apply diffs to the new branch." |
There was a problem hiding this comment.
Also worth mentioning this is only supported in Distributed Chroma / Chroma Cloud.
HammadB
reviewed
May 6, 2025
| "source": [ | ||
| "\n", | ||
| "PY_LANGUAGE = get_language(\"python\")\n", | ||
| "REPO_OWNER = \"jairad26\"\n", |
There was a problem hiding this comment.
bit strange to have your personal repo here
HammadB
reviewed
May 6, 2025
| "REPO_OWNER = \"jairad26\"\n", | ||
| "REPO_NAME = \"Django-WebApp\"\n", | ||
| "EXISTING_BRANCH = \"main\"\n", | ||
| "NEW_BRANCH = \"test1\"\n", |
There was a problem hiding this comment.
could we call this demo etc instead?
HammadB
reviewed
May 6, 2025
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### Chunker and Github Helpers\n", |
There was a problem hiding this comment.
Can we collapse / hide this region by default?
HammadB
reviewed
May 6, 2025
| " \"file_path\": file_path,\n", | ||
| " \"size\": chunk[\"size\"],\n", | ||
| " # \"node_types\": str([n[\"type\"] for n in chunk[\"nodes\"]]),\n", | ||
| " # \"node_names\": str([n[\"name\"] for n in chunk[\"nodes\"] if n[\"name\"]]),\n", |
There was a problem hiding this comment.
there is a bunch of commented out code here?
HammadB
reviewed
May 6, 2025
| "name": "stdout", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| " metadata document distance\n", |
There was a problem hiding this comment.
Can we make this output more legible?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes
This PR adds an example notebook to show forking usage with github repos and branching. It creates a custom chunker using tree sitter to generate embeddable chunks of code, and a github helper to pull file info, track diffs between branches. it then uses chroma cloud to index and adds a github repo into a collection, and based on the branch provided creates a fork of the collection, calculates the diff between the original branch and the new branch, deletes chunks from the new branch that correspond to the diffed files, and rechunks and reinserts them.
Test plan
How are these changes tested?
pytestfor python,yarn testfor js,cargo testfor rustDocumentation Changes
Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs section?