feat(tcfeed): read GitHub search as well as the subreddit - #114
Merged
Conversation
Adds the API behind https://github.com/search?q=stars:1000&type=repositories&s=updated&o=desc as a second source, merged with the reddit posts and deduplicated - a repository trending on reddit is exactly the kind that also turns up in a search sorted by recent activity, and cloning it twice in one run is the one thing worth avoiding. Asked through gh so it uses the token already on the machine. The HTML page is rate-limited hard for anyone not signed in, and parsing it would be a scraper of a page that changes shape without warning. The default query is `stars:1000` exactly as that URL has it, and it means *exactly* a thousand stars rather than a thousand or more. That reads like a typo and is not one: it is a narrow, oddly effective band - popular enough that somebody is watching, small enough that nobody has audited it yet - and `stars:>=1000` would return a different and far larger set. TCFEED_GH_QUERY overrides it, TCFEED_GH caps how many come back, and TCFEED_GH=0 turns the source off. Either source may now fail on its own. Reddit refuses this address often enough that making it fatal meant a run produced nothing on an afternoon when the search was answering perfectly; now whichever source came back is scanned, a source that failed says why rather than looking empty, and only losing every source that was asked is an error. Archived and forked repositories are dropped at the search rather than left for metadata() to reject one HTTP call later, because the search already knows. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ThreatCrush Security Scan67 finding(s) HIGH/CRITICAL: 11 | MEDIUM: 55 | LOW: 1
…and 17 more. Full results in the Security tab. Snippets are redacted; ThreatCrush never prints matched credential material. |
More than a thousand stars, sorted by most recently pushed, which is the band actually wanted: exactly-1000 was a sliver of a few hundred repositories that emptied out as they gained a star. The qualifier is `stars`, and it is worth being careful about. `starts:>1000` is not an error - it is a free-text search for the word, and it returns repositories with no stars at all, which is a query that looks like it worked. Noted in the header and beside the default. Practical consequence, measured over the first 20 results: 8 exceed the 300MB TOO_BIG_KB cap and are skipped (supabase 2371MB, openclaw 2458MB, metabase 2122MB), leaving 12 scannable. The size check downstream does considerably more work with this query than the feed ever gave it, because a repository with this many stars is often a monorepo. The cap is left where it is: an afternoon spent cloning a gigabyte is the thing it exists to prevent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Default query changed to
|
| Repo | Size |
|---|---|
| openclaw/openclaw | 2458 MB |
| supabase/supabase | 2371 MB |
| metabase/metabase | 2122 MB |
| twentyhq/twenty | 1559 MB |
| frappe/frappe | 793 MB |
| openclaw/Peekaboo | 632 MB |
| lightdash/lightdash | 486 MB |
| polarsource/polar | 461 MB |
12 scannable, 8 skipped. The cap is deliberately left where it is — an afternoon spent cloning a gigabyte is exactly what it exists to prevent — but it means the effective yield of this source is roughly 60% of TCFEED_GH, so set that number with the discard in mind.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the API behind
search?q=stars:1000&type=repositories&s=updated&o=descas a second source, merged with the reddit posts and deduplicated.Deduplicated because a repository trending on reddit is exactly the kind that also turns up in a search sorted by recent activity, and cloning it twice in one run is the one thing worth avoiding here.
stars:1000means exactly a thousandNot a thousand or more. That reads like a typo and is not one — it is the query in the URL, and it is a narrow, oddly effective band: popular enough that somebody is watching, small enough that nobody has audited it yet.
stars:>=1000returns a different and far larger set, so the distinction is documented rather than silently "corrected".TCFEED_GH_QUERYstars:1000TCFEED_GH250turns the source offAsked through
ghso it uses the token already on the machine. The HTML page is rate-limited hard for anyone not signed in, and parsing it would be a scraper of a page that changes shape without warning.Either source may now fail on its own
This is the part that matters more than the new source. Reddit refuses this address often enough that making it fatal meant a run produced nothing on an afternoon when the search was answering perfectly. Now whichever source came back is scanned, a source that failed says why rather than looking empty, and only losing every source that was asked is an error.
Archived and forked repositories are dropped at the search rather than left for
metadata()to reject one HTTP call later, because the search already knows.Verified
tsc --strictclean, and all three paths exercised against the live APIs withTCFEED_MAX=0so nothing was cloned:44 to consider — 40 from r/coolgithubprojects, 4 from search(5 requested, 1 dropped as archived)reddit: could not read r/… (HTTP 404)then4 to consider — 0 from …, 4 from search— run continuesno source answered, exit 1🤖 Generated with Claude Code