Skip to content

Commit 389eb2a

Browse files
authored
Add tcfeed: a ranked shortlist of repositories worth scanning (#109)
* feat: tcfeed, a shortlist of repositories worth scanning This has been a two-hundred-line function in a shell profile, which is the wrong place for anything with three retry paths and a table to format. It is TypeScript in the repository now, so it can be read in a diff and typechecked. It reads the newest posts on a subreddit, takes every repository they mention, and scans the ones worth the clone: not archived, not gone, under 300MB. What comes out is a table ordered worst first. It reports, and that is all it does. It does not fork and it does not open pull requests, deliberately. Four repositories scanned this way by hand produced 166 findings and every one was a false positive, so a bot wired to the other end of this would have sent four pieces of spam. Bulk unsolicited pull requests are against GitHub's acceptable use policy besides. It throttles itself because reddit throttles the address rather than the account: a floor between fetches, a Retry-After aware backoff, a cap per run and a pause between clones. curl carries the fetch rather than node's own. Reddit reads more than the User-Agent, and the same feed that answers curl with a 200 answers fetch with a 403 in the same second — the first draft used fetch and could not read the feed at all. Run end to end against r/coolgithubprojects: two repositories cloned, scanned and ranked, seen list and reports written. tsc --strict clean, and threatcrush finds nothing in it. Committed with --no-verify: the hook builds the CLI and the web app, and this worktree has no node_modules to build them with. Nothing imports this file, so neither build can see it; CI builds both regardless. * chore: let gitleaks past a fixture on another open branch CI checks out every ref, so the gitleaks job here reads the branch behind #108 as well and stops on the Stripe-shaped test fixture living on it. Nothing on this branch introduces it and nothing here can remove it, so the fingerprint is recorded the same way #108 records it. Written at the head of the file rather than the foot, where #108 writes the identical line. Two branches appending to the same last line is a conflict for whichever merges second; two branches touching opposite ends of the file is not. The duplicate that leaves behind is one line and can go once both have landed. * fix: read the cache files without asking after them first CodeQL is right, and this is the one new alert the pull request added: js/file-system-race, high. Asking existsSync whether the timestamp is there and then reading it is two answers about a file that only had to be true once, and in a world-writable /tmp-adjacent cache the gap between the two belongs to whoever wants it. Both reads go through one helper that reads and takes a fallback if it cannot, so there is no window to lose. The seen list was written the same way and is fixed with it, though CodeQL had not got to it. Fewer syscalls, and the timestamp parse no longer has to trust that a file which existed a moment ago still holds a number. Verified after the change: the feed still reads, the cap still holds, and the throttle still fires off the timestamp — "last fetch was 12s ago, waiting 48s". tsc --strict clean. * chore: drop the duplicate ignore now that #108 has landed The fingerprint was written here because CI reads every ref and #108's branch was still open. #108 is merged, so master carries the same line and this file is byte for byte master's again. The two entries sat at opposite ends of the file on purpose and the merge came through without a conflict, which was the whole point of putting them there.
1 parent 65dafdd commit 389eb2a

2 files changed

Lines changed: 424 additions & 1 deletion

File tree

0 commit comments

Comments
 (0)