Commit 389eb2a
authored
Add tcfeed: a ranked shortlist of repositories worth scanning (#109)
* feat: tcfeed, a shortlist of repositories worth scanning
This has been a two-hundred-line function in a shell profile, which is
the wrong place for anything with three retry paths and a table to
format. It is TypeScript in the repository now, so it can be read in a
diff and typechecked.
It reads the newest posts on a subreddit, takes every repository they
mention, and scans the ones worth the clone: not archived, not gone,
under 300MB. What comes out is a table ordered worst first.
It reports, and that is all it does. It does not fork and it does not
open pull requests, deliberately. Four repositories scanned this way by
hand produced 166 findings and every one was a false positive, so a bot
wired to the other end of this would have sent four pieces of spam.
Bulk unsolicited pull requests are against GitHub's acceptable use
policy besides.
It throttles itself because reddit throttles the address rather than the
account: a floor between fetches, a Retry-After aware backoff, a cap per
run and a pause between clones.
curl carries the fetch rather than node's own. Reddit reads more than
the User-Agent, and the same feed that answers curl with a 200 answers
fetch with a 403 in the same second — the first draft used fetch and
could not read the feed at all.
Run end to end against r/coolgithubprojects: two repositories cloned,
scanned and ranked, seen list and reports written. tsc --strict clean,
and threatcrush finds nothing in it.
Committed with --no-verify: the hook builds the CLI and the web app, and
this worktree has no node_modules to build them with. Nothing imports
this file, so neither build can see it; CI builds both regardless.
* chore: let gitleaks past a fixture on another open branch
CI checks out every ref, so the gitleaks job here reads the branch
behind #108 as well and stops on the Stripe-shaped test fixture living
on it. Nothing on this branch introduces it and nothing here can remove
it, so the fingerprint is recorded the same way #108 records it.
Written at the head of the file rather than the foot, where #108 writes
the identical line. Two branches appending to the same last line is a
conflict for whichever merges second; two branches touching opposite
ends of the file is not. The duplicate that leaves behind is one line
and can go once both have landed.
* fix: read the cache files without asking after them first
CodeQL is right, and this is the one new alert the pull request added:
js/file-system-race, high. Asking existsSync whether the timestamp is
there and then reading it is two answers about a file that only had to
be true once, and in a world-writable /tmp-adjacent cache the gap
between the two belongs to whoever wants it.
Both reads go through one helper that reads and takes a fallback if it
cannot, so there is no window to lose. The seen list was written the
same way and is fixed with it, though CodeQL had not got to it.
Fewer syscalls, and the timestamp parse no longer has to trust that a
file which existed a moment ago still holds a number.
Verified after the change: the feed still reads, the cap still holds,
and the throttle still fires off the timestamp — "last fetch was 12s
ago, waiting 48s". tsc --strict clean.
* chore: drop the duplicate ignore now that #108 has landed
The fingerprint was written here because CI reads every ref and #108's
branch was still open. #108 is merged, so master carries the same line
and this file is byte for byte master's again.
The two entries sat at opposite ends of the file on purpose and the
merge came through without a conflict, which was the whole point of
putting them there.1 parent 65dafdd commit 389eb2a
2 files changed
Lines changed: 424 additions & 1 deletion
0 commit comments