R2: re-sign every retry, bound every attempt, and give the signer a test bed

[anupsv]

Lands the SigV4 audit's remaining findings against
.github/scripts/download-r2-object.sh and .github/scripts/upload-r2-object.sh.

Base note. The task said to branch off fix/r2-sigv4-canonical-request so as
not to revert #818's fix. #818 landed on main as cfc644b before this work
started, so this branches off main (3e9f072), which already contains it — the
canonical-request construction is carried forward unchanged (it just moved inside
sign_request()).

---

F2 — retries reused the signature, and nothing bounded an attempt

amz_date was computed once and curl owned the retry (--retry 5 --retry-all-errors). curl replays the argv it was handed, so all six attempts
carried the same x-amz-date and the same Authorization: a retry that
cannot re-sign. R2 only accepts a signature inside its ~15 minute clock-skew
window, so a run that spends long enough retrying converts a transient 503 into a
permanent-looking 403 RequestTimeTooSkewed.

With no --max-time/--connect-timeout, a peer that accepts the connection and
then sends nothing triggered neither a retry nor a timeout: the transfer
"succeeded" with an empty body, or the job hung to its own 30-minute limit with no
diagnosis.

The transfer is now a bash loop:

• every attempt calls sign_request(), which recomputes amz_date,
  date_stamp, the canonical request, the string-to-sign and the signature;
• every attempt carries --connect-timeout 30 --max-time 600;
• six attempts as before — what changed is who retries, not how many times.
  R2_RETRY_DELAY_SECONDS (default 2, validated as a non-negative integer) lets
  a test exercise the loop's shape without its wall clock;
• the --output file is truncated at the top of each iteration. curl opens it
  lazily, so an attempt that dies at connect time never truncates it and a 503
  <Error> document could otherwise sit in front of the object the next attempt
  writes. The caller verifies a pinned sha256 and would report a golden mismatch
  for a transport artefact.

The status-gated error-body reporting from #810 is unchanged.

F4 — the upload payload hash broke on exotic paths

shasum -a 256 "${input_path}" escapes the filename and prefixes the digest
with a backslash when the path contains a backslash or a newline. Measured:

$ shasum -a 256 'back\slash.json' | awk '{print $1}'
\b7367c22dfc669fdf6f9fcdb91112e6aee109312a2fe68a1508b00dba48cc9cb    (65 chars)
$ shasum -a 256 < 'back\slash.json' | awk '{print $1}'
b7367c22dfc669fdf6f9fcdb91112e6aee109312a2fe68a1508b00dba48cc9cb     (64 chars)

That 65-character non-hex value went into both the signed canonical request
and the x-amz-content-sha256 header — self-consistent, and still rejected,
because R2 compares it against the body it received. A path starting with - was
parsed as options (Unknown option: d). Now shasum -a 256 < "${input_path}",
which has no filename in its output at all.

F5 — --location removed from both scripts

Verified against real curl 8.7.1 (macOS system curl, what the runners use) with
two local servers:

hop	authorization	x-amz-date	x-amz-content-sha256
cross-host, before redirect	PRESENT	PRESENT	PRESENT
cross-host, after 302	absent	PRESENT	PRESENT
same-host, after 302 to a different path	PRESENT (bound to the OLD path)	PRESENT	PRESENT

So a cross-host redirect delivers an unsigned request that still carries the
signing headers, and a same-host redirect delivers a signature for the wrong
path. R2 path-style issues no legitimate redirects, so following one could only
convert a loud failure into a confusing one.

F3 — the object key charset is now an enforced invariant

Nothing percent-encodes anywhere: the same raw bytes are signed into the canonical
request and handed to curl. For the keys actually in use ([A-Za-z0-9._/-])
identity encoding is exactly right. Outside that set it is not, and every way it
breaks is quiet: a space is curl exit 3, # truncates the URL at the fragment,
? turns the tail into a query string that the canonical request signs as empty,
and non-ASCII is percent-encoded on the wire after being signed raw.

Deliberately not an RFC 3986 encoder — the aws-cli branch encodes by
botocore's rules, and a second encoder here that disagreed with it would be a
fresh divergence between the two branches for the same key. object_path is
validated against ^[A-Za-z0-9._/-]+$ next to the existing empty/absolute/dot-
segment/control-character checks, so an unsupported key is a clear refusal before
any request instead of a 403 that reads like a credentials fault.

A companion test reads the literal *_R2_PATH keys out of benchmark.yml and
dflash-benchmark.yml and runs each one through the script, so a future key the
charset would reject fails in CI rather than on the runner.

F6 — the signed path now has an execution environment

download_with_aws_cli is silently preferred, so the bash signer only ever
ran on boxes without aws. That is exactly how two signing bugs shipped
undetected, and why M5-C's first ever execution of this signer was a production
hidden-golden fetch. R2_FORCE_SIGNED=1 skips the fallback.

A comment at each call site records the other half of the problem: if aws is
present but fails, the script falls through to the signed path silently — so
which implementation performed a given transfer varies run to run, and the only
record is the banner line each branch prints.

---

Tests

Tests/MLXFastTests/R2RequestExecutionTests.swift (new, 12 tests) executes the
real scripts, unmodified, with stub curl/date/aws first on M5-C's minimal
PATH (/usr/bin:/bin:/usr/sbin:/sbin), and asserts on the argv curl was actually
handed, attempt by attempt. The stub curl honours --retry in-process, replaying
its argv, so the pre-fix script still produces six attempts here — with identical
headers. The stub date advances 7 minutes per SigV4 clock read, so two attempts
straddle R2's skew window.

R2SignatureTests gains the two gaps the audit named:

• nothing asserted the string_to_sign CONSTRUCTION. The canonical-request
  test stops at the CR hash and the HMAC test starts from a hand-written
  string-to-sign whose last line is the literal deadbeef, so the line that
  assembles them could lose a newline, swap the date and the scope, or
  interpolate canonical_request where it means canonical_request_hash, and
  every existing assertion still passed. Pinned against the AWS SigV4
  documentation's worked example — its published CanonicalRequest, StringToSign
  and Signature, reproduced byte for byte.
• nothing composed the chain. CR → hash → STS → signature are each pinned
  separately, so a mismatch between links passes all of them. One pinned
  end-to-end signature per script now runs the shipped chain start to finish over
  R2-shaped inputs.

Both new tests splice the script's own assignment lines and hmac_hex()
definition into a shell, so they track the shipped construction rather than a
restatement of it. Every pinned value was recomputed independently with Python's
hmac/hashlib before being trusted.

swift test: 549 tests / 25 suites pass (535 / 24 on the parent commit).

Attack

Each fix was reverted in turn, with the tests left alone:

reintroduced defect	tests that fired
retries reuse one signature (signing hoisted out of the loop)	everyRetryIsSignedAfresh, everyUploadRetryIsSignedAfresh
curl owns the retry again (--retry 5 --retry-all-errors)	+ noAttemptDelegatesRetryToCurl, theOutputFileIsTruncatedBetweenAttempts
no --connect-timeout / --max-time	everyAttemptBoundsConnectAndTotalTime, aStalledAttemptTimesOutAndIsRetriedRatherThanSucceedingEmpty
--output not truncated between attempts	theOutputFileIsTruncatedBetweenAttempts, everyRetryIsSignedAfresh
shasum -a 256 "${path}" restored	thePayloadHashIsTheContentDigestForExoticInputPaths
--location restored	noAttemptFollowsRedirects
charset guard removed	unsignableObjectKeysAreRefusedBeforeAnyRequest
R2_FORCE_SIGNED removed	forceSignedBypassesAWorkingAWSCLI
string_to_sign swaps date and scope	theStringToSignMatchesTheAWSDocumentationWorkedExample, theWholeSigningChainReproducesOnePinnedEndToEndSignature
string_to_sign signs the request, not its hash	same two
credential scope region ≠ the region keyed into k_region	theWholeSigningChainReproducesOnePinnedEndToEndSignature, theEmittedRequestMatchesWhatWasSigned

The last row is the one worth reading. It is a defect between links, and under
it all three pre-existing tests still pass:

Pre-existing coverage (all of it, from #817/#818) under this attack:
   PASS noOpensslDigestParsesTheTextOutputFormat
   PASS theCanonicalRequestMatchesAnIndependentSigV4Implementation
   PASS theScriptSigningChainReproducesThePinnedSignature

Only the tests this PR adds catch it.

Honest scope note. One assertion in this PR is text-based rather than
executed: R2SignatureTests.noOpensslDigestParsesTheTextOutputFormat (pre-
existing, unchanged). Everything else runs the real scripts or splices their real
lines into a shell. noAttemptFollowsRedirects asserts on the argv the real
script handed curl — the consequence of --location was measured separately
with the two-server probe above, not asserted in CI, because reproducing a
cross-host redirect inside the test suite would need two listening hosts.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith with what you need. Autofix is disabled.}