Fix InputLen() guard bypass in streaming XDR decoders

[tamirms]

PR Checklist

PR Structure

• This PR has reasonably narrow scope (if not, break it down into smaller PRs).
• This PR avoids mixing refactoring changes with feature changes (split into two PRs
  otherwise).
• This PR's title starts with name of package that is most changed in the PR, ex.
  services/friendbot, or all or doc if the changes are broad or impact many
  packages.

Thoroughness

• This PR adds tests for the most critical parts of the new functionality or fixes.
• I've updated any docs (developer docs, .md
  files, etc... affected by this change). Take a look in the docs folder for a given service,
  like this one.

Release planning

• I've reviewed the changes in this PR and if I consider them worthwhile for being mentioned on release notes then I have updated the relevant CHANGELOG.md within the component folder structure. For example, if I changed horizon, then I updated (services/horizon/CHANGELOG.md. I add a new line item describing the change and reference to this PR. If I don't update a CHANGELOG, I acknowledge this PR's change may not be mentioned in future release notes.
• I've decided if this PR requires a new major/minor version according to
  semver, or if it's mainly a patch change. The PR is targeted at the next
  release branch if it's not a patch change.

What

Every xdr type generated by xdrgen has a DecodeFrom() function which contains a guards like:

  if il, ok := d.InputLen(); ok && uint(il) < uint(l) {
      return n, fmt.Errorf("length (%d) exceeds remaining input length (%d)", l, il)
  }

These fire before every variable-length allocation (slices, strings, opaque data), preventing attacker-controlled length fields from triggering unbounded memory allocations.

InputLen() delegates to the decoder's internal lenLeft interface:

  type lenLeft interface {
      Len() int
  }

When creating a decoder, go-xdr checks if the reader implements Len():

  func NewDecoder(r io.Reader) *Decoder {
      if l, ok := r.(lenLeft); ok {
          return &Decoder{r: r, l: l, ...}
      }
      return &Decoder{r: r, l: nil, ...}  // InputLen() returns (0, false)
  }

bytes.NewReader implements Len(), so guards are active. Streaming readers like bufio.Reader and zstd readers do not, so l is nil and all guards are silently skipped.

Problem

Two production decode paths create decoders wrapping streaming readers:

1. bufferedLedgerMetaReader — xdr3.NewDecoder(bufio.Reader) over the captive core pipe
2. compressxdr.XDRDecoder.ReadFrom — xdr.Unmarshal over a zstd streaming reader

In both cases, InputLen() returns (0, false) which means the allocation checks are bypassed.

These two cases are fixed by using bytes.NewReader.

Known limitations

[N/A]

[Copilot]

Pull request overview

This PR addresses a critical security vulnerability where XDR InputLen() guards were bypassed when decoding from streaming readers (bufio.Reader, zstd reader) that don't implement Len(). The fix adopts a buffer-first pattern: data is read/decompressed into []byte first, then decoded from bytes.NewReader which implements Len(), activating all 95 generated InputLen() guards.

Changes:

• Refactored ReadFrameLength to accept io.Reader instead of *xdr.Decoder and validate the magic bit (0x80000000)
• Implemented buffer-first pattern in bufferedLedgerMetaReader and compressxdr.XDRDecoder to ensure InputLen() guards are active
• Added max frame size guard (256 MB) for captive core pipe reader to prevent unbounded allocations

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
xdr/main.go	Refactored ReadFrameLength to accept io.Reader and validate magic bit; migrated to standard library errors
xdr/xdrstream.go	Updated Stream.ReadOne to use new ReadFrameLength signature; migrated to standard library errors
xdr/xdrstream_test.go	Updated tests to include magic bit (0x80000000) in frame headers
support/compressxdr/compress_xdr.go	Implemented buffer-first pattern: read all decompressed data into memory before XDR decoding
ingest/ledgerbackend/buffered_meta_pipe_reader.go	Implemented buffer-first pattern with reusable frame buffer and added 256 MB max frame size validation
ingest/ledgerbackend/buffered_meta_pipe_reader_test.go	Added comprehensive tests for frame reading, size validation, and multiple frames

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[leighmcculloch]

Couple questions but otherwise lgtm.