diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000000..aa9cb5d244 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,138 @@ +# PROJECT KNOWLEDGE BASE + +**Generated:** 2026-02-12 +**Commit:** 661961a5 +**Branch:** main + +## OVERVIEW + +Official Go SDK for Stellar blockchain. Provides transaction building (`txnbuild`), API clients (`horizonclient`, `rpcclient`), ledger ingestion (`ingest`), and XDR type definitions. Previously a monorepo—services moved to separate repos (Oct 2025). + +## STRUCTURE + +``` +go-stellar-sdk/ +├── txnbuild/ # Transaction building API (83 Go files) - START HERE for tx construction +├── clients/ # Network clients +│ ├── horizonclient/ # Horizon REST API client (primary) +│ ├── rpcclient/ # Stellar RPC client +│ ├── stellarcore/ # Direct stellar-core client +│ └── stellartoml/ # stellar.toml parser +├── ingest/ # Ledger data ingestion library +│ └── ledgerbackend/ # Captive Core, RPC backend, buffered storage +├── xdr/ # XDR types (MOSTLY GENERATED - see xdr/AGENTS.md) +├── support/ # Internal utilities (db, http, config, errors, logging) +├── historyarchive/ # History archive access +├── protocols/ # Protocol response types (horizon, rpc, stellarcore) +├── processors/ # ETL-style data processors +├── strkey/ # Stellar key encoding (G*, S*, M*, P*, C* addresses) +├── keypair/ # Cryptographic key pair operations +├── tools/ # Standalone CLI utilities (not library code) +└── exp/ # DEPRECATED experimental packages +``` + +## WHERE TO LOOK + +| Task | Location | Notes | +|------|----------|-------| +| Build transactions | `txnbuild/` | Start with `NewTransaction()` | +| Submit to Horizon | `clients/horizonclient/` | Use with txnbuild | +| Submit to RPC | `clients/rpcclient/` | Soroban smart contracts | +| Parse ledger data | `ingest/` | LedgerTransactionReader, CheckpointChangeReader | +| Run Captive Core | `ingest/ledgerbackend/` | CaptiveCoreConfig | +| XDR type helpers | `xdr/` | Hand-written helpers in non-generated files | +| Key encoding | `strkey/` | Decode/Encode/CheckValid | +| Sign transactions | `keypair/` | Full.Sign(), ParseAddress() | +| Network passphrases | `network/` | TestNetworkPassphrase, PublicNetworkPassphrase | +| Database utilities | `support/db/` | Session, batch insert builders | +| HTTP utilities | `support/http/` | Middleware, signed requests | + +## CODE MAP + +**Primary Public Interfaces:** + +| Package | Key Types/Funcs | Role | +|---------|-----------------|------| +| `txnbuild` | `Transaction`, `NewTransaction()`, `Operation` | Construct and sign transactions | +| `horizonclient` | `Client`, `DefaultTestNetClient` | Query Horizon, submit transactions | +| `rpcclient` | `Client` | Interact with Stellar RPC | +| `ingest` | `LedgerTransactionReader`, `CheckpointChangeReader` | Parse ledger data | +| `ledgerbackend` | `CaptiveCoreConfig`, `NewCaptive()` | Run embedded stellar-core | +| `xdr` | All Stellar protocol types | Network data structures | +| `keypair` | `Full`, `FromAddress`, `Parse()`, `Random()` | Key management | +| `strkey` | `Decode()`, `Encode()`, `CheckValid()` | Address encoding | + +## CONVENTIONS + +**Deviations from standard Go:** +- No top-level `cmd/` or `pkg/`—CLIs under `tools/`, libs at root +- Nested `internal/` dirs (e.g., `strkey/internal/`) instead of single `/internal` +- Many `main.go` files are NOT executables—check `package` declaration +- Line length limit: 140 chars (relaxed) +- Function length: up to 100 lines allowed + +**Error handling:** +- Mixed stdlib `fmt.Errorf("%w", err)` and `github.com/pkg/errors.Wrap()` +- Prefer `%w` for new code + +**Formatting:** +- Run `goimports -w .` before PR (preferred over plain gofmt) +- CI enforces via `./gofmt.sh` + +## ANTI-PATTERNS (THIS PROJECT) + +| Don't | Do Instead | +|-------|------------| +| Edit `xdr/xdr_generated.go` | Modify `.x` files, run `make xdr` | +| Edit `*.pb.go` files | Modify `.proto` files, run `make generate-proto` | +| Use `xdr.UnmarshalBinary()` on MarshalBinaryBase64 output | Use matching decode method | +| Wrap `io.EOF` in xdr streaming | Return bare `io.EOF` | +| Use deprecated `PathPayment` | Use `PathPaymentStrictReceive` | +| Use deprecated `AllowTrust` | Use `SetTrustLineFlags` | +| Use `horizon` client | Use `horizonclient` | + +**Generated code locations:** +- `xdr/xdr_generated.go` — XDR types (DO NOT EDIT) +- `gxdr/xdr_generated.go` — Alternative XDR (DO NOT EDIT) +- `*.pb.go` files — Protobuf (DO NOT EDIT) + +## COMMANDS + +```bash +# Format and lint (run before PR) +./gofmt.sh && ./gomod.sh && ./govet.sh && ./staticcheck.sh + +# Run tests +go test ./... # All tests (some need postgres) +go test ./txnbuild # Just txnbuild (safe) +go test -race -cover ./... # CI command + +# Regenerate XDR types +make xdr # Requires Docker + +# Regenerate proto +make generate-proto + +# Build all +go build ./... + +# Run specific tool +go run ./tools/ +``` + +## NOTES + +**Test dependencies:** +- Some tests require PostgreSQL 12+ (set `PGHOST`, `PGUSER`, etc.) +- `ingest/ledgerbackend` tests may need stellar-core binary +- Fuzz tests under `xdr/fuzz/` + +**Module path:** `github.com/stellar/go-stellar-sdk` + +**Go version:** 1.24+ (tests against 1.24 and 1.25) + +**Subdirectory AGENTS.md files exist for:** +- `txnbuild/AGENTS.md` — Transaction building details +- `xdr/AGENTS.md` — Generated vs manual code guidance +- `ingest/AGENTS.md` — Ingestion library patterns +- `clients/horizonclient/AGENTS.md` — Horizon client usage diff --git a/clients/horizonclient/AGENTS.md b/clients/horizonclient/AGENTS.md new file mode 100644 index 0000000000..51771949a5 --- /dev/null +++ b/clients/horizonclient/AGENTS.md @@ -0,0 +1,89 @@ +# horizonclient + +HTTP client for Stellar Horizon API. Query network state, submit transactions. + +## OVERVIEW + +Primary client for interacting with Horizon servers. Use with `txnbuild` for complete transaction workflow. + +## KEY TYPES + +| Type | Purpose | +|------|---------| +| `Client` | Main client; set `HorizonURL` | +| `DefaultTestNetClient` | Pre-configured testnet client | +| `DefaultPublicNetClient` | Pre-configured pubnet client | +| `AccountRequest` | Query account details | +| `TransactionRequest` | Query transactions | +| `Error` | Horizon error with problem details | + +## COMMON PATTERNS + +```go +// Use default testnet client +client := horizonclient.DefaultTestNetClient + +// Or configure custom +client := &horizonclient.Client{HorizonURL: "https://horizon.stellar.org"} + +// Get account +account, err := client.AccountDetail(horizonclient.AccountRequest{ + AccountID: "GABC...", +}) + +// Submit transaction (built with txnbuild) +resp, err := client.SubmitTransaction(tx) +// or from XDR +resp, err := client.SubmitTransactionXDR(base64XDR) + +// Stream ledgers +client.StreamLedgers(ctx, horizonclient.LedgerRequest{}, func(ledger horizon.Ledger) { + // handle ledger +}) +``` + +## REQUEST TYPES + +| Request | Endpoint | +|---------|----------| +| `AccountRequest` | `/accounts/{id}` | +| `AccountsRequest` | `/accounts` (list) | +| `TransactionRequest` | `/transactions` | +| `OperationRequest` | `/operations` | +| `EffectRequest` | `/effects` | +| `LedgerRequest` | `/ledgers` | +| `OfferRequest` | `/offers` | +| `TradeRequest` | `/trades` | +| `PathsRequest` | `/paths` | +| `OrderBookRequest` | `/order_book` | +| `AssetRequest` | `/assets` | +| `ClaimableBalanceRequest` | `/claimable_balances` | + +## ERROR HANDLING + +```go +resp, err := client.SubmitTransaction(tx) +if err != nil { + if hzErr, ok := err.(*horizonclient.Error); ok { + // Access Horizon problem details + fmt.Println(hzErr.Problem.Title) + fmt.Println(hzErr.Problem.Extras["result_codes"]) + } +} +``` + +## ANTI-PATTERNS + +| Don't | Do Instead | +|-------|------------| +| Use deprecated `horizon` package | Use `horizonclient` | +| Ignore `*horizonclient.Error` type | Type assert for detailed error info | +| Hardcode Horizon URLs | Use `DefaultTestNetClient` / `DefaultPublicNetClient` | + +## TESTING + +```bash +go test ./clients/horizonclient # Unit tests (mocked) +``` + +Mock available: `horizonclient.MockClient` for testing without network. diff --git a/ingest/AGENTS.md b/ingest/AGENTS.md new file mode 100644 index 0000000000..c9b2c3ac7a --- /dev/null +++ b/ingest/AGENTS.md @@ -0,0 +1,100 @@ +# ingest + +Ledger data ingestion library. Parse raw ledger data from Captive Core, RPC, or history archives. + +## OVERVIEW + +Build custom ingestion pipelines. Read transactions, state changes, and ledger metadata. Used by Horizon and custom analytics tools. + +## ARCHITECTURE + +``` + [ Your Code ] + | + [ Readers ] + / \ + [ChangeReader] [TransactionReader] + | | + CheckpointChange LedgerTransaction + Reader Reader + | + [ LedgerBackend ] + | + |---------|---------| + Captive Buffered RPC + Core Storage Backend +``` + +## KEY TYPES + +| Type | Purpose | +|------|---------| +| `LedgerTransactionReader` | Read transactions from a ledger | +| `CheckpointChangeReader` | Read state changes at checkpoints | +| `LedgerChangeReader` | Read changes within a single ledger | +| `Change` | Represents a ledger entry change (created/updated/removed) | +| `LedgerTransaction` | Transaction with result and metadata | + +## BACKENDS (`ledgerbackend/`) + +| Backend | Use Case | +|---------|----------| +| `CaptiveCoreBackend` | Run embedded stellar-core (most common) | +| `BufferedStorageBackend` | Read from pre-exported data lake | +| `RPCBackend` | Read from Stellar RPC | + +## COMMON PATTERNS + +```go +// Create Captive Core backend +backend, err := ledgerbackend.NewCaptive(ledgerbackend.CaptiveCoreConfig{ + BinaryPath: "/usr/bin/stellar-core", + NetworkPassphrase: network.TestNetworkPassphrase, + HistoryArchiveURLs: network.TestNetworkhistoryArchiveURLs, + Toml: captiveCoreToml, +}) +defer backend.Close() + +// Prepare ledger range +err = backend.PrepareRange(ctx, ledgerbackend.BoundedRange(1000, 2000)) + +// Read transactions +reader, err := ingest.NewLedgerTransactionReader(ctx, backend, passphrase, seq) +for { + tx, err := reader.Read() + if err == io.EOF { break } + // Process tx +} +``` + +## WHERE TO LOOK + +| Task | File | +|------|------| +| Read transactions | `ledger_transaction_reader.go` | +| Read state changes | `checkpoint_change_reader.go`, `ledger_change_reader.go` | +| Configure Captive Core | `ledgerbackend/captive_core_backend.go` | +| TOML config generation | `ledgerbackend/toml.go` | +| Buffered/lake backend | `ledgerbackend/buffered_storage_backend.go` | + +## ANTI-PATTERNS + +| Don't | Do Instead | +|-------|------------| +| Use `FeeChanges` directly | Use helper methods on `LedgerTransaction` | +| Ignore `PrepareRange` | Always prepare range before `GetLedger` | +| Skip `Close()` on backend | Defer close to avoid resource leaks | + +## TESTING + +```bash +go test ./ingest # Unit tests +go test ./ingest/ledgerbackend # Backend tests (may need stellar-core) +``` + +**Note:** Some tests require stellar-core binary or external services. + +## TUTORIALS + +Working examples in `ingest/tutorial/`: +- `expiring-sac-balances/` — Track expiring Soroban balances diff --git a/txnbuild/AGENTS.md b/txnbuild/AGENTS.md new file mode 100644 index 0000000000..8fbb37db55 --- /dev/null +++ b/txnbuild/AGENTS.md @@ -0,0 +1,68 @@ +# txnbuild + +Transaction building API for Stellar network. Construct, sign, and serialize transactions. + +## OVERVIEW + +Primary SDK entry point for building transactions. Use `NewTransaction()` with operations, sign with keypairs, submit via `horizonclient`. + +## KEY TYPES + +| Type | Purpose | +|------|---------| +| `Transaction` | Immutable signed transaction; call `Sign()` to add signatures | +| `TransactionParams` | Input to `NewTransaction()`: source, ops, fee, preconditions | +| `FeeBumpTransaction` | Wrapper to increase fee on existing transaction | +| `Operation` | Interface implemented by all 20+ operation types | +| `Account` | Interface for source account (sequence number management) | +| `Preconditions` | TimeBounds, LedgerBounds, MinSequence, etc. | + +## COMMON PATTERNS + +```go +// Build transaction +tx, err := txnbuild.NewTransaction(txnbuild.TransactionParams{ + SourceAccount: &account, + IncrementSequenceNum: true, + Operations: []txnbuild.Operation{&op}, + BaseFee: txnbuild.MinBaseFee, + Preconditions: txnbuild.Preconditions{TimeBounds: txnbuild.NewTimeout(300)}, +}) + +// Sign (returns NEW Transaction - immutable) +tx, err = tx.Sign(network.TestNetworkPassphrase, keypair) + +// Serialize for submission +xdrBase64, err := tx.Base64() +``` + +## WHERE TO LOOK + +| Task | File | +|------|------| +| Create transaction | `transaction.go` — `NewTransaction()` | +| Payment operation | `payment.go` | +| Path payments | `path_payment_strict_send.go`, `path_payment.go` | +| Account creation | `create_account.go` | +| Trust lines | `change_trust.go`, `set_trust_line_flags.go` | +| Offers/DEX | `manage_offer.go`, `manage_buy_offer.go` | +| Soroban invoke | `invoke_host_function.go` | +| Fee bumps | `transaction.go` — `NewFeeBumpTransaction()` | + +## ANTI-PATTERNS + +| Don't | Do Instead | +|-------|------------| +| Use `PathPayment` | Use `PathPaymentStrictReceive` (renamed) | +| Use `AllowTrust` | Use `SetTrustLineFlags` | +| Mutate Transaction after Sign | Transaction is immutable; Sign returns new instance | +| Forget `IncrementSequenceNum` | Set `true` unless managing sequence manually | + +## TESTING + +```bash +go test ./txnbuild # Unit tests (no external deps) +go test ./txnbuild -v # Verbose +``` + +Demo available: `go run ./txnbuild/cmd/demo` diff --git a/xdr/AGENTS.md b/xdr/AGENTS.md new file mode 100644 index 0000000000..3233c5b9ae --- /dev/null +++ b/xdr/AGENTS.md @@ -0,0 +1,76 @@ +# xdr + +Stellar XDR (External Data Representation) types. **MOSTLY GENERATED CODE.** + +## OVERVIEW + +Contains all Stellar protocol types: transactions, operations, ledger entries, assets, etc. The bulk is generated from `.x` IDL files—do NOT edit generated code. + +## GENERATED vs MANUAL + +| File | Type | Edit? | +|------|------|-------| +| `xdr_generated.go` | Generated | **NO** — 50k+ lines, regenerate via `make xdr` | +| `Stellar-*.x` | IDL source | Yes — modify these, then regenerate | +| All other `*.go` | Hand-written helpers | Yes | + +**Hand-written helpers** (safe to edit): +- `asset.go`, `account_id.go`, `muxed_account.go` — type helpers +- `json.go` — JSON marshaling customization +- `scval.go` — Soroban value helpers +- `ledger_key.go`, `ledger_entry.go` — ledger data helpers +- `transaction_envelope.go` — envelope manipulation +- `xdrstream.go` — streaming XDR reader + +## REGENERATION + +```bash +# Requires Docker +make xdr + +# Or step by step: +# 1. Edit xdr/Stellar-*.x files +# 2. Run make xdr (uses xdrgen via Docker) +# 3. Commit xdr_generated.go +``` + +## COMMON PATTERNS + +```go +// Decode base64 XDR +var envelope xdr.TransactionEnvelope +err := xdr.SafeUnmarshalBase64(base64Str, &envelope) + +// Encode to base64 +base64Str, err := xdr.MarshalBase64(envelope) + +// Stream reading (for large data) +reader := xdr.NewXdrStream(r) +for { + var entry xdr.LedgerEntry + err := reader.ReadOne(&entry) + if err == io.EOF { break } +} +``` + +## ANTI-PATTERNS + +| Don't | Do Instead | +|-------|------------| +| Edit `xdr_generated.go` | Edit `.x` files, run `make xdr` | +| `xdr.UnmarshalBinary()` on base64 data | Use `xdr.SafeUnmarshalBase64()` | +| Wrap `io.EOF` in stream reading | Return bare `io.EOF` for proper termination | +| Use deprecated `LedgerKey.LedgerKey()` | Use `LedgerEntryData.LedgerKey()` | + +## FUZZ TESTING + +Fuzz harnesses under `xdr/fuzz/`: +- `jsonclaimpredicate/` — JSON claim predicate fuzzing +- Corpus files for reproducible fuzzing + +## NOTES + +- XDR commit tracked in `xdr_commit_generated.txt` +- gxdr package (`../gxdr/`) is alternative XDR impl—also generated +- Many types have `MarshalBinary`/`UnmarshalBinary` for wire format +- JSON marshaling customized for Horizon API compatibility