diff --git a/assertions-book/assertions/aave-v3-suite.mdx b/assertions-book/assertions/aave-v3-suite.mdx index 59aee10..eeffd3f 100644 --- a/assertions-book/assertions/aave-v3-suite.mdx +++ b/assertions-book/assertions/aave-v3-suite.mdx @@ -23,7 +23,7 @@ We have mocked parts of the protocol in order to properly test the assertions an The Aave V3 assertion suite has been tested quite thoroughly given the early stage of the Credible Layer testing capabilities. We feel confident that the production assertions could be added to an Aave V3 deployment and add value to the security and robustness of the protocol. -We recommend running the assertions in a test environment against historical transactions to validate behavior before production promotion. This is standard best practice for protocol upgrades and does not just apply to assertions. +Before production promotion, create a release in the Phylax platform and review its automatic backtesting result. The platform tests the release against transactions from the last 20,000 blocks so you can identify benign activity that the assertion would have invalidated. See [How to Review Backtesting Results](/credible/backtesting). ## Here Be Dragons 🐲 diff --git a/assertions-book/previous-hacks/balancer-v2-stable-rate-exploit.mdx b/assertions-book/previous-hacks/balancer-v2-stable-rate-exploit.mdx index 180bacf..141a659 100644 --- a/assertions-book/previous-hacks/balancer-v2-stable-rate-exploit.mdx +++ b/assertions-book/previous-hacks/balancer-v2-stable-rate-exploit.mdx @@ -183,94 +183,11 @@ We chose **3x (300%)** as the threshold based on the following analysis: **Disclaimer:** This assertion is conceptual. Based on our analysis of the protocol mechanics, we have not discovered any legitimate operations that would cause rate changes exceeding 3x in a single transaction. However, we cannot guarantee that such edge cases do not exist. Thorough testing and analysis of real-world scenarios and edge cases is required before using this assertion in production. -## Backtesting Results +## Review the release backtest -We have built a custom backtesting framework that allows for running assertions against real-world historical transactions. +After adding this assertion to a project, create a release with `pcl apply`. The platform automatically tests the release against transactions from the last 20,000 blocks. Review every finding to confirm whether the assertion detected behavior you intend to prevent or would block a benign operation. -This tool is excellent for testing assertions we develop against known exploits and it's also what we used to verify the assertion's accuracy. - -We backtested the assertion against the actual exploit transaction on Ethereum mainnet: - -```bash -FOUNDRY_PROFILE=assertions pcl test --match-contract BatchSwapBacktest --ffi -vvv -``` - -### Test Configuration - -```solidity -contract BatchSwapBacktest is CredibleTestWithBacktesting { - address constant BALANCERV2_VAULT = 0xBA12222222228d8Ba445958a75a0704d566BF2C8; - uint256 constant END_BLOCK = 23717632; // Mainnet exploit block - uint256 constant BLOCK_RANGE = 1; - - function testBacktest_Balancer_BatchSwapOperations() public { - executeBacktest({ - targetContract: BALANCERV2_VAULT, - endBlock: END_BLOCK, - blockRange: BLOCK_RANGE, - assertionCreationCode: type(BatchSwapDeltaAssertion).creationCode, - assertionSelector: BatchSwapDeltaAssertion.assertionBatchSwapRateManipulation.selector, - rpcUrl: vm.envString("MAINNET_RPC_URL") - }); - } -} -``` - -### Results - -```text -========================================== - BACKTESTING CONFIGURATION -========================================== -Target Contract: 0xba12222222228d8ba445958a75a0704d566bf2c8 -Block Range: 23717632 to 23717632 -Assertion Selector: 0x5f3ce91c -========================================== - -Total transactions found: 3 - -=== TRANSACTION 1 === -Hash: 0x3e173ab0ba9183efa8a42caa783bdb5ec75daffcc8505cc1302009d11daf1ccf -Function: 0x60e087db ---- -Transaction gas cost: 14297996 -Assertion gas cost: 299665 - -Assertion function reverted: BatchSwap: Extreme pool rate manipulation detected -[ASSERTION_FAIL] VALIDATION FAILED ---- - -=== TRANSACTION 2 === -Hash: 0x6341ec5db92cab0cfd8c17bffab7b7194a591e20de8d14e6f5c5f0d338627a35 ---- -[SKIP] Assertion not triggered on this transaction ---- - -=== TRANSACTION 3 === -Hash: 0xac5837a3b4c17893725c0155b6c0ee24590b1e504487400a7aa4bd927606b24c ---- -[SKIP] Assertion not triggered on this transaction ---- - -========================================== - BACKTESTING SUMMARY -========================================== -Block Range: 23717632 - 23717632 -Total Transactions: 3 -Processed Transactions: 3 -Successful Validations: 2 -Failed Validations: 1 - -=== ERROR BREAKDOWN === -Protocol Violations (Assertion Failures): 1 -Unknown Errors: 0 - -Success Rate: 66% -!!! PROTOCOL VIOLATIONS DETECTED: 1 -================================ -``` - -**Result:** The assertion detected the exploit transaction, demonstrating how invariant-based protection can catch complex attacks, without requiring knowledge of the specific exploit mechanism. +The platform window moves with the chain, so an older exploit transaction may fall outside the release backtest. Keep a focused regression test for the known exploit and use the platform result to evaluate the release against recent protocol activity. See [How to Review Backtesting Results](/credible/backtesting) for the review workflow. ## Key Takeaways diff --git a/credible/agentic-pcl.mdx b/credible/agentic-pcl.mdx index 4d580a0..2b5f249 100644 --- a/credible/agentic-pcl.mdx +++ b/credible/agentic-pcl.mdx @@ -44,7 +44,7 @@ npx github:phylaxsystems/agent-skills#851cd274972efa6047393eaa61a935ed56f66b3b i The commit pin ensures that both commands install the reviewed skill versions shown in this guide. -The [agent-skills repository](https://github.com/phylaxsystems/agent-skills) also includes focused skills for invariant design, assertion implementation, testing, backtesting, trigger optimization, and troubleshooting. +The [agent-skills repository](https://github.com/phylaxsystems/agent-skills) also includes focused skills for invariant design, assertion implementation, testing, trigger optimization, and troubleshooting. Restart your agent after installation if it does not discover the new skills in the current session. diff --git a/credible/apply-assertions.mdx b/credible/apply-assertions.mdx index 2d8593b..836fe0c 100644 --- a/credible/apply-assertions.mdx +++ b/credible/apply-assertions.mdx @@ -108,11 +108,12 @@ pcl apply --yes After applying, complete deployment in the [Phylax platform](https://app.phylax.systems) using the [Deploy Assertions guide](/credible/deploy-assertions-dapp#step-4-deploy-the-assertion): 1. Navigate to your project -2. Click "Deployment" tab -3. Find your assertion and click "Proceed to Review" +2. Select **Releases** +3. Choose the release created by `pcl apply` 4. Review the release diff and selected target contracts -5. Select the target environment (Staging or Production) -6. Click "Deploy" and sign the transaction +5. Review **Assertion verification**, **Source code**, and **Backtesting**. See [Release Review Checks](./release-review-checks) for details. +6. Confirm the target environment declared in `credible.toml` +7. Connect the protocol manager wallet, select **Deploy to Staging** or **Deploy to Production**, and sign the transaction Creating a release through `pcl apply` does not make the assertion active by itself. The release is pending until an authorized protocol admin, also called a manager, signs the deployment transaction in the platform. The assertion becomes active only after that transaction is accepted on-chain and the configured timelock in the State Oracle contracts expires. Removals follow the same State Oracle timelock flow. diff --git a/credible/assertion-enforcer.mdx b/credible/assertion-enforcer.mdx index b16bb8a..0a6d8b8 100644 --- a/credible/assertion-enforcer.mdx +++ b/credible/assertion-enforcer.mdx @@ -141,7 +141,7 @@ Production assertion admission should account for: - Maximum gas or runtime budget - Expected read footprint - Deterministic behavior and read-only execution -- Historical backtesting and staging behavior +- Platform release-backtesting and staging behavior - Suitability for common acceleration paths, such as compact trace summaries or native helper functions These controls keep block-building resources predictable and help protocol teams understand which assertions are ready to promote from staging to production. @@ -157,7 +157,7 @@ Common controls include: - Runtime and read-footprint limits - Dedicated worker capacity for assertion execution - Timeout behavior with clear inclusion-policy consequences -- Staging and backtesting requirements before production promotion +- Staging and platform release-review requirements before production promotion Ingress filtering can also reduce repeated invalid submissions. When a transaction invalidates, the integration can record short-lived fingerprints for exact replays or repeated attempts against the same assertion and target contract or function. Broad fingerprints should use conservative TTLs and thresholds. diff --git a/credible/backtesting-reference.mdx b/credible/backtesting-reference.mdx deleted file mode 100644 index 6e7fcce..0000000 --- a/credible/backtesting-reference.mdx +++ /dev/null @@ -1,284 +0,0 @@ ---- -title: Backtesting Reference -description: API reference for backtesting configuration and options ---- - -This page provides detailed reference documentation for all backtesting configuration options, API methods, and data structures. - - -New to backtesting? Start with the [Backtesting Guide](/credible/backtesting) to understand the basics before diving into this reference. - - -## Configuration Parameters - -### `targetContract` - -The address of the contract ([Assertion Adopter](/credible/glossary#assertion-adopter)) to test assertions against. The backtesting tool will find all transactions that interact with this contract, including both direct calls and internal calls (e.g., through routers or aggregators). - -```solidity -targetContract: 0x5fd84259d66Cd46123540766Be93DFE6D43130D7 -``` - -### `endBlock` - -The last block in the range to test. When `endBlock` is greater than `blockRange`, the first tested block is `endBlock - blockRange + 1`. - -```solidity -endBlock: 31336940 // Test up to this block -``` - -### `blockRange` - -Number of blocks to test, including `endBlock`. When `endBlock` is greater than `blockRange`, the tool tests from `endBlock - blockRange + 1` through `endBlock`. - -```solidity -blockRange: 100 // Tests blocks 31336841 to 31336940 if endBlock is 31336940 -``` - - -To test against a specific transaction, set `blockRange: 1` with the block number containing that transaction. This is useful for validating assertions against known exploit transactions. See the [Balancer V2 Rate Manipulation Exploit](/assertions-book/previous-hacks/balancer-v2-stable-rate-exploit) for a real-world example. - - -### `assertionCreationCode` - -The bytecode of your assertion contract. Use `type(YourAssertion).creationCode` to get this value. - -```solidity -assertionCreationCode: type(MyAssertion).creationCode -``` - -### `assertionSelector` - -The function selector of the assertion function to trigger. This determines which assertion method runs during validation. - -```solidity -assertionSelector: MyAssertion.assertionBatchAccountHealth.selector -``` - -### `rpcUrl` - -The RPC endpoint URL to use for fetching blockchain data. To read it from an environment variable, pass the value with `vm.envString`. - -```solidity -rpcUrl: "https://sepolia.optimism.io" -// or -rpcUrl: vm.envString("MAINNET_RPC_URL") -``` - -### `forkByTxHash` - -Controls how the EVM state is forked for each transaction replay. This parameter is kept for interface compatibility but is effectively always `true` internally to ensure correct pre-transaction state. - -```solidity -forkByTxHash: true // Default and recommended -``` - -### `detailedBlocks` - -Controls logging verbosity. Currently reserved for future functionality. - -```solidity -detailedBlocks: false // Standard output -``` - -## API Reference - -### executeBacktest - -Executes a backtest with the provided configuration and returns detailed results. - -```solidity -function executeBacktest( - BacktestingTypes.BacktestingConfig memory config -) public returns (BacktestingTypes.BacktestingResults memory results) -``` - -**Parameters:** -- `config`: Configuration struct containing all backtesting parameters - -**Returns:** -- `results`: Results struct containing test execution statistics - -**Example:** -```solidity -BacktestingTypes.BacktestingResults memory results = executeBacktest( - BacktestingTypes.BacktestingConfig({ - targetContract: 0x5fd84259d66Cd46123540766Be93DFE6D43130D7, - endBlock: 31336940, - blockRange: 20, - assertionCreationCode: type(MyAssertion).creationCode, - assertionSelector: MyAssertion.assertionInvariant.selector, - rpcUrl: "https://sepolia.optimism.io", - detailedBlocks: false, - forkByTxHash: true - }) -); -``` - -### executeBacktestForTransaction - -Executes a backtest for a single transaction specified by its hash. This is useful for: -- Testing assertions against specific known transactions (such as historical exploits) -- Debugging staging findings (see [How to Plan Assertion Testing](/credible/testing-strategy) for more on the staging workflow) -- Quickly validating assertion behavior on a problematic transaction without specifying a block range - -```solidity -function executeBacktestForTransaction( - bytes32 txHash, - address targetContract, - bytes memory assertionCreationCode, - bytes4 assertionSelector, - string memory rpcUrl -) public returns (BacktestingTypes.BacktestingResults memory results) -``` - -**Parameters:** -- `txHash`: The transaction hash to backtest -- `targetContract`: The target contract address (assertion adopter) -- `assertionCreationCode`: The assertion contract creation code -- `assertionSelector`: The assertion function selector -- `rpcUrl`: The RPC URL to use - -**Returns:** -- `results`: Results struct containing test execution statistics - -**Example:** -```solidity -bytes32 exploitTxHash = 0x1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef; - -BacktestingTypes.BacktestingResults memory results = executeBacktestForTransaction( - exploitTxHash, - TARGET_CONTRACT, - type(MyAssertion).creationCode, - MyAssertion.assertionInvariant.selector, - vm.envString("MAINNET_RPC_URL") -); - -// Expect the assertion to catch the exploit -assertEq(results.assertionFailures, 1, "Assertion should catch the exploit"); -``` - -### BacktestingConfig - -Configuration struct for backtesting parameters. - -```solidity -struct BacktestingConfig { - address targetContract; // Contract to test - uint256 endBlock; // Latest block in range - uint256 blockRange; // Number of blocks to test - bytes assertionCreationCode; // Assertion contract bytecode - bytes4 assertionSelector; // Assertion function selector - string rpcUrl; // RPC endpoint - bool detailedBlocks; // Detailed logging (reserved) - bool forkByTxHash; // Fork at exact transaction -} -``` - -**Field Descriptions:** - -| Field | Type | Description | -|-------|------|-------------| -| `targetContract` | `address` | Contract address to test assertions against | -| `endBlock` | `uint256` | Most recent block in the test range | -| `blockRange` | `uint256` | Number of blocks to test backwards from `endBlock` | -| `assertionCreationCode` | `bytes` | Bytecode of the assertion contract | -| `assertionSelector` | `bytes4` | Function selector of the assertion to execute | -| `rpcUrl` | `string` | RPC endpoint URL for blockchain data | -| `detailedBlocks` | `bool` | Enable detailed logging (reserved for future use) | -| `forkByTxHash` | `bool` | Fork at exact transaction (kept for interface compatibility) | - -### BacktestingResults - -Results struct returned after backtesting execution. - -```solidity -struct BacktestingResults { - uint256 totalTransactions; // Total transactions found - uint256 processedTransactions; // Transactions validated - uint256 successfulValidations; // Passing validations - uint256 skippedTransactions; // Transactions where assertion wasn't triggered - uint256 assertionFailures; // Protocol violations detected - uint256 replayFailures; // Transactions that failed to replay - uint256 unknownErrors; // Unexpected errors -} -``` - -**Field Descriptions:** - -| Field | Type | Description | -|-------|------|-------------| -| `totalTransactions` | `uint256` | Total number of transactions found in the block range | -| `processedTransactions` | `uint256` | Number of transactions that were validated | -| `successfulValidations` | `uint256` | Transactions that passed validation | -| `skippedTransactions` | `uint256` | Transactions where the assertion wasn't triggered (function selector didn't match) | -| `assertionFailures` | `uint256` | Number of protocol violations detected (assertion reverted) | -| `replayFailures` | `uint256` | Transactions that failed to replay before assertion could execute | -| `unknownErrors` | `uint256` | Unexpected errors during execution | - -**Interpreting Results:** - -The `assertionFailures` field indicates how many transactions triggered assertion reverts. Since backtesting runs against historical transactions, review non-zero values to confirm whether they represent expected assertion behavior, logic that needs tuning, or a known exploit window. - -```solidity -// Check results -if (results.assertionFailures > 0) { - console.log("Assertion failures detected:", results.assertionFailures); - console.log("Review assertion logic, expected behavior, or gas usage"); -} - -// Log detailed results -console.log("Total transactions:", results.totalTransactions); -console.log("Successful validations:", results.successfulValidations); -console.log("Skipped transactions:", results.skippedTransactions); -console.log("Assertion failures:", results.assertionFailures); -``` - -**Common Causes of Assertion Failures:** -- **Assertion tuning needed:** Assertion logic flags legitimate protocol behavior -- **Gas limit exceeded:** Assertion ran out of gas (3,000,000 by default) during complex operations -- **Assertion bugs:** Logic errors in the assertion code -- **Known exploits:** If testing against historical exploit blocks, failures may be expected - -See the [Understanding Results](/credible/backtesting#understanding-results) section in the Backtesting Guide for detailed information on all result categories. - -## Example Configurations - -### Standard Configuration - -Basic configuration suitable for most backtesting scenarios: - -```solidity -BacktestingTypes.BacktestingConfig({ - targetContract: TARGET_CONTRACT, - endBlock: LATEST_BLOCK, - blockRange: 100, - assertionCreationCode: type(MyAssertion).creationCode, - assertionSelector: MyAssertion.assertionInvariant.selector, - rpcUrl: vm.envString("MAINNET_RPC_URL"), - detailedBlocks: false, - forkByTxHash: true -}) -``` - -**See the [Backtesting Guide](/credible/backtesting#common-configurations) for more configuration examples including debugging and quick test scenarios.** - -## Learn More - - - - Autogenerated API documentation for backtesting - - - Learn how to use backtesting to validate assertions - - - Basic testing guide for assertions - - - Automate your testing workflow - - - Use fuzz testing to test assertions - - diff --git a/credible/backtesting.mdx b/credible/backtesting.mdx index 3ec0837..09c6d60 100644 --- a/credible/backtesting.mdx +++ b/credible/backtesting.mdx @@ -1,242 +1,75 @@ --- -title: Backtesting -description: Test assertions against historical blockchain transactions +title: 'How to Review Backtesting Results' +description: 'Review historical transactions that your assertions would have invalidated before you deploy a release' --- -Backtesting runs your assertions against actual historical transactions from a specified block range. This helps ensure your assertions behave as expected with real transaction patterns before deployment. - - -If you're looking for the reference documentation on all configuration parameters, see the [Backtesting Reference](/credible/backtesting-reference). - +This guide shows you how to review the automatic backtest for a release, inspect a matched historical transaction, and decide whether an assertion is ready to deploy or needs further tuning. - -Backtesting is a development-time validation tool. Running a backtest does not deploy assertions, change the on-chain registry, or affect production enforcement. - +**Prerequisites**: -## How It Works - -Backtesting operates in two phases: - -1. **Transaction Fetching**: Identifies all transactions to the target contract in the block range using trace APIs (with automatic fallback) -2. **Transaction Validation**: Replays each transaction with your assertion enabled - -The tool automatically detects both direct calls and internal calls (e.g., when Contract A calls your target Contract B through a router or aggregator) using trace APIs. - -For validation, the tool forks the EVM at the exact transaction state and replays it against the specified assertion. - -## Example - -```solidity -// SPDX-License-Identifier: MIT -pragma solidity ^0.8.20; - -import {CredibleTestWithBacktesting} from "credible-std/CredibleTestWithBacktesting.sol"; -import {BacktestingTypes} from "credible-std/utils/BacktestingTypes.sol"; -import {MyAssertion} from "../assertions/src/MyAssertion.a.sol"; - -contract MyBacktestingTest is CredibleTestWithBacktesting { - function testHistoricalTransactions() public { - BacktestingTypes.BacktestingResults memory results = executeBacktest( - BacktestingTypes.BacktestingConfig({ - targetContract: 0x5fd84259d66Cd46123540766Be93DFE6D43130D7, // USDC on Optimism Sepolia - endBlock: 31336940, // Latest block to test - blockRange: 20, // Number of blocks to test - assertionCreationCode: type(MyAssertion).creationCode, - assertionSelector: MyAssertion.assertionInvariant.selector, - rpcUrl: "https://sepolia.optimism.io", - detailedBlocks: false, - forkByTxHash: true - }) - ); - - // Check results - assertEq(results.assertionFailures, 0, "Found protocol violations!"); - } -} -``` - -Backtesting tests inherit from `CredibleTestWithBacktesting` instead of `CredibleTest`. The `executeBacktest` function returns results in a `BacktestingTypes.BacktestingResults` struct. - - -Consider excluding backtesting from your CI/CD pipeline as it can take time to run. Run backtesting manually on demand to increase confidence in your assertions before deployment. - - -## Common Configurations - -### Standard Configuration - -```solidity -BacktestingTypes.BacktestingConfig({ - targetContract: TARGET_CONTRACT, - endBlock: LATEST_BLOCK, - blockRange: 100, - assertionCreationCode: type(MyAssertion).creationCode, - assertionSelector: MyAssertion.assertionInvariant.selector, - rpcUrl: vm.envString("MAINNET_RPC_URL"), - detailedBlocks: false, - forkByTxHash: true -}) -``` - -### Testing Against a Specific Block - -When you want to test your assertion against a specific known transaction (such as a historical exploit), set `blockRange: 1` with the block number containing that transaction: - -```solidity -BacktestingTypes.BacktestingConfig({ - targetContract: BALANCER_V2_VAULT, - endBlock: 23717632, // Block containing the exploit transaction - blockRange: 1, // Only test this single block - assertionCreationCode: type(BatchSwapDeltaAssertion).creationCode, - assertionSelector: BatchSwapDeltaAssertion.assertionBatchSwapRateManipulation.selector, - rpcUrl: vm.envString("MAINNET_RPC_URL"), - detailedBlocks: false, - forkByTxHash: true -}) -``` - -This is particularly useful for: -- Validating that your assertion catches known exploits -- Testing edge cases found in production -- Verifying assertion behavior on specific problematic transactions - - -See the [Balancer V2 Rate Manipulation Exploit](/assertions-book/previous-hacks/balancer-v2-stable-rate-exploit) for a real-world example of using backtesting to verify an assertion catches a historical exploit transaction. - - -### Testing a Single Transaction by Hash - -For debugging a staging finding or validating against a specific transaction, use `executeBacktestForTransaction`: - -```solidity -bytes32 txHash = 0x1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef; - -BacktestingTypes.BacktestingResults memory results = executeBacktestForTransaction( - txHash, - TARGET_CONTRACT, - type(MyAssertion).creationCode, - MyAssertion.assertionInvariant.selector, - vm.envString("MAINNET_RPC_URL") -); -``` - -This is particularly useful for: -- Debugging staging findings (see [How to Plan Assertion Testing](/credible/testing-strategy)) -- Quick validation of assertion behavior on a specific transaction -- Investigating incidents without needing to know the block number - -## Understanding Results - -When backtesting completes, transactions are categorized into different result types to help you quickly identify issues: - -### Result Categories - -**PASS** - Assertion passed successfully -- Transaction replayed and assertion validated without errors -- **Recommended Action:** None - everything is working as expected - -**SKIP** - Assertion not triggered -- Transaction called a function not monitored by your assertion (function selector didn't match) -- This is normal behavior - your assertion only triggers on specific function calls -- **Recommended Action:** None required, unless you expected this transaction to trigger your assertion -- **Note:** These are NOT assertion violations - they indicate transactions that aren't relevant to your assertion - -**REPLAY_FAIL** - Transaction replay failed -- Transaction failed to replay before the assertion could execute -- Common causes: - - **State dependencies:** Transaction depends on specific state that isn't present - - **Context requirements:** Transaction requires specific block context - - **Insufficient funds:** Sender balance changed between original execution and replay -- **Recommended Action:** Check the error message for details; this usually indicates an environmental issue rather than an assertion problem -- **Note:** These are NOT assertion violations - -**FAIL** - Assertion reverted -- Your assertion reverted when validating this transaction -- **Most Common Causes:** - - **False positive:** Your assertion logic incorrectly flags legitimate protocol behavior (most common) - - **Gas limit exceeded:** Assertion ran out of gas (3,000,000 by default) - - **Assertion bug:** Logic error in your assertion code - - **Known exploit:** If testing against a historical exploit block, this is expected behavior -- **Recommended Action:** - - Check if the transaction is a known exploit - if so, this confirms your assertion works correctly - - Review your assertion logic to ensure it properly handles this transaction pattern - - Check if the assertion is running out of gas and needs optimization - - Verify the transaction on a block explorer to understand what it does - -**ERROR** - Unexpected failure -- An error occurred that doesn't fit other categories -- May indicate RPC issues, assertion bugs, or unexpected contract behavior -- **Recommended Action:** Check the error message and retry; if persistent, [file a bug report](https://github.com/phylaxsystems/credible-std/issues). - -### Reading the Summary - -At the end of each backtest run, you'll see a summary like: - -```text -========================================== - BACKTESTING SUMMARY -========================================== -Block Range: 23697580 - 23697590 -Total Transactions: 15 -Processed Transactions: 15 -Successful Validations: 10 -Skipped Transactions: 3 -Failed Validations: 2 - -=== ERROR BREAKDOWN === -Protocol Violations (Assertion Failures): 1 -Replay Failures (Tx reverted before assertion): 1 - -Success Rate: 83% -================================ -``` - -## Running Tests - -```bash -# Set RPC URL environment variable -export RPC_URL="YOUR_RPC_URL" - -# Run backtesting tests -pcl test --ffi --match-test testHistoricalTransactions -``` - - -The `--ffi` flag is required to enable foreign function interface for RPC calls. - - -## Foundry Configuration - -Enable FFI in your Foundry profile: - -```toml -[profile.backtest-assertions] -src = "assertions/src" -test = "assertions/test/backtest" -out = "assertions/out" -libs = ["lib"] -solc = "0.8.29" -optimizer = true -optimizer_runs = 200 -ffi = true # To avoid setting --ffi in the test command -``` - - -See [CI/CD Integration](/credible/ci-cd-integration#foundry-profile-configuration) for complete Foundry profile configuration. - - -## Best Practices - -1. **Start small** - Test with 10-20 blocks first to verify your setup -2. **Use paid RPC providers** - For block ranges over 1,000 -3. **Check assertion failures** - Ensure `results.assertionFailures == 0` -4. **Run before deployment** - Validate behavior on real transactions -5. **Run manually** - Exclude from CI/CD to avoid long test runs -6. **Debug with single transactions** - Use `executeBacktestForTransaction` to investigate specific failures - -**Learn More:** -- [Backtesting Reference](/credible/backtesting-reference) - Complete configuration details -- [Testing Assertions](/credible/testing-assertions) - Basic testing guide -- [Fuzz Testing](/credible/fuzz-testing) - Test with random inputs -- [Troubleshooting](/credible/troubleshooting) - Common errors and solutions -- [Balancer V2 Rate Manipulation Exploit](/assertions-book/previous-hacks/balancer-v2-stable-rate-exploit) +- A project in the Phylax platform +- A release created with [`pcl apply`](./apply-assertions) +- Access to review that project's releases + +## What backtesting checks + +Backtesting is included in the platform's pre-release checks. When you create a release and open it for review, the platform tests every assertion in the release against transactions from the last 20,000 blocks. The results identify historical transactions that would have been prevented from settling if the current assertion set had been active. + +Use these findings to measure how an assertion behaves against real protocol activity before you deploy it. This is especially important for probabilistic or threshold-based protections, such as outflow circuit breakers and anomaly detection, where a threshold that is too restrictive can block benign behavior and one that is too permissive can miss harmful behavior. + +## Open the release + +In the platform, open your project, select **Releases**, and choose the release you want to review. The release overview shows the proposed assertion changes and three pre-release checks: assertion verification, source code, and backtesting. See [Release Review Checks](./release-review-checks) for the fields and purpose of each check. + +Review the release diff first so you know which contracts and assertions are included. Then check the status beside **Backtesting**. A passed check means the backtest found no historical transaction that the release would have invalidated. **Issue found** means at least one transaction needs your review. + + + Release overview for the MetaMorpho Top 5 Vaults project with assertion verification, source code, and backtesting checks + + +## Review a backtesting finding + +Select **Backtesting** to open the complete result. The summary shows the tested block range and whether the assertions invalidated any historical transactions. When the backtest finds an invalidation, the result identifies the matched transaction hash, assertion ID, block number, assertion adopter, and all assertion IDs watched during the replay. + +Start with the **Matched TX Hash**, **Matched Assertion ID**, and **Matched Assertion Adopter**. Together, these fields tell you which transaction was rejected, which assertion failed, and which protected contract it was evaluating. The attempt history records when the check ran, while **Transactions With Assertions** shows how many transactions in the range exercised the assertion set. + + + Backtesting result for a release showing one historical transaction that would have been invalidated + + +## Inspect the invalidated transaction + +Copy the **Matched TX Hash** and paste it into the block explorer for the project's network. Review the sender and recipient, decoded function call, calldata, transferred value, token transfers, event logs, and internal calls. Compare that activity with the assertion's intended invariant and the matched adopter address. The goal is to explain why the assertion would have rejected a transaction that previously settled: the transaction might contain behavior you want to prevent, or it might reveal a legitimate protocol operation that the assertion does not yet handle. + +Next, inspect the **Matched Incident Payload** in the backtesting result. The failure entry maps the revert data to the assertion function, assertion ID, and adopter address. For assertions that depend on cumulative flows, an outflow window, or anomaly-detection context, also expand the previous transactions and review the block environment. The matched transaction alone may look benign even though the preceding activity caused the assertion's threshold or model to fire. + + + Matched incident payload with revert data, assertion metadata, and previous transactions from the backtest + + +## Decide what to do next + +If the matched transaction represents behavior the assertion is intended to prevent, record why the finding is expected and continue reviewing the other release checks. A historical match can be useful evidence that the assertion detects the targeted behavior, but it does not prove that the assertion covers every harmful transaction. + +If the matched transaction is benign, treat it as a false positive. Adjust the assertion's thresholds, trigger scope, time window, or handling of normal protocol operations, then add the transaction as a regression case in your assertion tests. Create a new release with `pcl apply`; the platform will run the pre-release checks and backtest the updated assertion set again. + +If the matched transaction and assertion metadata do not explain the finding, inspect the **Matched Incident Payload** and expand the previous transactions. Decode the revert data, review the block environment, and confirm that the assertion ran against the intended adopter and state. Do not deploy the release until you can explain the finding from the transaction, assertion, and platform evidence. See [Troubleshooting](./troubleshooting) for common assertion failures. + +When the backtest passes, finish reviewing assertion verification and source code, then deploy the release to staging. Continue monitoring staging invalidations before promoting the assertion to production. A passing backtest confirms that the tested historical activity did not trigger the release; it is not a substitute for focused tests, known-exploit coverage, or staging observation. + +## Next steps + + + + Add focused and regression tests before creating another release + + + Create a new release after tuning an assertion + + + Deploy a reviewed release to staging + + + Review assertion verification, source code, and backtesting + + diff --git a/credible/ci-cd-integration.mdx b/credible/ci-cd-integration.mdx index 49551e0..cd68b6b 100644 --- a/credible/ci-cd-integration.mdx +++ b/credible/ci-cd-integration.mdx @@ -61,7 +61,7 @@ Use the `FOUNDRY_PROFILE: assertions` environment variable to ensure the correct ## Foundry Profile Configuration -Foundry profiles allow you to configure different compilation and testing settings for different parts of your project. For assertions, you typically want separate profiles for different types of tests to avoid running slow tests (like fuzz tests and backtests) every time you want to run quick unit tests locally. +Foundry profiles allow you to configure different compilation and testing settings for different parts of your project. For assertions, you typically want separate profiles for unit and fuzz tests so slower suites do not run every time you want quick local feedback. This configuration goes in your project's `foundry.toml` file, which you should be familiar with if you have used Foundry before. @@ -74,8 +74,7 @@ assertions/ ├── src/ # Assertion source code └── test/ ├── unit/ # Fast unit tests - ├── fuzz/ # Fuzz tests (slower) - └── backtest/ # Backtesting tests (slowest) + └── fuzz/ # Fuzz tests (slower) ``` ### Example foundry.toml @@ -119,17 +118,6 @@ solc = "0.8.29" optimizer = true optimizer_runs = 10000 -# Backtesting profile (requires FFI) -[profile.backtest-assertions] -src = "assertions/src" -test = "assertions/test/backtest" -out = "assertions/out" -libs = ["lib"] -solc = "0.8.29" -optimizer = true -optimizer_runs = 200 -ffi = true # Required for backtesting - # Fuzz configuration (applies to all profiles) [fuzz] runs = 256 @@ -140,7 +128,6 @@ max_test_rejects = 65536 - **Unit tests (`unit-assertions`)** - Fast feedback during development - **Fuzz tests (`fuzz-assertions`)** - Run separately when testing edge cases -- **Backtests (`backtest-assertions`)** - Run manually before deployment - **General (`assertions`)** - Runs all tests for comprehensive validation When using `pcl test` or setting `FOUNDRY_PROFILE=`, Foundry uses that profile's configuration instead of the default one. @@ -156,9 +143,6 @@ FOUNDRY_PROFILE=unit-assertions pcl test # Run fuzz tests FOUNDRY_PROFILE=fuzz-assertions pcl test -# Run backtests (slowest, run before deployment) -FOUNDRY_PROFILE=backtest-assertions pcl test --ffi - # Run all assertion tests FOUNDRY_PROFILE=assertions pcl test ``` @@ -176,4 +160,5 @@ If nothing is specified, the default profile is used. **Learn More:** - [Testing Assertions](/credible/testing-assertions) - Basic testing guide - [Fuzz Testing](/credible/fuzz-testing) - Test with random inputs +- [Review Backtesting Results](/credible/backtesting) - Review the automatic backtest attached to a platform release - [Foundry Book - Testing](https://book.getfoundry.sh/forge/tests) diff --git a/credible/credible-std-overview.mdx b/credible/credible-std-overview.mdx index 3bcbdb9..780522f 100644 --- a/credible/credible-std-overview.mdx +++ b/credible/credible-std-overview.mdx @@ -3,7 +3,7 @@ title: "credible-std Library" description: "Reference for credible-std contracts, helpers, specs, and testing utilities" --- -`credible-std` is the Solidity standard library for writing, testing, and backtesting Credible Layer assertions. It provides the base assertion contract, the `ph` interface for PhEVM cheatcodes, trigger registration helpers, snapshot-read helpers, typed state-change helpers, and Forge test utilities. +`credible-std` is the Solidity standard library for writing and testing Credible Layer assertions. It provides the base assertion contract, the `ph` interface for PhEVM cheatcodes, trigger registration helpers, snapshot-read helpers, typed state-change helpers, and Forge test utilities. For individual cheatcode signatures, see the [Cheatcodes API Reference](./cheatcodes-reference). For local assertion tests, see [Testing Assertions](./testing-assertions). @@ -22,18 +22,6 @@ The current Phylax app and Linea deployment support the V1 assertion specificati | Development dependency | `phylaxsystems/credible-std` | | Foundry remapping | `credible-std/=lib/credible-std/src/` | -Backtesting helpers require `ffi = true` in the relevant Foundry profile: - -```toml -[profile.backtesting] -src = "src" -out = "out" -libs = ["lib"] -ffi = true -gas_limit = 100000000 -test = "test" -``` - ## Core Contracts | Contract | Purpose | @@ -46,7 +34,6 @@ test = "test" | `StateChanges.sol` | Typed helpers for legacy state-change arrays. | | `ForkUtils.sol` | Helper functions for Reshiram-style snapshot reads and ERC20 balance delta workflows. | | `CredibleTest.sol` | Forge test base that exposes `cl.assertion(...)`. | -| `CredibleTestWithBacktesting.sol` | Test base for replaying historical transactions against assertions. | ## Assertion Helpers @@ -207,8 +194,6 @@ interface VmEx { } ``` -`CredibleTestWithBacktesting` extends the local test base for historical transaction replay. It looks for `credible-std/scripts/backtesting/transaction_fetcher.sh`; `CREDIBLE_STD_PATH` overrides the default dependency lookup path. - ## Related Pages @@ -218,8 +203,8 @@ interface VmEx { How to test assertions locally - - Backtesting configuration and behavior + + Review the automatic backtest attached to a platform release Build a first assertion with credible-std diff --git a/credible/dapp-projects.mdx b/credible/dapp-projects.mdx index 7d4a1e7..dc4b742 100644 --- a/credible/dapp-projects.mdx +++ b/credible/dapp-projects.mdx @@ -63,22 +63,14 @@ Once created, you can manage your project through the project dashboard: ### Deploy the Release Onchain - Review the release and link assertions to protected contracts -- Choose Staging or Production +- Confirm whether the release targets Staging or Production - Sign the deployment transaction Release overview showing the environment, status, verification checks, and assertions added by the release - - Assertion verification showing a passed assertion, its adopter contract, and function-call triggers - - - - Release backtesting result showing a historical transaction that would have been invalidated by the assertion set - - -For detailed instructions, see the [Deploy Assertions guide](/credible/deploy-assertions-dapp#step-4-deploy-the-assertion). +The platform checks assertion configuration, submitted source, and behavior against historical transactions before deployment. See [Release Review Checks](./release-review-checks) for the purpose and fields of each check, [How to Review Backtesting Results](./backtesting) for investigating a matched transaction, and [Deploy Assertions with the Platform](./deploy-assertions-dapp#step-4-deploy-the-assertion) for the complete deployment workflow. ### Project Settings diff --git a/credible/deploy-assertions-dapp.mdx b/credible/deploy-assertions-dapp.mdx index 86ac20b..9e1067b 100644 --- a/credible/deploy-assertions-dapp.mdx +++ b/credible/deploy-assertions-dapp.mdx @@ -79,22 +79,19 @@ After applying via `pcl`, complete deployment in the platform: 1. Navigate to your project -2. Click the "Deployment" tab +2. Select **Releases** -3. Find your assertion and click "Proceed to Review" +3. Choose the release created by `pcl apply` -4. Review the release diff +4. Review the release diff and confirm the target contracts and assertions -5. Review assertion details: - - Assertion name and parameters - - Target contract address - - Trigger conditions +5. Review **Assertion verification**, **Source code**, and **Backtesting**. See [Release Review Checks](./release-review-checks) for the fields and decisions in each check. -6. Choose environment: - - **Staging**: For testing (assertions run but don't block transactions) - - **Production**: For active protection (assertions block violating transactions) +6. Confirm the environment declared by the release: + - **Staging**: Assertions run but do not block transactions + - **Production**: Assertions block violating transactions -7. Click "Deploy" and sign the transaction +7. Connect the protocol manager wallet, select **Deploy to Staging** or **Deploy to Production**, and sign the transaction Deploying Assertion diff --git a/credible/execution-model.mdx b/credible/execution-model.mdx index b825aee..e82f6e2 100644 --- a/credible/execution-model.mdx +++ b/credible/execution-model.mdx @@ -154,7 +154,7 @@ protocol.withdraw(50); // Results in: "Expected 1 assertion to be executed, but 0 were executed" ``` -To verify triggers work correctly with real transactions, use [backtesting](/credible/backtesting). +To verify trigger behavior against recent real transactions, create a release and review its [automatic platform backtest](/credible/backtesting). ## Related Documentation @@ -166,7 +166,7 @@ To verify triggers work correctly with real transactions, use [backtesting](/cre Production trigger behavior - Test with real transactions + Review historical findings attached to a platform release Common errors diff --git a/credible/glossary.mdx b/credible/glossary.mdx index 63e6a32..7584f03 100644 --- a/credible/glossary.mdx +++ b/credible/glossary.mdx @@ -144,7 +144,7 @@ The PCL testing interface that extends Forge's testing capabilities with additio Learn more about testing assertions in the [Testing Guide](/credible/testing-assertions). ### Backtesting -A testing method that runs assertions against historical blockchain transactions from a specified block range. Backtesting helps ensure assertions behave as expected with real transaction patterns before deployment. +An automatic platform release check that tests every assertion in a release against transactions from the last 20,000 blocks. The result identifies historical transactions that the release would have invalidated so you can review false positives before deployment. Learn more: [Backtesting](/credible/backtesting) diff --git a/credible/interfaces-overview.mdx b/credible/interfaces-overview.mdx index cefd789..885bd13 100644 --- a/credible/interfaces-overview.mdx +++ b/credible/interfaces-overview.mdx @@ -42,7 +42,7 @@ The Credible Layer combines protocol components with Phylax-operated product sur | Assertion Enforcer | Executes active assertions and returns inclusion decisions during block production | Network enforcement | | Assertion DA | Stores assertion bytecode and source code for enforcers and public verification | Phylax-operated storage today | | `pcl` CLI | Builds, tests, authenticates, and submits assertion releases | Local developer tooling | -| Backtesting | Replays assertions against historical transactions before promotion | Phylax-operated development service | +| Backtesting | Tests a platform release against transactions from the last 20,000 blocks before deployment | Phylax-operated release review service | | Staging | Runs assertions against live network activity without dropping transactions | Phylax-operated validation environment | | Platform | Manages releases, invalidations, audit logs, collaborators, alerts, and project metadata | Phylax product | diff --git a/credible/invariant-to-assertion.mdx b/credible/invariant-to-assertion.mdx index ebadaab..86cda60 100644 --- a/credible/invariant-to-assertion.mdx +++ b/credible/invariant-to-assertion.mdx @@ -338,7 +338,7 @@ These are basic sanity checks. For production assertions, test edge cases like z See the [full test file](https://github.com/czepluch/ethereum-vault-connector/blob/master/assertions/test/unit/VaultSharePriceAssertion.t.sol) for the complete test suite covering all entry points and edge cases. -For thorough testing guidance including fuzz testing and backtesting against historical transactions, see [Testing Assertions](/credible/testing-assertions), [Fuzz Testing](/credible/fuzz-testing), and [Backtesting](/credible/backtesting). +For thorough testing guidance, see [Testing Assertions](/credible/testing-assertions), [Fuzz Testing](/credible/fuzz-testing), and [How to Review Backtesting Results](/credible/backtesting) for the automatic check attached to a platform release. ## Recap diff --git a/credible/release-review-checks.mdx b/credible/release-review-checks.mdx new file mode 100644 index 0000000..cbca000 --- /dev/null +++ b/credible/release-review-checks.mdx @@ -0,0 +1,79 @@ +--- +title: 'Release Review Checks' +description: 'Reference for assertion verification, source code, and backtesting checks attached to a platform release' +--- + +Release review checks report whether the assertions and submitted source in a release were verified and how the assertion set behaved against historical transactions. Use these results with the release diff before you authorize a staging or production deployment. + +## Checks at a glance + +| Check | What the platform reports | What you should confirm | +| --- | --- | --- | +| **Assertion verification** | Verification status, assertion ID, adopter, and registered triggers for each assertion contract | The release targets the intended contracts and every assertion uses the intended triggers | +| **Source code** | Source verification status for each submitted assertion ID | Every assertion in the release has a passing source result | +| **Backtesting** | Results from testing the release against transactions from the last 20,000 blocks | Any historical transaction that would have been invalidated is expected behavior rather than a false positive | + +These checks answer different questions. A passing result in one check does not replace review of the others. + +## Assertion verification + +The **Assertion verification** check reports the assertion contracts included in the release. Its summary shows the total, passed, and failed assertion counts. Each detail entry contains the following fields: + +| Field | Description | +| --- | --- | +| **Assertion ID** | Identifier for the assertion contract included in the release | +| **Status** | Verification result for that assertion | +| **Adopter** | Protected contract to which the assertion will be applied | +| **Trigger** | Registered function, selector, assertion trigger, or other trigger identifier | +| **Trigger type** | Trigger category, such as function call, transaction end, or cumulative outflow | +| **Used by** | Assertion function registered to the trigger, when applicable | + +Compare the assertion count, IDs, and adopter addresses with the release diff. Then inspect every trigger. A passing status confirms that the platform verified the submitted assertion configuration; it does not confirm that the invariant or trigger selection protects the intended behavior. + + + Assertion verification result showing a passed assertion, assertion ID, adopter contract, and function-call triggers + + +## Source code + +The **Source code** check reports the verification status for the submitted assertion source. The result contains one entry per assertion, with its **Assertion ID** and a **Passed** value. + +Match the assertion IDs against the assertion-verification result and release diff. Every assertion expected in the release should have a corresponding passing source entry. A passing source result does not mean that the assertion has been audited or that its business logic is correct; it reports only the platform's verification result for the submitted source. + +## Backtesting + +The **Backtesting** check tests every assertion in the release against transactions from the last 20,000 blocks. A passing result shows the tested block range, watched assertion IDs, number of transactions that exercised the assertions, and attempt history. + +An **Issue found** result additionally identifies the matched transaction, assertion, block, and adopter. It also includes the incident payload, previous transactions, block environment, and replayed transaction data needed to investigate the finding. + +See [How to Review Backtesting Results](./backtesting) for the complete investigation workflow in the platform. + +## Interpreting the overall review + +A release is ready for deployment review only after you have checked the release diff and examined all three results: + +- **All checks pass:** Confirm the target environment and continue to the wallet authorization step. +- **Assertion verification fails:** Identify the failed assertion, review its adopter and trigger details, and resolve the reported verification problem before creating another release. +- **Source code fails:** Identify the assertion ID without a passing source result and resolve the reported source-verification problem before creating another release. +- **Backtesting finds an issue:** Inspect and classify every matched transaction before deciding whether to tune the assertion or continue. + + +Passing release checks reduce deployment risk, but they do not prove that an assertion is correct, complete, or safe for production. Continue with focused tests and staging observation before production deployment. + + +## Related documentation + + + + Create a release with `pcl apply` + + + Investigate historical transactions matched by a release + + + Authorize a reviewed release for staging or production + + + Validate assertion behavior before deployment + + diff --git a/credible/testing-assertions.mdx b/credible/testing-assertions.mdx index 92ef5b0..ad755dd 100644 --- a/credible/testing-assertions.mdx +++ b/credible/testing-assertions.mdx @@ -6,7 +6,7 @@ description: Validate assertions locally with `CredibleTest`, realistic test pat Use this guide when you want to validate an assertion locally before staging or production. -For the higher-level testing workflow, see [How to Plan Assertion Testing](./testing-strategy). For historical transaction configuration fields, see the [Backtesting Reference](./backtesting-reference). +For the higher-level testing workflow, see [How to Plan Assertion Testing](./testing-strategy). After your local tests pass, create a release and review its [automatic platform backtest](./backtesting). Testing assertions is very similar to writing regular Forge tests. If you're familiar with Forge testing, you'll feel right at home. You get access to all the same testing utilities: diff --git a/credible/testing-strategy.mdx b/credible/testing-strategy.mdx index ba86633..333cf76 100644 --- a/credible/testing-strategy.mdx +++ b/credible/testing-strategy.mdx @@ -1,16 +1,16 @@ --- title: 'How to Plan Assertion Testing' -description: 'Plan local testing, backtesting, staging, and production promotion for assertions' +description: 'Plan local testing, platform release review, staging, and production promotion for assertions' --- -This guide shows you how to allocate assertion testing effort across local tests, backtesting, staging, and production promotion. +This guide shows you how to allocate assertion testing effort across local tests, platform release review, staging, and production promotion. **Summary** - Local testing validates core logic; staging validates real-world behavior - Deploy to staging early to observe behavior against real transaction flow - Real transactions expose scenarios that are difficult to predict locally -- Use backtesting to debug specific transactions with the smallest feedback loop +- Review the automatic release backtest before every staging or production deployment ## The Testing Philosophy @@ -34,7 +34,7 @@ Real-world transactions reveal edge cases that would take significant effort to | Phase | Effort | Goal | |-------|--------|------| | Local testing | ~20-30% | Validate core logic, catch obvious bugs, verify gas limits | -| Backtesting | ~10-20% | Sanity check before staging; debug specific transactions after | +| Release review | ~10-20% | Review verification, source code, and automatic backtesting before deployment | | Staging | ~50-60% | Discover edge cases, observe real-world behavior, iterate | The bulk of your validation happens in staging. Local testing catches the obvious issues quickly so you can deploy with confidence, but staging is where you discover the edge cases. @@ -62,47 +62,28 @@ See [Testing Assertions](/credible/testing-assertions) for local testing pattern **Leverage existing invariant tests.** If your protocol already has Forge invariant tests, the test setup and infrastructure can often be reused for assertion testing. This can significantly reduce the time spent on local test configuration. -## Backtesting: Two Key Uses +## Review platform backtesting before staging -Backtesting serves two distinct purposes in the development workflow: +When you create a release with `pcl apply`, the platform runs three release checks: assertion verification, source code, and backtesting. The backtesting service tests every assertion in the release against transactions from the last 20,000 blocks. -### 1. Pre-Staging Sanity Check +Open the release in the platform and review the backtesting result before you authorize a deployment. A passing result means the release did not invalidate any transaction in the tested window. An **Issue found** result identifies a historical transaction that the assertion would have prevented from settling. -Before deploying to staging, run a quick backtest against recent transactions: +For each finding, inspect the matched transaction, assertion ID, adopter, incident payload, previous transactions, and block environment. If the transaction is benign, tune the assertion and create a new release so the platform can check the updated assertion set. If you cannot explain a finding, do not deploy the release. -- Verifies expected behavior on normal protocol operations -- Catches trigger mismatches (assertion not running when expected) -- Identifies gas issues with realistic transaction complexity - -Keep block ranges small (10-100 blocks) for this sanity check. The goal is quick verification, not comprehensive coverage. - -For production promotion, expect a broader historical review. The exact window depends on protocol risk, transaction volume, network requirements, and the assertions being promoted. At minimum, include representative high-volume periods, known incident windows when applicable, and enough normal activity to measure assertion behavior on normal transactions. - -### 2. Debugging Staging Issues - -When staging reveals unexpected behavior, backtesting becomes your debugging tool: - -- Replay the specific transaction that triggered the issue locally -- Step through assertion logic with full visibility -- Fastest feedback loop for debugging - -This is often faster than trying to reproduce the scenario from scratch in a local test. - - -See [Backtesting](/credible/backtesting) for configuration and usage details. - +See [How to Review Backtesting Results](./backtesting) for the investigation workflow and [Release Review Checks](./release-review-checks) for the complete release-check reference. ## The Staging Workflow ```mermaid flowchart LR - LOCAL[Local Testing] --> BACKTEST[Backtesting
sanity check] - BACKTEST --> STAGING[Deploy to
Staging] + LOCAL[Local Testing] --> APPLY[Create a Release] + APPLY --> REVIEW[Review Platform Checks
and Backtesting] + REVIEW --> STAGING[Deploy to
Staging] STAGING --> MONITOR[Monitor] MONITOR --> ISSUE{Issues
Found?} - ISSUE --> |Yes| DEBUG[Backtest
specific tx] - DEBUG --> FIX[Fix & Redeploy] - FIX --> STAGING + ISSUE --> |Yes| DEBUG[Inspect the
Invalidation] + DEBUG --> FIX[Fix & Create
New Release] + FIX --> REVIEW ISSUE --> |No, over weeks| PROD[Promote to
Production] style PROD fill:#27ae60,color:#fff @@ -110,7 +91,7 @@ flowchart LR ### Deploy Early -Once local tests pass and backtesting shows no obvious issues, [deploy to staging](/credible/deploy-assertions-dapp). Don't wait for "perfect" local coverage—staging will reveal what you missed. Note that deployments have a [timelock](/credible/glossary#timelock) period before assertions become active, so starting early is beneficial. +Once local tests pass and you have reviewed every platform release check, [deploy to staging](/credible/deploy-assertions-dapp). Don't wait for "perfect" local coverage—staging will reveal what you missed. Note that deployments have a [timelock](/credible/glossary#timelock) period before assertions become active, so starting early is beneficial. ### Monitor Staging Behavior @@ -121,17 +102,17 @@ Check the [platform dashboard](https://app.phylax.systems) regularly for asserti Staging runs assertions but does NOT drop transactions, so you can observe behavior without risk. -### Debug with Backtesting +### Investigate staging invalidations When an issue appears in staging: 1. Identify the transaction that triggered unexpected behavior -2. Use backtesting to replay that exact transaction locally -3. Debug with full visibility into assertion execution (`-vvvv`) -4. Fix the issue and redeploy +2. Open the invalidation in the platform and inspect its assertion result, trace, and execution context +3. Add a focused local regression test that reproduces the assertion behavior +4. Fix the issue, create a new release, and review its automatic backtesting result -**Add regression tests for discovered edge cases.** When staging or backtesting reveals an unexpected scenario, create a local unit test that reproduces the behavior. This prevents regressions and builds a test suite informed by real-world usage patterns rather than speculation. +**Add regression tests for discovered edge cases.** When staging or platform backtesting reveals an unexpected scenario, create a local unit test that reproduces the behavior. This prevents regressions and builds a test suite informed by real-world usage patterns rather than speculation. ### Recommended Staging Duration @@ -148,7 +129,7 @@ Before promoting to production, verify: - Expected behavior observed on legitimate transactions - Sufficient transaction diversity to build confidence -- Historical backtesting covers the relevant protocol surfaces and known risky periods +- You reviewed every finding from the platform's automatic release backtest - Trigger frequency, runtime, and gas usage fit the network's production limits - You understand how the assertion behaves across different scenarios @@ -160,7 +141,7 @@ Moving to production means assertion results can affect transaction inclusion ac **Over-investing in local testing:** Trying to cover every edge case locally is inefficient. Staging does this better with real transactions. -**Skipping backtesting entirely:** A quick backtest catches obvious issues before staging and is essential for debugging staging discoveries. +**Skipping the release backtest:** Review the automatic platform result and investigate every finding before authorizing deployment. **Deploying directly to production:** Always validate in staging first. Production should only use assertions that have passed staging review. @@ -173,7 +154,7 @@ Moving to production means assertion results can affect transaction inclusion ac Local testing patterns and best practices - Test against historical transactions + Review the automatic backtest attached to a platform release Deploy assertions to staging or production diff --git a/credible/troubleshooting.mdx b/credible/troubleshooting.mdx index 92c10f0..103e060 100644 --- a/credible/troubleshooting.mdx +++ b/credible/troubleshooting.mdx @@ -21,11 +21,11 @@ This guide helps you diagnose and fix common errors when writing and testing ass ### "Expected 1 assertion to be executed, but 0 were executed" -This error means your assertion never ran on the target transaction. This can occur in production, testing, or backtesting. +This error means your assertion never ran on the target transaction. This can occur in local tests, staging, production, or a platform release backtest. **Common Causes:** -**In production or backtesting:** +**In staging, production, or a platform release backtest:** - The transaction that triggers the assertion fails or reverts before the assertion runs - The function selector in the trigger doesn't match the function being called @@ -115,7 +115,7 @@ Assertion execution has a default **3,000,000 gas limit**. Node operators can co **Symptoms:** - `OutOfGas` error when running tests -- "Unknown Revert Reason" with gas cost near the active assertion limit in backtesting results +- "Unknown Revert Reason" with gas cost near the active assertion limit in a platform release backtesting result - Assertion fails consistently on complex transactions **Common Causes:** @@ -324,15 +324,11 @@ FOUNDRY_PROFILE=assertions pcl test --match-contract ContractName - Check `foundry.toml` has assertions profile configured - Verify `remappings.txt` includes `credible-std/=lib/credible-std/src/` -## Backtesting Errors +## Release backtesting findings -For backtesting-specific errors, see the [Backtesting Guide](/credible/backtesting#understanding-results). +The platform runs backtesting automatically when you create a release. An **Issue found** result means at least one historical transaction from the last 20,000 blocks would have been invalidated by the release. -**Common Backtesting Issues:** - -- **ASSERTION_FAIL** - Assertion reverted (may be false positive, [gas limit exceeded](#gas-limit-exceeded), or known exploit) -- **UNKNOWN_ERROR** - Unexpected failure (check error message, may be RPC issue) -- **NEEDS_REVIEW** - Transaction called a function not monitored by your assertion, or failed to replay +Open **Backtesting** in the release and review the matched transaction hash, assertion ID, adopter, incident payload, previous transactions, and block environment. If the finding is benign, tune the assertion and create a new release. If the evidence does not explain the finding, do not deploy the release. See [How to Review Backtesting Results](./backtesting) for the complete workflow. ## Debugging diff --git a/credible/trust-model.mdx b/credible/trust-model.mdx index c730745..69db385 100644 --- a/credible/trust-model.mdx +++ b/credible/trust-model.mdx @@ -57,7 +57,7 @@ Useful policy questions include: ## Assurance Practices -Teams build confidence by validating assertions locally, backtesting against historical transactions, running assertions in staging, and promoting to production only after sufficient review. See [How to Plan Assertion Testing](/credible/testing-strategy) for the recommended workflow. +Teams build confidence by validating assertions locally, reviewing the automatic backtesting check attached to each platform release, running assertions in staging, and promoting to production only after sufficient review. See [How to Plan Assertion Testing](/credible/testing-strategy) for the recommended workflow. ## Next Steps diff --git a/docs.json b/docs.json index 0c47dc0..15d64e4 100644 --- a/docs.json +++ b/docs.json @@ -69,6 +69,7 @@ "pages": [ "credible/invariant-to-assertion", "credible/apply-assertions", + "credible/release-review-checks", "credible/deploy-assertions-dapp", "credible/manual-verification" ] @@ -155,7 +156,6 @@ "pages": [ "credible/cli-reference", "credible/cheatcodes-reference", - "credible/backtesting-reference", "credible/credible-std-overview" ] }, diff --git a/images/backtesting-payload.png b/images/backtesting-payload.png new file mode 100644 index 0000000..5379625 Binary files /dev/null and b/images/backtesting-payload.png differ diff --git a/images/backtesting-ui.png b/images/backtesting-ui.png new file mode 100644 index 0000000..c352784 Binary files /dev/null and b/images/backtesting-ui.png differ diff --git a/images/release-overview.png b/images/release-overview.png new file mode 100644 index 0000000..c7bd358 Binary files /dev/null and b/images/release-overview.png differ