feat: Add Solparq - #55
Merged
Merged
Conversation
… archives - Implemented CLI structure using Clap for commands: list, summary, schema, and scan. - Added configuration and argument parsing for local and S3 archive sources. - Developed functionality to summarize archives, output schemas, and scan for transaction rows. - Introduced output formatting for JSON, JSONL, and CSV. - Created tests for CLI argument validation and archive operations. - Updated README with usage instructions for the new solparq-read tool.
…pport - Added support for new archive tables: entries, gsfa_hot, and token_owner_activity. - Introduced ArchiveManifest and ArchiveManifestTable for better tracking of archived data. - Updated DbTables to include new tables and their configurations. - Enhanced the ClickHouse client to handle streaming and counting rows for new tables. - Implemented logic to create and write manifests for local and S3 archive locations. - Improved error handling and logging for table checks and archiving processes. - Updated tests to cover new functionality and ensure proper integration of new tables.
…w and update versions to 0.3.0
…anagement - Changed the naming convention of the done marker files to include a .txt extension. - Improved the logic for handling existing archive bundles without done markers, ensuring they are removed before recreating the archive. - Introduced a new ArchiveRunTable struct to encapsulate details about archive tables in the ArchiveRunReport. - Enhanced the dashboard rendering to include a list of detected archive tables. - Added tests for the new functionality, including the removal of incomplete archive bundles and rendering of S3 output locations.
…olparq-initial
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…olparq-initial
- Added a comprehensive set of metrics to track the health and performance of the archiving process, including gauges, counters, and histograms. - Introduced new structs for labeling metrics based on archive kind, validation categories, and known gaps. - Implemented methods to observe and export metrics, ensuring they are updated during archive operations. - Enhanced the AppState struct to maintain metrics and health status, allowing for better monitoring and diagnostics. - Updated the server to expose metrics in the OpenMetrics format and improved health check responses. - Added tests to validate the metrics endpoint and ensure correct behavior across various scenarios, including error handling and archive skips.
- Added functionality to repair overcount transaction mismatches in ClickHouse via deduplication of partitions. - Introduced `MismatchRepair` struct to track repair outcomes, including optimized partitions and slot counts before and after repair. - Enhanced `ValidationReport` to categorize transaction mismatches into undercount and overcount ranges. - Updated metrics to expose counts of mismatched slots and repair outcomes for monitoring. - Modified configuration to allow enabling mismatch repair and specify local transaction tables for clustered deployments. - Added tests to validate the new mismatch direction logic and ensure metrics are correctly reported.
…shboard and Prometheus
- Added `--dry-run` flag to allow planning and validating archives without making any changes. - Implemented logic to skip ClickHouse deletions and archive cleanups during dry runs. - Enhanced reporting to indicate when an archive would be created without actual file writes. - Updated metrics to classify dry run skips and prevent inflating row counters. - Introduced build identity metrics to expose version and git SHA in the dashboard and metrics endpoint. - Updated README and documentation to reflect new dry run functionality and build info.
… update manifest structure
…cally-distinct rows
… solparq-initial
Mctursh
reviewed
Jul 20, 2026
Mctursh
left a comment
Mctursh
left a comment
Contributor
There was a problem hiding this comment.
Went through this focused on the data paths (archive, restore, delete). The structure is clean and the local staging + done-marker resume is nice. Three things I'd want fixed before merge, all on the S3 and delete side. Left them inline.
--delete-archived-data-rangecan delete slots that were never archived.- The S3 restore matches columns by position while the local restore matches by name, so S3 can silently mis-column on schema drift.
- A failed S3 export logs the query, and the query has the S3 access key and secret in it.
Rest of the crate read solid, so these are pretty contained.
Contributor
Author
Hey thanks for the feedback, I'll review and include, thanks. |
Contributor
|
@Mctursh your comments are valid, but we're going to push a separate PR for them so we can address them without blocking this. We'll make sure they get in to v0.6.0! |
This was referenced Jul 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request adds the new
superbank-solparqpackage to the workspace, which provides Parquet archiving and reading capabilities for Superbank ClickHouse tables. It updates project documentation, build configuration, and CI to support the new binaries and their release process, and introduces a set of engineering guidelines for the new package.Major additions and changes:
1. New Parquet archiver and reader package
crates/superbank-solparqto the workspace, which builds two binaries:superbank-solparq(the archiver daemon) andsuperbank-solparq-read(Parquet archive reader). The package includes its ownCargo.toml, abuild.rsfor embedding version/git info, and a dedicatedAGENTS.mdwith engineering rules. [1] [2] [3] [4]2. Build and release pipeline updates
.goreleaser.yamlto build, package, and archive the newsuperbank-solparqandsuperbank-solparq-readbinaries for both x86_64 and aarch64 Linux. [1] [2].github/workflows/build-binaries.ymlto upload the new binaries as artifacts, and updated release workflow to exclude certain tags. [1] [2]3. Documentation and project structure
README.mdandAGENTS.mdto document the new package, its binaries, and usage (including restore and inspection of Parquet archives). Also updated development/build instructions and repository layout. [1] [2] [3] [4] [5] [6] [7]Cargo.tomlto include the new package.These changes collectively introduce Parquet archive support to Superbank, improve developer workflow, and ensure the new functionality is documented and included in builds and releases.