Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Fix Fog View Connection Cargo.toml dependencies * Implement create_multi_view_query_data API for Fog View Router (#2115) * Implement create_multi_view_store_query_data API * Address remoun's comments * Implement MultiViewStoreQuery API for Fog View (#2133) * Create MultiViewStoreQuery API for Fog View Store * Implement multi_view_store_query in Fog View Store * Use mut_decryption_error * Implement enclave APIs that facilitate Fog View Router dynamic store discovery (#2200) * Add client_init and client_connect enclave methods * Move multiview query construction to ake enclave * Implement Remoun's suggestions * Implement second round of remoun's suggestions * Remove duplicate wasi files * Re-add older wasi version * Implement query / dynamic shard component discovery in Fog View Router (#2189) * Remove FogViewStoreUri * Implement core Fog Router Service logic * Refactor module structure and add tests * Use where clauses * Implement nick's suggestions * Implement second round of nick's suggestions * Implement lint suggestions * Implement james's suggestions * Fix naming error * Fix whitespace * Split Fog View API into Client-facing and Store (Fog Router) APIs (#2265) * Add FogViewStoreUri * Split Fog View API in two and implement * Implement nick's suggestions * Create helper methods to obliviously collate shard responses (#2252) * Create helper methods that obliviously collate shard responses * Implement chris's suggestions * Fix error string * Fix lint * Sort dependencies * Run format * fix newline * Collate shard response in constant time (obliviously) (#2250) * Add backend decrypt method to ake enclave * Add enclave methods for shard response collation * Refactor response to always include view store uri * Use shard responses to fill router response * Implement Nick's suggestions and fix CI errors * Implement epoch sharding via trait (#2350) * Use sharding strategy to tell view what blocks to process (#2351) * Return minimum highest processed block to client (#2387) * Add config option for sharding strategy (#2352) * Add readiness API to ShardingStrategy (#2353) * Add readiness API to ShardingStrategy * Implement readiness check per ShardingStrategy * Change MVQR error to accommodate different types * Set not ready message * Pull out query processing logic to separate method * Add report fetching logic to FVR (#2454) * Fix responder ID usage in FVR (#2446) * Create different types of View Servers for store and client-facing (#2447) * Messages for noise protocol exchanges with explicit nonces (#2461) * attest::NonceMessage, EnclaveNonceMessage, add handling to attest-ake * Rustfmt fixes * Minor DRY cleanups around sessions and auth messages (#2462) * attest::NonceMessage, EnclaveNonceMessage, add handling to attest-ake * Rustfmt fixes * Minor DRY cleanups around sessions and auth messages * Update FVR client for integration test (#2448) * Create gRPC FogViewRouterAdminAPI (#2360) * Create FogViewRouterAdminUri (#2361) * update cargo lock * Sealed messages for clients with backends (#2449) * Implement client message to sealed message API for backends * Update Fog View enclave to build with new AKE enclave API * Rust type aliases don't protect against passing the aliased type, use a new type instead * Re-add comment ASCII art * Additional plumbing of new SealedClientMessage type * Move SealedClientMessage, finish implementing in fog view router * Properly update the lockfiles * Fix clippy lint * Update lockfiles again after rebase Co-authored-by: Andrew Wygle <[email protected]> * Implement shard management API sans auth (#2354) * Implement shard management API * Implement James's suggestions * Re-apply 2463. (#2547) * Re-apply PR #2473 (was accidentally removed) (#2548) * Reapply changes from #2473 * Add inbound frontend connection support crypto-ake-enclave (#2549) * Add frontends * Fix traits on NonceSession * Make constants and comments slightly friendlier/more consistent (#2550) * Re-apply 2463. (#2547) * Fix traits on NonceSession * Minor comment and constant consistency/friendliness changes * Fix compile error. * Fix deleted Eq from NonceSession. * fix cargo lock * Sort deps * Fix cargo lock files * Spurious commit to trigger CI * Remove `get_nonce` from NonceSession, rename `peek_nonce`->`nonce` (#2638) Also sort Cargo.toml files due to precommit hook. * Replace client with nonce paradigm for backends (#2625) * "Replace client with nonce paradigm for backends" * Sort cargo deps * Fix clone * Fix clippy * Modify MVQ to use NonceMessage (#2617) * Fix view proto * Save work * Implement nick's suggestions + fix lint * Fix fog conformance OOM failure by reducing `frontends` allocation (#2660) * Fix CI and remove Cargo.lock changes * Fix CI and remove Cargo.lock changes * Create query_backend enclave methods (#2627) * Create query_backend enclave methods * Implement nick's suggestions * Rename to QueryStore * Implement nonce based auth for Fog View Store (#2653) * Implement nonce based auth for Fog View Store * Implement nonce based auth * Close gRPC channel when client is dropped (#2692) * Implement nonce based auth for Fog View Store * Implement nonce based auth * Close gRPC channel when client is dropped * Close gRPC channel when client is dropped (#2694) * Create fog view router test utils (#2698) * Create fog view router test utils * Implement nick's suggestions * Add block range to MVQ response (#2720) * Add block range to MVQ response * Implement James's suggestions * Update BlockTracker to use ShardingStrategy (#2711) * Create MVQ internal type (#2721) * Create MVQ internal type * Remove explicit vec type * Add unknown status * Implement nick's suggestions * Implement new highest processed block count logic (#2722) * Implement highest processed block count logic based on gaps * Implement James's suggestions * Fix comment * Set block-related data more accurately (#2726) * Set block-related data more accurately * Change last known block logic * Implement nick's suggested test cases * Create Fog View Router integration test (#2375) * Create FVR integration tests * Implement nick's suggestions * Calculate last known block txo count and add tests (#2766) * Calculate last known block txo count and add tests * Implement Nick's suggestions * Implement James's suggestions * Collate remaining QueryResponse fields (#2781) * Create types module * Create shared data * Implement Nick's suggestions * Implement James's suggestions * Clean up response creation (#2783) * Make query batch logic compatible with sharding strategy (#2845) * Fix DbFetcher logic and modify recovery db method to use BlockRange * Implement Remoun's suggestions * Re-submit 2803 (#2866) * Replace ciphertext delta with explicit length field * Implement Nick's suggestions * Make payload_length fixed int * Make FVR integration test more complex (#2804) * Make FVR integration test more complex * Implement Nick's + Andrew's suggestions * Add unary apis to FogViewRouter (#2839) * Add unary apis to FogViewRouter * Implement Nick's suggestions * Account for epochs in block tracker highest processed block count logic (#2857) * Account for epochs in block tracker highest processed block count logic * Create and use block range method for ingress keys * Skip a key if the shard's epoch doesn't overlap * Implement Chris's suggestion * Incorporate missed blocks into calculation for highest processed block count (#2858) * Incorporate missed blocks into calculation for highest processed block count * Implement Nick's suggestions * Fix blank query sealing issue (#2904) * Fix blank query sealing issue * Implement Nick's suggestions * Rename seal message * Remove FogViewApi from FogViewService (#2852) * Remove FogViewApi from FogViewService * Fix tests, parametrize, create util methods * Remove FogViewRouterAdminUri * Modify fog conformance and local tests * Implement Nick's suggestions * Implement James's suggestions * Rename tests (#2905) * Rename tests * Implement Nick's suggestions * Implement James's suggestions * Create InternalTxOutSearchResult (#2880) * Create FixedTxOutSearchResult * Replace TxOutSearchResult with FixedTxOutSearchResult * Implement tx out search result conversion * Fill TxOutSearchResult field and update tests * Implement Nick's suggestions * Implement James's comments * Fix response processing loop logic (#2946) * Implement prometheus metrics for FVR (#2976) * Remove unnecessary logs * Create helper fn for response status * Create methods that don't rely on ctx * Implement correct prometheus metrics for FVR * Implement nick's suggestions * implement eran's suggestions * fix: bump postgresql chart version * fix: update scheme * Implement optelemetry tracing for FVR (#2979) * Implement optelemetry tracing for FVR * implement eran's suggestions * Merge master * Fix lock files * Ensure shards provide correct blocks (#2981) * Add sharding_strategies field to config * Plumb block range from config down to shard processor * Throw error if block ranges do not match * Fix python tests * Implement James's and Andrew's suggestions * Fix tests * Run cargo fmt * Fix python tests * Fix default parse * Add admin uri env var * Fix typo * Make Fog View Store return externally-addressable URI to Router (#3081) * Fix store uri by using responder id * Add log for uri * Fix query param * Fix test urls * Use host and responder id in router service * Add logging for ResponderId in router and store * Add more logging * Add more logging * Re-add urls push * Remove logging * Fix clippy * Fix clippy * Fix e_tx_out_store (#3082) * Fix e_tx_out_store * Make default payload_length == 0 * Run format * Fix clippy * Implement James's suggestions * Make find_record constant time (#3097) * Make find_record constant time * Implement Chris's suggestion * Improve error logging for epoch sharding parsing (#3141) * Improve error logging for epoch sharding parsing * Fix string format * Fix clippy lint re: expect statement * Make shard uris delimiter be a comma (#3142) * Improve error logging for epoch sharding parsing * Fix string format * Fix clippy lint re: expect statement * Make shard uris delimiter be a comma * Fix clap feature param * Add uri logging * Add uri logging * Fix deps * Remove custom readiness check * Fix bad merge * Use mc-rand * Re-add edl dep * Re-add all deps taken out by merge * Update pem in mc-fog-view-server * Update cargo lock * Update Cargo.lock * Implement Nick's suggestions * Fix modified test * Fix lint * Add MC_SHARDING_STRATEGY env var * Pull in protobuf changes from milliec/ledger-router-dev (#2889) * Pull in protobuf changes from milliec/ledger-router-dev * Move AddShardRequest to fog_common.proto, resolving #2888 * Clarifying comments and names, implementing changes requested for PR. * Fixing old copy/paste mistakes with 'view' * Ledger enclave support for router and store (#2896) * Pulling changes in from milliec/ledger-router-dev * Run cargo fmt * Making requested changes and clarifications to Fog ledger router comments. * Additional comment fixes * Fix CI lint * PR feedback nits Co-authored-by: Andrew Wygle <[email protected]> * URIs for Fog Ledger Router (#2897) * URIs for Fog Ledger Router * Use AdminUri for Fog Ledger Router Admin URI * PR feedback from James * Protobuf conversion functions for Fog Ledger Router (#2891) * URIs for Fog Ledger Router * Use AdminUri for Fog Ledger Router Admin URI * Protobuf conversion functions for Fog Ledger Router * Key Image Router Service (#2898) * Key Image Router Service * Remove unneeded dead_code annotation * Update fog/ledger/server/src/error.rs Co-authored-by: Nick Santana <[email protected]> * Update fog/ledger/server/src/key_image_router_service.rs Co-authored-by: Nick Santana <[email protected]> * Clean up commented-out code Co-authored-by: Nick Santana <[email protected]> * Fix misnamed type in a comment Co-authored-by: Nick Santana <[email protected]> * Address PR feedback around logging and comments. * Address error in loop termination logic. * Parameterize allowed number of retries for query loop * Update based on changes from previous PRs * Don't create 'groups' in `mod` or `use` declarations. Co-authored-by: NotGyro <[email protected]> Co-authored-by: Nick Santana <[email protected]> * Oblivious collation of key image store responses (#2913) * Oblivious collation of key image store responses * Refactor tests to be more readable, per PR feedback * Update fog/ledger/enclave/impl/src/oblivious_utils.rs Co-authored-by: Nick Santana <[email protected]> Co-authored-by: Nick Santana <[email protected]> * Fog Ledger Router Admin service (#2920) * Fog Ledger Router Admin service * Sort itertools properly in Cargo.toml * Fix redundant mod declaration * Update to use local SVC_COUNTERS * Key Image Router Server + Binary (#2921) * Ledger enclave support for router and store (#2896) * Pulling changes in from milliec/ledger-router-dev * Run cargo fmt * Making requested changes and clarifications to Fog ledger router comments. * Additional comment fixes * Fix CI lint * PR feedback nits Co-authored-by: Andrew Wygle <[email protected]> * Fog Ledger Router Admin service * Sort itertools properly in Cargo.toml * Key Image Router Server + Binary * Update router config for parameterized retries * Remove dead_code declaration on router service constructor * Changes due to rebase * normalize naming - ledger router, key image store * Linting fixes * Accept code review suggestions * Updates for GRPCIO 0.12 * Remove some unwraps in ledger_router binary Co-authored-by: Emily C <[email protected]> * Key image store service and server for router/store system (#2954) * Ledger enclave support for router and store (#2896) * Pulling changes in from milliec/ledger-router-dev * Run cargo fmt * Making requested changes and clarifications to Fog ledger router comments. * Additional comment fixes * Fix CI lint * PR feedback nits Co-authored-by: Andrew Wygle <[email protected]> * Fog Ledger Router Admin service * Sort itertools properly in Cargo.toml * Key Image Router Server + Binary * Update router config for parameterized retries * Changes due to rebase * normalize naming - ledger router, key image store * Linting fixes * Accept code review suggestions * Updates for GRPCIO 0.12 * Remove some unwraps in ledger_router binary * Pulling changes in from milliec/ledger-router-dev * Making requested changes and clarifications to Fog ledger router comments. * PR feedback nits * Key Image Router Service * Update fog/ledger/server/src/key_image_router_service.rs Co-authored-by: Nick Santana <[email protected]> * Address PR feedback around logging and comments. * Parameterize allowed number of retries for query loop * Fog Ledger Router Admin service * Sort itertools properly in Cargo.toml * Key Image Router Server + Binary * Key image store changes pulled in from milliec/ledger-router-dev * Cargo fmt * Run clippy * Sort itertools properly in Cargo.toml * Rebase and update to match current fog ledger router branch * Remove unused dependencies in fog-ledger-server Co-authored-by: Nick Santana <[email protected]> * Apply suggestions - Remove unused deps Co-authored-by: Nick Santana <[email protected]> * Apply suggestions from comments Applying suggestions from @nick-mobilecoin's review Co-authored-by: Nick Santana <[email protected]> * Resolving some code quality issues in direct_key_image_store_check() * Cargo fmt * key image server and key image service moved to router server and router service * Apply suggestions from code review Co-authored-by: Sam Dealy <[email protected]> Co-authored-by: Nick Santana <[email protected]> --------- Co-authored-by: Andrew Wygle <[email protected]> Co-authored-by: Nick Santana <[email protected]> Co-authored-by: Sam Dealy <[email protected]> * Implement Sharding Strategy (#2873) * Ledger enclave support for router and store (#2896) * Pulling changes in from milliec/ledger-router-dev * Run cargo fmt * Making requested changes and clarifications to Fog ledger router comments. * Additional comment fixes * Fix CI lint * PR feedback nits Co-authored-by: Andrew Wygle <[email protected]> * Fog Ledger Router Admin service * Sort itertools properly in Cargo.toml * Key Image Router Server + Binary * Update router config for parameterized retries * Changes due to rebase * normalize naming - ledger router, key image store * Linting fixes * Accept code review suggestions * Updates for GRPCIO 0.12 * Remove some unwraps in ledger_router binary * Pulling changes in from milliec/ledger-router-dev * Making requested changes and clarifications to Fog ledger router comments. * PR feedback nits * Key Image Router Service * Update fog/ledger/server/src/key_image_router_service.rs Co-authored-by: Nick Santana <[email protected]> * Address PR feedback around logging and comments. * Parameterize allowed number of retries for query loop * Fog Ledger Router Admin service * Sort itertools properly in Cargo.toml * Key Image Router Server + Binary * Key image store changes pulled in from milliec/ledger-router-dev * Cargo fmt * Run clippy * Sort itertools properly in Cargo.toml * Rebase and update to match current fog ledger router branch * Remove unused dependencies in fog-ledger-server Co-authored-by: Nick Santana <[email protected]> * Apply suggestions - Remove unused deps Co-authored-by: Nick Santana <[email protected]> * Apply suggestions from comments Applying suggestions from @nick-mobilecoin's review Co-authored-by: Nick Santana <[email protected]> * Resolving some code quality issues in direct_key_image_store_check() * Cargo fmt * key image server and key image service moved to router server and router service * Fog Ledger Router Admin service * Key Image Router Server + Binary * Update router config for parameterized retries * Changes due to rebase * normalize naming - ledger router, key image store * Linting fixes * Accept code review suggestions * Implement ShardingStrategy into DbFetcher * Split out DbFetcher creation and starting for ease of integration * Add a DbFetcher to the ledger store server. * Start the db_fetcher in the Fog Ledger Server to reflect the new behavior * Run cargo fmt * Run cargo clippy * Other clippy and fmt changes * Sort dependencies * Remove unimplemented! that snuck in from other branch * Update and sync with latest Fog ledger router * Remove some unwraps * Lint again * rebase fixups * PR feedback --------- Co-authored-by: Emily C <[email protected]> Co-authored-by: Nick Santana <[email protected]> * Implement non-KeyImage Functionality for Fog Ledger Router (#3084) * Ledger enclave support for router and store (#2896) * Pulling changes in from milliec/ledger-router-dev * Run cargo fmt * Making requested changes and clarifications to Fog ledger router comments. * Additional comment fixes * Fix CI lint * PR feedback nits Co-authored-by: Andrew Wygle <[email protected]> * Fog Ledger Router Admin service * Sort itertools properly in Cargo.toml * Key Image Router Server + Binary * Update router config for parameterized retries * Changes due to rebase * normalize naming - ledger router, key image store * Linting fixes * Accept code review suggestions * Updates for GRPCIO 0.12 * Pulling changes in from milliec/ledger-router-dev * Making requested changes and clarifications to Fog ledger router comments. * PR feedback nits * Key Image Router Service * Update fog/ledger/server/src/key_image_router_service.rs Co-authored-by: Nick Santana <[email protected]> * Address PR feedback around logging and comments. * Parameterize allowed number of retries for query loop * Fog Ledger Router Admin service * Sort itertools properly in Cargo.toml * Key Image Router Server + Binary * Key image store changes pulled in from milliec/ledger-router-dev * Cargo fmt * Run clippy * Sort itertools properly in Cargo.toml * Rebase and update to match current fog ledger router branch * Implement ShardingStrategy into DbFetcher * Add a DbFetcher to the ledger store server. * Run cargo fmt * Other clippy and fmt changes * Sort dependencies * Remove unimplemented! that snuck in from other branch * Update and sync with latest Fog ledger router * Refactor router dispatch into match statement Uses ugly names from rust-protobuf. rust-protobuf version 3.* will do the Rusty thing and put the generated enums into modules, but we're on version 2.* and they do the C thing of really_long_snake_case_names. * Add merkle proof service to router server * Add untrusted tx out service to router server * Add block service to router server * Fixups from rebase --------- Co-authored-by: Emily C <[email protected]> Co-authored-by: Nick Santana <[email protected]> * Porting existing Fog Ledger tests to Router (#3138) * Port existing tests to router server binary, except key image * Port key image test in router_connection.rs to streaming API * Fixups from rebase * remove logging statements * All Ledger tests now use portpicker to select ports * Fog router support for the unary API (#3123) * Cargo fmt * Ensure unary key image service gets started for router server. * Improving comment clarity. * Apply suggestions removing unnecessary comments Co-authored-by: Nick Santana <[email protected]> --------- Co-authored-by: Nick Santana <[email protected]> * Apply suggestions from code review Co-authored-by: Nick Santana <[email protected]> * Pull request feedback --------- Co-authored-by: Emily C <[email protected]> Co-authored-by: Nick Santana <[email protected]> * Test key image retrieval via unary API on the router (#3163) * Port existing tests to router server binary, except key image * Port key image test in router_connection.rs to streaming API * Fixups from rebase * remove logging statements * All Ledger tests now use portpicker to select ports * Fog router support for the unary API (#3123) * Cargo fmt * Ensure unary key image service gets started for router server. * Improving comment clarity. * Apply suggestions removing unnecessary comments Co-authored-by: Nick Santana <[email protected]> --------- Co-authored-by: Nick Santana <[email protected]> * Test key image retrieval via unary API on the router. * Cargo fmt * Fix a merging mistake. * Fix some additional merge mistakes * Sort dependencies --------- Co-authored-by: Andrew Wygle <[email protected]> Co-authored-by: Nick Santana <[email protected]> * Fix up Clippy issues from new Rust version * Make Fog Ledger Store return externally-addressable URI to Router (#3192) * Port fog local network and conformance tests to use ledger router (#3144) * Initial port of python tests, needs cleanup * Fix ledger and watcher DB paths in conformance * Fixups from rebase and add path arguments to key image store * Pull request feedback * Rebase fixups for unary test * Rename new_from_config to new, drop old new * Fog Ledger Router integration test (#3188) * Router integration test * Clippy updates after rebase * Implement Drop for TestEnvironment * Eliminate temp dirs after test * Remove unused GRPC arguments * Load environment config from JSON file. Might delete later. * Revert "Load environment config from JSON file. Might delete later." This reverts commit a677b59. * Remove magic numbers * Remove client-facing API from key-image store (#3217) * Remove client-facing API from key-image store * Cargo fmt * Remove fog_ledger_key_images_test from connection.rs (equivalent exists in router_connection.rs) * Cargo fmt * Pull out old code for the single-server design which will not be used anymore. * Fix cargo.toml. * Optelemetry tracing for FLR (#3264) * Optelemetry tracing for FLR * Cargo fmt * Apply James' suggested span names Co-authored-by: James Cape <[email protected]> * Cargo fmt --------- Co-authored-by: James Cape <[email protected]> * Process all blocks, but only add our blocks to the ORAM (#3203) (#3262) * Prometheus metrics for FLR (#3267) * Prometheus metrics for FLR * Cargo fmt * Post-rebase cleanups * Use default readiness check for FLR * Remove mc-crypto-rand from FLR branch * Address PR feedback * Clean up Cargo.locks * Add MC_SHARDING_STRATEGY --------- Co-authored-by: Sam Dealy <[email protected]> Co-authored-by: Sam Dealy <[email protected]> Co-authored-by: James Cape <[email protected]> Co-authored-by: Aaron Weisberg <[email protected]> Co-authored-by: Sam Dealy <[email protected]> Co-authored-by: Emily C <[email protected]> Co-authored-by: Nick Santana <[email protected]>
- Loading branch information
8 people
authored
Apr 24, 2023
1 parent
f6b702f
commit 7298a6b