Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Fog Ledger Router #3312

Merged
merged 143 commits into from
Apr 24, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
143 commits
Select commit Hold shift + click to select a range
c22781a
Fix Fog View Connection Cargo.toml dependencies
samdealy Jul 13, 2022
262eea6
Implement create_multi_view_query_data API for Fog View Router (#2115)
samdealy Jun 14, 2022
d4a7a61
Implement MultiViewStoreQuery API for Fog View (#2133)
samdealy Jun 23, 2022
ed6cd60
Implement enclave APIs that facilitate Fog View Router dynamic store …
samdealy Jul 8, 2022
c801436
Remove duplicate wasi files
samdealy Jul 13, 2022
d6b4d59
Re-add older wasi version
samdealy Jul 13, 2022
cc7d8be
Implement query / dynamic shard component discovery in Fog View Route…
samdealy Jul 15, 2022
84017c7
Split Fog View API into Client-facing and Store (Fog Router) APIs (#2…
samdealy Jul 19, 2022
7366bee
Create helper methods to obliviously collate shard responses (#2252)
samdealy Aug 10, 2022
01104d2
Collate shard response in constant time (obliviously) (#2250)
samdealy Aug 10, 2022
e3039f7
Implement epoch sharding via trait (#2350)
samdealy Aug 10, 2022
290bb5b
Use sharding strategy to tell view what blocks to process (#2351)
samdealy Aug 12, 2022
3a33948
Return minimum highest processed block to client (#2387)
samdealy Aug 12, 2022
f8fb892
Add config option for sharding strategy (#2352)
samdealy Aug 13, 2022
a329df2
Add readiness API to ShardingStrategy (#2353)
samdealy Aug 30, 2022
7989e7e
Add report fetching logic to FVR (#2454)
samdealy Aug 30, 2022
3ba7998
Fix responder ID usage in FVR (#2446)
samdealy Aug 31, 2022
f4d0ff7
Create different types of View Servers for store and client-facing (#…
samdealy Sep 1, 2022
2149b65
Messages for noise protocol exchanges with explicit nonces (#2461)
Sep 2, 2022
71e4bcd
Minor DRY cleanups around sessions and auth messages (#2462)
Sep 2, 2022
fa685ca
Update FVR client for integration test (#2448)
samdealy Sep 2, 2022
c59c8a4
Create gRPC FogViewRouterAdminAPI (#2360)
samdealy Sep 7, 2022
045270c
Create FogViewRouterAdminUri (#2361)
samdealy Sep 7, 2022
856128e
update cargo lock
samdealy Sep 13, 2022
789ffb8
Sealed messages for clients with backends (#2449)
awygle Sep 14, 2022
53ee79c
Implement shard management API sans auth (#2354)
samdealy Sep 14, 2022
0d0a4a1
Re-apply 2463. (#2547)
Sep 21, 2022
2c7845b
Re-apply PR #2473 (was accidentally removed) (#2548)
Sep 21, 2022
42a425a
Add inbound frontend connection support crypto-ake-enclave (#2549)
Sep 28, 2022
f1eee8a
Make constants and comments slightly friendlier/more consistent (#2550)
Sep 28, 2022
e253d13
fix cargo lock
samdealy Sep 28, 2022
3072986
Sort deps
samdealy Sep 28, 2022
027189e
Fix cargo lock files
samdealy Sep 30, 2022
2a01478
Spurious commit to trigger CI
samdealy Sep 30, 2022
8100f96
Remove `get_nonce` from NonceSession, rename `peek_nonce`->`nonce` (#…
awygle Oct 3, 2022
52e5fb6
Replace client with nonce paradigm for backends (#2625)
samdealy Oct 3, 2022
0ef6ef3
Modify MVQ to use NonceMessage (#2617)
samdealy Oct 5, 2022
1982d80
Merge master
samdealy Oct 5, 2022
a1b2bae
Fix fog conformance OOM failure by reducing `frontends` allocation (#…
samdealy Oct 6, 2022
96beb05
Create query_backend enclave methods (#2627)
samdealy Oct 8, 2022
2ae2565
Implement nonce based auth for Fog View Store (#2653)
samdealy Oct 11, 2022
51e2f40
Close gRPC channel when client is dropped (#2692)
samdealy Oct 11, 2022
a76ae5e
Close gRPC channel when client is dropped (#2694)
samdealy Oct 12, 2022
b493bf4
Create fog view router test utils (#2698)
samdealy Oct 14, 2022
31f2bd9
Add block range to MVQ response (#2720)
samdealy Oct 19, 2022
a310ba3
Update BlockTracker to use ShardingStrategy (#2711)
samdealy Oct 19, 2022
508443c
Create MVQ internal type (#2721)
samdealy Oct 19, 2022
fb8b136
Implement new highest processed block count logic (#2722)
samdealy Oct 19, 2022
b979c78
Set block-related data more accurately (#2726)
samdealy Oct 21, 2022
0526f2d
Create Fog View Router integration test (#2375)
samdealy Oct 24, 2022
f13b617
Calculate last known block txo count and add tests (#2766)
samdealy Oct 26, 2022
91f18ba
Collate remaining QueryResponse fields (#2781)
samdealy Nov 2, 2022
45222b7
Clean up response creation (#2783)
samdealy Nov 2, 2022
b446019
Merge master
samdealy Nov 7, 2022
37f6699
Make query batch logic compatible with sharding strategy (#2845)
samdealy Nov 14, 2022
caa4a37
Re-submit 2803 (#2866)
samdealy Nov 14, 2022
39f9060
Make FVR integration test more complex (#2804)
samdealy Nov 14, 2022
83105b3
Add unary apis to FogViewRouter (#2839)
samdealy Nov 15, 2022
ec3fea3
Merge branch 'master' into feature/fog-view-router
samdealy Nov 28, 2022
83f0f00
Account for epochs in block tracker highest processed block count log…
samdealy Nov 29, 2022
0c0e74b
Incorporate missed blocks into calculation for highest processed bloc…
samdealy Nov 29, 2022
4d13d24
Fix blank query sealing issue (#2904)
samdealy Dec 1, 2022
eea10fd
Remove FogViewApi from FogViewService (#2852)
samdealy Dec 2, 2022
864092d
Rename tests (#2905)
samdealy Dec 2, 2022
2b52ce5
Create InternalTxOutSearchResult (#2880)
samdealy Dec 16, 2022
7cb27dc
Fix response processing loop logic (#2946)
samdealy Dec 16, 2022
f32748f
Implement prometheus metrics for FVR (#2976)
samdealy Jan 9, 2023
5ba23ca
fix: bump postgresql chart version
aweis89 Jan 10, 2023
199bcac
fix: update scheme
aweis89 Jan 10, 2023
fd52c2d
Implement optelemetry tracing for FVR (#2979)
samdealy Jan 10, 2023
da78cd3
Merge master
samdealy Jan 11, 2023
c89f398
Merge master
samdealy Jan 12, 2023
151b934
Fix lock files
samdealy Jan 12, 2023
bed11dc
Ensure shards provide correct blocks (#2981)
samdealy Jan 13, 2023
fa3fe5f
Merge branch 'master' into feature/fog-view-router
samdealy Feb 2, 2023
09374b5
Merge branch 'master' into feature/fog-view-router
samdealy Feb 2, 2023
4c91f1a
Merge branch 'master' into feature/fog-view-router
samdealy Feb 3, 2023
1a70ffd
Merge branch 'master' into feature/fog-view-router
samdealy Feb 3, 2023
a87e64d
Make Fog View Store return externally-addressable URI to Router (#3081)
samdealy Feb 6, 2023
db728a5
Fix e_tx_out_store (#3082)
samdealy Feb 7, 2023
f059730
Merge branch 'master' into feature/fog-view-router
samdealy Feb 7, 2023
4d85726
Merge branch 'master' into feature/fog-view-router
samdealy Feb 8, 2023
2a83307
Make find_record constant time (#3097)
samdealy Feb 9, 2023
5acf879
Merge branch 'master' into feature/fog-view-router
samdealy Feb 9, 2023
ddae9b2
Merge branch 'master' into feature/fog-view-router
samdealy Feb 10, 2023
d851ac7
Merge branch 'master' into feature/fog-view-router
samdealy Feb 15, 2023
8450dd1
Merge branch 'master' into feature/fog-view-router
samdealy Feb 15, 2023
84adb6a
Improve error logging for epoch sharding parsing (#3141)
samdealy Feb 21, 2023
c72955f
Make shard uris delimiter be a comma (#3142)
samdealy Feb 21, 2023
ff7c49e
Merge branch 'master' into feature/fog-view-router
samdealy Feb 21, 2023
42367f2
Merge branch 'master' into feature/fog-view-router
samdealy Feb 23, 2023
9f23f5b
Add uri logging
samdealy Feb 27, 2023
46da66e
Add uri logging
samdealy Feb 27, 2023
4cdbf71
Merge branch 'master' into feature/fog-view-router
samdealy Mar 17, 2023
4cf66b9
Fix deps
samdealy Mar 15, 2023
4a8f2d7
Remove custom readiness check
samdealy Feb 23, 2023
2bb14a9
Fix bad merge
samdealy Mar 17, 2023
16dc1eb
Merge branch 'master' into feature/fog-view-router
samdealy Mar 22, 2023
5b53ed1
Merge branch 'master' into feature/fog-view-router
samdealy Mar 23, 2023
391d2b9
Merge branch 'master' into feature/fog-view-router
samdealy Mar 24, 2023
afb3779
Merge branch 'master' into feature/fog-view-router
samdealy Mar 27, 2023
77b2da1
Merge branch 'master' into feature/fog-view-router
samdealy Mar 29, 2023
0c82d6a
Use mc-rand
samdealy Mar 29, 2023
747f000
Re-add edl dep
samdealy Mar 29, 2023
4d8954b
Re-add all deps taken out by merge
samdealy Mar 29, 2023
56808e2
Update pem in mc-fog-view-server
samdealy Mar 29, 2023
039d4b4
Update cargo lock
samdealy Mar 29, 2023
9441857
Merge branch 'master' into feature/fog-view-router
samdealy Apr 17, 2023
12ba8fc
Update Cargo.lock
samdealy Apr 17, 2023
15099e3
Implement Nick's suggestions
samdealy Apr 18, 2023
283a4cb
Fix modified test
samdealy Apr 18, 2023
3a63e44
Merge branch 'master' into feature/fog-view-router
samdealy Apr 20, 2023
ebfff7a
Fix lint
samdealy Apr 20, 2023
bcd23ef
Add MC_SHARDING_STRATEGY env var
samdealy Apr 21, 2023
27c0901
Pull in protobuf changes from milliec/ledger-router-dev (#2889)
NotGyro Dec 7, 2022
e55d642
Ledger enclave support for router and store (#2896)
NotGyro Dec 14, 2022
cab7272
URIs for Fog Ledger Router (#2897)
awygle Dec 14, 2022
07a4c20
Protobuf conversion functions for Fog Ledger Router (#2891)
awygle Dec 14, 2022
d8965ca
Key Image Router Service (#2898)
awygle Jan 3, 2023
44b095e
Oblivious collation of key image store responses (#2913)
awygle Jan 9, 2023
908782d
Fog Ledger Router Admin service (#2920)
awygle Jan 9, 2023
381832d
Update to use local SVC_COUNTERS
awygle Jan 17, 2023
c8c6673
Key Image Router Server + Binary (#2921)
awygle Jan 24, 2023
dfe738a
Key image store service and server for router/store system (#2954)
NotGyro Feb 7, 2023
38a6033
Implement Sharding Strategy (#2873)
awygle Feb 9, 2023
cd90439
Implement non-KeyImage Functionality for Fog Ledger Router (#3084)
awygle Feb 9, 2023
a19363a
Porting existing Fog Ledger tests to Router (#3138)
awygle Feb 27, 2023
2275c3e
Test key image retrieval via unary API on the router (#3163)
NotGyro Feb 28, 2023
640c497
Fix up Clippy issues from new Rust version
awygle Mar 1, 2023
a3385ff
Make Fog Ledger Store return externally-addressable URI to Router (#3…
awygle Mar 2, 2023
a99be82
Port fog local network and conformance tests to use ledger router (#3…
awygle Mar 2, 2023
37ce735
Fog Ledger Router integration test (#3188)
awygle Mar 17, 2023
538c78c
Remove client-facing API from key-image store (#3217)
NotGyro Mar 20, 2023
fa9ced2
Optelemetry tracing for FLR (#3264)
NotGyro Mar 21, 2023
81ea818
Process all blocks, but only add our blocks to the ORAM (#3203) (#3262)
awygle Mar 23, 2023
08019e7
Prometheus metrics for FLR (#3267)
NotGyro Mar 23, 2023
3081b9d
Post-rebase cleanups
awygle Mar 28, 2023
eda00fb
Use default readiness check for FLR
awygle Mar 29, 2023
c476076
Remove mc-crypto-rand from FLR branch
awygle Apr 11, 2023
45ca0b4
Address PR feedback
awygle Apr 20, 2023
b521939
Clean up Cargo.locks
awygle Apr 20, 2023
fee2eb8
Add MC_SHARDING_STRATEGY
awygle Apr 21, 2023
6447985
Merge branch 'master' into feature/fog-ledger-router
awygle Apr 24, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 5 additions & 0 deletions fog/api/proto/fog_common.proto
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,8 @@ message BlockRange {
/// One-past-the-end of the range
uint64 end_block = 2;
}

message AddShardRequest {
// The shard's URI in string format.
string shard_uri = 1;
}
90 changes: 90 additions & 0 deletions fog/api/proto/ledger.proto
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,96 @@ import "fog_common.proto";
package fog_ledger;
option go_package = "mobilecoin/api";

import "google/protobuf/empty.proto";

////
// Ledger router API
////

service LedgerAPI {
rpc Request(stream LedgerRequest) returns (stream LedgerResponse) {}
}

service LedgerRouterAdminAPI {
// Adds a shard to the Fog Ledger Router's list of shards to query.
rpc AddShard(fog_common.AddShardRequest) returns (google.protobuf.Empty) {}
}

/// Fulfills requests sent by the Fog Ledger Router. This is not meant to fulfill requests sent directly by the client.
service KeyImageStoreAPI {
/// This is called to perform IX key exchange with the enclave before calling GetOutputs.
rpc Auth(attest.AuthMessage) returns (attest.AuthMessage) {}
/// Input should be an encrypted MultiKeyImageStoreRequest, result is an encrypted response.
rpc MultiKeyImageStoreQuery(MultiKeyImageStoreRequest) returns (MultiKeyImageStoreResponse) {}
}

message LedgerRequest {
oneof request_data {
attest.AuthMessage auth = 1;
attest.Message check_key_images = 2;
// TODO: Fill in block query service and merkle proof service.
// Potentially untrusted_tx_out_service? To be decided.
}
}

message LedgerResponse {
oneof response_data {
attest.AuthMessage auth = 1;
attest.Message check_key_image_response = 2;
// TODO: Fill in block query service and merkle proof service.
// Potentially untrusted_tx_out_service? To be decided.
}
}

// Identical to FogViewStoreDecryptionError
message FogLedgerStoreDecryptionError {
/// The FogLedgerStoreUri for the specific Fog Ledger Store that
/// tried to decrypt the MultiKeyImageStoreRequest and failed.
/// The client should subsequently authenticate with the machine
/// described by this URI.
string store_uri = 1;

/// An error message that describes the decryption error.
string error_message = 2;
}

// Identical to MultiViewStoreQueryRequest
message MultiKeyImageStoreRequest {
/// A list of queries encrypted for Fog Ledger Stores.
repeated attest.NonceMessage queries = 1;
}


/// The status associated with a MultiKeyImageStoreQueryResponse
enum MultiKeyImageStoreResponseStatus {
/// Ensure default value (unfilled status) doesn't falsely appear to be a success
UNKNOWN = 0;
/// The Fog Ledger Store successfully fulfilled the request.
SUCCESS = 1;
/// The Fog Ledger Store is unable to decrypt a query within the MultiKeyImageStoreRequest. It needs to be authenticated
/// by the router.
AUTHENTICATION_ERROR = 2;
/// The Fog Ledger Store is not ready to service a MultiLedgerStoreQueryRequest. This might be because the store has
/// not loaded enough blocks yet.
NOT_READY = 3;
}

message MultiKeyImageStoreResponse {
/// Optional field that gets set when the Fog Ledger Store is able to decrypt a query
/// included in the MultiKeyImageStoreRequest and create a query response for that
// query. This is an encrypted CheckKeyImagesResponse.
attest.NonceMessage query_response = 1;

/// The FogLedgerStore for the specific Fog Ledger Store that
/// tried to decrypt the MultiLedgerStoreQueryRequest and failed.
/// The client should subsequently authenticate with the machine
/// described by this URI.
string store_uri = 2;

/// Status that gets returned when the Fog Ledger Store services a MultiKeyImageStoreRequest.
MultiKeyImageStoreResponseStatus status = 3;
}

////
// Merkle proofs
////
Expand Down
2 changes: 1 addition & 1 deletion fog/api/proto/view.proto
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ service FogViewRouterAPI {

service FogViewRouterAdminAPI {
// Adds a shard to the Fog View Router's list of shards to query.
rpc addShard(AddShardRequest) returns (google.protobuf.Empty) {}
rpc addShard(fog_common.AddShardRequest) returns (google.protobuf.Empty) {}
}

message AddShardRequest {
Expand Down
23 changes: 22 additions & 1 deletion fog/api/src/conversions.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@
//
// Contains helper methods that enable conversions for Fog Api types.

use crate::{fog_common, ingest_common, view::MultiViewStoreQueryRequest};
use crate::{
fog_common, ingest_common, ledger::MultiKeyImageStoreRequest, view::MultiViewStoreQueryRequest,
};
use mc_api::ConversionError;
use mc_attest_api::attest;
use mc_attest_enclave_api::{EnclaveMessage, NonceSession};
Expand Down Expand Up @@ -30,6 +32,25 @@ impl From<Vec<attest::NonceMessage>> for MultiViewStoreQueryRequest {
}
}

impl From<Vec<EnclaveMessage<NonceSession>>> for MultiKeyImageStoreRequest {
fn from(enclave_messages: Vec<EnclaveMessage<NonceSession>>) -> MultiKeyImageStoreRequest {
enclave_messages
.into_iter()
.map(|enclave_message| enclave_message.into())
.collect::<Vec<attest::NonceMessage>>()
.into()
}
}

impl From<Vec<attest::NonceMessage>> for MultiKeyImageStoreRequest {
fn from(attested_query_messages: Vec<attest::NonceMessage>) -> MultiKeyImageStoreRequest {
let mut multi_key_image_store_request = MultiKeyImageStoreRequest::new();
multi_key_image_store_request.set_queries(attested_query_messages.into());

multi_key_image_store_request
}
}

impl From<&common::BlockRange> for fog_common::BlockRange {
fn from(common_block_range: &common::BlockRange) -> fog_common::BlockRange {
let mut proto_block_range = fog_common::BlockRange::new();
Expand Down
8 changes: 8 additions & 0 deletions fog/ledger/connection/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,17 @@ rust-version = { workspace = true }
[dependencies]
# mobilecoin
mc-api = { path = "../../../api" }
mc-attest-ake = { path = "../../../attest/ake" }
mc-attest-core = { path = "../../../attest/core" }
mc-attest-verifier = { path = "../../../attest/verifier" }
mc-blockchain-types = { path = "../../../blockchain/types" }
mc-common = { path = "../../../common", features = ["log"] }
mc-crypto-keys = { path = "../../../crypto/keys" }
mc-crypto-noise = { path = "../../../crypto/noise" }
mc-rand = "1.0"
mc-transaction-core = { path = "../../../transaction/core" }
mc-util-grpc = { path = "../../../util/grpc" }
mc-util-serial = { path = "../../../util/serial" }
mc-util-uri = { path = "../../../util/uri" }

# fog
Expand All @@ -25,10 +30,13 @@ mc-fog-types = { path = "../../types" }
mc-fog-uri = { path = "../../uri" }

# third-party
aes-gcm = "0.10.1"
displaydoc = { version = "0.2", default-features = false }
futures = "0.3"
grpcio = "0.12.1"
protobuf = "2.27.1"
retry = "2.0"
sha2 = { version = "0.10", default-features = false }

[dev-dependencies]
mc-common = { path = "../../../common", features = ["loggers"] }
3 changes: 3 additions & 0 deletions fog/ledger/connection/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,6 @@ pub use merkle_proof::{FogMerkleProofGrpcClient, OutputError, OutputResultExtens

mod untrusted;
pub use untrusted::FogUntrustedLedgerGrpcClient;

mod router_client;
pub use router_client::LedgerGrpcClient;
Loading