Introduce O(n) canonicalization algorithm

[evanlinjin]

Fixes #1665
Replaces #1659

Description

Previously, getting the canonical history of transactions/UTXOs required calling TxGraph::get_chain_position on each transaction. This was highly inefficient and resulted in an O(n^2) algorithm. The situation is especially problematic when we have many unconfirmed conflicts.

This PR introduces an O(n) algorithm to determine the canonical set of transactions in TxGraph. The algorithm's premise is as follows:

1. If transaction A is determined to be canonical, all of A's ancestors must also be canonical.
2. If transaction B is determined to be NOT canonical, all of B's descendants must also be NOT canonical.
3. If a transaction is anchored in the best chain, it is canonical.
4. If a transaction conflicts with a canonical transaction, it is NOT canonical.
5. A transaction with a higher last-seen has precedence.
6. Last-seen values are transitive. A transaction's collective last-seen value is the max of it's last-seen value and all of it's descendants.

We maintain two mutually-exclusive txid sets: canoncial and not_canonical.

Imagine a method mark_canonical(A) that is based on premise 1 and 2. This method will mark transaction A and all of it's ancestors as canonical. For each transaction that is marked canonical, we can iterate all of it's conflicts and mark those as non_canonical. If a transaction already exists in canoncial or not_canonical, we can break early, avoiding duplicate work.

This algorithm iterates transactions in 3 runs.

1. Iterate over all transactions with anchors in descending anchor-height order. For any transaction that has an anchor pointing to the best chain, we call mark_canonical on it. We iterate in descending-height order to reduce the number of anchors we need to check against the ChainOracle (premise 1). The purpose of this run is to populate non_canonical with all transactions that directly conflict with anchored transactions and populate canonical with all anchored transactions and ancestors of anchors transactions (transitive anchors).
2. Iterate over all transactions with last-seen values, in descending last-seen order. We can call mark_canonical on all of these that do not already exist in canonical or not_canonical.
3. Iterate over remaining transactions that contains anchors (but not in the best chain) and have no last-seen value. We treat these transactions in the same way as we do in run 2.

Benchmarks

Thank you to @ValuedMammal for working on this.

$ cargo bench -p bdk_chain --bench canonicalization

Benchmark results (this PR):

many_conflicting_unconfirmed::list_canonical_txs                                                                            
                        time:   [709.46 us 710.36 us 711.35 us]
many_conflicting_unconfirmed::filter_chain_txouts                                                                            
                        time:   [712.59 us 713.23 us 713.90 us]
many_conflicting_unconfirmed::filter_chain_unspents                                                                            
                        time:   [709.95 us 711.16 us 712.45 us]
many_chained_unconfirmed::list_canonical_txs                                                                             
                        time:   [2.2604 ms 2.2641 ms 2.2680 ms]
many_chained_unconfirmed::filter_chain_txouts                                                                             
                        time:   [3.5763 ms 3.5869 ms 3.5979 ms]
many_chained_unconfirmed::filter_chain_unspents                                                                             
                        time:   [3.5540 ms 3.5596 ms 3.5652 ms]
nested_conflicts_unconfirmed::list_canonical_txs                                                                            
                        time:   [660.06 us 661.75 us 663.60 us]
nested_conflicts_unconfirmed::filter_chain_txouts                                                                            
                        time:   [650.15 us 651.36 us 652.71 us]
nested_conflicts_unconfirmed::filter_chain_unspents                                                                            
                        time:   [658.37 us 661.54 us 664.81 us]

Benchmark results (master): https://github.com/evanlinjin/bdk/tree/fix/1665-master-bench

many_conflicting_unconfirmed::list_canonical_txs                                                                             
                        time:   [94.618 ms 94.966 ms 95.338 ms]
many_conflicting_unconfirmed::filter_chain_txouts                                                                             
                        time:   [159.31 ms 159.76 ms 160.22 ms]
many_conflicting_unconfirmed::filter_chain_unspents                                                                             
                        time:   [163.29 ms 163.61 ms 163.96 ms]

# I gave up running the rest of the benchmarks since they were taking too long.

Notes to the reviewers

• PLEASE MERGE feat(chain,wallet)!: Transitive ChainPosition #1733 BEFORE THIS PR! We had to change the signature of ChainPosition to account for transitive anchors and unconfirmed transactions with no last-seen value.

• The canonicalization algorithm is contained in /crates/chain/src/canonical_iter.rs.

• Since the algorithm requires traversing transactions ordered by anchor height, and then last-seen values, we introduce two index fields in TxGraph; txs_by_anchor and txs_by_last_seen. Methods insert_anchor and insert_seen_at are changed to populate these index fields.

• An ADR is added: docs/adr/0003_canonicalization_algorithm.md. This is based on the work in Architectural Decision Records #1592.

Changelog notice

• Added: Introduce an O(n) canonicalization algorithm. This logic is contained in /crates/chain/src/canonical_iter.rs.
• Added: Indexing fields in TxGraph; txs_by_anchor_height and txs_by_last_seen. Pre-indexing allows us to construct the canonical history more efficiently.
• Removed: TxGraph methods: try_get_chain_position and get_chain_position. This is superseded by the new canonicalization algorithm.

Checklists

All Submissions:

• I've signed all my commits
• I followed the contribution guidelines
• I ran cargo fmt and cargo clippy before committing

New Features:

• I've added tests for the new feature
• I've added docs for the new feature

Bugfixes:

• This pull request breaks the existing API
• I've added tests to reproduce the issue which are now passing
• I'm linking the issue being fixed by this PR

[notmandatory]

Per call today @evanlinjin will open a new PR for the 1.0.0-beta milestone that only makes expected breaking changes for Wallet, likely only the ChainPosition enum. Then the algorithm improvements in this PR can be released in a future non-breaking patch version.

[notmandatory]

On further discussion today at release planning call this PR was moved back into the 1.0 milestone if it can be completed and reviewed in time. If not it will have to wait for a 2.0 milestone because it required breaking changes to chain crate APIs that are exposed in the Wallet API.

Alternatively we could remove the following Wallet functions and move any error types from chain into the core module which should insulate the wallet crate from this chain crate API changes.

    /// Get a reference to the inner [`TxGraph`].
    pub fn tx_graph(&self) -> &TxGraph<ConfirmationBlockTime> {
        self.indexed_graph.graph()
    }

    /// Get a reference to the inner [`KeychainTxOutIndex`].
    pub fn spk_index(&self) -> &KeychainTxOutIndex<KeychainKind> {
        &self.indexed_graph.index
    }

    /// Get a reference to the inner [`LocalChain`].
    pub fn local_chain(&self) -> &LocalChain {
        &self.chain
    }

[evanlinjin]

@notmandatory how does this solution compare with just giving wallet a major version bump?

[notmandatory]

Removing the above functions and moving required chain error type to core should allow us to do breaking chain crate api changes without having to do a major wallet crate release. I also don't see why Wallet users need to access the inner chain types directly instead of using higher level functions like transactions(), or we can add new helper functions without breaking any APIs.

But all that said if it's less risky to just keep everything as is and do a 2.0 release in 6 mo or so I'd be fine with that too.

[ValuedMammal]

ACK 956d0a9

[nymius]

ACK 956d0a9

[oleonardolima]

utACK 956d0a9

[jirijakes]

ACK 956d0a9

🔥

[notmandatory]

Thanks @evanlinjin and @ValuedMammal for all the work on getting this coded, documented, and benchmark tested, and to everyone who helped with review!