upgrade Job Declaration and Mining Protocols #121

plebhash · 2025-01-17T23:05:13Z

close #120

changes are divided by commits:

replace "async" terminology with new JDP "modes": Coinbase-only and Full-Template
drop ShortIds and SipHash (as discussed in Make more clear how we use the block header to calculate the txs short ids (job declarator) #99)
add prev_hash field to DeclareMiningJob message
add version field to SubmitSolution message
clarify description of nbits field of SubmitSolution message
remove extranonce_size, min_ntime and version fields from SetCustomMiningJob message

for easier review, I recommend this Markdown rendering on my fork

06-Job-Declaration-Protocol.md

Sjors · 2025-01-24T07:35:27Z

I don't think it's wise to have two messages in the spec called SubmitSolution, defined in two different places and with different content. Imo they should either be unified and defined in one place, or one needs a new name.

06-Job-Declaration-Protocol.md

plebhash · 2025-01-24T13:57:45Z

I don't think it's wise to have two messages in the spec called SubmitSolution, defined in two different places and with different content. Imo they should either be unified and defined in one place, or one needs a new name.

I don't think it's possible to unify them, as one is meant for TP and the other for JDS (and each needs different parts of the solution)

I'm not opposed to renaming either one of them or both, but it would be good to get input from @Fi3 and @TheBlueMatt on this

05-Mining-Protocol.md

Fi3 · 2025-01-28T08:10:51Z

05-Mining-Protocol.md

@@ -470,8 +469,6 @@ The `mining_job_token` provides the information for the pool to authorize the cu
 | coinbase_tx_outputs         | B0_64K         | Bitcoin transaction outputs to be included as the last outputs in the coinbase transaction                                                                            |
 | coinbase_tx_locktime        | U32            | The locktime field in the coinbase transaction                                                                                                                        |
 | merkle_path                 | SEQ0_255[U256] | Merkle path hashes ordered from deepest                                                                                                                               |
-| extranonce_size             | U16            | Size of extranonce in bytes that will be provided by the downstream node                                                                                              |


why you remove this? Is this ok with braiins as well @jakubtrnka?

from discussions on Google Doc:

@TheBlueMatt said:

I do kinda wonder if we shouldn't drop all the extranonce size accounting (for custom work)... Like, the pool doesn't care what the extranonce size is, as long as its extranonce_prefix is honored. The extranonce prefix has a strict limit of 32 bytes, the server is expected to build the coinbase tx from the extranonce, the length is never given on the wire, why do we care here?

For extended channels, the extranonce length is included in coinbase_prefix, so the pool needs to know the extranonce length when it goes to send that. The same is not true anywhere in custom work. IMO the field should be dropped.

then @Fi3 said:

make totally sense, this is a very big change btw. I think that we should coordinate it at least with braiins if we decide to do it.

ok just make sure that braiins is ok with this change

Fi3 · 2025-01-28T08:11:04Z

05-Mining-Protocol.md

@@ -470,8 +469,6 @@ The `mining_job_token` provides the information for the pool to authorize the cu
 | coinbase_tx_outputs         | B0_64K         | Bitcoin transaction outputs to be included as the last outputs in the coinbase transaction                                                                            |
 | coinbase_tx_locktime        | U32            | The locktime field in the coinbase transaction                                                                                                                        |
 | merkle_path                 | SEQ0_255[U256] | Merkle path hashes ordered from deepest                                                                                                                               |
-| extranonce_size             | U16            | Size of extranonce in bytes that will be provided by the downstream node                                                                                              |
-| min_ntime                   | OPTION[u32]    | TBD: Can custom job ever be future?                                                                                                                                   |


why you remove this? Is this ok with braiins as well @jakubtrnka?

it's duplicated... if you look above, you'll see that there's already a min_ntime field in SetCustomMiningJob

this field has data type OPTION[u32] and the description is TBD: Can custom job ever be future?

so the real purpose of this duplicate is not very clear... in case there's some meaningful motivation and we cannot remove it, we should at least clarify this motivation in the spec

Fi3 · 2025-01-28T08:15:08Z

06-Job-Declaration-Protocol.md

@@ -75,45 +125,40 @@ Notably, if the pool intends to change the space it requires for coinbase transa
 | request_id                          | U32       | Unique identifier for pairing the response                                                                                                                                                                                                                                                                                                                               |
 | mining_job_token                    | B0_255    | Token that makes the client eligible for committing a mining job for approval/transaction declaration or for identifying custom mining job on mining connection.                                                                                                                                                                                                         |
 | coinbase_output_max_additional_size | U32       | The maximum additional serialized bytes which the pool will add in coinbase transaction outputs. See discussion in the Template Distribution Protocol's CoinbaseOutputDataSize message for more details.                                                                                                                                                                 |
-| async_mining_allowed                | BOOL      | If true, the mining_job_token can be used immediately on a mining connection in the SetCustomMiningJob message, even before DeclareMiningJob and DeclareMiningJob.Success messages have been sent and received. If false, Job Declarator MUST use this token for DeclareMiningJob only. <br>This MUST be true when SetupConnection.flags had REQUIRES_ASYNC_JOB_MINING set. |


how the upstream communicate if the token is for coinbase-only mode or for full-template mode?

by simply allowing the connection to be established or not

when JDC sends SetupConnection, the DECLARE_TX_DATA bit flag is used to signal that it consents to revealing the txs... then it's up to JDS to decide whether this is ok or not

so we have 4 possible scenarios:

JDS allows Coinbase-only mode JDS does NOT allow Coinbase-only mode

JDC sets DECLARE_TX_DATA SetupConnection.Success SetupConnection.Success

JDC does NOT set DECLARE_TX_DATA SetupConnection.Success SetupConnection.Error (unsupported-feature-flags)

do you feel this still leaves room for unspecified scenarios?

I guess this is heavily based on the assumption that JDS will operate EITHER under Coinbase-only OR Full-Template, but not BOTH at the same time.

If we want to allow for JDS to operate under both at the same time, then I see how DECLARE_TX_DATA is not sufficient, as JDC wouldn't know under which mode the token is supposed to be used.

So I guess the question is: does this scenario make sense? Can JDS ever allow for both Coinbase-only and Full-Template at the same time? TBH it doesn't really make much sense in my head, but maybe I have some blindspot.

Fi3 · 2025-01-28T08:24:09Z

06-Job-Declaration-Protocol.md

-| tx_short_hash_nonce         | U64                   | A unique nonce used to ensure tx_short_hash collisions are uncorrelated across the network                                                                                                                                                                                                                                                                                                                                   |
-| tx_short_hash_list          | SEQ0_64K[SHORT_TX_ID] | Sequence of SHORT_TX_IDs. Inputs to the SipHash functions are transaction hashes from the mempool. Secret keys k0, k1 are derived from the first two little-endian 64-bit integers from the SHA256(tx_short_hash_nonce), respectively (see bip-0152 for more information). Upstream node checks the list against its mempool. Does not include the coinbase transaction (as there is no corresponding full data for it yet). |
-| tx_hash_list_hash           | U256                  | SHA256 hash of the list of full txids, concatenated in the same sequence as they are declared in the_short_hash_list                                                                                                                                                                                                                                                                                                         |
+| prev_hash                   | U256                  | Chain tip that the template is based on. JDS SHOULD accept templates based on different tips, as long as they are on the best block height. JDS MUST reject templates based on old tips.                                                                                                                                                                                                                                |


can we make this field optional?

what would be the rationale?

I assume you want to give room for a scenario where JDS is allowed to ignore prev_hash on DeclareMiningJob, which leaves us with the following possibilities:

JDS cares about prev_hash JDS does NOT care about prev_hash

DeclareMiningJob sets prev_hash JDS verifies prev_hash JDS ignores prev_hash

or maybe DeclareMiningJob.Error to discourage unnecessary bandwidth consumption

DeclareMiningJob does NOT set prev_hash DeclareMiningJob.Error (invalid-job-param-value-prev-hash) -

is that correct?

I'm not sure how it will work. Let's say that the pool reject the work cause it is in a old tip. Isn't it the same that just rejecting the work? When a client get a share rejection and the share is good it wait a little bit and if it do not see a new prev hash it just change pool. I don't understand what this field add (in that scenario). I agree that it could be useful in particular cases; for example when we are in a fork, so the pool know which phash need to use to verify shares. But for the 99.9% of the time I don't see how this can be useful. Also note that we should support the case where JDS and pool do not communicate.

The prev_hash here is needed just to let the JDS check that the declared job is built on a valid tip (and not on an old one).

06-Job-Declaration-Protocol.md

@GitGab19

as agreed between @GitGab19, @TheBlueMatt, @Fi3 and @plebhash, we are dropping the "async" terminology in preference of coinbase vs template "modes". async terminology was causing confusion and leading to conflicting interpretation of the specs. now, coinbase vs template modes make it explicitly clear how each side (JDC vs JDS) handles job declaration, especially with regards to: - knowledge about the txdata contained in the template - ability to propagate a valid block the original semantics of async is now expressed with the notion of "optimistic mining"

as discussed in stratum-mining#99

allows JDS to reject work on old tips, or with potential double spends

the version field was rolled during mining, and therefore has to be informed to JDS when the solution is being submitted. apparently it was removed by mistake on commit ce1edfb

Fi3 · 2025-01-28T15:00:25Z

I don't think it's wise to have two messages in the spec called SubmitSolution, defined in two different places and with different content. Imo they should either be unified and defined in one place, or one needs a new name.

make sense

…ningJob Co-authored-by: Gabriele Vernetti <[email protected]>

Co-authored-by: Gabriele Vernetti <[email protected]>

plebhash force-pushed the upgrade-jdp branch 3 times, most recently from d7dbd0b to 22fb563 Compare January 17, 2025 23:22

plebhash commented Jan 21, 2025

View reviewed changes

06-Job-Declaration-Protocol.md Outdated Show resolved Hide resolved

plebhash force-pushed the upgrade-jdp branch 6 times, most recently from c3baaca to 5583c11 Compare January 23, 2025 14:59

plebhash changed the title ~~[WIP] upgrade Job Declaration Protocol~~ [WIP] upgrade Job Declaration and Mining Protocols Jan 23, 2025

Sjors reviewed Jan 24, 2025

View reviewed changes

06-Job-Declaration-Protocol.md Show resolved Hide resolved

Sjors reviewed Jan 24, 2025

View reviewed changes

06-Job-Declaration-Protocol.md Show resolved Hide resolved

GitGab19 reviewed Jan 24, 2025

View reviewed changes

plebhash force-pushed the upgrade-jdp branch from d7d67d8 to 9c0c248 Compare January 24, 2025 13:21

plebhash marked this pull request as ready for review January 24, 2025 13:57

plebhash changed the title ~~[WIP] upgrade Job Declaration and Mining Protocols~~ upgrade Job Declaration and Mining Protocols Jan 24, 2025

Fi3 reviewed Jan 28, 2025

View reviewed changes

05-Mining-Protocol.md Outdated Show resolved Hide resolved

Fi3 reviewed Jan 28, 2025

View reviewed changes

06-Job-Declaration-Protocol.md Outdated Show resolved Hide resolved

plebhash force-pushed the upgrade-jdp branch from c111b71 to 251067a Compare January 28, 2025 12:46

plebhash added 4 commits January 28, 2025 10:26

drop ShortIds and SipHash..

c496164

as discussed in stratum-mining#99

add prev_hash field to DeclareMiningJob..

393f82e

allows JDS to reject work on old tips, or with potential double spends

add version field to SubmitSolution..

ace4fd4

the version field was rolled during mining, and therefore has to be informed to JDS when the solution is being submitted. apparently it was removed by mistake on commit ce1edfb

plebhash added 3 commits January 28, 2025 10:26

clarify description of nbits field of SubmitSolution

420feac

remove extranonce_size field from SetCustomMiningJob

b27e4e6

remove min_ntime from SetCustomMiningJob message

7b6faaa

plebhash force-pushed the upgrade-jdp branch from 251067a to b16de1e Compare January 28, 2025 13:26

plebhash and others added 2 commits January 28, 2025 12:15

clarify coinbase_prefix and coinbase_suffix descriptions of DeclareMi…

25f708f

…ningJob Co-authored-by: Gabriele Vernetti <[email protected]>

rename tx_hash_list to tx_ids_list

dd64e06

Co-authored-by: Gabriele Vernetti <[email protected]>

plebhash force-pushed the upgrade-jdp branch from b16de1e to dd64e06 Compare January 28, 2025 15:15

This was referenced Jan 29, 2025

JDS connects to wrong RPC port on Integration Tests stratum-mining/stratum#1418

Open

CI migration: MG to Integration/Unit Tests stratum-mining/stratum#1121

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

upgrade Job Declaration and Mining Protocols #121

upgrade Job Declaration and Mining Protocols #121

plebhash commented Jan 17, 2025 •

edited

Loading

Sjors commented Jan 24, 2025

plebhash commented Jan 24, 2025

Fi3 Jan 28, 2025

plebhash Jan 28, 2025

Fi3 Jan 28, 2025

Fi3 Jan 28, 2025

plebhash Jan 28, 2025

Fi3 Jan 28, 2025

plebhash Jan 28, 2025

plebhash Jan 28, 2025 •

edited

Loading

Fi3 Jan 28, 2025

plebhash Jan 28, 2025 •

edited

Loading

Fi3 Jan 28, 2025

GitGab19 Jan 28, 2025

Fi3 commented Jan 28, 2025

	JDS allows Coinbase-only mode	JDS does NOT allow Coinbase-only mode
JDC sets `DECLARE_TX_DATA`	`SetupConnection.Success`	`SetupConnection.Success`
JDC does NOT set `DECLARE_TX_DATA`	`SetupConnection.Success`	`SetupConnection.Error` (`unsupported-feature-flags`)

	JDS cares about `prev_hash`	JDS does NOT care about `prev_hash`
`DeclareMiningJob` sets `prev_hash`	JDS verifies `prev_hash`	JDS ignores `prev_hash` or maybe `DeclareMiningJob.Error` to discourage unnecessary bandwidth consumption
`DeclareMiningJob` does NOT set `prev_hash`	`DeclareMiningJob.Error` (`invalid-job-param-value-prev-hash`)	-

upgrade Job Declaration and Mining Protocols #121

Are you sure you want to change the base?

upgrade Job Declaration and Mining Protocols #121

Conversation

plebhash commented Jan 17, 2025 • edited Loading

Sjors commented Jan 24, 2025

plebhash commented Jan 24, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

plebhash Jan 28, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

plebhash Jan 28, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Fi3 commented Jan 28, 2025

plebhash commented Jan 17, 2025 •

edited

Loading

plebhash Jan 28, 2025 •

edited

Loading

plebhash Jan 28, 2025 •

edited

Loading