Precise hashrate calculation algorithm

[mlgodskz]

After conducting a small experimental statistical study, I derived an algorithm for calculating hashrate based on the shares received from the miner. On a sample of > 1,000-2,000 shares, this algorithm provides hashrate determination accuracy within a 1% margin of error. I'd like to hear the opinions of esteemed colleagues. I'll clarify in advance: I'm not prepared to participate in complex mathematical discussions :-) I derived it experimentally, working with hash generators and their corresponding statistical graphs, not mathematically on paper.

1. I take the shares sent by the miner over a period of time, say 10 minutes.
2. I calculate the difficulty (share_diff = max_target / hash) for each.
3. I find the share corresponding to the (1 - 1/e) percentile (this is approximately the 63rd percentile) in terms of difficulty.
4. I multiply the difficulty of the share found in step 3 by the total number of shares.
5. I multiply the result from step 4 by 1/e (approximately 0.37).
6. I multiply by 2^32 and get the number of hashes that were required to try in order to find all these shares.
7. I divide by the number of seconds (for 10 minutes, this is 600 seconds) and get the hashes per second.

If someone understands code better:

def calculate_hashrate(shares_list, time_period_seconds=600):
    max_target = 0x00000000FFFF0000000000000000000000000000000000000000000000000000
    difficulties = np.array([max_target / hash_value for hash_value in shares_list])
    percentile_difficulty = np.percentile(difficulties, (1 - 1/np.e) * 100)
    total = percentile_difficulty * len(difficulties) * (1/np.e) * 2**32
    return total / time_period_seconds

Tnx

[pavlenex]

Tagging @Fi3 @rrybarczyk @GitGab19 @plebhash :)

[mlgodskz]

...Naturally, the smaller the sample, the lower the accuracy, and the larger the sample, the higher the accuracy. The best results are obtained when the number of shares for analysis exceeds several thousand. And from this arises a practical question:

• In your opinion, what is the optimal number of shares per minute from a single worker? And why?

Thank you.

[plebhash]

it's hard to give you a precise number, as we don't have a lot of practical experience on testing SRI at scale

but at a theoretical level, this is a multidimensional analysis, so the answer to your question depends on a few other questions:

• how many workers do you expect your system to handle on average?
• how many workers should your system be able to handle as a max limit? (ideally infinite but that's not realistic)
• how powerful is your system?

in the end of the day, this will boil down to your system's capacity to handle the load.

• more samples per worker incur in a bigger load (as a price for higher accuracy)
• more workers incur in a bigger load (as a price for higher scalability)

so there will be some sweet spot that will allow you to find the best trade-off between accuracy, scalability, and system capacity that still allows you to operate under acceptable conditions

note: I still didn't have a look at the algorithm on the top of this discussion. This is just an answer to the specific question on this thread.

[GitGab19]

@mlgodskz is there any particular reason to fix the time variable to 600 seconds (10 min)?

[mlgodskz]

10 minutes is an arbitrary value, simply for example. Ultimately, it's not the time interval that matters, but the number of hashes (shares) for analysis. If we receive 600 shares in one hour, we'll divide the calculated "total" hashes by (600 * 6) seconds.

[plebhash]

by looking at the algorithm on the top of this Discussion, I think it might be overcomplicating things

---

Here's a summarized probabilistic modeling for estimating worker hashrate.

I wrote this last year with help from @storopoli, which is a PhD in computational statistics and contributor to the rust-bitcoin ecosystem.

It is heavily based on the Rosenfeld Paper (which is kind of the "bible" of statistics of bitcoin mining).

Let's establish the following variables:

• $\hat{h}$: estimated worker hashrate (in Hashes / s)
• $S$: number of valid shares submitted by the worker
• $d$: difficulty of individual share
• $D$: difficulty threshold that establishes the validity of a share

from the Rosenfeld paper (section 1.1), we know that a share with difficulty $d > D$ is found with probability:

$$P(d \geq D) = \frac{2^{16} - 1}{2^{48}D} \approx \frac{1}{2^{32}D}$$

if we collect shares for a time window $t$ (in seconds), then we expect the total of valid shares to be:

$$S = \frac{\hat{h}t}{2^{32}D}$$

so the estimated hashrate $\hat{h}$ is:

$$\hat{h} = \frac{S2^{32}D}{t}$$

as $t \rightarrow \infty$, then the expectation $E[\hat{h}] \rightarrow E[h]$, and we can trust $\hat{h}$ as an accurate estimation of $h$.

[GitGab19]

I completely agree with this model.

To get a better accuracy and precision my suggestion would be to maintain a sum of all share_diff(s) received by a specific worker.
So every time you receive a share, you compute its share_diff and you add this value to the current sum of that worker.

In that case, the final adjusted formula would be:

$$\hat{h} = \frac{2^{32}S}{t}$$

where $S$ is the cumulative sum of all share_diff(s) submitted by the worker during a specific time period.

[mlgodskz]

Have I understood correctly that you propose using the formula: hashrate = (number_of_shares × 2^32 × difficulty) / time, where difficulty is the ratio of max_target / target (Downstream)? If so, I agree that such formula has the right to exist. And it essentially forms the basis of the algorithm I proposed. However, there's a nuance: the target for Downstream also needs to be calculated somehow, doesn't it? The beauty of my proposed approach is that it calculates hashrate based solely on the shares submitted by the worker. I'm not insisting on the exclusive correctness of my approach right now - I just want to make sure I understand you correctly.

[mlgodskz]

In that case, the final adjusted formula would be:
h ^ = 2 32 S t
where S is the cumulative sum of all share_diff(s) submitted by the worker during a specific time period.

We've already tried this approach. It doesn't work, especially over short periods, because shares can vary significantly in difficulty, and sometimes even a single very difficult share can completely skew the hashrate measurement over a period of an hour or more. However (and this just occurred to me) - we don't necessarily have to calculate miner rewards based solely on hashrate and work time. Perhaps we should consider rewarding based on the cumulative difficulty of submitted shares or something similar. At the very least, this could be a topic for discussion with the client :) Moreover: rewarding based on cumulative share difficulty is not only the simplest approach but also the most logical. Because in the end, we're giving rewards based on cumulative share difficulty anyway, since we multiply hashrate by work time, essentially doing the reverse transformations :-) Thank you.

[plebhash]

Have I understood correctly that you propose using the formula: hashrate = (number_of_shares × 2^32 × difficulty) / time, where difficulty is the ratio of max_target / target (Downstream)?

yes, that is essentially this formula:

(renaming the $N$ variable to avoid confusion with the one proposed by @GitGab19 )

$$\hat{h} = \frac{N2^{32}D}{t}$$

things to note here:

• $N$ is the number of valid shares submitted by the worker
• you never look at any individual share's difficulty $d$
• you only care about the difficulty threshold $D$ and the number $N$ of shares whose difficulty meet this criteria ($d \geq D$)
• consumes less memory and computation, with the tradeoff of less accuracy

or with a more concrete example:

$D=10$
$d_{i=0} = 10$ (valid... exact value of $d_i$ doesn't really matter, only matters that $d_i \geq D$)
$d_{i=1} = 20$ (valid... exact value of $d_i$ doesn't really matter, only matters that $d_i \geq D$)
$d_{i=2} = 30$ (valid... exact value of $d_i$ doesn't really matter, only matters that $d_i \geq D$)
$d_{i=3} = 5$ (invalid)
$N=3$

$$\hat{h} = \frac{N2^{32}D}{t} = \frac{3 \cdot 2^{32}\cdot 10}{t}$$

---

but there's also an alternative approach proposed by @GitGab19, which is kinda similar but slightly different:

let $S$ be the sum of all individual difficulties $d_i$ of all valid shares (where $d_i \geq D$):

$$ S = \sum_{i}^{N} d_i $$

you can estimate $\hat{h}$ as:

$$\hat{h} = \frac{2^{32}S}{t}$$

things to note here:

• you still have a total of $N$ shares
• you still have a difficulty threshold $D$ as a criteria for share validity, but it's not used in the formula
• more accuracy for $\hat{h}$, with the tradeoff of consuming more memory and computation

or with a more concrete example:

$D=10$
$d_{i=0} = 10$ (valid... exact value of $d_i$ matters)
$d_{i=1} = 20$ (valid... exact value of $d_i$ matters)
$d_{i=2} = 30$ (valid... exact value of $d_i$ matters)
$d_{i=3} = 5$ (invalid)
$N=3$

$$ S = \sum_{i}^{N} d_i = \sum_{i=0,1,2}^{} d_i = 10 + 20 + 30 = 60$$

$$\hat{h} = \frac{2^{32}S}{t} = \frac{2^{32} \cdot 60}{t}$$

---

I hope this clarifies things for you!

[plebhash]

However, there's a nuance: the target for Downstream also needs to be calculated somehow, doesn't it?

if we were to translate this question in terms of the formulae above, it would be:

how to calculate the ideal $D$?

the SRI Translator Proxy deals with this by establishing:

• min_individual_miner_hashrate (hashes/s of the weakest miner that will be connecting)
• shares_per_minute (target number of shares per minute the miner should be sending)

these parameters are used to establish $D$ (we call this algorithm vardiff, and we're aiming to modularize it so it can be used on any implementation via #1229), and can be used for both approaches described above

the exact values for the parameters are related to this question: #1441 (comment)

[coleFD]

Share submissions is a Poisson process (as is the number of bitcoin blocks over a period of time), and 2k samples in a poisson distribution has a stdev of 2.2%, so 68% of miners' submissions are expected to vary by 2.2% minimum. For that reason, something seems off to me