chore: Reduce block proving timeout worst case

[sergerad]

Closes #1855.

Also adds a commit missed in last block proving PR (return new_tip instead of vec).

[sergerad]

@Mirko-von-Leipzig regarding trace structure:

inner

INFO     ┝━ prove_block.generate [ 73.3µs | 14.39% ] block.number: 180
ERROR    ┝━ 🚨 [error]: block proving attempt timed out, retrying | target: "miden-store" | block.number: 180 | attempt: 1 | elapsed: deadline has elapsed

outer

INFO     prove_block [ 339µs | 67.55% / 100.00% ] block.number: 181
INFO     ┝━ prove_block.generate [ 110µs | 32.45% ] block.number: 181
ERROR    ┕━ 🚨 [error]:  | error: block proving overall timeout (0ns) exceeded

Error: Component store failed

Caused by:
    0: failed while serving store component
    1: proof scheduler fatal error
    2: block proving overall timeout (0ns) exceeded
    3: deadline has elapsed

[Mirko-von-Leipzig]

Code lgtm, spans are still not quite what I would expect.

To be more specific: we should almost never be using the event! style macros. We should be storing almost all information in the span, including if that span error'd.

An error happens? Mark the span as an error, don't emit an error!.

Want to emit some values? Attach it to the span. I would only expect to see event! macros if there is some sort of tiny subevent that happens within the span where it doesn't make sense to attach it. An example would be committing a block, and you might want to trace the transactions being committed so you emit

for tx in block.transactions() {
    info!(transaction.id, "transaction committed");
}

The sort of span tree I would expect is something like (names are just placeholders):

prove_block (%block.number, %block.commitment, etc)
  - remote_prover (%retry=0, %retry.limit=N, error=%..)
  - remote_prover (%retry=1, ..)
  - save_proof
  - ...

[sergerad]

@Mirko-von-Leipzig note that I have removed the error!() events for the transient failures so all of those would only be logged once the parent span closes (after overall timeout).

 INFO     prove_block [ 1.56ms | 58.62% / 100.00% ] block.number: 34
 INFO     ┝━ prove_attempt [ 494µs | 5.74% / 31.76% ] attempt: 1 | error: hi
 INFO     │  ┕━ prove_block.generate [ 405µs | 26.03% ] block.number: 34
 ERROR    │     ┕━ 🚨 [error]:  | error: transient error: hi
 INFO     ┝━ prove_attempt [ 150µs | 6.24% / 9.61% ] attempt: 2 | error: hi
 INFO     │  ┕━ prove_block.generate [ 52.5µs | 3.37% ] block.number: 34
 ERROR    │     ┕━ 🚨 [error]:  | error: transient error: hi
 ERROR    ┕━ 🚨 [error]:  | error: block proving overall timeout (10s) exceeded

Spans without failure (notice the proven_tip gap fill after in-sequence block arrives)

INFO     prove_block [ 176µs | 33.17% / 100.00% ] block.number: 79 | proven_chain_tip: 38
INFO     ┝━ store.block_store.save_proof [ 14.2µs | 8.08% ] block.number: 79 | proof_size: 0
INFO     ┝━ mark_proven_and_advance_sequence [ 31.1µs | 17.71% ]
INFO     ┕━ prove_attempt [ 72.1µs | 12.90% / 41.04% ] attempt: 1
INFO        ┕━ prove_block.generate [ 49.5µs | 25.56% / 28.14% ] block.number: 79
INFO           ┕━ prove [ 4.54µs | 2.58% ]
INFO     prove_block [ 202µs | 35.71% / 100.00% ] block.number: 62 | proven_chain_tip: 38
INFO     ┝━ store.block_store.save_proof [ 15.6µs | 7.73% ] block.number: 62 | proof_size: 0
INFO     ┝━ mark_proven_and_advance_sequence [ 39.7µs | 19.64% ]
INFO     ┕━ prove_attempt [ 74.6µs | 9.34% / 36.92% ] attempt: 1
INFO        ┕━ prove_block.generate [ 55.8µs | 25.50% / 27.58% ] block.number: 62
INFO           ┕━ prove [ 4.21µs | 2.08% ]
INFO     prove_block [ 859µs | 0.00% / 100.00% ] block.number: 39 | proven_chain_tip: 84
INFO     ┝━ store.block_store.save_proof [ 17.6µs | 2.05% ] block.number: 39 | proof_size: 0
INFO     ┝━ mark_proven_and_advance_sequence [ 56.0µs | 6.52% ]
INFO     ┕━ prove_attempt [ 785µs | 0.00% / 91.43% ] attempt: 1
INFO        ┕━ prove_block.generate [ 785µs | 90.83% / 91.43% ] block.number: 39
INFO           ┕━ prove [ 5.17µs | 0.60% ]