refactor task bridge to handle async tasks concurrently, preventing potential duplicate submission#600
Conversation
JannikSt
left a comment
There was a problem hiding this comment.
Nice! This looks a lot more solid. Same for this change on the task bridge I would actually run this on a real GPU now with a proper workload for testing before merging this.
Also what do you think of using FuturesUnordered instead of the JoinSet that requires tokio unstable?
|
@JannikSt going to copy paste my response here: I'm using |
JannikSt
left a comment
There was a problem hiding this comment.
minor comment - otherwise good to go!
| CARGO_TERM_COLOR: always | ||
| CARGO_INCREMENTAL: 0 | ||
| RUSTFLAGS: "-C debuginfo=0" | ||
| RUSTFLAGS: "-C debuginfo=0 --cfg tokio_unstable" |
There was a problem hiding this comment.
this is no longer required right?
handle_file_validationwhich includes the on-chain call tosubmitWork()) was being submitted multiple times for the same work (identified by hash). this was likely also happening for file upload as well. a potential cause of this is because the call tohandle_file_validationwas beingtokio::spawned, so it was possible for multiple submission tasks for the same hash to be startedTaskBridgesuch that all futures run concurrently inside oneselect!{..}statement instead of being spawnedhandle_file_validationinside a tokioJoinMap, which stores the futures with the file hash as the key. thus duplicatehandle_file_validationcalls cannot run at the same time.closes #596