You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm working on a new feature to resume interrupted file transfers at a later time. For this feature to work, I need to ensure that the value in my uploadedBytes variable matches exactly what the API, that is receiving this file, has written to the disk. However, I'm facing issues with my current implementation.
When aborting the upload with this.#abortSignal, some bytes reported by GOT's uploadProgress event are never saved to the server's disk. As ongoingRequestPromise.cancel() closes the connection, I assume these bytes might be lost in various places, such as in the network card buffer or the server's buffer, or discarded because the server's CancellationToken was cancelled before they could be written to the disk.
New approach
To address this, I decided to gracefully destroy the file stream, assuming that if I manage to close the file stream gracefully, GOT will read all buffered bytes from the stream, send them, and then fulfill the promise. Although the abort operation won't be immediate, in my scenario, this is acceptable as long as I'm sure that the server safely writes all sent bytes to its disk and my uploadedBytes variable remains accurate.
Nodified code
this.#fileHandle =awaitfs.open(this.#fullFilePath,fsConsts.O_RDONLY);this.#fileReadStream =this.#fileHandle.createReadStream({start: this.chunkStartPosition,end: this.getChunkEndPosition(),});// This event handler is newthis.#abortSignal.addEventListener(`abort`,()=>{// I tried every combination of below but no luck herethis.#fileReadStream?.push(null);this.#fileReadStream?.close();this.#fileReadStream?.emit(`end`);this.#fileReadStream?.unpipe();this.#fileReadStream?.destroy();});letuploadedBytes=0;constongoingRequestPromise=gotChunkInstance.post(this.#uploadUrl,{body: this.#fileReadStream,headers: {'content-length': this.chunkLengthInBytes.toString(),},});ongoingRequestPromise.on(`uploadProgress`,progress=>{uploadedBytes=progress.transferred;});awaitongoingRequestPromise;
Issue
This correctly stops the upload gracefully, but unfortunately, ongoingRequestPromise is never fulfilled. If I change my abort signal handler to this.#fileReadStream?.destroy(new Error());, an error is thrown, and the connection is closed immediately, making it no different from the initial approach using ongoingRequestPromise.cancel();.
Questions
Is this a bug?
Is there a different way to achieve what I want?
I'm using GOT 14.2.x and NodeJS 20.x.x
Thank you for your help!
The text was updated successfully, but these errors were encountered:
When refactoring my code for this feature I forgot to change the header from content-length to 'Transfer-Encoding': 'chunked'. With that change, GOT gracefully closes the request, and all data is correctly processed by the server.
I'm working on a new feature to resume interrupted file transfers at a later time. For this feature to work, I need to ensure that the value in my
uploadedBytes
variable matches exactly what the API, that is receiving this file, has written to the disk. However, I'm facing issues with my current implementation.Simplified code:
Server-Side Pseudocode
The server code handling this request can be represented with the following pseudocode:
Problem
When aborting the upload with
this.#abortSignal
, some bytes reported by GOT'suploadProgress
event are never saved to the server's disk. AsongoingRequestPromise.cancel()
closes the connection, I assume these bytes might be lost in various places, such as in the network card buffer or the server's buffer, or discarded because the server'sCancellationToken
was cancelled before they could be written to the disk.New approach
To address this, I decided to gracefully destroy the file stream, assuming that if I manage to close the file stream gracefully, GOT will read all buffered bytes from the stream, send them, and then fulfill the promise. Although the abort operation won't be immediate, in my scenario, this is acceptable as long as I'm sure that the server safely writes all sent bytes to its disk and my
uploadedBytes
variable remains accurate.Nodified code
Issue
This correctly stops the upload gracefully, but unfortunately,
ongoingRequestPromise
is never fulfilled. If I change my abort signal handler tothis.#fileReadStream?.destroy(new Error());
, an error is thrown, and the connection is closed immediately, making it no different from the initial approach usingongoingRequestPromise.cancel();
.Questions
I'm using GOT 14.2.x and NodeJS 20.x.x
Thank you for your help!
The text was updated successfully, but these errors were encountered: