Streaming get headers#32
Conversation
Streaming decompression approach that stops after reading the header and script, skipping the demo stream and statistics entirely. 4-25x faster and up to 92% less memory than skipPackets on tested replays. Deduplicates header parsing into a shared static parseHeader method. Does not include winningAllyTeamIds or statistics (requires full file).
fcaps
force-pushed
the
streaming_get_headers
branch
from
85a8e1f to
146887c
Compare
p2004a
left a comment
p2004a
left a comment
There was a problem hiding this comment.
This isn't streaming decompression but it duplicates a bunch of code to extract header only. It doesn't close #30 which also targets lowering overall memory usage and unblocking the even loop a bit during decompression.
I don't think there is urgency to merge this workaround instead of reworking the rest of the implementation properly.
|
Yeah, i was kinda only using it for my use-case: fast indexing/filtering demofiles without the commands, but if your current indexer (BAR-DB?) has no issues with keeping up, then for sure, reworking it all would make more sense. For me the memory usage when processing the whole demofile is negligible, since i have to hold all used messages in memory anyways for later usage. |
2 participants
Hey,
a working slimfit version of skipPackets, it has no statistics (and no winningTeam), but it is fast and could run on a casio watch.
Sadly the statistics-packets are at the end of the file and we would need to decompress the whole file for the winning team, but this is a start imo. related #30
closes: #30
Cheers
PS: I directly branched from #31 , so you need to only review the second commit here