Skip to content

Conversation

@kartoFlane
Copy link
Member

This PR is primarily motivated by itb-community/ftldat-rs#1, which (among other things) asked for ftldat-rs to be published as a standalone package that other Rust projects could interact with.

This represented an opportunity to reduce duplication in both ftldat-rs and itb-io projects, since both of those projects had to include their own copies of files that allowed them to export Lua bindings.

As a result, I created https://github.com/itb-community/itb-rs-lua, which includes ftldat-rs as a dependency (crate), and fully incorporates https://github.com/itb-community/itb-io-rs, with an improved project structure that should be easier to extend in the future.

Changes in file sizes:

Old:  ftldat.dll + itb_io.dll = 395kB + 541kB = 936kB
New:  itb_rs_lua.dll                          = 696kB

While preparing ftldat-rs for publishing, I also overhauled the way it loads files. During reading, files are now memory-mapped, rather than read into a buffer. Also, the in-memory representation of the .dat file now only stores a list of entry headers, rather than their full contents, which dramatically reduces memory usage. In practice, there aren't any noticeable speed improvements, but ITB's memory usage no longer spikes into over 350MB range during startup.

The fact that the archive file is now memory-mapped means that is important to properly destroy each FtlDat instance (by invoking FtlDat:destroy), as it actually holds system resources now.

Since files cannot be overwritten while there exists a lock on them, we cannot overwrite resources.dat while an FtlDat instance exists. To resolve this issue, when writing to a file, FtlDat instances first write to a temporary file, release system resources, and then replace the destination file with the temporary file. This means that once the FtlDat:write method is invoked, that instance becomes invalid and cannot be used anymore.
This also means that when there are multiple FtlDat instances created from the same source file at once, it is impossible to overwrite their source file. In practice, this shouldn't be an issue, as the mod loader is the only client of FtlDat, and provides hooks that mods use to apply their desired changes.

@kartoFlane kartoFlane added the enhancement New feature or request label Apr 17, 2023
@kartoFlane kartoFlane marked this pull request as draft April 17, 2023 00:24
@kartoFlane kartoFlane changed the title Draft: Merge ftldat.dll and itb_io.dll into a single .dll Merge ftldat.dll and itb_io.dll into a single .dll Apr 17, 2023
@kartoFlane kartoFlane marked this pull request as ready for review April 23, 2023 12:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants