arma3pbo

This repository is a small way on how to pack folders into pbo files. This Python script is implemented with TDD. Also we are frickling around with ctypes, many bytearrays and pretty low-level-file-system-style topics. Was pretty fun to implement it!

License

"Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)" https://creativecommons.org/licenses/by-sa/4.0/

Requirements

You must know the principles of TDD and be skilled in python programming.

Project anatomy

This project is written with the TDD principles. So the tests are found in the tests folder. A great way on how to run them is via any IDE where my favorite choice is PyCharm (Professional).
The production code is found in the src folder where an python module is placed which has the same name as the project.

Development, the anatomy of a pbo file (bytelevel)

The Head-, Body- and Footblock are seperated by a 0-byte.

Head (pbo_head.py)

All .pbo files are beginning with an

arma3pbo/src/arma3pbo/pbo/pbo_head.py

Line 9 in a57ccc2

head_start_bytecode = bytearray(b"\0sreV\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0")

After then, all the so-called pboprefixes are added. This happens by adding the following bytes:

arma3pbo/src/arma3pbo/pbo/pbo_head.py

Lines 13 to 19 in a57ccc2

	def create_header_bytecode(key, value):
	bytes_from_entry_fields = bytearray()
	bytes_from_entry_fields.extend(key.encode("utf-8"))
	bytes_from_entry_fields.extend(b"\0")
	bytes_from_entry_fields.extend(value.encode("utf-8"))
	bytes_from_entry_fields.extend(b"\0")
	return bytes_from_entry_fields

These Prefixes are optional and are only added if a file $<filename>$ with content lies in the project root of the folder.
Lastly the pbofile must know what files with what metadata is found in the project structure. A great source for that was the wiki from BI (https://community.bistudio.com/wiki/PBO_File_Format). An Entry is added with the following code:

arma3pbo/src/arma3pbo/pbo/pbo_head.py

Lines 21 to 44 in a57ccc2

	@staticmethod
	def header_entry_from_file(base_path, file_path):
	complete_path = base_path + "/" + file_path
	return HeaderEntry(
	filename=tools.convert_to_winPath(file_path),
	mimetype=ctypes.c_uint32(0),
	original_size=ctypes.c_uint32(os.path.getsize(complete_path)),
	reserved=ctypes.c_uint32(0),
	timestamp=ctypes.c_uint32(int(os.path.getmtime(complete_path))),
	data_size=ctypes.c_uint32(int(os.path.getsize(complete_path)))
	)
	@staticmethod
	def create_bytecode_from_entry(entry: HeaderEntry):
	bytes_from_entry_fields = bytearray()
	bytes_from_entry_fields.extend(entry.filename.encode("utf-8"))
	bytes_from_entry_fields.extend(b"\0")
	bytes_from_entry_fields.extend(bytearray(entry.mimetype))
	bytes_from_entry_fields.extend(bytearray(entry.original_size))
	bytes_from_entry_fields.extend(bytearray(entry.reserved))
	bytes_from_entry_fields.extend(bytearray(entry.timestamp))
	bytes_from_entry_fields.extend(bytearray(entry.data_size))
	return bytes_from_entry_fields

Important to know is that all the numbers are saved in little endian uint32

Body (pbo_body.py)

This part is easy. Here all the files from the project are added with their contents. Its possible to compress these files, but its not really necesarry anymore. Arma 3 was written nearly 1,5 decaded ago where network speed and optimizing attempts were valued higher than performance. Nowadays this isn't really a problem anymore and in addition to that the engine must decompress the whole thing which may slightly decrese performance.
The only important thing here is that the bytecode of the files are in the same order as in the header-entrys. I am just looping twice over the file array here.

Foot (pbo_foot.py)

The last 20 bytes are an SHA1 Hash-Value based on the preceding bytes.

arma3pbo/src/arma3pbo/pbo/pbo_foot.py

Lines 5 to 7 in a57ccc2

	@staticmethod
	def get_hash(bytes):
	return hashlib.sha1(bytes).digest()

How to use

This packs a folder to a pbo file:

python3 src/arma3pbo/main.py build -i <input-folder> -o <output-file>