RFC: Custom Packet differenciation (aka packet ids) for NGC

[Green-Sky]

Toxcore made the choice of not requiring a packet id system in the api for custom packets.

c-toxcore/toxcore/tox.h

Lines 4744 to 4771 in 9592d59

	/**
	* Send a custom packet to the group.
	*
	* If lossless is true the packet will be lossless. Lossless packet behaviour is comparable
	* to TCP (reliability, arrive in order) but with packets instead of a stream.
	*
	* If lossless is false, the packet will be lossy. Lossy packets behave like UDP packets,
	* meaning they might never reach the other side or might arrive more than once (if someone
	* is messing with the connection) or might arrive in the wrong order.
	*
	* Unless latency is an issue or message reliability is not important, it is recommended that you use
	* lossless packets.
	*
	* The message length may not exceed TOX_MAX_CUSTOM_PACKET_SIZE. Larger packets
	* must be split by the client and sent as separate packets. Other clients can
	* then reassemble the fragments. Packets may not be empty.
	*
	* @param group_number The group number of the group the packet is intended for.
	* @param lossless True if the packet should be lossless.
	* @param data A byte array containing the packet data.
	* @param length The length of the packet data byte array.
	*
	* @return true on success.
	*/
	bool tox_group_send_custom_packet(
	const Tox *tox, Tox_Group_Number group_number, bool lossless,
	const uint8_t data[], size_t length,
	Tox_Err_Group_Send_Custom_Packet *error);

As such, here are a couple of possibilities, ideally one of which we choose to be THE ONLY way. There are however implicit compatibilities between different options.

static packet ids

integer packet id

A0 - 16bit fixed integer

Every packet is prefixed with a 16bit integer packet id. This allows for a total of 65536 possibilities.
This also allows specify subranges for individual use and tox-offical "spec", like eg the first 1024 ids or more.

A1 and A2 - 24bit and 32bit fixed integer

Like option A0, but larger number space that is very unlikely to ever be exhausted/needed.

dual integer packet id

B0 - 8/16bit switched integer

Here we let the first bit decide whether the id is 8bit or 16bit (effectively 7bit and 15bit ranges). This allows for us to save an extra byte for packet types we deem profiting from this. eg. filetransfers will send a lot of data and ack type packets that can collectively waste bytes in the gigabytes ranges or more each year. Since the first bit is the switch, this only allows for 128 different ids, so they would be rare. It also cuts the larger id into 32768 possibilities. Which is probably still sufficient. Individually the ranges act similar to option A0 (and A1 and A2).

B1 and B2 - 8/24bit and 8/32bit switched integer

Like option B0, but larger second range. (also see option A1 and A2)

large integer packet id with namespace subdivision(s)

C0 - 64bit (48bit namespace + 8bit packet id + 8bit packet version)

This is conceptually like options A0, A1 and A2, but assignes different bytes different meanings.
The first 6 bytes are used as a user/org specific namespace (aka magic), the next byte specifies the packet id and the last byte a packet version, for future compatibility checks.
The advantage here is the clear separation between individuals and protocol changes.
The disadvantage is the size, 64bit just to identify a packet type seems excessive.

C1 - 32bit (16bit namespace + 8bit packet id + 8bit packet version)

Same as option C0 but with a more reasonable namespace size. Still pretty large.

C2 - 24bit (12bit namespace + 12bit packet id)

Same as option C0 but with an even smaller namespace (4096) and smaller and combined packet id + packet version space (4096). The packet id and version amalgamation also means less space is wasted for the unlikely case of version filling the full 256 possibilities.

C3 - 8/24bit and 8/32bit switched namespaced integer

This is a combination of options B0 (B1, B2) and C2/C1. Where the 8bit(7bit) numbers are tox-official small packet id designated and the larger variant is option C2 or C1 with some bits moved around (either namespace or packet id(/version) has 1bit less).

dynamic packet ids

initial mapping announce

D0 - full packet id mapping exchanged at/after connection / before use

Here we exchange all the custom packets we understand and which ephemeral packet id to prefix each packet with. This might take multiple mapping announce packets. Then the other side knows which ephemeral id they should use to make data be treated as a certain packet type. Each packet type will be specified by a 64bit UUID (parts of which is a namespace). If a mapping announce packet is not ACKed, we resend the packet up to 3 times after each timeout.
The mapping of a UUID to a ephemeral id is static, once announced. Later mapping announces can still add new mappings.

TODO: more variations of D0 ?

E0 - toxext (modified)

see https://github.com/toxext/toxext/blob/master/DESIGN.md
With toxext, one sides sends UUID + MY_IDENTIFIER and the other side replies with YOUR_IDENTIFIER.
The toxext magic will not be necessary, since we will designate every other use as off-spec.

---

If you have ideas for more variants, please let me know so I can add them to the list 📬 .
At some point we will have to discuss it more and settle on one of the options 🗣️ .
Timeline for this is to be settled on one of the options by the time v0.2.19 released (so very soon).

[zoff99]

dont forget to establish a package registry ala https://github.com/zoff99/toxcore_custom_packets_registry

[zoff99]

also i am using packages with these magic first 6 bytes already: 0x667788113435
so mark those as "already registered" please.
depending on how many bytes it will be: 0x6677 if it will be the first 2

[Green-Sky]

also i am using packages with these magic first 6 bytes already: 0x667788113435 so mark those as "already registered" please. depending on how many bytes it will be: 0x6677 if it will be the first 2

yep, was my plan.

dont forget to establish a package registry ala https://github.com/zoff99/toxcore_custom_packets_registry

will do once we settle on the packet id type.

[Green-Sky]

Ok, ngc got a release. Pinging so we can move this along.
I'd like to hear the opinion from @nurupo @JFreegman @iphydf @robinlinden and everyone interested :)