Message/PresenceMessage/Annotation size

[sacOO7]

• Extension to https://github.com/ably/specification/pull/326/files

• Added a clear specification to eliminate ambiguity when calculating sizes for each Message type.

• This also aids in annotating the code with the correct specification.

• Link to internal conversation => https://ably-real-time.slack.com/archives/C03JDBVM5MY/p1749642052882309

[sacOO7]

If data has type boolean or number then what should be it's size?

[sacOO7]

Same will be valid for PresenceMessage and Annotation

[sacOO7]

Also, couldn't find ably-js impl. for calculating PresenceMessage size, seems it's handled in a generic way as a part of
message.ts#getMessageSize

function getMessageSize(msg: WireMessage) {
  let size = 0;
  if (msg.name) {
    size += msg.name.length;
  }
  if (msg.clientId) {
    size += msg.clientId.length;
  }
  if (msg.extras) {
    size += JSON.stringify(msg.extras).length;
  }
  if (msg.data) {
    size += Utils.dataSizeBytes(msg.data);     
  }
  return size;
}

// TM6b  => This seems wrong. : (
// dataSizeBytes only checks for number, boolean, string or binary

export function dataSizeBytes(data: string | number | boolean | Bufferlike): number {
  if (Platform.BufferUtils.isBuffer(data)) {
    return Platform.BufferUtils.byteLength(data);
  }
  if (typeof data === 'string') {
    return Platform.Config.stringByteSize(data);
  }
  if (typeof data === 'number') {
    return 8;
  }
  if (typeof data === 'boolean') {
    return 1;
  }
  throw new Error(
    `Expected input of Utils.dataSizeBytes to be a string, a number, a boolean or a buffer, but was: ${typeof data}`,
  );
}

[sacOO7]

Calculation for msg.data seems wrong, because as per TM6b => The size of an @Object@ or @Array@ @data@ property is its string length after being JSON-stringified

[VeskeR]

If data has type boolean or number then what should be it's size?

This is not an accepted data type for regular Messages or PresenceMessage.
Don't know about Annotation messages, but the message size calculation for it should be discussed in a separate conversation below.

[VeskeR]

Calculation for msg.data seems wrong, because as per TM6b => The size of an @object@ or @array@ @DaTa@ property is its string length after being JSON-stringified

It is calculated correctly. The message size is calculated on the encoded (and encrypted) data, as it requires data to be represented as a string or buffer:
https://github.com/ably/ably-js/blob/d55afe78340f1f16d17f7ec9eeffc15bf6563667/src/common/lib/types/message.ts#L91-L93

That said, I can see the spec is missing the part that the size calculation should take place after encoding and encryption. This is something that should be added.

[sacOO7]

If data field after encoding is either string or buffer, then we don't need TM6b right

[VeskeR]

Uhhh, it seems like it's not that straightforward.
Based on the ably-js implementation comment, the message should be encoded and encrypted before calculating its size (which happens here). However, it should not apply protocol-specific encoding - such as representing binary data as a Base64 string in the JSON protocol (TM6c). That kind of encoding takes place at a later stage in ably-js, after message size calculation.

So currently, the spec for message size calculation kind of dances around this logic. It says that Object and Array data should be JSON-stringified in TM6b (per RSL4c3 and RSL4d3), but states that Base64 encoding should not be applied for binary data (RSL4d1 should not apply). At the same time, it doesn’t mention that the data should be encrypted (if applicable) before size calculation if client was configured to do so.

Ideally, all of these nuances should be clarified and stated explicitly in the spec for message size calculation. However, I’m not sure this should be our priority right now. Based on Simon’s comment below, it’s not a requirement to have this implemented for Annotations, and we haven’t planned to update this for Message and PresenceMessage.

Would opening a separate issue that outlines the scope of the needed changes suffice in this case?

[sacOO7]

Created separate issue #336, meanwhile you can review updated spec PR.

[VeskeR]

If data has type boolean or number then what should be it's size?

This is not an accepted data type for regular Messages or PresenceMessage.
Don't know about Annotation messages, but the message size calculation for it should be discussed in a separate conversation below.

[VeskeR]

Calculation for msg.data seems wrong, because as per TM6b => The size of an @object@ or @array@ @DaTa@ property is its string length after being JSON-stringified

It is calculated correctly. The message size is calculated on the encoded (and encrypted) data, as it requires data to be represented as a string or buffer:
https://github.com/ably/ably-js/blob/d55afe78340f1f16d17f7ec9eeffc15bf6563667/src/common/lib/types/message.ts#L91-L93

That said, I can see the spec is missing the part that the size calculation should take place after encoding and encryption. This is something that should be added.

[sacOO7]

We don't have Non-ASCII characters included in extras right

[SimonWoolf]

we definitely can, they can contain arbitrary data, eg in a push payload

[sacOO7]

Thanks will update for all string spec points 👍

[sacOO7]

updated

[SimonWoolf]

There's no reason that name, clientId, data, and json-encoded-extras strings should use different calculations for their sizes. If we want to specify "UTF-8 encoded length in bytes" for a string data (which I agree is the correct interpretation, we want to bound the encoded size over the wire), we should use that same wording on the others for consistency.

(if ably-js doesn't do this that's technically a bug, though not one that matters much. if the difference changes the result then it wasn't grossy over the limit and it can just get rejected by the server, shrug)

[sacOO7]

Okay, then it will be better to be specific for each spec point 👍
I guess we will need to update all string specific spec points.

[sacOO7]

updated