File tree Expand file tree Collapse file tree 1 file changed +17
-0
lines changed
doc/architectural_decisions Expand file tree Collapse file tree 1 file changed +17
-0
lines changed Original file line number Diff line number Diff line change @@ -34,6 +34,23 @@ documents allowed by `event-model`
3434Wherever Kafka is mentioned above, the actual implementation may be a Kafka-like (e.g. RedPanda).
3535```
3636
37+ ### Alternatives considered
38+
39+ Encoding bluesky documents into JSON and then wrapping them in the
40+ [ ` json_json.fbs ` flatbuffers schema] ( https://github.com/ess-dmsc/streaming-data-types/blob/58793c3dfa060f60b4a933bc085f831744e43f17/schemas/json_json.fbs )
41+ was considered.
42+
43+ We chose ` msgpack ` instead of json strings + flatbuffers because:
44+ - It is more standard in the bluesky community (e.g. it is the default used in ` bluesky-kafka ` )
45+ - Bluesky events will be streamed to a dedicated topic, which is unlikely to be confused with data
46+ using any other schema.
47+
48+ Performance/storage impacts are unlikely to be noticeable for bluesky documents, but nonetheless:
49+ - ` msgpack ` -encoded documents are 30-40% smaller than ` json ` + flatbuffers
50+ for a typical bluesky document
51+ - ` msgpack ` -encoding messages is ~ 5x faster than ` json ` + flatbuffers encoding
52+ for a typical bluesky document.
53+
3754## Justification & Consequences
3855
3956We will stream bluesky documents to Kafka, encoded using ` msgpack-numpy ` .
You can’t perform that action at this time.
0 commit comments