Structured cbor #3036

JesusMcCloud · 2025-07-05T05:03:02Z

This PR introduces structured CBOR encoding and decoding

Encoding from/to `CborElement`

Bytes can be decoded into an instance of CborElement with the [Cbor.decodeFromByteArray] function by either manually
specifying [CborElement.serializer()] or specifying [CborElement] as generic type parameter.
It is also possible to encode arbitrary serializable structures to a CborElement through [Cbor.encodeToCborElement].

Since these operations use the same code paths as regular serialization (but with specialized serializers), the config flags
behave as expected

Newly introduced CBOR-specific structures

[CborPrimitive] represents primitive CBOR elements, such as string, integer, float boolean, and null.
CBOR byte strings are also treated as primitives
Each primitive has a [value][CborPrimitive.value]. Depending on the concrete type of the primitive, it maps
to corresponding Kotlin Types such as String, Int, Double, etc.
Note that Cbor discriminates between positive ("unsigned") and negative ("signed") integers!
CborPrimitive is itself an umbrella type (a sealed class) for the following concrete primitives:
- [CborNull] mapping to a Kotlin null
- [CborBoolean] mapping to a Kotlin Boolean
- [CborInt] which is an umbrella type (a sealed class) itself for the following concrete types
  (it is still possible to instantiate it as the invoke operator on its companion is overridden accordingly):
  - [CborPositiveInt] represents all Long numbers ≥0
  - [CborNegativeInt] represents all Long numbers <0
- [CborString] maps to a Kotlin String
- [CborFloat] maps to Kotlin Double
- [CborByteString] maps to a Kotlin ByteArray and is used to encode them as CBOR byte string (in contrast to a list
  of individual bytes)
[CborList] represents a CBOR array. It is a Kotlin [List] of CborElement items.
[CborMap] represents a CBOR map/object. It is a Kotlin [Map] from CborElement keys to CborElement values.
This is typically the result of serializing an arbitrary

Example

bf                                 # map(*)
   61                              #   text(1)
      61                           #     "a"
   cc                              #   tag(12)
      1a 0fffffff                  #     unsigned(268,435,455)
   d8 22                           #   base64 encoded text, tag(34)
      61                           #     text(1)
         62                        #       "b"
                                   #     invalid length at 0 for base64
   20                              #   negative(-1)
   d8 38                           #   tag(56)
      61                           #     text(1)
         63                        #       "c"
   d8 4e                           #   typed array of i32, little endian, twos-complement, tag(78)
      42                           #     bytes(2)
         cafe                      #       "\xca\xfe"
                                   #     invalid data length for typed array
   61                              #   text(1)
      64                           #     "d"
   d8 5a                           #   tag(90)
      cc                           #     tag(12)
         6b                        #       text(11)
            48656c6c6f20576f726c64 #         "Hello World"
   ff                              #   break

Decoding it results in the following CborElement (shown in manually formatted diagnostic notation):

CborMap(tags=[], content={  
    CborString(tags=[],   value=a) = CborPositiveInt( tags=[12],     value=268435455),  
    CborString(tags=[34], value=b) = CborNegativeInt( tags=[],       value=-1),  
    CborString(tags=[56], value=c) = CborByteString(  tags=[78],     value=h'cafe),  
    CborString(tags=[],   value=d) = CborString(      tags=[90, 12], value=Hello World)  
})

Implementation Details

I tried to stick to the existing CBOR codepaths as closely as possible, and the approach to add tags directly to CborElements is the most pragmatic way of getting expressiveness and convenient use. It does come with a caveat (also taken from the Readme:

Tags are properties of CborElements, and it is possible to mixing arbitrary serializable values with CborElements that
contain tags inside a serializable structure. It is also possible to annotate any [CborElement] property
of a generic serializable class with @ValueTags.
This can lead to asymmetric behavior when serializing and deserializing such structures!

The test cases (and comments in the test cases reflect this

Closing Remarks

I also fixed a faulty hex input test vector that I introduced myself, last year, if I pieced it together correctly (see here) and I amended the benchmarks. (see here).

Since the commits from here will be squashed anyways, I did not care for a clean history.

JesusMcCloud · 2025-07-08T10:28:59Z

Full disclosure: This PR incorporates code from a draft generated by Junie (albeit an impressive draft that saved a day of work). This is not a dumb copypasta of AI-generated code. Even if it were already feature-complete It would still not yet be marked ready for review because we have yet to review everything internally. I also want to stress that "we" is not a euphemism. There will be at least two of us reviewing and discussing internally, almost certainly with additional input from other humans in the process of readying this PR.

pdvrieze

I've reviewed the code at a general level. There is a lot of repetition, so I've only commented on the first case, not every one. I guess some of the AI generation is visible in the details.

formats/cbor/commonMain/src/kotlinx/serialization/cbor/Cbor.kt

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborElement.kt

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/CborElementSerializers.kt

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborElement.kt

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/CborElementSerializers.kt

JesusMcCloud · 2025-08-07T11:29:28Z

Performance seems to be OK (fromBytes and toBytes are the baseline on my machine):

Metric / Benchmark	`fromBytes`	`fromStruct`	`structFromBytes`	`toBytes`	`structToBytes`	`toStruct`
Average (ops/ms)	1205.615 ± 20.541	1545.814 ± 50.743	2896.728 ± 74.485	2089.013 ± 30.152	1442.766 ± 32.257	2581.397 ± 32.497
Min	1186.023	1458.225	2796.131	2066.499	1404.482	2550.026
Max	1229.778	1581.420	2960.572	2125.658	1475.015	2619.815
Stdev	13.586	33.563	49.267	19.944	21.336	21.495
CI low (99.9 %)	1185.075	1495.071	2822.244	2058.861	1410.509	2548.900
CI high (99.9 %)	1226.156	1596.557	2971.213	2119.165	1475.023	2613.893

My hot takes:

Deserialising from a structure is fast enough since it is in the same ballpark as deserialising from bytes
Deserialising into a generic CBOR structure takes twice the time than directly deserialising, which is fine, given that we instantiate much more as even primitives need a containing class and an array of tags
Serialising a generic CBOR structure to bytes is faster but in the same ballpark as generic to-byte serialisation of arbitrary serializable data
Serializing to a CBOR structure is slower than to bytes, but OK enough, since it's in the same ballpark and we instantiate more

JesusMcCloud · 2025-08-07T12:12:51Z

I just noticed something that looks weird to me. See this test case here that is failing and closely compare expected vs actual.

the byte string is wrapped twice for the reference. I know there were some discussions, but I don't recall them, so I have to ask: why? did I mess this up last year or is this intentional? Because the way I see it, were' wrapping a bytearray instead of encoding it differently
EDIT: the test vector is faulty as this comparison fails the same way

whyoleg

First round of review from my side :)
Mostly reviewed API surface, will take a look on the implementation details later

whyoleg · 2025-08-08T11:34:55Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborElement.kt

+    }
+}
+@Serializable(with = CborIntSerializer::class)
+public sealed class CborInt<T : Any>(


According to RFC 8949, integers are:

type 0 - 0..2^64-1

type 1 - -2^64..-1

This means that only ULong can really fit both positive and negative values.

So I believe that it will be more correct to have a single CborInt implementation, with an API similar to:

public class CborInt(sign: Int, absoluteValue: ULong, tags) { constructor(value: Long) fun toLong(): Long fun toLongOrNull(): Long? }

where:

sign is: -1, 0, 1

toLong - will throw if can't fit - probably a very rare case

Good catch! I agree. The constructors with the current signature (simply passing a number) should stay for convenience.

whyoleg · 2025-08-08T11:37:07Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborElement.kt

+ * traditional methods like [List.get] or [List.size] to obtain CBOR elements.
+ */
+@Serializable(with = CborListSerializer::class)
+public class CborList(


It might make sense to call it CborArray, as spec says, it's array, and Json format also calls it JsonArray because it's called array in spec :)

Sadly, there is an annotation by the very same name ins the same package. Now moving the cbor elements is easily done, but it will probably not be very intuitive (thinking of autocompletion here).

In any case: Good point but not my call!

Oh, yes, how did I miss that :(

But this is on my anyhow because git blame says "Prünster" for the whole file containing the @CborArray annotation. So my short-sighted naming is to blame.

whyoleg · 2025-08-08T11:39:31Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborElement.kt

+     * See [RFC 8949 3.4. Tagging of Items](https://datatracker.ietf.org/doc/html/rfc8949#name-tagging-of-items).
+     */
+    @OptIn(ExperimentalUnsignedTypes::class)
+    public var tags: ULongArray = tags


I believe we should make the implementation in a way, that it will be not mutable

We can make it so that there is a val with a custom getter that delegates to the actual field, but the tags should still be simply written to a CborElement as they are collected upon deserialisation, to avoid serialization overhead. Actually, the elements should probably be mutable too internally so the whole container structure that now collects tags and elements for Cbor structures can go away.

In the end this would mean that we have:

an internal var making it possible to collect tags during deserialisation, but a public val tags with a custom getter and no backing field

an internal val elements for array and map so we can also collect elements as we go and get rid of another instantiation, but a public val elements: List<CborElement>

This should boost performance for deserialisation significantly

Of course, this is related to #3036 (comment), but it would also be nice to compare it to how JSON works with lists. Maybe there is some pattern there that could be used inside the CBOR implementation.

whyoleg · 2025-08-08T11:44:19Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborDecoder.kt

+     * This method is allowed to invoke only as the part of the whole deserialization process of the class,
+     * calling this method after invoking [beginStructure] or any `decode*` method will lead to unspecified behaviour.
+     */
+    public fun decodeCborElement(): CborElement


I believe that there should also be CborEncoder.encoderCborElement as in Json

That just delegates to encodeToByteArray<CborElement>, right?!

Most probably to encodeSerializableValue(CborElementSerializer, element), but yes, just for symmetry

whyoleg · 2025-08-08T11:55:24Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/CborElementSerializers.kt

+    )
+
+/*need to expose writer to access encodeTag()*/
+internal fun Encoder.asCborEncoder() = this as? CborWriter


I do think that we should probably make it possible to cast only to CborEncoder here so that we use the same public API as the user.
As far as I see, the only thing needed is encodeTags—so is this a sign that encodeTags might be exposed on the CborEncoder interface?
I'm not sure, though, how useful it could be in reality, but it seems like it should be possible to encode tags manually.

You are probably on to something here. I should investigate tag handling some more

whyoleg · 2025-08-08T18:39:16Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborElement.kt

+@Serializable(with = CborPositiveIntSerializer::class)
+public class CborPositiveInt(
+    value: ULong,
+    vararg tags: ULong


This vararg feels very unfortunate. For example, currently, it's possible to call CborPositiveInt(1u, 2u, 3u), and it's really hard to tell what's going on here.
UlongArray will probably be better

Overall, it's applied to all other declarations; e.g., CborString("1", 1, 2, 3) doesn't feel better.

I agree with the first part, but the second part is simply in line with how the tagging annotations behave and it is convenient for the user (much more so that ulongArrrayOf().
Yet, I am aware that it should be consistent, so making ints work differently from everything else is also not really nice… In the end, don't really have an opinion. Sort it out internally, it's a straight-forward refactor anyways.

What would help in such cases is a language feature that forces parameter name specification at the call site, but that is not going to happen anytime soon, if ever.

Just in case, for me additional tags field looks very similar to how annotations are applied currently.
Additionally, with Collection Literals it will look like CborString("hello", [1u, 2u, 3u]) or with Named-only parameters it could even be forced to call it like CborString("hello", tags = [1u, 2u, 3u])

Those two features would help, but in the meantime, we need something else for a solution

whyoleg · 2025-08-08T18:50:23Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborElement.kt

+ * The whole hierarchy is [serializable][Serializable] only by [Cbor] format.
+ */
+@Serializable(with = CborElementSerializer::class)
+public sealed class CborElement(


What do you think about the alternative design:

sealed class CborElement { abstract val tags: ULongArray abstract fun withTags(tags: ULongArray): CborElement } // and all other class CborString(value: String): CborElement() { override fun withTags(tags: ULongArray): CborString }

Or, even having CborElement have no tags at all, but have a wrapper element CborTagged?

sealed class CborElement // and all other class CborString(value: String): CborElement() class CborTagged(element: CborElement, tags: ULongArray): CborElement

Both variants may apply some performance penalty, though.
It's just that, reading the spec and current implementation of CborElementSerializers, the current placement of tags seems unfortunate and does not result in straightforward logic.

The first one solves the issues nicely and it would work well with the internally mutable tags array s.t. it would avoid instantiation overhead just for modifying tags. BUT, that would be a footgun for the consumer.

I have thought about CborTagged, also because of the asymmetric serialization/deserialization behaviour, which really is not all that nice, but also very niche (why would anyone in their right mind do this??)

When I started with the implementation, I was imagining how the most straight-forward and usable way would look like: Tags are attached to something and have a hierarchy. An array is enough to represent this and makes for a nice API that mirrors the pattern of @ValueTags, @ObjectTags, and @KeyTags. Now you could get rid of that pesky asymmetry, if you disallow those annotations on CborElement. Then I'd assume we'd be back to straightforward logic again?!

Bottom line: there are issues that need sorting out. The inconsistency issue is documented in the readme, so I was not trying to cover anything up, but I wanted unbiased opinions and fresh ideas because I could not get to a satisfactory solution by myself. It worked once already. I just hope Leonid checks the code first, and the comments later ;-)

whyoleg · 2025-08-09T08:58:36Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborElement.kt

By the way, all new declarations/APIs should probably have an @ExperimentalSerializationApi

I think it is OK to not annotate them all, since the whole CBOR part is formally experimental. Although people probably don't perceive it so without annotations.

IMO, it will just feel a bit odd if those are not annotated, while all other declarations in CBOR are annotated.
I would treat that as if the CborElement hierarchy is stable (or stabilized), but other parts are not(yet).

pdvrieze reviewed Jul 8, 2025

View reviewed changes

JesusMcCloud mentioned this pull request Jul 23, 2025

Use Map delegate for COSE/JOSE data structures a-sit-plus/signum#335

Open

JesusMcCloud changed the base branch from master to dev August 5, 2025 11:26

JesusMcCloud force-pushed the feature/structuredCbor branch 2 times, most recently from 674e7ef to aea3f4b Compare August 7, 2025 10:12

JesusMcCloud added 17 commits August 7, 2025 12:13

WIP strucutred cbor

1d3574c

proper when for decoding

d561069

baseline tagging

5857ef0

some cleanups

718422f

cleanups

6984f69

tree encoding half-done

998f01e

tree encoding close

1a46b06

polish encoder

46c8258

visibility fixes

f6bc421

more checks

890b284

WIP decode from CborElement

255f250

more AI slop

4e15cc2

cleanup after Junie

834fa67

clean up more

d13ec50

fix structural issues

9986e7b

fix map size regression

2a61e81

streamlining and cleanups

342500b

JesusMcCloud force-pushed the feature/structuredCbor branch 2 times, most recently from cd963c7 to c89fd46 Compare August 7, 2025 11:27

JesusMcCloud force-pushed the feature/structuredCbor branch from 7cdbe4d to 9b3f0e5 Compare August 8, 2025 13:22

JesusMcCloud added 2 commits August 8, 2025 16:35

benchmarks

f681698

add more tests

b154640

JesusMcCloud added 3 commits August 8, 2025 16:35

clarify faulty test vector

397c5c2

fix test vector

ebd87c9

Fix Tagging and Simplify

8d49f2c

JesusMcCloud force-pushed the feature/structuredCbor branch from e3c207b to 8d49f2c Compare August 8, 2025 14:35

JesusMcCloud added 3 commits August 8, 2025 16:45

finalize api

6ae839d

APIDUMP

f9dc353

docs + apidump

259966f

JesusMcCloud marked this pull request as ready for review August 8, 2025 15:54

whyoleg reviewed Aug 8, 2025

View reviewed changes

whyoleg reviewed Aug 9, 2025

View reviewed changes

Structured cbor #3036

Are you sure you want to change the base?

Structured cbor #3036

Uh oh!

Conversation

JesusMcCloud commented Jul 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Encoding from/to CborElement

Newly introduced CBOR-specific structures

Example

Implementation Details

Closing Remarks

Uh oh!

JesusMcCloud commented Jul 8, 2025

Uh oh!

pdvrieze left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JesusMcCloud commented Aug 7, 2025

Uh oh!

JesusMcCloud commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

whyoleg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JesusMcCloud commented Jul 5, 2025 •

edited

Loading

Encoding from/to `CborElement`

JesusMcCloud commented Aug 7, 2025 •

edited

Loading