|
| 1 | + |
| 2 | + |
| 3 | ++++ |
| 4 | +title = "We Prefer Extensions Over Any" |
| 5 | +weight = 93 |
| 6 | +description = "Covers why the Protobuf team regrets Any as a replacement for extensions, and why extensions are preferred for most use cases" |
| 7 | +type = "docs" |
| 8 | ++++ |
| 9 | + |
| 10 | +When proto3 was introduced, it removed general-purpose |
| 11 | +[extensions](/programming-guides/editions#extensions) |
| 12 | +(keeping them only for custom options) and introduced |
| 13 | +[`google.protobuf.Any`](/programming-guides/editions#any) |
| 14 | +as their replacement. |
| 15 | + |
| 16 | +Unfortunately, `Any` has enough design flaws that that this is a decision the |
| 17 | +Protobuf team regrets. We have no plans to formally deprecate `Any`, and it |
| 18 | +remains supported, but we recommend using extensions instead of `Any` for nearly |
| 19 | +all use cases. |
| 20 | + |
| 21 | +* [Protobuf Editions](/editions/overview) restored full |
| 22 | + support for extensions starting in Edition 2023. |
| 23 | +* For new schemas, **prefer extensions over `Any`** whenever extensions can |
| 24 | + satisfy the use case (see [Proto Best |
| 25 | + Practices](/best-practices/dos-donts#avoid-any)). |
| 26 | +* Reserve `Any` for the uncommon case where horizontal infrastructure needs to |
| 27 | + propagate completely arbitrary messages and declaring an extension for each |
| 28 | + possible type is infeasible. |
| 29 | + |
| 30 | +## What Extensions and `Any` Are {#what} |
| 31 | + |
| 32 | +Both mechanisms solve the same dependency problem: letting a container message |
| 33 | +carry a submessage whose `.proto` file the container does not import. Instead of |
| 34 | +the container depending on every possible payload type, each payload depends on |
| 35 | +the container. |
| 36 | + |
| 37 | +### Extensions {#what-extensions} |
| 38 | + |
| 39 | +An [extension](/programming-guides/editions#extensions) |
| 40 | +is a field defined outside of its container message, using a field number from |
| 41 | +an `extensions` range reserved by the container: |
| 42 | + |
| 43 | +```proto |
| 44 | +// media/user_content.proto |
| 45 | +edition = "2024"; |
| 46 | + |
| 47 | +package media; |
| 48 | + |
| 49 | +message UserContent { |
| 50 | + string id = 1; |
| 51 | + |
| 52 | + extensions 100 to 199 [ |
| 53 | + declaration = { |
| 54 | + number: 100, |
| 55 | + full_name: ".kittens.kitten_video", |
| 56 | + type: ".kittens.Video" |
| 57 | + } |
| 58 | + ]; |
| 59 | +} |
| 60 | +``` |
| 61 | + |
| 62 | +```proto |
| 63 | +// kittens/video_ext.proto |
| 64 | +edition = "2024"; |
| 65 | + |
| 66 | +package kittens; |
| 67 | + |
| 68 | +import "kittens/video.proto"; |
| 69 | +import "media/user_content.proto"; |
| 70 | + |
| 71 | +extend media.UserContent { |
| 72 | + Video kitten_video = 100; |
| 73 | +} |
| 74 | +``` |
| 75 | + |
| 76 | +In generated code, the extension is accessed as a typed submessage stored |
| 77 | +directly on the parent message: |
| 78 | + |
| 79 | +```cpp |
| 80 | +media::UserContent content; |
| 81 | +content.MutableExtension(kittens::kitten_video)->set_title("Cute kittens"); |
| 82 | + |
| 83 | +if (content.HasExtension(kittens::kitten_video)) { |
| 84 | + const kittens::Video& video = content.GetExtension(kittens::kitten_video); |
| 85 | +} |
| 86 | +``` |
| 87 | + |
| 88 | +On the wire, an extension is identical to a regular field with the same field |
| 89 | +number, type, and cardinality. With [extension |
| 90 | +declarations](/programming-guides/extension_declarations), |
| 91 | +the container can list the number, name, and type name of each extension in one |
| 92 | +place without importing the payload `.proto` file. |
| 93 | + |
| 94 | +### `google.protobuf.Any` {#what-any} |
| 95 | + |
| 96 | +[`google.protobuf.Any`](/reference/protobuf/google.protobuf#any) |
| 97 | +is a wrapper message defined in `google/protobuf/any.proto`: |
| 98 | + |
| 99 | +```proto |
| 100 | +syntax = "proto3"; |
| 101 | + |
| 102 | +package google.protobuf; |
| 103 | + |
| 104 | +message Any { |
| 105 | + string type_url = 1; |
| 106 | + bytes value = 2; |
| 107 | +} |
| 108 | +``` |
| 109 | + |
| 110 | +Instead of extending the container message, the container declares a field of |
| 111 | +type `google.protobuf.Any`. Packing a message into `Any` serializes the inner |
| 112 | +message into `value` and writes a URI containing the inner message's fully |
| 113 | +qualified name into `type_url`: |
| 114 | + |
| 115 | +```cpp |
| 116 | +media::UserContent content; |
| 117 | +kittens::Video video; |
| 118 | +video.set_title("Cute kittens"); |
| 119 | +content.mutable_payload()->PackFrom(video); |
| 120 | + |
| 121 | +if (content.payload().Is<kittens::Video>()) { |
| 122 | + kittens::Video unpacked; |
| 123 | + if (content.payload().UnpackTo(&unpacked)) { |
| 124 | + // use unpacked |
| 125 | + } |
| 126 | +} |
| 127 | +``` |
| 128 | + |
| 129 | +## Why Proto3 Replaced Extensions with `Any` {#why-introduced} |
| 130 | + |
| 131 | +In proto2, extensions had two real problems: |
| 132 | + |
| 133 | +1. There was no built-in way to coordinate extension numbers on a container |
| 134 | + message without importing the extensions. Because `extend` blocks lived in |
| 135 | + separate `.proto` files, two teams could pick the same field number on a |
| 136 | + shared container message and silently collide on the wire. |
| 137 | +2. Proto3 tried to simplify the language so that new runtimes could represent |
| 138 | + messages as plain [open |
| 139 | + structs](/design-decisions/opaque-apis) with minimal |
| 140 | + runtime machinery. Extensions required an `ExtensionRegistry`, extension |
| 141 | + storage on messages, and parameterized accessor APIs in every language. |
| 142 | + |
| 143 | +`Any` avoided both: on the binary wire it is an ordinary message with a `string` |
| 144 | +and a `bytes` field, its discriminator is the globally unique message name |
| 145 | +rather than a coordinated field number, and runtime support could be limited to |
| 146 | +`PackFrom` / `UnpackTo` helpers plus special-case handling in |
| 147 | +[ProtoJSON](/programming-guides/json#any) and TextFormat. |
| 148 | + |
| 149 | +The open-struct premise was abandoned (see [We Prefer Opaque |
| 150 | +APIs](/design-decisions/opaque-apis)). |
| 151 | + |
| 152 | +Risks of extension number collisions are still a concern with extensions, but |
| 153 | +can be mitigated in most cases by using [extension |
| 154 | +declarations](/programming-guides/extension_declarations). |
| 155 | + |
| 156 | +## Why We Regret `Any` as a Replacement for Extensions {#flaws} |
| 157 | + |
| 158 | +### Type Names on the Binary Wire Prevent Renaming and Repackaging {#type-url-lock-in} |
| 159 | + |
| 160 | +A core property of the Protobuf binary wire format is that message names, field |
| 161 | +names, and package names do not appear on the wire. Because the binary format |
| 162 | +identifies fields by numeric tag, you can rename a field, rename a message, or |
| 163 | +move a message to a different `.proto` package without breaking wire |
| 164 | +compatibility with deployed binaries or stored data. |
| 165 | + |
| 166 | +`Any` breaks this property. Because `PackFrom()` writes the fully qualified |
| 167 | +message name into `type_url` on the binary wire, **any message packed into an |
| 168 | +`Any` cannot safely be renamed or moved to a different `.proto` package**: |
| 169 | + |
| 170 | +* If a message is renamed or moved to a new package, `Is<T>()` and |
| 171 | + `UnpackTo()` compare the new descriptor's `full_name()` against the old |
| 172 | + `type_url` string and reject existing payloads, even though the serialized |
| 173 | + bytes in `value` are wire-compatible. |
| 174 | +* At the same time, new binaries start writing the new `type_url` string, |
| 175 | + which older binaries and rollback targets do not recognize. |
| 176 | + |
| 177 | +Worse, **`Any` is not opt-in**. A message author does not declare that their |
| 178 | +message may be packed into an `Any`. Any caller in another repository or service |
| 179 | +can pass an arbitrary message to `PackFrom()` without the message owner's |
| 180 | +knowledge, and because packing can happen through generic or reflective code |
| 181 | +paths, static analysis cannot prove that a message type is never stored in an |
| 182 | +`Any`. This is a very spooky property that makes it difficult to know if |
| 183 | +renaming a message might not be safe. |
| 184 | + |
| 185 | +Extensions identify fields on the wire by their numeric tag, just like regular |
| 186 | +fields. Renaming an extension, renaming the message type it holds, or moving |
| 187 | +either across `.proto` packages is safe on the binary wire. |
| 188 | + |
| 189 | +### Wire Incompatibility with Regular Fields {#wire-incompatibility} |
| 190 | + |
| 191 | +An extension has the exact same binary wire encoding as a regular field with the |
| 192 | +same field number, type, and cardinality. That makes the boundary between |
| 193 | +regular fields and extensions reversible: |
| 194 | + |
| 195 | +* If a regular field on a container message is causing unwanted build |
| 196 | + dependencies, it can be moved out into an `extend` block with the same field |
| 197 | + number and type without changing a single byte on the wire. |
| 198 | +* Conversely, a widely used extension can be promoted directly into a regular |
| 199 | + field on the container message. |
| 200 | + |
| 201 | +`Any` wraps its payload inside a submessage containing `type_url` (field 1) and |
| 202 | +`value` (field 2). It is wire-incompatible with a normal submessage field, so a |
| 203 | +field cannot be migrated into or out of an `Any` without breaking the wire |
| 204 | +format. |
| 205 | + |
| 206 | +### Wire Size Overhead {#wire-size} |
| 207 | + |
| 208 | +An extension tag on the wire is a single varint encoding the field number and |
| 209 | +wire type: 2 bytes for field numbers up to 2047, and at most 5 bytes for the |
| 210 | +largest valid field numbers. |
| 211 | + |
| 212 | +Every `Any` message carries the full `type.googleapis.com/package.MessageName` |
| 213 | +string on the wire in addition to the nested length prefixes. A typical |
| 214 | +`type_url` adds 40 to 80 bytes per instance. For small messages or `repeated |
| 215 | +google.protobuf.Any` fields, the repeated `type_url` strings often take more |
| 216 | +space than the payload itself. |
| 217 | + |
| 218 | +### In-Memory Cost and Eager Serialization {#serialization} |
| 219 | + |
| 220 | +Because `Any` stores its payload as a `bytes value` field rather than as a typed |
| 221 | +submessage in the message tree, working with `Any` in memory is clumsy and |
| 222 | +expensive: |
| 223 | + |
| 224 | +* `PackFrom()` immediately serializes the inner message into a `bytes` buffer, |
| 225 | + even if the outer message is only passed around in memory. |
| 226 | +* In most languages (including C++, Go, Python, and open-source Java), calling |
| 227 | + `UnpackTo()` allocates a separate message object and parses `value` from |
| 228 | + scratch on every read, unless the caller maintains a separate cache by hand. |
| 229 | + By contrast, an extension is stored directly inside the parent message (and |
| 230 | + shares its arena in C++), so reading `GetExtension()` is a pointer lookup. |
| 231 | + |
| 232 | +A common misconception is that `Any` is cheaper for intermediate proxies that do |
| 233 | +not care about the payload because `value` stays unparsed until `UnpackTo()` is |
| 234 | +called. Extensions already have this property: if a binary does not link in or |
| 235 | +register an extension, the bytes remain unparsed in the unknown field set and |
| 236 | +are forwarded intact on serialization. Even when a binary *does* link in the |
| 237 | +extension, runtimes that support lazy fields defer parsing the extension until |
| 238 | +the first `GetExtension()` call and then cache the parsed submessage in place. |
| 239 | + |
| 240 | +### Broken `MergeFrom`, Equality, and Reflection Semantics {#semantics} |
| 241 | + |
| 242 | +To Protobuf, an `Any` is just a two-field message (`string type_url = 1; bytes |
| 243 | +value = 2;`), not a typed container. Standard message operations act on those |
| 244 | +two scalar fields rather than on the packed message inside: |
| 245 | + |
| 246 | +#### `MergeFrom` {#merging} |
| 247 | + |
| 248 | +When two messages with a submessage extension are merged via `MergeFrom()`, the |
| 249 | +extension submessages are merged recursively field by field, and distinct |
| 250 | +extensions on the same container coexist. |
| 251 | + |
| 252 | +When two `Any` messages are merged, `MergeFrom()` merges `type_url` and `value` |
| 253 | +as ordinary scalar fields, overwriting `value` instead of merging the inner |
| 254 | +message. Worse, because `google/protobuf/any.proto` is defined with `syntax = |
| 255 | +"proto3"`, both `type_url` and `value` have [implicit |
| 256 | +presence](/design-decisions/implicit-presence#merging): |
| 257 | + |
| 258 | +1. Suppose `A` is an `Any` packed with a non-empty `Bar` message (`type_url = |
| 259 | + "type.googleapis.com/Bar"`, `value = "\x08\x01"`). |
| 260 | +2. Suppose `B` is an `Any` packed with a default, empty `Quux` message. Because |
| 261 | + an all-default message serializes to 0 bytes, `B` has `type_url = |
| 262 | + "type.googleapis.com/Quux"` and `value = ""`. |
| 263 | +3. Calling `A.MergeFrom(B)` overwrites `A.type_url` with |
| 264 | + `"type.googleapis.com/Quux"`, but because `B.value` is empty (`""`), |
| 265 | + implicit presence treats `B.value` as unset and does not overwrite |
| 266 | + `A.value`. |
| 267 | + |
| 268 | +The merged `Any` now has `Quux`'s `type_url` paired with `Bar`'s serialized wire |
| 269 | +bytes. |
| 270 | + |
| 271 | +#### Equality {#equality} |
| 272 | + |
| 273 | +Standard message equality on `Any` compares the raw `value` byte string rather |
| 274 | +than the logical inner message. As explained in [Serialization Is Not |
| 275 | +Canonical](/programming-guides/serialization-not-canonical), |
| 276 | +two logically equal Protobuf messages can serialize to different byte sequences |
| 277 | +(for example, due to map entry ordering or unknown fields), causing two `Any` |
| 278 | +messages holding equivalent payloads to compare unequal. |
| 279 | + |
| 280 | +#### Reflection and Generic Traversal {#reflection} |
| 281 | + |
| 282 | +Unless a tool links in the packed message's descriptor and writes custom |
| 283 | +`Any`-unpacking code, reflection walkers, field-path utilities, and redaction |
| 284 | +passes only see a `string type_url` and an opaque `bytes value`. Extensions have |
| 285 | +normal `FieldDescriptor` entries and are visited by standard reflection walkers |
| 286 | +without special cases. |
| 287 | + |
| 288 | +### Poor Fit for Mobile and Lite Runtimes {#lite} |
| 289 | + |
| 290 | +In mobile, embedded, and web client builds (`LITE_RUNTIME`), Protobuf strips |
| 291 | +descriptors and symbol names to reduce binary size and avoid leaking unreleased |
| 292 | +feature names in shipped client binaries. |
| 293 | + |
| 294 | +`Any` relies on runtime type names to construct and check `type_url`: |
| 295 | + |
| 296 | +* In **JavaLite**, `MessageLite` does not carry descriptor names, so |
| 297 | + `Any.pack()`, `Any.unpack()`, and `Any.is()` are omitted from the generated |
| 298 | + `Any` class altogether. Callers must construct and compare `type_url` |
| 299 | + strings manually. |
| 300 | +* In **C++ Lite**, `Any::PackFrom<T>()` and `UnpackTo<T>()` only work because |
| 301 | + the code generator emits a `FullMessageName()` string literal on every |
| 302 | + message class, which forces the binary to retain the `.proto` package and |
| 303 | + message name for every type used with `Any`. |
| 304 | + |
| 305 | +Extensions use numeric tags and generated extension identifiers, which work in |
| 306 | +`LITE_RUNTIME` builds and support tree shaking without retaining descriptor |
| 307 | +strings. |
| 308 | + |
| 309 | +### Type URLs Do Not Provide Dynamic Schema Discovery {#type-urls} |
| 310 | + |
| 311 | +The `type_url` field was originally formatted as a URL |
| 312 | +(`type.googleapis.com/...`) on the theory that runtimes might fetch unknown |
| 313 | +schemas over HTTP at parse time. Protobuf never implemented remote type fetching |
| 314 | +and explicitly warns against contacting `type_url` endpoints in `any.proto` as a |
| 315 | +security problem. Every parser simply strips everything up to the last `/`, and |
| 316 | +unpacking an `Any` in JSON, TextFormat, or reflection still requires the target |
| 317 | +descriptor to be linked or loaded into the local binary's descriptor pool, just |
| 318 | +like an extension. |
| 319 | + |
| 320 | +## What to Use Instead {#instead} |
| 321 | + |
| 322 | +### Regular Submessage Fields or `oneof` {#oneof} |
| 323 | + |
| 324 | +When the set of possible payload types is bounded, known to the container |
| 325 | +author, and importing them directly into the container `.proto` does not cause |
| 326 | +dependency problems, use a regular submessage field or a `oneof`. A `oneof` |
| 327 | +makes the set of alternatives explicit in one place and generates the cleanest |
| 328 | +API in every language. |
| 329 | + |
| 330 | +### Extensions with Extension Declarations {#use-extensions} |
| 331 | + |
| 332 | +When payload definitions need to live in separate `.proto` files so the |
| 333 | +container does not depend on them, use |
| 334 | +[extensions](/programming-guides/editions#extensions): |
| 335 | + |
| 336 | +* Add [extension |
| 337 | + declarations](/programming-guides/extension_declarations) |
| 338 | + to the container's `extensions` range so all allocated extension numbers and |
| 339 | + type names are listed in one place and checked by `protoc` without importing |
| 340 | + the payload `.proto` files. |
| 341 | +* If central code-review ownership on the container `.proto` file would be a |
| 342 | + bottleneck for teams adding extensions, put the extendable container (or a |
| 343 | + dedicated extension-carrier submessage) in a separate `.proto` file with |
| 344 | + broader ownership. |
| 345 | +* If your container message is currently in a `syntax = "proto3"` file (which |
| 346 | + does not allow `extensions` ranges), migrate that file to [Protobuf |
| 347 | + Editions](/editions/overview). Moving a file from |
| 348 | + `proto3` to Edition 2023 is wire-compatible, and remaining `proto3` files |
| 349 | + can still import it. |
| 350 | + |
| 351 | +### When `Any` Is Still Appropriate {#when-any} |
| 352 | + |
| 353 | +The one case where `Any` is still the right tool is **unbounded horizontal |
| 354 | +infrastructure**: systems that transport, log, or store completely arbitrary |
| 355 | +messages across independent domains (for example, a generic RPC error detail |
| 356 | +list like `google.rpc.Status.details`, a generic task queue payload, or a dead |
| 357 | +letter queue) where declaring an extension on the envelope for every message |
| 358 | +type in the ecosystem is infeasible. |
| 359 | + |
| 360 | +Whenever you do use `Any`, treat the package name and message name of every |
| 361 | +packed message as a permanently frozen part of your wire protocol. |
0 commit comments