Skip to content

Commit c4bd0df

Browse files
Add design decision doc explaining why we prefer extensions over Any
PiperOrigin-RevId: 1010489362
1 parent 0fefb7b commit c4bd0df

5 files changed

Lines changed: 369 additions & 3 deletions

File tree

‎content/best-practices/dos-donts.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -142,7 +142,9 @@ If you plan to use `Any`, first double-check if extensions can satisfy the use
142142
case instead. If yes, prefer using extensions.
143143

144144
`Any` was created in Proto3 to replace extensions. However, it has a number of
145-
design flaws. For most use cases, prefer using extensions.
145+
design flaws (see
146+
[We Prefer Extensions Over Any](/design-decisions/extensions-over-any)).
147+
For most use cases, prefer using extensions.
146148

147149
The main use case that `Any` fulfills is the uncommon scenario in which
148150
horizontal infrastructure needs to propagate completely arbitrary messages, and

‎content/design-decisions/_index.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,3 +23,5 @@ things develop in the broader ecosystem context.
2323
Presence](/design-decisions/implicit-presence)
2424
* [Insertion Points and Generic
2525
Services](/design-decisions/insertion-points-and-generic-services)
26+
* [Prefer Extensions Over
27+
Any](/design-decisions/extensions-over-any)
Lines changed: 361 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,361 @@
1+
2+
3+
+++
4+
title = "We Prefer Extensions Over Any"
5+
weight = 93
6+
description = "Covers why the Protobuf team regrets Any as a replacement for extensions, and why extensions are preferred for most use cases"
7+
type = "docs"
8+
+++
9+
10+
When proto3 was introduced, it removed general-purpose
11+
[extensions](/programming-guides/editions#extensions)
12+
(keeping them only for custom options) and introduced
13+
[`google.protobuf.Any`](/programming-guides/editions#any)
14+
as their replacement.
15+
16+
Unfortunately, `Any` has enough design flaws that that this is a decision the
17+
Protobuf team regrets. We have no plans to formally deprecate `Any`, and it
18+
remains supported, but we recommend using extensions instead of `Any` for nearly
19+
all use cases.
20+
21+
* [Protobuf Editions](/editions/overview) restored full
22+
support for extensions starting in Edition 2023.
23+
* For new schemas, **prefer extensions over `Any`** whenever extensions can
24+
satisfy the use case (see [Proto Best
25+
Practices](/best-practices/dos-donts#avoid-any)).
26+
* Reserve `Any` for the uncommon case where horizontal infrastructure needs to
27+
propagate completely arbitrary messages and declaring an extension for each
28+
possible type is infeasible.
29+
30+
## What Extensions and `Any` Are {#what}
31+
32+
Both mechanisms solve the same dependency problem: letting a container message
33+
carry a submessage whose `.proto` file the container does not import. Instead of
34+
the container depending on every possible payload type, each payload depends on
35+
the container.
36+
37+
### Extensions {#what-extensions}
38+
39+
An [extension](/programming-guides/editions#extensions)
40+
is a field defined outside of its container message, using a field number from
41+
an `extensions` range reserved by the container:
42+
43+
```proto
44+
// media/user_content.proto
45+
edition = "2024";
46+
47+
package media;
48+
49+
message UserContent {
50+
string id = 1;
51+
52+
extensions 100 to 199 [
53+
declaration = {
54+
number: 100,
55+
full_name: ".kittens.kitten_video",
56+
type: ".kittens.Video"
57+
}
58+
];
59+
}
60+
```
61+
62+
```proto
63+
// kittens/video_ext.proto
64+
edition = "2024";
65+
66+
package kittens;
67+
68+
import "kittens/video.proto";
69+
import "media/user_content.proto";
70+
71+
extend media.UserContent {
72+
Video kitten_video = 100;
73+
}
74+
```
75+
76+
In generated code, the extension is accessed as a typed submessage stored
77+
directly on the parent message:
78+
79+
```cpp
80+
media::UserContent content;
81+
content.MutableExtension(kittens::kitten_video)->set_title("Cute kittens");
82+
83+
if (content.HasExtension(kittens::kitten_video)) {
84+
const kittens::Video& video = content.GetExtension(kittens::kitten_video);
85+
}
86+
```
87+
88+
On the wire, an extension is identical to a regular field with the same field
89+
number, type, and cardinality. With [extension
90+
declarations](/programming-guides/extension_declarations),
91+
the container can list the number, name, and type name of each extension in one
92+
place without importing the payload `.proto` file.
93+
94+
### `google.protobuf.Any` {#what-any}
95+
96+
[`google.protobuf.Any`](/reference/protobuf/google.protobuf#any)
97+
is a wrapper message defined in `google/protobuf/any.proto`:
98+
99+
```proto
100+
syntax = "proto3";
101+
102+
package google.protobuf;
103+
104+
message Any {
105+
string type_url = 1;
106+
bytes value = 2;
107+
}
108+
```
109+
110+
Instead of extending the container message, the container declares a field of
111+
type `google.protobuf.Any`. Packing a message into `Any` serializes the inner
112+
message into `value` and writes a URI containing the inner message's fully
113+
qualified name into `type_url`:
114+
115+
```cpp
116+
media::UserContent content;
117+
kittens::Video video;
118+
video.set_title("Cute kittens");
119+
content.mutable_payload()->PackFrom(video);
120+
121+
if (content.payload().Is<kittens::Video>()) {
122+
kittens::Video unpacked;
123+
if (content.payload().UnpackTo(&unpacked)) {
124+
// use unpacked
125+
}
126+
}
127+
```
128+
129+
## Why Proto3 Replaced Extensions with `Any` {#why-introduced}
130+
131+
In proto2, extensions had two real problems:
132+
133+
1. There was no built-in way to coordinate extension numbers on a container
134+
message without importing the extensions. Because `extend` blocks lived in
135+
separate `.proto` files, two teams could pick the same field number on a
136+
shared container message and silently collide on the wire.
137+
2. Proto3 tried to simplify the language so that new runtimes could represent
138+
messages as plain [open
139+
structs](/design-decisions/opaque-apis) with minimal
140+
runtime machinery. Extensions required an `ExtensionRegistry`, extension
141+
storage on messages, and parameterized accessor APIs in every language.
142+
143+
`Any` avoided both: on the binary wire it is an ordinary message with a `string`
144+
and a `bytes` field, its discriminator is the globally unique message name
145+
rather than a coordinated field number, and runtime support could be limited to
146+
`PackFrom` / `UnpackTo` helpers plus special-case handling in
147+
[ProtoJSON](/programming-guides/json#any) and TextFormat.
148+
149+
The open-struct premise was abandoned (see [We Prefer Opaque
150+
APIs](/design-decisions/opaque-apis)).
151+
152+
Risks of extension number collisions are still a concern with extensions, but
153+
can be mitigated in most cases by using [extension
154+
declarations](/programming-guides/extension_declarations).
155+
156+
## Why We Regret `Any` as a Replacement for Extensions {#flaws}
157+
158+
### Type Names on the Binary Wire Prevent Renaming and Repackaging {#type-url-lock-in}
159+
160+
A core property of the Protobuf binary wire format is that message names, field
161+
names, and package names do not appear on the wire. Because the binary format
162+
identifies fields by numeric tag, you can rename a field, rename a message, or
163+
move a message to a different `.proto` package without breaking wire
164+
compatibility with deployed binaries or stored data.
165+
166+
`Any` breaks this property. Because `PackFrom()` writes the fully qualified
167+
message name into `type_url` on the binary wire, **any message packed into an
168+
`Any` cannot safely be renamed or moved to a different `.proto` package**:
169+
170+
* If a message is renamed or moved to a new package, `Is<T>()` and
171+
`UnpackTo()` compare the new descriptor's `full_name()` against the old
172+
`type_url` string and reject existing payloads, even though the serialized
173+
bytes in `value` are wire-compatible.
174+
* At the same time, new binaries start writing the new `type_url` string,
175+
which older binaries and rollback targets do not recognize.
176+
177+
Worse, **`Any` is not opt-in**. A message author does not declare that their
178+
message may be packed into an `Any`. Any caller in another repository or service
179+
can pass an arbitrary message to `PackFrom()` without the message owner's
180+
knowledge, and because packing can happen through generic or reflective code
181+
paths, static analysis cannot prove that a message type is never stored in an
182+
`Any`. This is a very spooky property that makes it difficult to know if
183+
renaming a message might not be safe.
184+
185+
Extensions identify fields on the wire by their numeric tag, just like regular
186+
fields. Renaming an extension, renaming the message type it holds, or moving
187+
either across `.proto` packages is safe on the binary wire.
188+
189+
### Wire Incompatibility with Regular Fields {#wire-incompatibility}
190+
191+
An extension has the exact same binary wire encoding as a regular field with the
192+
same field number, type, and cardinality. That makes the boundary between
193+
regular fields and extensions reversible:
194+
195+
* If a regular field on a container message is causing unwanted build
196+
dependencies, it can be moved out into an `extend` block with the same field
197+
number and type without changing a single byte on the wire.
198+
* Conversely, a widely used extension can be promoted directly into a regular
199+
field on the container message.
200+
201+
`Any` wraps its payload inside a submessage containing `type_url` (field 1) and
202+
`value` (field 2). It is wire-incompatible with a normal submessage field, so a
203+
field cannot be migrated into or out of an `Any` without breaking the wire
204+
format.
205+
206+
### Wire Size Overhead {#wire-size}
207+
208+
An extension tag on the wire is a single varint encoding the field number and
209+
wire type: 2 bytes for field numbers up to 2047, and at most 5 bytes for the
210+
largest valid field numbers.
211+
212+
Every `Any` message carries the full `type.googleapis.com/package.MessageName`
213+
string on the wire in addition to the nested length prefixes. A typical
214+
`type_url` adds 40 to 80 bytes per instance. For small messages or `repeated
215+
google.protobuf.Any` fields, the repeated `type_url` strings often take more
216+
space than the payload itself.
217+
218+
### In-Memory Cost and Eager Serialization {#serialization}
219+
220+
Because `Any` stores its payload as a `bytes value` field rather than as a typed
221+
submessage in the message tree, working with `Any` in memory is clumsy and
222+
expensive:
223+
224+
* `PackFrom()` immediately serializes the inner message into a `bytes` buffer,
225+
even if the outer message is only passed around in memory.
226+
* In most languages (including C++, Go, Python, and open-source Java), calling
227+
`UnpackTo()` allocates a separate message object and parses `value` from
228+
scratch on every read, unless the caller maintains a separate cache by hand.
229+
By contrast, an extension is stored directly inside the parent message (and
230+
shares its arena in C++), so reading `GetExtension()` is a pointer lookup.
231+
232+
A common misconception is that `Any` is cheaper for intermediate proxies that do
233+
not care about the payload because `value` stays unparsed until `UnpackTo()` is
234+
called. Extensions already have this property: if a binary does not link in or
235+
register an extension, the bytes remain unparsed in the unknown field set and
236+
are forwarded intact on serialization. Even when a binary *does* link in the
237+
extension, runtimes that support lazy fields defer parsing the extension until
238+
the first `GetExtension()` call and then cache the parsed submessage in place.
239+
240+
### Broken `MergeFrom`, Equality, and Reflection Semantics {#semantics}
241+
242+
To Protobuf, an `Any` is just a two-field message (`string type_url = 1; bytes
243+
value = 2;`), not a typed container. Standard message operations act on those
244+
two scalar fields rather than on the packed message inside:
245+
246+
#### `MergeFrom` {#merging}
247+
248+
When two messages with a submessage extension are merged via `MergeFrom()`, the
249+
extension submessages are merged recursively field by field, and distinct
250+
extensions on the same container coexist.
251+
252+
When two `Any` messages are merged, `MergeFrom()` merges `type_url` and `value`
253+
as ordinary scalar fields, overwriting `value` instead of merging the inner
254+
message. Worse, because `google/protobuf/any.proto` is defined with `syntax =
255+
"proto3"`, both `type_url` and `value` have [implicit
256+
presence](/design-decisions/implicit-presence#merging):
257+
258+
1. Suppose `A` is an `Any` packed with a non-empty `Bar` message (`type_url =
259+
"type.googleapis.com/Bar"`, `value = "\x08\x01"`).
260+
2. Suppose `B` is an `Any` packed with a default, empty `Quux` message. Because
261+
an all-default message serializes to 0 bytes, `B` has `type_url =
262+
"type.googleapis.com/Quux"` and `value = ""`.
263+
3. Calling `A.MergeFrom(B)` overwrites `A.type_url` with
264+
`"type.googleapis.com/Quux"`, but because `B.value` is empty (`""`),
265+
implicit presence treats `B.value` as unset and does not overwrite
266+
`A.value`.
267+
268+
The merged `Any` now has `Quux`'s `type_url` paired with `Bar`'s serialized wire
269+
bytes.
270+
271+
#### Equality {#equality}
272+
273+
Standard message equality on `Any` compares the raw `value` byte string rather
274+
than the logical inner message. As explained in [Serialization Is Not
275+
Canonical](/programming-guides/serialization-not-canonical),
276+
two logically equal Protobuf messages can serialize to different byte sequences
277+
(for example, due to map entry ordering or unknown fields), causing two `Any`
278+
messages holding equivalent payloads to compare unequal.
279+
280+
#### Reflection and Generic Traversal {#reflection}
281+
282+
Unless a tool links in the packed message's descriptor and writes custom
283+
`Any`-unpacking code, reflection walkers, field-path utilities, and redaction
284+
passes only see a `string type_url` and an opaque `bytes value`. Extensions have
285+
normal `FieldDescriptor` entries and are visited by standard reflection walkers
286+
without special cases.
287+
288+
### Poor Fit for Mobile and Lite Runtimes {#lite}
289+
290+
In mobile, embedded, and web client builds (`LITE_RUNTIME`), Protobuf strips
291+
descriptors and symbol names to reduce binary size and avoid leaking unreleased
292+
feature names in shipped client binaries.
293+
294+
`Any` relies on runtime type names to construct and check `type_url`:
295+
296+
* In **JavaLite**, `MessageLite` does not carry descriptor names, so
297+
`Any.pack()`, `Any.unpack()`, and `Any.is()` are omitted from the generated
298+
`Any` class altogether. Callers must construct and compare `type_url`
299+
strings manually.
300+
* In **C++ Lite**, `Any::PackFrom<T>()` and `UnpackTo<T>()` only work because
301+
the code generator emits a `FullMessageName()` string literal on every
302+
message class, which forces the binary to retain the `.proto` package and
303+
message name for every type used with `Any`.
304+
305+
Extensions use numeric tags and generated extension identifiers, which work in
306+
`LITE_RUNTIME` builds and support tree shaking without retaining descriptor
307+
strings.
308+
309+
### Type URLs Do Not Provide Dynamic Schema Discovery {#type-urls}
310+
311+
The `type_url` field was originally formatted as a URL
312+
(`type.googleapis.com/...`) on the theory that runtimes might fetch unknown
313+
schemas over HTTP at parse time. Protobuf never implemented remote type fetching
314+
and explicitly warns against contacting `type_url` endpoints in `any.proto` as a
315+
security problem. Every parser simply strips everything up to the last `/`, and
316+
unpacking an `Any` in JSON, TextFormat, or reflection still requires the target
317+
descriptor to be linked or loaded into the local binary's descriptor pool, just
318+
like an extension.
319+
320+
## What to Use Instead {#instead}
321+
322+
### Regular Submessage Fields or `oneof` {#oneof}
323+
324+
When the set of possible payload types is bounded, known to the container
325+
author, and importing them directly into the container `.proto` does not cause
326+
dependency problems, use a regular submessage field or a `oneof`. A `oneof`
327+
makes the set of alternatives explicit in one place and generates the cleanest
328+
API in every language.
329+
330+
### Extensions with Extension Declarations {#use-extensions}
331+
332+
When payload definitions need to live in separate `.proto` files so the
333+
container does not depend on them, use
334+
[extensions](/programming-guides/editions#extensions):
335+
336+
* Add [extension
337+
declarations](/programming-guides/extension_declarations)
338+
to the container's `extensions` range so all allocated extension numbers and
339+
type names are listed in one place and checked by `protoc` without importing
340+
the payload `.proto` files.
341+
* If central code-review ownership on the container `.proto` file would be a
342+
bottleneck for teams adding extensions, put the extendable container (or a
343+
dedicated extension-carrier submessage) in a separate `.proto` file with
344+
broader ownership.
345+
* If your container message is currently in a `syntax = "proto3"` file (which
346+
does not allow `extensions` ranges), migrate that file to [Protobuf
347+
Editions](/editions/overview). Moving a file from
348+
`proto3` to Edition 2023 is wire-compatible, and remaining `proto3` files
349+
can still import it.
350+
351+
### When `Any` Is Still Appropriate {#when-any}
352+
353+
The one case where `Any` is still the right tool is **unbounded horizontal
354+
infrastructure**: systems that transport, log, or store completely arbitrary
355+
messages across independent domains (for example, a generic RPC error detail
356+
list like `google.rpc.Status.details`, a generic task queue payload, or a dead
357+
letter queue) where declaring an extension on the envelope for every message
358+
type in the ecosystem is infeasible.
359+
360+
Whenever you do use `Any`, treat the package name and message name of every
361+
packed message as a permanently frozen part of your wire protocol.

‎content/programming-guides/editions.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1510,7 +1510,8 @@ If you want to limit contained messages to a small number of types and to
15101510
require permission before adding new types to the list, consider using
15111511
[extensions](#extensions) with
15121512
[extension declarations](/programming-guides/extension_declarations)
1513-
instead of `Any` message types.
1513+
instead of `Any` message types. For most use cases,
1514+
[we prefer extensions over `Any`](/design-decisions/extensions-over-any).
15141515

15151516
## Oneof {#oneof}
15161517

‎content/reference/protobuf/google.protobuf.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ In most cases, there are better options:
5050

5151
* For new messages, it's better to use regular explicit-presence fields
5252
(`optional` in proto2/proto3, regular field in edition >= 2023).
53-
* Extensions are generally a better option than `Any` fields.
53+
* [Extensions are generally a better option than `Any` fields](/design-decisions/extensions-over-any).
5454

5555
## Any {#any}
5656

0 commit comments

Comments
 (0)