"Token Types" chapter of Idiomatic Rust #2921

tall-vase · 2025-09-22T09:30:37Z

Materials on "token types."

The "Branded tokens" section of this one requires a little extra scrutiny. It may be the case it is too complex for this stage, or that the explanation isn't suitable for the audience/instructors. Either way, I'm eager for input.

google-cla · 2025-09-22T09:30:42Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

The previous STYLE.md advised against using speaker notes as a 'script' but didn't provide clear examples of what that meant. This could be interpreted by LLMs as disallowing any kind of instructor prompt. This change clarifies the guideline by: 1. Distinguishing between long-form, verbatim 'scripts' (bad) and short, actionable 'teaching prompts' (good). 2. Providing concrete 'good' and 'bad' examples to make the distinction unambiguous for future contributors.

src/idiomatic/leveraging-the-type-system/token-types.md

…all-vase/comprehensive-rust into idiomatic/typesystem-tokens

gribozavr · 2025-09-28T17:22:38Z

STYLE.md

+  >
+  > - Note: The `Ext` suffix is a common convention.
+  > - Ask: What happens if the `use` statement is removed?
+  > - Demo: Comment out the `use` statement to show the compiler error.


I think this content already exists in the main branch? Please rebase to hide the spurious diff.

src/idiomatic/leveraging-the-type-system/token-types.md

gribozavr · 2025-09-28T17:35:02Z

src/idiomatic/leveraging-the-type-system/token-types.md

+
+<details>
+
+- Token types let us use the privacy tools of types and modules to control when


This explanation is a bit too abstract and sounds very similar to how we introduce the typestate pattern, "How can we ensure that only valid operations are allowed on a value based on its current state?", except that we're not talking about a specific state. Could we make it more concrete? Should we merge the two sections? Should this section be ordered before typestate but after newtype?

My thinking is that newtype wraps a concrete piece of data, and represents a proof that the data satisfies a predicate.

What I see in the token types section suggests that a token type represents a proof that some data elsewhere, or the program control flow, satisfies a certain predicate.

The current progression seems to be newtype -> typestate -> token types.

This does not seem ideal. First we introduce a foundational concept (Newtype), then jump to a complex application (the generic serializer in Typestate), and then present a simpler concept (token types). It seems to me that token types are a more straightforward extension of the initial newtype pattern.

We can't quite do that if we're covering branded types, as that depends on the Borrowck invariants work (introduction of PhantomData, use of Lifetimes to enforce invariants). If we cut the Branded types content, then this re-ordering would be feasible.

Sorry, I tried referencing those sections, but I don't understand. The token types slide does not depend on PhantomData or complex use of lifetimes. Could you explain this issue in more detail?

To clarify, I don't mean moving the entirety of this PR between newtype and typestate. Only this slide.

src/idiomatic/leveraging-the-type-system/token-types.md

src/idiomatic/leveraging-the-type-system/token-types/mutex-guard.md

src/idiomatic/leveraging-the-type-system/token-types/branded-tokens.md

src/idiomatic/leveraging-the-type-system/token-types.md

gribozavr · 2025-09-28T19:18:44Z

src/idiomatic/leveraging-the-type-system/token-types/branded-tokens.md

+  make sure cells can't "escape" a context where we know where cyclic operations
+  are safe.
+
+</details>


This is all technically correct, but too dense for a slide. I think we need to have a more gentle flow, broken down into multiple slides, starting with motivation (in this example it is not clear what the token is achieving). When we move on to explain the implementation, we should first start with a sketch that has the right structure but omits some of the details that make the code correct for the sake of brevity (e.g., the variance mechanics).

Here's an idea, but you can of course come up with your own.

Motivation: if you decide to use the AdminToken example in the previous slide, you can build a segue to this idea. Remind the students that the web server processes multiple requests, but only one at a time. We could use one user's credentials to log in and get an AdminToken, stash it in a global variable somewhere, and then process other user's message deletion request using that token.

=> Idea for a solution: the Board::delete_message() function should only accept a token that was returned by Board::login() by the same object.

You made great improvements, but the example is still abstract.

Co-authored-by: Dmitri Gribenko <[email protected]>

gribozavr · 2025-10-03T17:22:56Z

src/idiomatic/leveraging-the-type-system/token-types/permission-tokens.md

+
+# Permission Tokens
+
+Token types work well as a trivial "proof of checked permission."


Suggested change

Token types work well as a trivial "proof of checked permission."

Token types work well as a proof of checked permission.

gribozavr · 2025-10-03T17:24:10Z

src/idiomatic/leveraging-the-type-system/token-types.md

+- Ask: What is the purpose of the `proof: ()` field here?
+
+  Without `proof: ()`, `Token` would have no private fields and users would be
+  able to construct values of `Token` arbitrarily.


It would be nice to prompt the instructor to try to construct the token in main and show that the compiler does not allow doing that.

gribozavr · 2025-10-03T17:25:31Z

src/idiomatic/leveraging-the-type-system/token-types.md

+  until they've performed a specific task.
+
+  We can do this by defining a type the API consumer cannot construct on their
+  own, through privacy tools of structs and modules.


It would be nice remind students that the newtype pattern is based on the same idea.

gribozavr · 2025-10-03T17:26:44Z

src/idiomatic/leveraging-the-type-system/token-types/mutex-guard.md

+# Token Types with Data: Mutex Guards
+
+Sometimes, a token type needs additional data. A mutex guard is an example of
+permission + data.


Suggested change

permission + data.

a permission + data.

gribozavr · 2025-10-03T17:28:34Z

src/idiomatic/leveraging-the-type-system/token-types/mutex-guard.md

+  `Mutex` while the underlying `Mutex` keeps that data private from the user.
+
+- If `mutex.lock()` does not return a `MutexGuard`, you don't have permission to
+  change the value within the mutex.


What's important to emphasize here is that not only the user does not have permission (which is also true, e.g., for C++ mutexes), the user does not have a way to access the data (which sets Rust mutexes apart).

gribozavr · 2025-10-03T17:49:10Z

src/idiomatic/leveraging-the-type-system/token-types/branded-03-impl.md

+        // We want a function whose lifetime is specific to each time
+        // `new` is called, not tied to any one data structure other than


Suggested change

// We want a function whose lifetime is specific to each time

// `new` is called, not tied to any one data structure other than

// We want a function whose lifetime is specific to each call to

// `new`, not tied to any one data structure other than

gribozavr · 2025-10-03T17:51:31Z

src/idiomatic/leveraging-the-type-system/token-types/branded-tokens.md

+  make sure cells can't "escape" a context where we know where cyclic operations
+  are safe.
+
+</details>


You made great improvements, but the example is still abstract.

gribozavr · 2025-10-03T17:56:39Z

src/idiomatic/leveraging-the-type-system/token-types/branded-01-motivation.md

+- [`GhostCell`](https://plv.mpi-sws.org/rustbelt/ghostcell/paper.pdf) is a
+  prominent user of this, later slides will touch on it.
+
+</details>


Next, once we have shown the general idea, I'd suggest transitioning to a concrete API design problem (not an abstract one like here).

gribozavr · 2025-10-03T18:41:53Z

src/idiomatic/leveraging-the-type-system/token-types/branded-03-impl.md

+- The underlying Branded Data Structure we're going to use here is just a
+  `Vec<u8>` (the data) and an `InvariantLifetime`.
+
+- The constructor for this type will take **data for the `Vec<u8`** plus a


Suggested change

- The constructor for this type will take **data for the `Vec<u8`** plus a

- The constructor for this type will take **data for the `Vec<u8>`** plus a

gribozavr · 2025-10-03T18:42:29Z

src/idiomatic/leveraging-the-type-system/token-types/branded-03-impl.md

+  Expect not much, it's "for" in the sense of "forall" from mathematics.
+
+- The `for<'a> [trait bound that uses 'a]` binding of `'a` means the lifetime is
+  "self contained."


I think we can explain this in more depth, rather than through an analogy.

gribozavr · 2025-10-03T18:53:18Z

src/idiomatic/leveraging-the-type-system/token-types/permission-tokens.md

+  get a `AdminToken` to perform administrator actions within a specific
+  environment (here, a chat client).
+
+  Once the permissions are gained, we can call a `add_moderator` function.


Suggested change

Once the permissions are gained, we can call a `add_moderator` function.

Once the permissions are gained, we can call the `add_moderator` function.

tall-vase added 3 commits September 11, 2025 08:08

Initial skeleton for token types

b7e8128

Reframe around mutexguard and a gentle walkthrough of branded tokens

1a7d356

Another editing pass, focused on the branded tokens slide

d7f3984

tall-vase added the deep_dives/idiomatic label Sep 22, 2025

gribozavr and others added 6 commits September 22, 2025 11:44

Formatting pass

ba2ceda

Initial skeleton for token types

400c336

Reframe around mutexguard and a gentle walkthrough of branded tokens

a70aa6b

Another editing pass, focused on the branded tokens slide

29872e3

Formatting pass

bc6abb0

mgeisler reviewed Sep 22, 2025

View reviewed changes

src/idiomatic/leveraging-the-type-system/token-types.md Show resolved Hide resolved

tall-vase added 4 commits September 22, 2025 19:06

Merge branch 'idiomatic/typesystem-tokens' of github.com-mainmatter:t…

aa3b402

…all-vase/comprehensive-rust into idiomatic/typesystem-tokens

fix merge artefact

d47ca92

fix test errors

a23df16

Address lints

dfebb37

gribozavr self-requested a review September 28, 2025 19:11

gribozavr requested changes Sep 28, 2025

View reviewed changes

gribozavr assigned tall-vase Sep 28, 2025

gribozavr added the waiting for author Waiting on the issue author to reply label Sep 28, 2025

tall-vase and others added 7 commits October 1, 2025 11:29

Update src/idiomatic/leveraging-the-type-system/token-types.md

6c2157d

Co-authored-by: Dmitri Gribenko <[email protected]>

Apply suggestions from review

146a30f

Co-authored-by: Dmitri Gribenko <[email protected]>

Apply feedback to tokens/mutex slide

63e40dc

Rewrite token types speaker notes and correct mutex explanation

1adee3c

Address further feedback

a680dd8

Fix compilation of branded tokens pt 1

36da55f

Editing pass

af6523c

gribozavr requested changes Oct 3, 2025

View reviewed changes

gribozavr reviewed Oct 3, 2025

View reviewed changes


		<details>

		- Token types let us use the privacy tools of types and modules to control when


		# Permission Tokens

		Token types work well as a trivial "proof of checked permission."

	Token types work well as a trivial "proof of checked permission."
	Token types work well as a proof of checked permission.

		// We want a function whose lifetime is specific to each time
		// `new` is called, not tied to any one data structure other than

	- The constructor for this type will take data for the `Vec<u8` plus a
	- The constructor for this type will take data for the `Vec<u8>` plus a

	Once the permissions are gained, we can call a `add_moderator` function.
	Once the permissions are gained, we can call the `add_moderator` function.

"Token Types" chapter of Idiomatic Rust #2921

Are you sure you want to change the base?

"Token Types" chapter of Idiomatic Rust #2921

Uh oh!

Conversation

tall-vase commented Sep 22, 2025

Uh oh!

google-cla bot commented Sep 22, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!