Skip to content

Commit a09aa0e

Browse files
hardbytehardbyte
committed
feat(python): DX improvements and codegen pipeline refactor
Squashed from 18 commits on fix/python-dx-improvements: - Improve Python DX type metadata and architecture docs - Compact enum representation via discriminated unions - Fix Python name collisions and enum docstrings - Trim reflected Python metadata to irreducible cases - Add format option to Python Config - Refactor: move type mapping from schema layer to Python backend - Refactor: split compiler pipeline into codegen crate - Refactor: move SymbolId from raw schema types to compiler-owned side table - Assign schema IDs in compiler path - Tighten client generation docs - Various fixes (hash truncation, stuttering, serializer aliases, flat class names, field_0 in externally-tagged, name collisions, doctest cargo collision, Rust formatting)
1 parent 586e447 commit a09aa0e

185 files changed

Lines changed: 4531 additions & 5501 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Cargo.lock‎

Lines changed: 9 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ members = [
77
"reflectapi-demo/clients/rust/generated",
88
"reflectapi-derive",
99
"reflectapi-schema",
10+
"reflectapi-schema-codegen",
1011
]
1112
resolver = "2"
1213

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@
3636
- [Crates.io](https://crates.io/crates/reflectapi) - Package information and versions
3737
- [API Documentation](https://docs.rs/reflectapi) - Complete API reference
3838
- [User Guide](docs/src/SUMMARY.md) - Tutorials and examples (build locally with `mdbook serve` in `docs/`)
39-
- [Architecture](docs/architecture.md) - System design and internals
39+
- [Architecture](docs/src/architecture.md) - System design and internals
4040

4141
## Development notes
4242

‎docs/architecture.md‎

Lines changed: 0 additions & 533 deletions
This file was deleted.

‎docs/src/SUMMARY.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
# Summary
22

3-
[Introduction](./introduction.md)
3+
- [Introduction](./introduction.md)
4+
- [Architecture](./architecture.md)
45

56
# Getting Started
67

78
- [Quick Start](./getting-started/quick-start.md)
89
- [Installation](./getting-started/installation.md)
910
- [Client Libraries](./clients/README.md)
10-

‎docs/src/architecture.md‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
# Architecture
2+
3+
## Overview
4+
5+
ReflectAPI has three layers:
6+
7+
1. Rust types and handler functions define the API surface.
8+
2. Reflection builds a `Schema`, which is the interchange format between the server side and code generators.
9+
3. Codegen backends transform that schema into language-specific clients or an OpenAPI document.
10+
11+
The workspace is split accordingly:
12+
13+
- `reflectapi-schema`: raw schema types and raw-schema transforms
14+
- `reflectapi-schema-codegen`: compiler-owned IDs, normalization pipeline, semantic IR
15+
- `reflectapi-derive`: `#[derive(Input, Output)]` macros
16+
- `reflectapi`: reflection traits, builder, runtime integrations, codegen backends
17+
- `reflectapi-cli`: CLI wrapper around codegen
18+
- `reflectapi-demo`: snapshot and integration tests
19+
- `reflectapi-python-runtime`: runtime support for generated Python clients
20+
21+
## Reflection Model
22+
23+
Reflection starts from the `Input` and `Output` traits in `reflectapi/src/traits.rs`. Derived implementations and hand-written impls register types into a `Typespace` and return `TypeReference`s that point at those definitions.
24+
25+
The top-level `Schema` in `reflectapi-schema/src/lib.rs` contains:
26+
27+
- `functions`: endpoint definitions
28+
- `input_types`: types seen in request positions
29+
- `output_types`: types seen in response positions
30+
31+
Input and output types stay separate at schema-construction time so the same Rust name can have different request and response shapes. Some backends later consolidate them into a single naming domain.
32+
33+
## Schema and IDs
34+
35+
`SymbolId` and `SymbolKind` live in `reflectapi-schema-codegen/src/symbol.rs`. They are compiler identifiers, not part of the stable JSON contract.
36+
37+
Key points:
38+
39+
- raw `Schema`, `Function`, and type/member definitions do not store symbol IDs
40+
- `build_schema_ids()` in `reflectapi-schema-codegen/src/ids.rs` assigns IDs in a compiler-owned side table
41+
- the schema root now uses `SymbolKind::Schema`
42+
- the schema root path includes the `__schema__` sentinel to avoid colliding with a user-defined type of the same name
43+
44+
That keeps `reflectapi.json` wire-focused while normalization and semantic analysis still get stable identities.
45+
46+
## Type Metadata
47+
48+
Every reflected type is one of:
49+
50+
- `Primitive`
51+
- `Struct`
52+
- `Enum`
53+
54+
`Primitive.fallback` lets a backend substitute a simpler representation when it does not natively model the original Rust type. Examples in the current codebase include pointer-like wrappers falling back to `T`, and ordered collections falling back to unordered equivalents or vectors.
55+
56+
Language-specific metadata is carried by `LanguageSpecificTypeCodegenConfig` in `reflectapi-schema/src/codegen.rs`:
57+
58+
- Rust metadata is serialized when present, for example extra derives on generated Rust types.
59+
- Python type mappings are backend-local in `reflectapi/src/codegen/python.rs`, not schema annotations.
60+
61+
## Normalization
62+
63+
Normalization lives in `reflectapi-schema-codegen/src/normalize.rs`.
64+
65+
There are two parts:
66+
67+
1. A mutable normalization pipeline over raw `Schema`
68+
2. A `Normalizer` that converts the resulting schema into `SemanticSchema`
69+
70+
The configurable pipeline is built with `PipelineBuilder`. The convenience constructors are:
71+
72+
- `NormalizationPipeline::standard()`
73+
Runs type consolidation, naming resolution, and circular dependency resolution.
74+
- `NormalizationPipeline::for_codegen()`
75+
Skips consolidation and naming, and only runs circular dependency resolution.
76+
77+
After the pipeline runs, `Normalizer` performs:
78+
79+
1. symbol discovery
80+
2. type resolution
81+
3. dependency analysis
82+
4. semantic validation
83+
5. semantic IR construction
84+
85+
`SemanticSchema` provides resolved, deterministic views of functions and types and is defined in `reflectapi-schema-codegen/src/semantic.rs`.
86+
87+
## Backend Behavior
88+
89+
Backends do not all consume the schema in the same way.
90+
91+
### TypeScript
92+
93+
The TypeScript backend in `reflectapi/src/codegen/typescript.rs` consolidates raw schema types and renders directly from the raw schema.
94+
95+
### Rust
96+
97+
The Rust backend in `reflectapi/src/codegen/rust.rs` also works primarily from the raw schema after consolidation.
98+
99+
### Python
100+
101+
The Python backend in `reflectapi/src/codegen/python.rs` uses both representations:
102+
103+
1. `schema.consolidate_types()` runs first
104+
2. `validate_type_references()` checks raw references
105+
3. `Normalizer::normalize_with_pipeline(...)` builds `SemanticSchema` using a pipeline that skips consolidation and naming
106+
4. rendering uses semantic ordering and symbol information, while still consulting raw schema details where the backend needs original field/type shapes
107+
108+
Python-specific type support is driven by backend-local mappings keyed by canonical Rust type name. Those mappings are static codegen knowledge, not part of the shared schema contract.
109+
110+
### OpenAPI
111+
112+
The OpenAPI backend in `reflectapi/src/codegen/openapi.rs` walks the raw schema directly.
113+
114+
## Runtime-Specific Types
115+
116+
ReflectAPI includes special API-facing types whose semantics matter to codegen:
117+
118+
- `reflectapi::Option<T>`: three-state optional value for PATCH-like APIs
119+
- `reflectapi::Empty`: explicit empty request/response body type
120+
- `reflectapi::Infallible`: explicit “no error payload” type
121+
122+
The Python backend treats these as runtime-provided abstractions rather than generated models.
123+
124+
## Testing and Validation
125+
126+
`reflectapi-demo` is the main regression suite.
127+
128+
The snapshot harness in `reflectapi-demo/src/tests/assert.rs` generates five artifacts per test:
129+
130+
- raw schema JSON
131+
- TypeScript client output
132+
- Rust client output
133+
- OpenAPI output
134+
- Python client output
135+
136+
The workspace also contains compile-pass and compile-fail tests driven by `trybuild`.
137+
138+
This architecture chapter is intended to describe the code paths that exist in the repository, not an aspirational future design.

‎docs/src/clients/README.md‎

Lines changed: 74 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -1,70 +1,95 @@
11
# Client Generation
22

3-
`reflectapi` automatically generates type-safe client libraries for multiple programming languages from your Rust API server. This section covers how to generate and use clients in different languages.
3+
`reflectapi` can generate client code from a reflected schema JSON file.
44

5-
## Supported Languages
5+
## Supported Outputs
66

7-
| Language | Status |
8-
|------------|----------|
9-
| TypeScript | ✅ Stable |
10-
| Rust | ✅ Stable |
11-
| Python | ✅ Experiemental |
7+
| Output | Status | Notes |
8+
|--------|--------|-------|
9+
| TypeScript | Stable | Single generated file |
10+
| Rust | Stable | Single generated file |
11+
| Python | Experimental | Package-style output with `__init__.py` and `generated.py` |
1212

13-
## Code Generation Workflow
13+
OpenAPI generation is also supported by the CLI, but it is documented separately as an API description format rather than a client library.
1414

15-
See demo project setup [https://github.com/thepartly/reflectapi/tree/main/reflectapi-demo](https://github.com/thepartly/reflectapi/tree/main/reflectapi-demo)
15+
## Workflow
1616

17-
1. **Define your API server** using `reflectapi` traits and builder
18-
2. **Generate schema** as JSON from your Rust application
19-
3. **Run the CLI** to generate client libraries
20-
4. **Use the clients** in your applications with full type safety
17+
1. Define your API server using `reflectapi` derives and the builder API.
18+
2. Write the schema JSON from your Rust application.
19+
3. Run `reflectapi codegen` for the target language.
20+
4. Commit or consume the generated client code from your application.
2121

22-
```bash
23-
# Generate schema (from your Rust app)
24-
cargo run # Your app should save reflectapi-schema.json
22+
The CLI defaults to `reflectapi.json` if `--schema` is omitted. The demo project uses that filename. If your application writes a different filename such as `reflectapi-schema.json`, pass that path explicitly.
2523

26-
# Generate clients
27-
cargo run reflectapi codegen --language typescript --schema reflectapi-schema.json --output clients/typescript
28-
cargo run reflectapi codegen --language python --schema reflectapi-schema.json --output clients/python
29-
cargo run reflectapi codegen --language rust --schema reflectapi-schema.json --output clients/rust
24+
```bash
25+
# Create output directories first. TypeScript and Rust write a single file
26+
# unless the output path already exists as a directory or ends with a slash.
27+
mkdir -p clients/typescript clients/python clients/rust
28+
29+
# Generate TypeScript client -> clients/typescript/generated.ts
30+
cargo run --bin reflectapi -- codegen \
31+
--language typescript \
32+
--schema reflectapi.json \
33+
--output clients/typescript/
34+
35+
# Generate Python client -> clients/python/__init__.py and generated.py
36+
cargo run --bin reflectapi -- codegen \
37+
--language python \
38+
--schema reflectapi.json \
39+
--output clients/python/ \
40+
--python-sync
41+
42+
# Generate Rust client -> clients/rust/generated.rs
43+
cargo run --bin reflectapi -- codegen \
44+
--language rust \
45+
--schema reflectapi.json \
46+
--output clients/rust/
3047
```
3148

32-
## Common Features
49+
If you installed the CLI separately, replace `cargo run --bin reflectapi --` with `reflectapi`.
50+
51+
## Output Shape
52+
53+
The generators do not all emit the same file layout:
54+
55+
| Output | Files written by the generator |
56+
|--------|--------------------------------|
57+
| TypeScript | `generated.ts` |
58+
| Rust | `generated.rs` |
59+
| Python | `__init__.py`, `generated.py` |
60+
61+
The demo repository includes extra project scaffolding around some generated clients, but that scaffolding is not produced by `reflectapi codegen` itself.
62+
63+
## Language Behavior
3364

34-
All generated clients share these characteristics:
65+
### TypeScript
3566

36-
### Type Safety
37-
- Native type definitions for each language
38-
- Compile-time or runtime type checking
39-
- IDE support with autocompletion
67+
- Uses generated TypeScript types and function wrappers.
68+
- Uses a `fetch`-based default client implementation.
69+
- Parses JSON responses, but does not generate runtime schema validators today.
70+
- Supports custom client implementations via the generated client interface.
4071

41-
### Extensibility
42-
- Default base client implementation is provided
43-
- Which can be replaced or extended with features, such opentelemetry instrumentation or playwrite tracing
72+
### Python
4473

45-
### Error Handling
46-
- Structured error types
47-
- Network error handling
74+
- Generates Pydantic-based models and client code.
75+
- Generates an async client by default.
76+
- Adds a sync client only when `--python-sync` is passed.
77+
- Uses `reflectapi_runtime` for client base classes and runtime helpers.
4878

49-
### Async Support
50-
- Modern async/await patterns
79+
### Rust
5180

52-
### Documentation
53-
- Generated from your Rust documentation
54-
- Type information in IDE tooltips
81+
- Generates typed async client methods.
82+
- Integrates with `reflectapi::rt::Client`.
83+
- Supports optional tracing instrumentation through `--instrument`.
84+
- Generates serde-compatible types and request helpers for JSON-based transport.
5585

56-
### HTTP Client Libraries
86+
## Shared Characteristics
5787

58-
| Language | HTTP Library | Features |
59-
|----------|--------------|----------|
60-
| TypeScript | fetch API | Native browser/Node.js support |
61-
| Python | httpx | Async/sync, HTTP/2, connection pooling |
62-
| Rust | reqwest | Async, HTTP/2, TLS, middleware |
88+
The generated clients all aim to provide:
6389

64-
### Serialization
90+
- Types derived from the Rust-reflected schema
91+
- Function wrappers with generated documentation
92+
- Structured handling of application errors versus transport/protocol failures
93+
- Good IDE support through generated type information
6594

66-
| Language | Serialization | Validation |
67-
|----------|---------------|------------|
68-
| TypeScript | JSON.parse/stringify | Runtime type checking |
69-
| Python | Pydantic | Schema validation, type coercion |
70-
| Rust | JSON or MessagePack (serde) | Compile-time, zero-cost |
95+
They do not currently all provide the same runtime validation guarantees or the same runtime transport abstractions, so those details should be considered language-specific.

‎docs/src/getting-started/installation.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,9 +13,11 @@ cargo add reflectapi --features builder,axum,json,chrono
1313
Install the CLI tool to generate client libraries:
1414

1515
```bash
16-
cargo install reflectapi-cli --help
16+
cargo install reflectapi-cli
1717
```
1818

19+
This installs the `reflectapi` binary.
20+
1921
## Next Steps
2022

2123
- **New users**: Follow the [Quick Start](./quick-start.md) guide

0 commit comments

Comments
 (0)