more concise readme

Author: yaugenst-flex

tidy3d/config/README.md

@@ -1,40 +1,20 @@
 # `tidy3d.config` Architecture
 
-The configuration subsystem wires together runtime defaults, environment
-overrides, profile files, and optional plugin hooks so that `from tidy3d.config
-import config` always exposes an up-to-date view of the active settings. This
-document focuses on the developer-facing design so you can confidently extend
-or debug the module.
+`tidy3d.config` combines defaults, environment overrides, profile files, and plugin sections so `config` always reflects the active settings. This note is aimed at contributors who need to extend or debug the module.
 
 ## Big Picture
 
-- **Pydantic schemas define sections.** Every built-in section lives in
-  `sections.py` and is registered via the `@register_section` decorator.
-- **A central manager composes state.** `ConfigManager` merges defaults,
-  environment variables, builtin/user profiles, and runtime overrides. It also
-  applies side-effect handlers whenever values change.
-- **Persistence is layered.** `ConfigLoader` handles filesystem reads/writes;
-  `serializer.py` keeps TOML files annotated with descriptions and stable key
-  ordering.
-- **Registries are dynamic.** `registry.py` tracks section schemas and handler
-  callables and notifies the manager whenever new items arrive (including from
-  plugins).
-- **Legacy wrappers exist for compatibility.** `legacy.py` exposes the previous
-  API surface while delegating to the modern manager.
-
-## Lifecycle Overview
-
-1. Importing `tidy3d.config` loads `sections.py`, which registers every built-in
-   section and handler.
-2. `ConfigManager` instantiates, attaches itself to the registry, and loads the
-   effective tree by merging builtin profiles, `config.toml`, profile overrides,
-   environment variables, and any runtime overrides.
-3. Handlers declared with `@register_handler` run to push settings into global
-   side effects (log level, environment variables, cache directories, etc.).
-4. The public `config` object is a `LegacyConfigWrapper` that proxies to the
-   active manager but still honours legacy attributes.
-5. Runtime mutations go through `ConfigManager.update_section`, which triggers a
-   reload+reapply cycle so memoized models always represent the latest state.
+- Section schemas live in `sections.py` and register via `register_section`.
+- `ConfigManager` merges builtin defaults, saved files, environment overrides, and runtime edits, then runs section handlers.
+- `ConfigLoader` handles disk IO while `serializer.py` preserves comments and key order inside TOML files.
+- `registry.py` tracks sections and handlers so late imports (plugins, tests) attach automatically.
+- `legacy.py` keeps the historical API working by delegating to the manager.
+
+## Runtime Flow
+
+1. Importing `tidy3d.config` registers built-in sections and handlers.
+2. `ConfigManager` attaches to the registry, loads builtin and user profiles, applies environment overrides, and composes the effective tree.
+3. Handlers push side effects (logging level, env vars, cache dirs). Calls to `update_section` reload the tree and re-run the relevant handlers.
 
 ## Component Map
 
@@ -64,120 +44,38 @@ flowchart LR
     env_vars["Environment variables"] --> loader_py
     builtin_profiles["profiles.py<br/>BUILTIN_PROFILES"] --> manager_py
     runtime_overrides["Runtime overrides"] --> manager_py
-    plugins["register_plugin(... )<br/>(plugin imports)"] --> registry_py
+    plugins["register_plugin(...)<br/>(plugin imports)"] --> registry_py
     registry_py --> manager_py
 ```
 
 ## Module Reference
 
-### `sections.py`
-
-- Defines concrete `ConfigSection` subclasses (Pydantic models) for built-in
-  sections (`logging`, `simulation`, `microwave`, `adjoint`, `web`,
-  `local_cache`, `plugins` container).
-- Decorated with `@register_section("name")` so the schema is discoverable at
-  import time. For plugins, use `@register_plugin("plugin_name")`.
-- Optional `@register_handler("name")` functions apply side effects after the
-  manager reloads (e.g., update logger globals, export environment variables).
-- Fields can include `json_schema_extra={"persist": True}` to mark values that
-  should be written to disk by default.
-
-### `registry.py`
-
-- Holds the global section and handler dictionaries.
-- Exposes decorators (`register_section`, `register_plugin`, `register_handler`)
-  and accessors (`get_sections`, `get_handlers`).
-- Provides `attach_manager()` so the active `ConfigManager` is notified whenever
-  new sections or handlers register. This allows late imports (plugins, tests)
-  to automatically appear without manual refresh.
-
-### `manager.py`
-
-- `ConfigManager` orchestrates the entire system:
-  - Attaches to the registry and caches per-section Pydantic model instances.
-  - Loads data through `ConfigLoader` and merges layers using `deep_merge`.
-  - Tracks runtime overrides in memory per profile.
-  - Filters persisted values (`_filter_persisted`) so `config.save()` writes only
-    annotated fields unless `include_defaults=True`.
-  - Invokes handlers after every reload or targeted update.
-  - Exposes helper accessors (`plugins`, `profiles`, `get_section`, `format`,
-    etc.).
-- `SectionAccessor` proxies dot-attribute access back into `update_section`.
-- Normalizes profile names so builtin aliases like `"prod"` resolve
-  consistently.
-
-### `loader.py`
-
-- Resolves the configuration directory (`resolve_config_directory`) based on
-  platform, `$TIDY3D_BASE_DIR`, and legacy fallbacks.
-- Reads/writes `config.toml` and profile overrides atomically, including
-  temporary file + backup behaviour.
-- Parses environment variables into nested dictionaries (`load_environment_overrides`).
-- Supplies dictionary utilities (`deep_merge`, `deep_diff`) used throughout
-  the manager.
-- Coordinates with `serializer.build_document` to preserve inline comments and
-  descriptions when writing TOML.
-
-### `serializer.py`
-
-- Looks up field descriptions from registered Pydantic models so TOML files
-  stay annotated.
-- Converts nested dictionaries into `tomlkit` documents with stable formatting
-  and comment preservation—critical for user-friendly diffs.
-
-### `profiles.py`
-
-- Maintains the shipped profile presets (`default`, `prod`, `dev`, `uat`,
-  `pre`, `nexus`). The manager merges these into the baseline before applying
-  user overrides.
-
-### `legacy.py`
-
-- Provides backwards-compatible attribute access (`LegacyConfigWrapper`,
-  `LegacyEnvironment`, etc.) that forward to the modern manager.
-- Emits `DeprecationWarning`s to encourage migrations while keeping older code
-  functional.
-
-## Adding a New Section
-
-1. Define a Pydantic model in `sections.py` (or your plugin) that inherits from
-   `ConfigSection` and decorate it with `@register_section("name")`.
-2. Optionally decorate a function with `@register_handler("name")` to apply
-   side effects whenever the section changes.
-3. Add field descriptions and `json_schema_extra={"persist": True}` where
-   appropriate so they show up in the generated docs and persisted config.
-4. Import the module somewhere reachable so registration happens as part of
-   normal startup (built-ins live in `sections.py`; plugins should register at
-   import time).
-
-`ConfigManager` will automatically detect the new section, expose it under
-`config.<name>`, and include it in persistence and documentation without extra
-wiring.
-
-## Handler Side Effects
-
-- Handlers receive the fully validated Pydantic model for the section.
-- Only the sections explicitly updated trigger their handler; call
-  `config.reload_config()` or `ConfigManager._apply_handlers()` manually if you
-  need to reapply everything (usually done during initialization).
-- Handlers must be robust to repeated calls because they run on every reload.
-
-## Persistence and Serialization Notes
-
-- Only fields marked with `json_schema_extra={"persist": True}` are written by
-  default. Call `config.save(include_defaults=True)` to flush the entire model
-  tree to disk (useful for debugging).
-- TOML output preserves descriptions derived from field docstrings, so keep the
-  docstrings concise and informative.
-- The loader writes files atomically and stores a `.bak` during the swap to
-  prevent data loss. Beware of direct edits to `_loader._docs`; tests can use
-  `ConfigLoader` to manipulate the cached documents safely.
-
-## Debugging Tips
-
-- `config.format()` (or `print(config)`) renders a Rich tree of the current
-  effective configuration—useful when verifying merges.
-- To inspect persisted values without env overrides, call
-  `ConfigManager._compose_without_env()` in a debugger.
-- The registry exposes `get_sections()` / `get_handlers()` for quick sanity
-  checks that your plugin registered correctly.
+- `sections.py` - Pydantic models for built-in sections (logging, simulation, microwave, adjoint, web, local cache, plugin container) registered via `register_section`. The bundled models inherit from the internal `ConfigSection` helper, but external code can use plain `BaseModel` subclasses. Optional handlers perform side effects. Fields mark persistence with `json_schema_extra={"persist": True}`.
+- `registry.py` - Stores section and handler registries and notifies the attached manager so new entries appear immediately.
+- `manager.py` - `ConfigManager` caches validated models, tracks runtime overrides per profile, filters persisted fields, exposes helpers such as `plugins`, `profiles`, and `format`. `SectionAccessor` routes attribute access to `update_section`.
+- `loader.py` - Resolves the config directory, loads `config.toml` and `profiles/<name>.toml`, parses environment overrides, and writes atomically through `serializer.build_document`.
+- `serializer.py` - Builds stable TOML documents with descriptive comments derived from section docstrings.
+- `profiles.py` - Supplies builtin profiles merged ahead of user overrides.
+- `legacy.py` - Implements backward-compatible wrappers and deprecation warnings around the manager.
+
+## Extending the System
+
+1. Define a Pydantic model and decorate it with `register_section`. Built-in sections use `ConfigSection`, but the decorator accepts any `BaseModel`.
+2. Optionally define a handler with `register_handler` for side effects that must track the section.
+3. Ensure the module imports during startup so registration happens automatically.
+
+## Handler Rules
+
+- Handlers receive the validated section model.
+- They must tolerate repeated calls and only run for sections that changed unless you trigger `config.reload_config()` for a full pass.
+
+## Persistence Notes
+
+- Only fields tagged with `persist` write by default. Call `config.save(include_defaults=True)` to emit the full tree.
+- `ConfigLoader` writes files atomically and leaves a `.bak` backup while swapping.
+
+## Debugging
+
+- `config.format()` prints the composed tree - handy for verifying merges.
+- Inspect `_compose_without_env()` in a debugger to view the persisted state only.
+- `get_sections()` and `get_handlers()` confirm that new registrations landed.