pydantic v2 (wip)

add somefiles

replacing v1

more replacements

copy methods and ordering of basemodel

update get_submodels_by_hash

basemodel done (except for docs)

basemodel and modespec done

slowly but surely..

progress

next batch

going going

getting started on medium

more refactoring

new structure for medium.py

add medium

why it no work

fix medium

upgrade material library

most of it

first pass

add pydantic-settings

fixes to validators, mutable assignment, ...:

remove skip_if_fields_missing

cleaning up types & type serializaton

fix printing and serialization of autograd types

more type serialization updates

Fix traced ndarray serialization

fix equality check in basemodel

fix some v1 leftovers

fix some tests

fix equality check

sim_data tests passing

make serializer more robust and another fix for equality comparison

fix warn if none validators

Check for pydantic v2 ValidationErrors

Canonicalize coordinate handling of unstructured datasets

Simulation data tests passing

material library tests passing

fix bad name

fix material libray -> library

lotsa fixes, tests_web & test_package passing

fix remaining web test warnings

everything importable for doctests

rebase, wip

fix mutation

safer ndarray coercion and expressions fix

working on post init validation

fix multiphysics medium attribute lookup

remove test script from vcs

doctests passing

fix NedeljkovicSorefMashanovich

fix non-component tests

fix tracer serialization

remove unnecessary to_static call

the smallest changes really do take the longest

all of test_IO passing

fix caching and copy update

docstring for __init_subclass__

passing: IO, base, beam, boundaries, custom, eme, field_projection, geometry, log

passing: grid, grid_spec, heat, heat_charge, layerrefinement, lumped_element, medium

passing: meshgenerate, microwave, mode

passing: monitor, packaging, parameter_perturbation, perturbation_medium, scene, sidewall

passing: expressions

passing: smatrix, array_factor, design, dispersion_fitter, microwave, mode_solver

wip: adjoint & invdes

passing: source, structure, time_modulation

passing: types

add helpers to filter model fields

add tests for new basemodel helpers

small fixes

wip: autograd

fix waveguide

passing: autograd

rebase fixes

no more post-init validators

fix dataarray json schema

rework array constraints

wip: adjoint

ruff

chore(github): improve script  to determine test scope (#2510)

* chore(github): add script  to determine whether a PR was already approved

This step results in a full test suite run if an already approved PR receives changes to prevent breaking the main branch.

* cancel parallel triggers of the remote tests

* fix group assignment

* add python version to group assignment

fix bug when inserting snapping points near sim min boundary

fix contains check for lumped element, which was too strict (#2511)

fix[web]: use dedicated endpoint for `web.delete_old()`

Upgrade ruff 0.5.5 -> 0.11.11

Run `ruff check . --fix`

Run `ruff format .`

Move test-specifc `ruff.toml` to `per-file-ignores`

Sort test imports

Import `Literal` from `typing`

Force `from __future__ import annotations` import

Rewrite `dict()` as literal (2x faster)

Disallow function calls in default arguments

Be intentional about exception context

Upgrade all (compatible) generics to built-in types

Sort `__all__` blocks

Enable `RUF` ruleset

Prefer tuple unpacking over concatenation

Forbid implicit optional

No explicit string concatenation

Remove unnecessary `pass` statements

Remove unnecessary parantheses on raised exceptions

Disallow relative imports from parent modules

Remove unused import aliases

Add lint rule comments

Revert "Disallow function calls in default arguments"

This reverts commit 12a0a6a.

Fix some stragglers

Updated ruff version in workflow

Lint `data/` dirs

rebase fixes

sentinel pattern for function calls in default arguments

rebase fixes

fix python 3.9 compat

same behavior as v1 for `FreqArray`

Also updating FieldDataset in monitor_data/charge.py

Make `FreqArray` pure `ArrayFloat1D`

remove old material_libray.py

replace deprecated symbols

disable adjoint plugin

pin pydantic>2.8

fixing warnings

fix modesolver validators

Updating TriangularGridDataset.normal_axis and TemperatureData validator

Warning adjoint removal in apidocs

pin pydantic>=2.9, fix literal types

Author: yaugenst-flex

docs/api/plugins/adjoint.rst

@@ -1,6 +1,11 @@
 Plugins
 =======
 
+.. warning::
+
+   |:warning:| The 'adjoint' plugin (legacy JAX-based adjoint plugin)
+   was deprecated in Tidy3D 'v2.7.0' and is disabled as of 'v2.9.0'.
+
 .. toctree::
     :hidden:
 
@@ -10,7 +15,6 @@ Plugins
     ./smatrix
     ./resonance
     ./autograd
-    ./adjoint
     ./invdes
     ./design
     ./waveguide
@@ -23,8 +27,7 @@ Plugins
 .. include:: /api/plugins/smatrix.rst
 .. include:: /api/plugins/resonance.rst
 .. include:: /api/plugins/autograd.rst
-.. include:: /api/plugins/adjoint.rst
 .. include:: /api/plugins/invdes.rst
 .. include:: /api/plugins/design.rst
 .. include:: /api/plugins/waveguide.rst
-.. include:: /api/plugins/microwave.rst
+.. include:: /api/plugins/microwave.rst

docs/api/plugins/index.rst

@@ -34,7 +34,7 @@ numpy = "*"
 matplotlib = "*"
 shapely = "^2.0"
 pandas = "*"
-pydantic = "^2.0"
+pydantic = ">=2.9,<3"
 PyYAML = "*"
 dask = "*"
 toml = "*"
@@ -291,7 +291,12 @@ required-imports = ["from __future__ import annotations"]
 
 [tool.pytest.ini_options]
 # TODO: remove --assert=plain when https://github.com/scipy/scipy/issues/22236 is resolved
-addopts = "--cov=tidy3d --doctest-modules -n auto --dist worksteal --assert=plain -m 'not numerical' "
+# TODO(yaugenst-flex): Revisit adjoint plugin for pydantic v2
+addopts = """
+--doctest-modules -n auto --dist worksteal --assert=plain -m 'not numerical' \
+--ignore=tests/test_plugins/test_adjoint.py \
+--ignore=tidy3d/plugins/adjoint/
+"""
 markers = [
   "numerical: marks numerical tests for adjoint gradients that require running simulations (deselect with '-m \"not numerical\"')",
 ]

poetry.lock

@@ -24,6 +24,7 @@
 
 # Store an example of every minor release simulation to test updater in the future
 SIM_DIR = "tests/sims"
+SIM_STATIC = SIM.to_static()
 
 
 @pytest.fixture
@@ -69,14 +70,14 @@ def test_simulation_load_export(split_string, tmp_path):
     major, minor, patch = __version__.split(".")
     path = os.path.join(tmp_path, f"simulation_{major}_{minor}_{patch}.json")
     path_hdf5 = os.path.join(tmp_path, f"simulation_{major}_{minor}_{patch}.h5")
-    SIM.to_file(path)
-    SIM.to_hdf5(path_hdf5)
+    SIM_STATIC.to_file(path)
+    SIM_STATIC.to_hdf5(path_hdf5)
     SIM2 = td.Simulation.from_file(path)
     SIM_HDF5 = td.Simulation.from_hdf5(path_hdf5)
-    assert set_datasets_to_none(SIM)._json_string == SIM2._json_string, (
+    assert set_datasets_to_none(SIM_STATIC)._json_string == SIM2._json_string, (
         "original and loaded simulations are not the same"
     )
-    assert SIM == SIM_HDF5, "original and loaded from hdf5 simulations are not the same"
+    assert SIM_STATIC == SIM_HDF5, "original and loaded from hdf5 simulations are not the same"
 
 
 def test_simulation_load_export_yaml(tmp_path):
@@ -104,30 +105,30 @@ def test_component_load_export_yaml(tmp_path):
 
 def test_simulation_load_export_hdf5(split_string, tmp_path):
     path = str(tmp_path / "simulation.hdf5")
-    SIM.to_file(path)
+    SIM_STATIC.to_file(path)
     SIM2 = td.Simulation.from_file(path)
-    assert SIM == SIM2, "original and loaded simulations are not the same"
+    assert SIM_STATIC == SIM2, "original and loaded simulations are not the same"
 
 
 def test_simulation_load_export_hdf5_gz(split_string, tmp_path):
     path = str(tmp_path / "simulation.hdf5.gz")
-    SIM.to_file(path)
+    SIM_STATIC.to_file(path)
     SIM2 = td.Simulation.from_file(path)
-    assert SIM == SIM2, "original and loaded simulations are not the same"
+    assert SIM_STATIC == SIM2, "original and loaded simulations are not the same"
 
 
 def test_simulation_load_export_hdf5_explicit(split_string, tmp_path):
     path = str(tmp_path / "simulation.hdf5")
-    SIM.to_hdf5(path)
+    SIM_STATIC.to_hdf5(path)
     SIM2 = td.Simulation.from_hdf5(path)
-    assert SIM == SIM2, "original and loaded simulations are not the same"
+    assert SIM_STATIC == SIM2, "original and loaded simulations are not the same"
 
 
 def test_simulation_load_export_hdf5_gz_explicit(split_string, tmp_path):
     path = str(tmp_path / "simulation.hdf5.gz")
-    SIM.to_hdf5_gz(path)
+    SIM_STATIC.to_hdf5_gz(path)
     SIM2 = td.Simulation.from_hdf5_gz(path)
-    assert SIM == SIM2, "original and loaded simulations are not the same"
+    assert SIM_STATIC == SIM2, "original and loaded simulations are not the same"
 
 
 def test_simulation_load_export_pckl(tmp_path):
@@ -192,7 +193,7 @@ def test_validation_speed(tmp_path):
         for i in range(n):
             new_structure = SIM.structures[0].copy(update={"name": str(i)})
             new_structures.append(new_structure)
-        S = SIM.copy(update={"structures": new_structures})
+        S = SIM.copy(update={"structures": tuple(new_structures)})
 
         S.to_file(path)
         time_start = time()
@@ -226,7 +227,9 @@ def test_simulation_updater(sim_file):
 def test_yaml(tmp_path):
     path = str(tmp_path / "simulation.json")
     SIM.to_file(path)
+    SIM.to_file("simulation.json")
     sim = td.Simulation.from_file(path)
+
     path1 = str(tmp_path / "simulation.yaml")
     sim.to_yaml(path1)
     sim1 = td.Simulation.from_yaml(path1)

pyproject.toml

@@ -3,8 +3,8 @@
 from __future__ import annotations
 
 import matplotlib.pyplot as plt
-import pydantic.v1 as pydantic
 import pytest
+from pydantic import ValidationError
 
 import tidy3d as td
 
@@ -17,27 +17,27 @@ def test_apodization():
 
 
 def test_end_lt_start():
-    with pytest.raises(pydantic.ValidationError):
+    with pytest.raises(ValidationError):
         _ = td.ApodizationSpec(start=2, end=1, width=0.2)
 
 
 def test_no_width():
-    with pytest.raises(pydantic.ValidationError):
+    with pytest.raises(ValidationError):
         _ = td.ApodizationSpec(start=1, end=2)
-    with pytest.raises(pydantic.ValidationError):
+    with pytest.raises(ValidationError):
         _ = td.ApodizationSpec(start=1)
-    with pytest.raises(pydantic.ValidationError):
+    with pytest.raises(ValidationError):
         _ = td.ApodizationSpec(end=2)
 
 
 def test_negative_times():
-    with pytest.raises(pydantic.ValidationError):
+    with pytest.raises(ValidationError):
         _ = td.ApodizationSpec(start=-2, end=-1, width=0.2)
 
-    with pytest.raises(pydantic.ValidationError):
+    with pytest.raises(ValidationError):
         _ = td.ApodizationSpec(start=1, end=2, width=-0.2)
 
-    with pytest.raises(pydantic.ValidationError):
+    with pytest.raises(ValidationError):
         _ = td.ApodizationSpec(start=1, end=2, width=0)