feat: `RF GUI <-> python client` interoperabilty #2663

daquinteroflex · 2025-07-15T19:52:52Z

Not particularly pleased with this implementation, but can't think of a cleaner option whilst mantaining backwards compatiblity. We should properly design this though and looking forward to the webapi refactor one day.

There are more things that maybe would be worth changing, but unless we truly break backwards compatibility and have to update several notebooks... Maybe that is TBC in a different PR, for now this is just about standardising the simulation and data flow.

So ultimately, I think this should be close to achieving the goal of having a self-contained TerminalComponentModelerData as a serializable and deserializable data structure. However, again, unless we break backwards compatibility with batch_data on xComponentModeler, we cannot have fully pure functions. However, I have tried to at least "purify" the batch_data usage wherever it is called so that we can replace it in the future from simulation.batch_data.

Open to chatting about the exact data structures chosen for the SMatrixData types, it was mainly done by chatting with Damian and enabling backwards compatibility. It was renamed MicrowaveXData to reduce the scope and compatibility of these changes, really we should revisit this with a holistic view and willing to break compatibility. However, having these strong data types should in any case support the migration effort whatever we do onwards.

Considerations:

Implemement minimal changes, whilst enabling interoperability
Eventually enable web.run(TerminalComponentModeler) -> TerminalComponentModelerData, by using a local_run.run(TerminalComponentModeler) -> TerminalComponentModelerData

Again, I'm not super happy with it, but feel it's a bit better and can help us move forward. Open to suggestions to improve or feel free to propose changes directly.

TODO

Add coverage tests if we're all happy with the implementation structure
We need to debate when we merge this within the release flow, 2.9.0 or 2.10.0rc1
Update submodules to develop
Add depreciation warnings on TerminalComponentModeler classmethods that run internally.
batch_cached is a mutation aberration that we should remove sooner than later.

Greptile Summary

This PR implements interoperability between the RF GUI and Python client for Terminal Component Modeling through a significant architectural restructuring. The changes focus on standardizing data flow and S-matrix computations while maintaining backward compatibility with existing implementations.

Key changes include:

New data structures (MicrowavePortSimulationData, MicrowaveSMatrixData, TerminalComponentModelerData)
Dedicated local_run.py module for S-matrix computations
RF-specific web API endpoints and task types
Port impedance validation utilities
RF license warning validators
Version bump to 2.10.0rc1

Confidence score: 2/5

This PR requires significant attention before merging but provides a foundation for future improvements
Score reflects:
- Acknowledged technical debt in implementation
- Circular dependencies in data.py that need resolution
- Mixing of web methods in plugin data types that violates architectural principles
- However, core RF computations and data structures are well-implemented with proper physics validation
Files needing attention:
- tidy3d/plugins/smatrix/data/data.py: Resolve circular dependencies with local_run.py
- tidy3d/plugins/smatrix/component_modelers/terminal.py: Clean up web method coupling
- tidy3d/plugins/smatrix/local_run.py: Complete batch_data decoupling

greptile-apps

Reviewing changes made in this pull request

tidy3d/plugins/smatrix/component_modelers/base.py

tidy3d/web/api/container.py

tidy3d/plugins/smatrix/utils.py

tidy3d/web/api/tidy3d_stub.py

tidy3d/plugins/smatrix/component_modelers/terminal.py

tidy3d/plugins/smatrix/run.py

tidy3d/plugins/smatrix/component_modelers/terminal.py

dmarek-flex · 2025-07-20T22:05:27Z

The failing tests is due to the random data, which was previously circumvented using monkeypatch to ignore the check.

To fix the breaking tests, you need to update this monkeypatch part with something that will ignore that check.

greptile-apps

_{7 files reviewed, 3 comments}

_{Edit Code Review Bot Settings | Greptile}

tests/test_plugins/smatrix/test_terminal_component_modeler.py

tidy3d/plugins/smatrix/data/data.py

schemas/TerminalComponentModeler.json

greptile-apps

_{8 files reviewed, 2 comments}

_{Edit Code Review Bot Settings | Greptile}

tidy3d/plugins/smatrix/local_run.py

greptile-apps

_{18 files reviewed, no comments}

_{Edit Code Review Bot Settings | Greptile}

greptile-apps

_{18 files reviewed, 1 comment}

_{Edit Code Review Bot Settings | Greptile}

greptile-apps · 2025-07-22T10:19:16Z

tidy3d/web/core/task_core.py

        Returns
        -------
        folders : [Folder]
            List of folders


style: Documentation inconsistent with type hint. Return type shows '[]' but docstring indicates '[Folder]'

Suggested change

Returns

-------

folders : [Folder]

List of folders

def list(cls, projects_endpoint: str = "tidy3d/projects") -> list[Folder]:

yaugenst-flex

Thanks @daquinteroflex! Left a couple of comments but overall I would say if it works we'll go with it. Depends a bit on how many breaking API changes we can introduce here. In general I would probably try to put many more methods into the data model. In some sense these are pretty similar to the monitor data in components, which also do a lot of domain-specific postprocessing.

yaugenst-flex · 2025-07-22T11:01:10Z

tests/test_plugins/smatrix/test_component_modeler.py

+    # BREAK this test because it introduces mutability which shouldn't exist
+    # assert modeler2.batch_cached == modeler2.batch == batch


But that seems to be a problem with the implementation and not with the test...?

Sure maybe it's probably the type hint that can fix it when it rehydrates

Renabled and passing

yaugenst-flex · 2025-07-22T11:33:00Z

tidy3d/plugins/smatrix/component_modelers/base.py

@@ -168,8 +174,10 @@ def to_file(self, fname: str) -> None:
        super(AbstractComponentModeler, self).to_file(fname=fname)  # noqa: UP008


The noqa here should go, as should the parameters to super()

So I think a lot of this stuff is from the privous implementation

tidy3d/plugins/smatrix/component_modelers/base.py

yaugenst-flex · 2025-07-22T11:42:24Z

tidy3d/plugins/smatrix/component_modelers/terminal.py

+    @staticmethod
+    def _set_port_data_array_attributes(data_array: PortDataArray) -> PortDataArray:
+        """Helper to set additional metadata for ``PortDataArray``."""
+        data_array.name = "Z0"
+        return data_array.assign_attrs(units=OHM, long_name="characteristic impedance")


Ooof.. is this really necessary? Why does the array need to be mutated?

This is from my original implementation. My reasoning was that there are many types of data that can be represented by the same DataArray types, since they have the same dimensions or coordinates.

Is it better to create a new DataArray type just for impedance values, or change this method to copy and modify the input?

Yeah. In principle you're right of course but this is exactly the reason why we have so many different array types. We'll have to rethink that at some point but I think for now it'd be better to just introduce a separate type.

tidy3d/plugins/smatrix/data/data.py

tidy3d/plugins/smatrix/local_run.py

yaugenst-flex · 2025-07-22T12:00:35Z

tidy3d/plugins/smatrix/local_run.py

+    )
+
+
+def construct_smatrix(


This seems like it should be a method of ComponentModelerData. Actually this is my impression for most of the functions in here.

So I was thinking it'd be good if we can separate the "data construction" methods from the data containers and the simualtion definitions. ie. it'd be nice if they're self contained functions that just take in local.run(ComponentModeler) -> ComponentModelerData to avoid the mixing up of stuff in the original implementation. I've moved them to a separate file because this way we can reuse them for backwards and onwards compatibilty without having to recreate a TerminalComponentModelerData class inside TerminalComponentModeler to use this method directly for now.

Also would have expected most of these methods to be part of ComponentModelerData, since that is how these things are mostly organized for MonitorData. I don't have a strong preference either way. But shouldn't you also move get_antenna_metrics_data and _monitor_data_at_port_amplitude to this file as well then?

Do we have a general guideline on those separations that are (or will be) applied to the entire tidy3d?

Will reply privately to both this and this, as related to future plans as far as I understand them

Also would have expected most of these methods to be part of ComponentModelerData, since that is how these things are mostly organized for MonitorData. I don't have a strong preference either way. But shouldn't you also move get_antenna_metrics_data and _monitor_data_at_port_amplitude to this file as well then?

yaugenst-flex · 2025-07-22T12:04:54Z

tidy3d/plugins/smatrix/utils.py

+    return z_matrix
+
+
+def check_port_impedance_sign(Z_numpy: np.ndarray):


This seems like it should be a validator somewhere.

Agreed, the problem is backwards compatibility on how this method gets used dynamically in construct_smatrix

When a WavePort is present, the port impedance is calculated in a post-processing step. Does it still make sense to make this a validator?

yaugenst-flex · 2025-07-22T12:06:08Z

tidy3d/web/core/task_core.py

@@ -37,15 +37,15 @@ class Folder(Tidy3DResource, Queryable, extra=Extra.allow):
    )

    @classmethod
-    def list(cls) -> []:
+    def list(cls, projects_endpoint: str = "tidy3d/projects") -> []:


Return type hint should be list, not []

So the endpoints code itself has not been finished because it depends on whatver implementation we choose to do, so should have clarfied this is still WIP or can remove it for now

dmarek-flex · 2025-07-23T13:38:38Z

tidy3d/plugins/smatrix/data/data.py

rename modeler_data.py?

dmarek-flex

Just made a first pass, and the basic idea looks good to me. My main uncertainty is the the usage of the post-processing functionality present in local_run.py. Ignoring backwards compatibility, can you explain why that would be better than having them as members in the TerminalComponentModelerData?

dmarek-flex · 2025-07-23T13:41:03Z

tidy3d/plugins/smatrix/data/data.py

+        "Stores the computed S-matrix and reference impedances for the terminal ports"
+        from tidy3d.plugins.smatrix.local_run import construct_smatrix
+
+        terminal_port_data = construct_smatrix(simulation=self.simulation)


Think it would be better to directly use self.data now

But I also see the problem, you would need to add a method for going from dict[TerminalPortType, SimulationData] to dict[str, SimulationData] which would be compatible to how the batch_data is used.

So aiming for somthing lke

sim_data = modeler_data.data[port_in]

and/or

def compose_terminal_modeler_data( modeler: TerminalComponentModeler, batch_data: BatchData = None, ) -> TerminalComponentModelerData: port_to_sim_data_map = { port_i: batch_data[modeler.get_task_name(port=port_i)] for port_i in modeler.ports } port_simulation_data = PortSimulationData(data=port_to_sim_data_map) return TerminalComponentModelerData(modeler=modeler, data=port_simulation_data)

dmarek-flex · 2025-07-23T13:47:48Z

tidy3d/plugins/smatrix/utils.py

+    return z_matrix
+
+
+def check_port_impedance_sign(Z_numpy: np.ndarray):


When a WavePort is present, the port impedance is calculated in a post-processing step. Does it still make sense to make this a validator?

dmarek-flex · 2025-07-23T14:23:37Z

tidy3d/plugins/smatrix/component_modelers/base.py

@@ -35,6 +35,11 @@
 class AbstractComponentModeler(ABC, Tidy3dBaseModel):
    """Tool for modeling devices and computing port parameters."""

+    name: str = pd.Field(


Just needs update for title and description

dmarek-flex · 2025-07-23T14:36:24Z

tidy3d/plugins/smatrix/data/data.py

+        description="Reference impedance for each port used in the S-parameter calculation. "
+        "This is optional and may not be present if not specified or computed.",
+    )
+    data: TerminalPortDataArray = pd.Field(


rename to s

dmarek-flex · 2025-07-23T14:42:41Z

tidy3d/plugins/smatrix/data/data.py

+class MicrowavePortSimulationData(Tidy3dBaseModel):
+    """Stores raw simulation data from each microwave port-specific simulation."""
+
+    data: dict[TerminalPortType, SimulationData] = pd.Field(


I think this should change to dict[str, SimulationData] for two reasons:

One to make it easier to take the place of batch_data in some of these helper functions for postprocessing. Since I think in every post processing function the batch_data is mainly used as a mapping from a task name to the simulation data.

More importantly, a single Port may be associated with more than one SimulationData, like when we support multimodal WavePort where each mode will correspond with a different simulation.

So how does the current implementation of sound like? Achieves that goal

PortReferenceType = Union[str, TerminalPortType] # TODO Debate this class PortSimulationData(Tidy3dBaseModel): # TODO Debate typing.TypedDict instead, I prefer """Stores raw simulation data from each microwave port-specific simulation.""" data: dict[PortReferenceType, SimulationData] = pd.Field(

dmarek-flex · 2025-07-23T14:48:56Z

tidy3d/plugins/smatrix/local_run.py

+    )
+
+
+def construct_smatrix(


Also would have expected most of these methods to be part of ComponentModelerData, since that is how these things are mostly organized for MonitorData. I don't have a strong preference either way. But shouldn't you also move get_antenna_metrics_data and _monitor_data_at_port_amplitude to this file as well then?

tidy3d/plugins/smatrix/data/data.py

daquinteroflex changed the title ~~feat: RF GUI endpoints demo~~ feat: RF GUI<->python client interoperabilty Jul 18, 2025

daquinteroflex changed the title ~~feat: RF GUI<->python client interoperabilty~~ feat: RF GUI <-> python client interoperabilty Jul 18, 2025

daquinteroflex force-pushed the dario/rf_endpoints branch 2 times, most recently from 0e00e0d to 49f3708 Compare July 18, 2025 11:23

daquinteroflex marked this pull request as ready for review July 18, 2025 15:38

greptile-apps bot reviewed Jul 18, 2025

View reviewed changes

daquinteroflex marked this pull request as draft July 21, 2025 13:11

daquinteroflex marked this pull request as ready for review July 21, 2025 20:10

greptile-apps bot reviewed Jul 21, 2025

View reviewed changes

tests/test_plugins/smatrix/test_terminal_component_modeler.py Outdated Show resolved Hide resolved

tidy3d/plugins/smatrix/data/data.py Outdated Show resolved Hide resolved

schemas/TerminalComponentModeler.json Show resolved Hide resolved

daquinteroflex marked this pull request as draft July 21, 2025 20:14

daquinteroflex marked this pull request as ready for review July 22, 2025 09:00

greptile-apps bot reviewed Jul 22, 2025

View reviewed changes

tidy3d/plugins/smatrix/local_run.py Outdated Show resolved Hide resolved

tidy3d/plugins/smatrix/local_run.py Outdated Show resolved Hide resolved

daquinteroflex requested a review from yaugenst-flex July 22, 2025 09:17

daquinteroflex marked this pull request as draft July 22, 2025 09:23

daquinteroflex marked this pull request as ready for review July 22, 2025 09:23

greptile-apps bot reviewed Jul 22, 2025

View reviewed changes

yaugenst-flex reviewed Jul 22, 2025

View reviewed changes

daquinteroflex requested review from dbochkov-flexcompute and dmarek-flex July 22, 2025 19:05

dmarek-flex reviewed Jul 23, 2025

View reviewed changes

tidy3d/plugins/smatrix/data/data.py Outdated

Copy link

Contributor

dmarek-flex Jul 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rename modeler_data.py?

daquinteroflex reacted with thumbs up emoji

dmarek-flex reviewed Jul 23, 2025

View reviewed changes

dmarek-flex mentioned this pull request Jul 23, 2025

add selective simulation capabilities to TerminalComponentModeler #2573

Merged

daquinteroflex force-pushed the dario/rf_endpoints branch 3 times, most recently from 69b2f32 to a676b07 Compare July 31, 2025 14:36

daquinteroflex added 3 commits July 31, 2025 16:43

Parametrized folder endpoints

98b8749

circular import still a problm

73d868c

Remove circular

8acd1a6

daquinteroflex added 3 commits July 31, 2025 16:43

Starting to remove incompatibiltiies

faf0199

Add tasks to webapi

912464d

Further refactor

9e89977

daquinteroflex force-pushed the dario/rf_endpoints branch from a676b07 to 9e89977 Compare July 31, 2025 17:10

daquinteroflex added 6 commits August 1, 2025 10:51

Avoid rename for diff

b939abe

Preparing to further refactor

f395827

Progress

5533c59

Before refactoring smatrix on modeler data

63bc725

still in progress

a161de7

First structural refactor complete

2b4e107

daquinteroflex marked this pull request as draft August 1, 2025 12:46

flexcompute deleted a comment from github-actions bot Aug 1, 2025

daquinteroflex changed the base branch from develop to dario/release-2.9.0 August 1, 2025 13:02

daquinteroflex added 4 commits August 1, 2025 15:15

Cleanup usage of rf endpoint

31b7a83

Stable time to try out tests and notebooks

a313e71

Importing, but definitely broken tests

81eaf26

Just add port simulation map

324b721

		# BREAK this test because it introduces mutability which shouldn't exist
		# assert modeler2.batch_cached == modeler2.batch == batch

		@@ -168,8 +174,10 @@ def to_file(self, fname: str) -> None:
		super(AbstractComponentModeler, self).to_file(fname=fname) # noqa: UP008

		return z_matrix


		def check_port_impedance_sign(Z_numpy: np.ndarray):

feat: RF GUI <-> python client interoperabilty #2663

Are you sure you want to change the base?

feat: RF GUI <-> python client interoperabilty #2663

Uh oh!

Conversation

daquinteroflex commented Jul 15, 2025 • edited by greptile-apps bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence score: 2/5

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dmarek-flex commented Jul 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

yaugenst-flex left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmarek-flex Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yaugenst-flex Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daquinteroflex Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

feat: `RF GUI <-> python client` interoperabilty #2663

feat: `RF GUI <-> python client` interoperabilty #2663

daquinteroflex commented Jul 15, 2025 •

edited by greptile-apps bot

Loading

dmarek-flex commented Jul 20, 2025 •

edited

Loading

dmarek-flex Jul 22, 2025 •

edited

Loading

yaugenst-flex Jul 22, 2025 •

edited

Loading

daquinteroflex Jul 22, 2025 •

edited

Loading

daquinteroflex Jul 22, 2025 •

edited

Loading