Skip to content

feat: RF GUI <-> python client interoperabilty #2663

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 16 commits into
base: dario/release-2.9.0
Choose a base branch
from

Conversation

daquinteroflex
Copy link
Collaborator

@daquinteroflex daquinteroflex commented Jul 15, 2025

Not particularly pleased with this implementation, but can't think of a cleaner option whilst mantaining backwards compatiblity. We should properly design this though and looking forward to the webapi refactor one day.

There are more things that maybe would be worth changing, but unless we truly break backwards compatibility and have to update several notebooks... Maybe that is TBC in a different PR, for now this is just about standardising the simulation and data flow.

So ultimately, I think this should be close to achieving the goal of having a self-contained TerminalComponentModelerData as a serializable and deserializable data structure. However, again, unless we break backwards compatibility with batch_data on xComponentModeler, we cannot have fully pure functions. However, I have tried to at least "purify" the batch_data usage wherever it is called so that we can replace it in the future from simulation.batch_data.

Open to chatting about the exact data structures chosen for the SMatrixData types, it was mainly done by chatting with Damian and enabling backwards compatibility. It was renamed MicrowaveXData to reduce the scope and compatibility of these changes, really we should revisit this with a holistic view and willing to break compatibility. However, having these strong data types should in any case support the migration effort whatever we do onwards.

Considerations:

  • Implemement minimal changes, whilst enabling interoperability
  • Eventually enable web.run(TerminalComponentModeler) -> TerminalComponentModelerData, by using a local_run.run(TerminalComponentModeler) -> TerminalComponentModelerData

Again, I'm not super happy with it, but feel it's a bit better and can help us move forward. Open to suggestions to improve or feel free to propose changes directly.

TODO

  • Add coverage tests if we're all happy with the implementation structure
  • We need to debate when we merge this within the release flow, 2.9.0 or 2.10.0rc1
  • Update submodules to develop
  • Add depreciation warnings on TerminalComponentModeler classmethods that run internally.
  • batch_cached is a mutation aberration that we should remove sooner than later.

Greptile Summary

This PR implements interoperability between the RF GUI and Python client for Terminal Component Modeling through a significant architectural restructuring. The changes focus on standardizing data flow and S-matrix computations while maintaining backward compatibility with existing implementations.

Key changes include:

  • New data structures (MicrowavePortSimulationData, MicrowaveSMatrixData, TerminalComponentModelerData)
  • Dedicated local_run.py module for S-matrix computations
  • RF-specific web API endpoints and task types
  • Port impedance validation utilities
  • RF license warning validators
  • Version bump to 2.10.0rc1

Confidence score: 2/5

  1. This PR requires significant attention before merging but provides a foundation for future improvements
  2. Score reflects:
    • Acknowledged technical debt in implementation
    • Circular dependencies in data.py that need resolution
    • Mixing of web methods in plugin data types that violates architectural principles
    • However, core RF computations and data structures are well-implemented with proper physics validation
  3. Files needing attention:
    • tidy3d/plugins/smatrix/data/data.py: Resolve circular dependencies with local_run.py
    • tidy3d/plugins/smatrix/component_modelers/terminal.py: Clean up web method coupling
    • tidy3d/plugins/smatrix/local_run.py: Complete batch_data decoupling

@daquinteroflex daquinteroflex changed the title feat: RF GUI endpoints demo feat: RF GUI<->python client interoperabilty Jul 18, 2025
@daquinteroflex daquinteroflex changed the title feat: RF GUI<->python client interoperabilty feat: RF GUI <-> python client interoperabilty Jul 18, 2025
@daquinteroflex daquinteroflex force-pushed the dario/rf_endpoints branch 2 times, most recently from 0e00e0d to 49f3708 Compare July 18, 2025 11:23
@daquinteroflex daquinteroflex marked this pull request as ready for review July 18, 2025 15:38
Copy link

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewing changes made in this pull request

@dmarek-flex
Copy link
Contributor

dmarek-flex commented Jul 20, 2025

The failing tests is due to the random data, which was previously circumvented using monkeypatch to ignore the check.

To fix the breaking tests, you need to update this monkeypatch part with something that will ignore that check.

@daquinteroflex daquinteroflex marked this pull request as draft July 21, 2025 13:11
@daquinteroflex daquinteroflex marked this pull request as ready for review July 21, 2025 20:10
Copy link

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

7 files reviewed, 3 comments

Edit Code Review Bot Settings | Greptile

@daquinteroflex daquinteroflex marked this pull request as draft July 21, 2025 20:14
@daquinteroflex daquinteroflex marked this pull request as ready for review July 22, 2025 09:00
Copy link

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

8 files reviewed, 2 comments

Edit Code Review Bot Settings | Greptile

@daquinteroflex daquinteroflex marked this pull request as draft July 22, 2025 09:23
@daquinteroflex daquinteroflex marked this pull request as ready for review July 22, 2025 09:23
Copy link

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

18 files reviewed, no comments

Edit Code Review Bot Settings | Greptile

Copy link

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

18 files reviewed, 1 comment

Edit Code Review Bot Settings | Greptile

Comment on lines 43 to 46
Returns
-------
folders : [Folder]
List of folders
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

style: Documentation inconsistent with type hint. Return type shows '[]' but docstring indicates '[Folder]'

Suggested change
Returns
-------
folders : [Folder]
List of folders
def list(cls, projects_endpoint: str = "tidy3d/projects") -> list[Folder]:

Copy link
Collaborator

@yaugenst-flex yaugenst-flex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @daquinteroflex! Left a couple of comments but overall I would say if it works we'll go with it. Depends a bit on how many breaking API changes we can introduce here. In general I would probably try to put many more methods into the data model. In some sense these are pretty similar to the monitor data in components, which also do a lot of domain-specific postprocessing.

Comment on lines 411 to 412
# BREAK this test because it introduces mutability which shouldn't exist
# assert modeler2.batch_cached == modeler2.batch == batch
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But that seems to be a problem with the implementation and not with the test...?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure maybe it's probably the type hint that can fix it when it rehydrates

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renabled and passing

@@ -168,8 +174,10 @@ def to_file(self, fname: str) -> None:
super(AbstractComponentModeler, self).to_file(fname=fname) # noqa: UP008
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The noqa here should go, as should the parameters to super()

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I think a lot of this stuff is from the privous implementation

Comment on lines 262 to 259
@staticmethod
def _set_port_data_array_attributes(data_array: PortDataArray) -> PortDataArray:
"""Helper to set additional metadata for ``PortDataArray``."""
data_array.name = "Z0"
return data_array.assign_attrs(units=OHM, long_name="characteristic impedance")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ooof.. is this really necessary? Why does the array need to be mutated?

Copy link
Contributor

@dmarek-flex dmarek-flex Jul 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is from my original implementation. My reasoning was that there are many types of data that can be represented by the same DataArray types, since they have the same dimensions or coordinates.

Is it better to create a new DataArray type just for impedance values, or change this method to copy and modify the input?

Copy link
Collaborator

@yaugenst-flex yaugenst-flex Jul 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. In principle you're right of course but this is exactly the reason why we have so many different array types. We'll have to rethink that at some point but I think for now it'd be better to just introduce a separate type.

)


def construct_smatrix(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like it should be a method of ComponentModelerData. Actually this is my impression for most of the functions in here.

Copy link
Collaborator Author

@daquinteroflex daquinteroflex Jul 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I was thinking it'd be good if we can separate the "data construction" methods from the data containers and the simualtion definitions. ie. it'd be nice if they're self contained functions that just take in local.run(ComponentModeler) -> ComponentModelerData to avoid the mixing up of stuff in the original implementation. I've moved them to a separate file because this way we can reuse them for backwards and onwards compatibilty without having to recreate a TerminalComponentModelerData class inside TerminalComponentModeler to use this method directly for now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also would have expected most of these methods to be part of ComponentModelerData, since that is how these things are mostly organized for MonitorData. I don't have a strong preference either way. But shouldn't you also move get_antenna_metrics_data and _monitor_data_at_port_amplitude to this file as well then?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have a general guideline on those separations that are (or will be) applied to the entire tidy3d?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will reply privately to both this and this, as related to future plans as far as I understand them

Also would have expected most of these methods to be part of ComponentModelerData, since that is how these things are mostly organized for MonitorData. I don't have a strong preference either way. But shouldn't you also move get_antenna_metrics_data and _monitor_data_at_port_amplitude to this file as well then?

return z_matrix


def check_port_impedance_sign(Z_numpy: np.ndarray):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like it should be a validator somewhere.

Copy link
Collaborator Author

@daquinteroflex daquinteroflex Jul 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, the problem is backwards compatibility on how this method gets used dynamically in construct_smatrix

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When a WavePort is present, the port impedance is calculated in a post-processing step. Does it still make sense to make this a validator?

@@ -37,15 +37,15 @@ class Folder(Tidy3DResource, Queryable, extra=Extra.allow):
)

@classmethod
def list(cls) -> []:
def list(cls, projects_endpoint: str = "tidy3d/projects") -> []:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Return type hint should be list, not []

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the endpoints code itself has not been finished because it depends on whatver implementation we choose to do, so should have clarfied this is still WIP or can remove it for now

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rename modeler_data.py?

Copy link
Contributor

@dmarek-flex dmarek-flex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just made a first pass, and the basic idea looks good to me. My main uncertainty is the the usage of the post-processing functionality present in local_run.py. Ignoring backwards compatibility, can you explain why that would be better than having them as members in the TerminalComponentModelerData?

"Stores the computed S-matrix and reference impedances for the terminal ports"
from tidy3d.plugins.smatrix.local_run import construct_smatrix

terminal_port_data = construct_smatrix(simulation=self.simulation)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think it would be better to directly use self.data now

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But I also see the problem, you would need to add a method for going from dict[TerminalPortType, SimulationData] to dict[str, SimulationData] which would be compatible to how the batch_data is used.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So aiming for somthing lke

sim_data = modeler_data.data[port_in]

and/or

def compose_terminal_modeler_data(
    modeler: TerminalComponentModeler,
    batch_data: BatchData = None,
) -> TerminalComponentModelerData:
    port_to_sim_data_map = {
        port_i: batch_data[modeler.get_task_name(port=port_i)] for port_i in modeler.ports
    }
    port_simulation_data = PortSimulationData(data=port_to_sim_data_map)
    return TerminalComponentModelerData(modeler=modeler, data=port_simulation_data)

return z_matrix


def check_port_impedance_sign(Z_numpy: np.ndarray):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When a WavePort is present, the port impedance is calculated in a post-processing step. Does it still make sense to make this a validator?

@@ -35,6 +35,11 @@
class AbstractComponentModeler(ABC, Tidy3dBaseModel):
"""Tool for modeling devices and computing port parameters."""

name: str = pd.Field(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just needs update for title and description

description="Reference impedance for each port used in the S-parameter calculation. "
"This is optional and may not be present if not specified or computed.",
)
data: TerminalPortDataArray = pd.Field(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rename to s

class MicrowavePortSimulationData(Tidy3dBaseModel):
"""Stores raw simulation data from each microwave port-specific simulation."""

data: dict[TerminalPortType, SimulationData] = pd.Field(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should change to dict[str, SimulationData] for two reasons:

  1. One to make it easier to take the place of batch_data in some of these helper functions for postprocessing. Since I think in every post processing function the batch_data is mainly used as a mapping from a task name to the simulation data.
  2. More importantly, a single Port may be associated with more than one SimulationData, like when we support multimodal WavePort where each mode will correspond with a different simulation.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So how does the current implementation of sound like? Achieves that goal

PortReferenceType = Union[str, TerminalPortType]  # TODO Debate this

class PortSimulationData(Tidy3dBaseModel):  # TODO Debate typing.TypedDict instead, I prefer
    """Stores raw simulation data from each microwave port-specific simulation."""

    data: dict[PortReferenceType, SimulationData] = pd.Field(

)


def construct_smatrix(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also would have expected most of these methods to be part of ComponentModelerData, since that is how these things are mostly organized for MonitorData. I don't have a strong preference either way. But shouldn't you also move get_antenna_metrics_data and _monitor_data_at_port_amplitude to this file as well then?

@daquinteroflex daquinteroflex marked this pull request as draft August 1, 2025 12:46
@flexcompute flexcompute deleted a comment from github-actions bot Aug 1, 2025
@daquinteroflex daquinteroflex changed the base branch from develop to dario/release-2.9.0 August 1, 2025 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants