Morphology model #282

dkeller9 · 2025-07-14T05:49:27Z

This branch includes a minimal set of commits to illustrate the data model, and is branched off of a more recent main. The primary files affected are: model.py, types.py, and morphology.py (although this is intended just to give context to the data model). "ReconstructionCellMorphology" is changed to "CellMorphology" because now the reconstruction morphologies are a subtype of cell morphology. Because the different sub-clases of morphology were not made programmatic subclasses of cell morphology for the sake of maintaining compatibility with how this structure is processed, the extra data needed for the subclasses is in the "extended_data" field. A text description of all the different morphology components is in: https://openbraininstitute.sharepoint.com/:w:/r/sites/OBI-Scientificstaff/_layouts/15/Doc.aspx?sourcedoc=%7BEEE86C08-A09D-469E-9849-CD85599E4026%7D&file=General_schema.docx&action=default&mobileredirect=true&DefaultItemOpen=1

app/db/model.py

eleftherioszisis · 2025-07-14T06:44:54Z

app/db/model.py

+    pipeline_state: Mapped[PipelineType | None] = mapped_column(
+        Enum(PipelineType, name="pipeline_type"), nullable=True
+    )
+    is_related_to: Mapped[list[uuid.UUID] | None] = mapped_column(


This should be a relationship, not an array of ids because there will be no referential integrity if these ids are deleted.

Yes, perhaps this could be handled through the is_derived_from source and target table we discussed previously, in which case the is_related_field could be removed. But even if that is done, deletion of an id means that it would still need to be removed from the relationship table.

I am not sure I understand. I read the document but I don't know what the relationship is . Is it a derivation ?

@jdcourcol this was preserved from the current implementation (we discussed in one of the first meetings). In my understanding it is the relationship of raw morphology - linked to a repaired - linked to a clone etc.

eleftherioszisis · 2025-07-14T06:45:48Z

app/db/model.py

+        ARRAY(UUID(as_uuid=True)), nullable=True
+    )
+    score_dict: Mapped[dict | None] = mapped_column(JSONB, nullable=True)
+    provenance: Mapped[dict | None] = mapped_column(JSONB, nullable=True)


provenance should also be a relationship. We shouldn't be able to add aribtrary stuff in here. How is it planned to be used?

I think this might actually be difficult to structure as a formal relationship but perhaps @lidakanari has an opinion?

Is that the configuration how the morphology has been reconstructed ? Usually we use the regular provenance pattern.
For instance, you have an entity for the "synthesis" configuration with an asset w/ all the parameters, and a derivation relationship between the 2.

Is this only for synthesized morphologies? This seems to be general here. Perhaps it is the provenance, such as published in another dataset, DOI, or paper? @dkeller9 could you please clarify here?

For synthesized I agree with @jdcourcol but here it seems general and I see another one for synthesized?

From the general schema document: "[required] provenance (dict): synthesis inputs e.g., parameters, distributions, amorphology etc. This is from Eleftherios and required to reproduce. We should ask him what kind of object it is." @eleftherioszisis

For example, in circuits, provenance is papers and related artifacts (see below):

I had the impression that we want to have something similar for morphologies. For experimental morphologies that would be related to papers, or raw data if the cell has been produced by another neuron morphology. For synthesized morphologies, that would be papers, inputs etc. Is this what we need for provenance?

These aspects of provenance are already handled by the publication linkage table and the activity table, in the case of derived morphologies. I think what was originally meant by "provenance" as noted in the General Schema was the synthesis parameters. But apparently these are also handled in a separate table.

app/db/types.py

app/schemas/morphology.py

eleftherioszisis · 2025-07-14T07:05:47Z

app/schemas/morphology.py

+    """
+
+    protocol_design: str
+    staining_method: str


This should be constrained with a StrEnum

@lidakanari what staining methods should we give as options?

Yes, this should be Enum. I will compile a list of options. I am afraid it may be restricted to the staining I am aware of, so we should have a strategy to expand this in the future @eleftherioszisis

eleftherioszisis · 2025-07-14T07:06:13Z

app/schemas/morphology.py

+        Whether data has been corrected for shrinkage
+    """
+
+    protocol_design: str


This should be constrained with a StrEnum

@lidakanari what items should be in the protocol_design StrEnum?

I think the protocol design is added when we have a respective document for this. We only have a full protocol document for LNMC, and the rest are from papers. There was a list shared with @jdcourcol last year, but I'm not sure what is best to do here, should we register protocols, link to existing papers?

app/db/model.py

jdcourcol · 2025-07-22T07:48:11Z

app/db/model.py

+
+    # Attributes from ComputationallySynthesized
+
+    provenance: Mapped[str]


Coming back to this, I don't understand what is that string.

These are the parameters that go into the synthesis algorithm from @eleftherioszisis

I don't get how one string can be a list of parameter values.

We need feedback from @eleftherioszisis on what this should be. Probably a JSON to represent a dictionary.

then I don't understand why this is not the input of an activity

If indeed it is tracked in activity, then we can remove it. Can you confirm @eleftherioszisis ?

app/db/model.py

jdcourcol · 2025-07-23T06:25:33Z

app/db/model.py

+class ComputationallySynthesizedMorphologyMethod(MorphologyProtocol):
+    __tablename__ = "computationally_synthesized_morphology_protocol"
+    id: Mapped[uuid.UUID] = mapped_column(ForeignKey("morphology_protocol.id"), primary_key=True)
+    method: Mapped[str] # This 'method' might need further typing based on your vocabulary


why not "name" ? and why not a property on the base "MorphologyProtocol" class ?

My understanding is that this is particular to the SynthesizedMorphology class. But perhaps it could be changed to name, if @lidakanari thinks it is okay.

app/db/model.py

jdcourcol · 2025-07-23T06:28:05Z

app/db/model.py

+class ModifiedMorphologyMethod(MorphologyProtocol):
+    __tablename__ = "modified_morphology_protocol"
+    id: Mapped[uuid.UUID] = mapped_column(ForeignKey("morphology_protocol.id"), primary_key=True)
+    method: Mapped[MethodsType] # Assuming MethodsType is an Enum or similar


above it is a string. Here is it a enum ?

Yes, it is polymorphic because the methods can vary considerably.

jdcourcol · 2025-07-23T06:28:34Z

app/db/model.py

+    has_been_corrected_for_shrinkage: Mapped[bool | None]
+
+    __mapper_args__ = {
+        "polymorphic_identity": "experimental", # A string identifier for this type


"polymorphic_identity": "experimental" -> "polymorphic_identity": tablename

Can't this identifier be user-defined as has been done here?

You need to add "polymorphic_on": "type" which is the intended column to be used here for sqlalchemy to decide which hierarchy of tables to update when an operation is made.

In addition, in MorphologyProtocol the type must be constrained by a MorphologyProtocolType StreEnum the values of which should be the children tables, e.g.

class MorphologyProtocolType(StrEnum): experimental_morphology_protocol = auto()

and then in MorphologyProtocol type: Mapped[MorphologyProtocolType], and in the children tables:

class ExperimentalMorphologyProtocol(MorphologyProtocol): __tablename__ = MorphologyProtocolType.experimental_morphology_protocol.value .... __mapper_args__ = { "polymorphic_on": "type" "polymorphic_identity": __tablename__

This is fixed in the latest push.

jdcourcol · 2025-07-23T06:32:11Z

app/db/model.py

+    __tablename__ = "morphology_protocol"
+    protocol_document: Mapped[str | None]
+    protocol_design: Mapped[str]
+    type: Mapped[str]  # Discriminator column for polymorphism


do we need that ? @GianlucaFicarelli ?

jdcourcol · 2025-07-23T06:32:41Z

app/db/model.py

+
+class MorphologyProtocol(Identifiable): # Inherit from Identifiable for primary key and timestamps
+    __tablename__ = "morphology_protocol"
+    protocol_document: Mapped[str | None]


I don't understand what that is ? a URL ? a DOI ?

I think it is the url pointing to the protocol document. But perhaps we should upload the document into the system and have this point to it? What do you think?

jdcourcol · 2025-07-23T06:32:49Z

app/db/model.py

+class MorphologyProtocol(Identifiable): # Inherit from Identifiable for primary key and timestamps
+    __tablename__ = "morphology_protocol"
+    protocol_document: Mapped[str | None]
+    protocol_design: Mapped[str]


same here ?

app/db/model.py

dkeller9 added 2 commits July 12, 2025 15:29

only db and schema

846b608

cleanup

e7a06e4

dkeller9 requested a review from jdcourcol July 14, 2025 05:49

dkeller9 added 2 commits July 14, 2025 08:31

linting

71b41cc

made types auto

c3ed72a

eleftherioszisis reviewed Jul 14, 2025

View reviewed changes

dkeller9 added 6 commits July 14, 2025 10:26

changed how pipelinetypes is referred to

bf32ac6

more linting

e6ab103

fixed import

8092632

fixed MorphologyStructureType

b1a03c7

more linting

1e9ae38

removed redundant imports

2340de3

jdcourcol reviewed Jul 15, 2025

View reviewed changes

app/db/model.py Outdated Show resolved Hide resolved

jdcourcol reviewed Jul 15, 2025

View reviewed changes

app/db/model.py Outdated Show resolved Hide resolved

dkeller9 added 4 commits July 15, 2025 14:53

reorganized types

8ab30b7

removed scoredict and separate metadata table

274ca8e

change morphology method t protocol

76ea9a7

augmented types

78a0f66

jdcourcol reviewed Jul 22, 2025

View reviewed changes

app/db/model.py Outdated Show resolved Hide resolved

jdcourcol reviewed Jul 22, 2025

View reviewed changes

app/db/model.py Outdated Show resolved Hide resolved

jdcourcol reviewed Jul 22, 2025

View reviewed changes

app/db/model.py Outdated Show resolved Hide resolved

small changes in names

25ae533

jdcourcol reviewed Jul 23, 2025

View reviewed changes

app/db/model.py Outdated Show resolved Hide resolved

jdcourcol reviewed Jul 23, 2025

View reviewed changes

app/db/model.py Outdated Show resolved Hide resolved

jdcourcol reviewed Jul 23, 2025

View reviewed changes

app/db/model.py Outdated Show resolved Hide resolved

jdcourcol reviewed Jul 23, 2025

View reviewed changes

app/db/model.py Outdated Show resolved Hide resolved

jdcourcol reviewed Jul 23, 2025

View reviewed changes

app/db/model.py Outdated Show resolved Hide resolved

dkeller9 added 2 commits July 23, 2025 09:41

small changes in field names

1102fb0

fixed table names

835cc3c


		# Attributes from ComputationallySynthesized

		provenance: Mapped[str]

Morphology model #282

Are you sure you want to change the base?

Morphology model #282

Uh oh!

Conversation

dkeller9 commented Jul 14, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eleftherioszisis Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eleftherioszisis Jul 14, 2025 •

edited

Loading