Exposing the rendered neuron population for comparison with the original neuron populations.

[napakalas]

The edges and nodes rendered on the Flatmap may differ in representation from the original data in SCKAN. These differences could arise due to aliasing or missing nodes.

Therefore, a mechanism is needed to expose the rendered connectivity on the Flatmap-server, allowing it to be further utilized by the Flatmap-viewer.

Proposed methods to expose the rendered neuron population:

• Add entries to MBTiles (e.g., rendered_connectivities). Then, implement a method in the Flatmap-server (e.g., rendered_connectivities), allowing access via ../rendered_connectivities.
• Alternatively, include the rendered neuron population as a new entry in pathways, eliminating the need to modify the Flatmap-server.

Format of rendered neuron population information :

[
    {
        "id": "ilxtr:neuron-type-keast-16",
        "connectivity":
        [
            [["UBERON:0000988", []], ["UBERON:0007632", []]],
            [["UBERON:0000988", []], ["UBERON:0001891", []]],
            [["UBERON:0001891", []], ["UBERON:0002625",[]]]
        ]
    },
   ...
]

[dbrnz]

What nodes are you proposing to use in the edge information? Just those that were actually used from SCKAN's connectivity path?

Also note that the actually SCKAN connectivity is already available to the viewer via its queryKnowledge() method, from both the knowledge and the connectivity_nodes tables in the knowledge store.

[dbrnz]

What I have wondered about passing to the viewer is a mapping from feature id to neuron path ids, although this would need to be used with an anatomical hierarchy (e.g. Heart ==> Nodes in Heart (from hierarchy) ==> Path Ids (from new table) which would get all paths involved with the heart. The place to put this (and your proposed table) would be in index.mbtiles -- there's no need for a new table, as it can be stored as another (name, value) pair in the existing metadata table, with value a JSON representation of a dictionary.

[napakalas]

What nodes are you proposing to use in the edge information? Just those that were actually used from SCKAN's connectivity path?

Yes, the edge information displays the nodes used from the SCKAN connectivity and the nodes that have been aliased, ensuring they reflect the nodes in the rendered flatmap.

What I have wondered about passing to the viewer is a mapping from feature id to neuron path ids, although this would need to be used with an anatomical hierarchy (e.g. Heart ==> Nodes in Heart (from hierarchy) ==> Path Ids (from new table) which would get all paths involved with the heart.

For the anatomical hierarchy, can the node structure (e.g., ('UBERON:0002165', ('UBERON:0002080',))) be used to determine the structure, along with the SVGSource, or is there another approach I should consider?

[dbrnz]

Yes, the edge information displays the nodes used from the SCKAN connectivity and the nodes that have been aliased, ensuring they reflect the nodes in the rendered flatmap.

Maybe a simple (made up?) example would be useful, showing the difference between a SCKAN path and a rendered path in terms of connectivity nodes.

For the anatomical hierarchy, can the node structure (e.g., ('UBERON:0002165', ('UBERON:0002080',))) be used to determine the structure, along with the SVGSource, or is there another approach I should consider?

Not sure of what you mean by SVGSource but the anatomical hierarchy is a tree of UBERON/ILX terms.

The node structure though can add bottom layers to the hierarchy tree -- your example implies that ('UBERON:0002165', ('UBERON:0002080',))) is part-of UBERON:0002080 which in turn is part-of higher levels in the tree.

In general, (A, (B, C, ... Z)) implies (A, (B, C, ... Z)) < (B, (C, ... Z)) < (C, (... Z)) < ... < Z and then Z is in the standard hierarchy.

[napakalas]

Maybe a simple (made up?) example would be useful, showing the difference between a SCKAN path and a rendered path in terms of connectivity nodes.

Here is an example for ilxtr:neuron-type-keast-13 in human female. In the source flatmap there are no [UBERON:0001893", []] and ["UBERON:0001894", []] so the edges connected to them are bypassed to other nodes.

{
    "id": "ilxtr:neuron-type-keast-13",
    "sckan_connectivity":
    [
        [["UBERON:0001880", []], ["UBERON:0001893", []]],
        [["UBERON:0001893", []], ["UBERON:0001894", []]],
        [["UBERON:0001894", []], ["UBERON:0001891", []]],
        [["UBERON:0001891", []], ["UBERON:0000988", []]],
        [["UBERON:0000988", []], ["UBERON:0007632", []]]
    ],
    "rendered_connectivity":
    [
        [["UBERON:0001880", []], ["UBERON:0001891", []]],
        [["UBERON:0001891", []], ["UBERON:0000988", []]],
        [["UBERON:0000988", []], ["UBERON:0007632", []]]
    ]
}

In general, (A, (B, C, ... Z)) implies (A, (B, C, ... Z)) < (B, (C, ... Z)) < (C, (... Z)) < ... < Z and then Z is in the standard hierarchy.

Pretty clear with this one, and doable, thanks.

[dbrnz]

What about:

1. The SCKAN node is [A, [B]] and because the map has no A, we go up the term list and find feature B -- should the rendered connectivity node be [A, [B]] or [B, []]?
2. The SCKAN node is [A, []] and the map doesn't have A but has an alias for it called B, which the map has -- should the rendered connectivity node be [A, []] or [B, []]?

[napakalas]

1. The SCKAN node is [A, [B]] and because the map has no A, we go up the term list and find feature B -- should the rendered connectivity node be [A, [B]] or [B, []]?

[B, []]

2. The SCKAN node is [A, []] and the map doesn't have A but has an alias for it called B, which the map has -- should the rendered connectivity node be [A, []] or [B, []]?

[B, []]

[napakalas]

Here is the rendered connectivity and rendered hierarchy format. I also included examples for unbranched-10, keast-2, aacar-4, aacar-10a and femrep/76 in the json file (rendered_connectivity_example.json)

{
    "rendered-connectivities":
    [
        {
            "id": "ilxtr:sparc-nlp/femrep/76",
            "connectivity": 
            [ 
                 [["UBERON:0005303", []], ["UBERON:0016508", []]], 
                 [["ILX:0793832", []], ["UBERON:0012648",[]]],
                  ... 
            ],
            "rendered-nodes":
            [
                {
                    "rendered-node": ["UBERON:0016508", []],
                    "featureIds":[1338]
                },
                {
                    "rendered-node": ["UBERON:0005303", []],
                    "segmentIds":[1467],
                    "nerveIds":[1910]
                },
                {
                    "rendered-node": ["ILX:0793832", []],
                    "featureIds": [1832, 1831]
                },
                ...
            ]
        }
    ],
    "rendered-hierarchy":
    [
        {
            "id": "UBERON:0001258",
            "children": 
             [
                  ["UBERON:0002384", ["UBERON:0001258"]], 
                  ["UBERON:0001135", ["UBERON:0001258"]]
             ],
            "paths": ["ilxtr:neuron-type-keast-2"]
        },
        ...
    ]
    "rendered-nodes":
    {
        "1338":
        {
            "id": "ganglion_4-1",
            "featureId": 1338,
            "path-ids": ["ilxtr:sparc-nlp/femrep/76", "ilxtr:neuron-type-keast-2"],
            "nodes": [["UBERON:0002014", []], ["UBERON:0016508", []]],
            "rendered-node": ["UBERON:0016508", []]
        },
        "1351":
        {
            "id": "ganglion_19-1",
            "featureId": 1351,
            "path-ids": ["ilxtr:sparc-nlp/femrep/76"],
            "nodes": [["ILX:0794732", []]],
            "rendered-node": ["ILX:0793082", []]
        },
        ...
    }
 }

[napakalas]

Rename top-level JSON key from 'rendered-nodes' to 'rendered-features' for better specificity. rendered_connectivity_example.json

{
    "rendered-connectivities":
    [
        {
            "id": "ilxtr:sparc-nlp/femrep/76",
            "connectivity": 
            [ 
                 [["UBERON:0005303", []], ["UBERON:0016508", []]], 
                 [["ILX:0793832", []], ["UBERON:0012648",[]]],
                  ... 
            ],
            "rendered-nodes":
            [
                {
                    "rendered-node": ["UBERON:0016508", []],
                    "featureIds":[1338]
                },
                {
                    "rendered-node": ["UBERON:0005303", []],
                    "segmentIds":[1467],
                    "nerveIds":[1910]
                },
                {
                    "rendered-node": ["ILX:0793832", []],
                    "featureIds": [1832, 1831]
                },
                ...
            ]
        }
    ],
    "rendered-hierarchy":
    [
        {
            "id": "UBERON:0001258",
            "children": 
             [
                  ["UBERON:0002384", ["UBERON:0001258"]], 
                  ["UBERON:0001135", ["UBERON:0001258"]]
             ],
            "paths": ["ilxtr:neuron-type-keast-2"]
        },
        ...
    ]
    "rendered-features":
    {
        "1338":
        {
            "id": "ganglion_4-1",
            "featureId": 1338,
            "path-ids": ["ilxtr:sparc-nlp/femrep/76", "ilxtr:neuron-type-keast-2"],
            "nodes": [["UBERON:0002014", []], ["UBERON:0016508", []]],
            "rendered-node": ["UBERON:0016508", []]
        },
        "1351":
        {
            "id": "ganglion_19-1",
            "featureId": 1351,
            "path-ids": ["ilxtr:sparc-nlp/femrep/76"],
            "nodes": [["ILX:0794732", []]],
            "rendered-node": ["ILX:0793082", []]
        },
        ...
    }
 }

[dbrnz]

@napakalas looking at the above JSON and PR #122's code I think it is unnecessarily complicated. Is it not sufficient to extend the paths records in the pathways metadata entry of a map's index.mbtiles by adding a new connectivity field that represents rendered connectivity nodes, along with each feature that is a connectivity node having another property, say also called connectivity that is a list of tuples in the form [PATH_ID, RENDERED_NODE]?

For instance, the existingilxtr:sparc-nlp/femrep/76 record would have connectivity added as follows:

"ilxtr:sparc-nlp/femrep/76":
{
    "lines": [5856, 5857, 5858, 5859, 5860, 5846, 5847, 5848, 5849, 5850, 5851, 5852, 5853, 5854, 5855],
    "nerves": [1926],
    "nodes": [1356, 1357, 1294, 1293, 1845, 1847, 1369],
    "models": "ilxtr:sparc-nlp/femrep/76",
    "centrelines": ["hypogastric_n"],
    "connectivity": 
    [ 
        [["UBERON:0005303", []], ["UBERON:0016508", []]], 
        [["ILX:0793832", []], ["UBERON:0012648",[]]],
        ... 
    ]
},

and assuming the feature with the GeoJSON id 1356 corresponds to the rendered node ["UBERON:0005303", []], its properties would include:

"connectivity": [["ilxtr:sparc-nlp/femrep/76", ["UBERON:0005303", []]], ...]

I also fail to see why the anatomical hierarchy of a specific map has a dependency on neuron paths -- the existing mechanism, managed by MapTermGraph in the viewer's code may need some extension to automatically include the implied hierarchy of a connectivity node but that should be all.

[dbrnz]

In order to find features from competency query results in the viewer, the relationship between SCKAN connectivity nodes and map features has to be available. This could be done by adding another new field to a feature's properties, another list of tuples in the form [PATH_ID, SCKAN_NODE]. We would end up with the above, but renamed to say rendered, and a new list, called say sckan, that contains the path ids and SCKAN nodes used to locate the feature. (There's no need to include connectivity in the field's name).

[napakalas]

Confirm that:

• connectivity will be added to each path in Pathways.
• rendered and sckan will be added to the appropriate features in Annotations.

Is this correct?

[dbrnz]

connectivity will be added to each path in Pathways.

The place where this should happen is inside pathways.ResolvedPath with as_dict including a connectivity entry.

rendered and sckan will be added to the appropriate features in Annotations

No. The place to do this is when you've found features for a node using features_for_anatomical_node(). I have in fact just extended the PropertyMixin to add append_property(), so pull/merge main and you can then feature.append_property('rendered', (path_id, node)) (ditto for sckan). It will create a list valued property (if it doesn't exist) and append values to it. rendered and scan also need adding to the EXPORTED_FEATURE_PROPERTIES list.

[napakalas]

Some commits have been pushed:

• The sckan and rendered attributes are now added to features when calling __route_graph_from_connectivity(), and they appear in each feature under /annotations.
• The connectivity attribute has been added to ResolvedPath and now appears in /pathways for each path.
• The hierarchy node for rendered nodes is now exposed in /node-hierarchy. The format follows the MapTermGraph mechanism but includes only the nodes and links attributes. Here's an example:

{
    "nodes":
    [
        {
            "id": ["LX:0791153", []]
        },
        {
            "id": ["ILX:0784721", []]
        },
        ...
    ]
    "links":
    [
        {
            "source": ["UBERON:0000483", ["UBERON:0001263"]],
            "target": ["UBERON:0001263", []]
        },
        {
            "source":["UBERON:0004677", ["UBERON:0006450"]],
            "target":["UBERON:0006450", []]
        },
        ...
    ]
}