epic: Index

[farhoud]

A module that takes a graph and index it in a database.

[mehdibalouchi]

@amrhssn had a meeting with Farhoud and Ramin about interfaces. the conclusion was to have the interfaces based on the usage on the Digest and Retrieve side.
Two interfaces, one for Retrieve, one for Digest

• Interface with Digest will be a pub/sub model for producing and consuming nodes of the lattice
  • Traverse starts with a file name and a function name
  • On each step, Digest will produce a Node, an Edge, and a path to the root.
  • Index on the other side, consumes each node, extending necessary indices.
• Interface with Retrieve will be a flat representation of a subset of all nodes
  • On each query, the Index returns a set of nodes with their embeddings.
  • The Retrieve module can request for expansion on nodes. The index module will respond with a set of neighbors and edges

feel free to make fun of it

[amrhssn]

@mehdibalouchi
Great stuff! Thanks for the update.
We should have a meeting about enrichment, and also about node and edge attributes.

I did some initial experiments, and as expected, the naive way of returning the top similarity score between the query and data embeddings doesn't work well.
We should enrich the data in a smart way in the Index module. Also, we should do some post-processing and re-ranking after computing the top-k similar nodes/edges.

The first interface for the Retrieve module is good but I also need all other attributes for pre/post-processing the initial results.

About the second interface, I'd say we wait and put off the extra engineering after we're happy with one end-to-end cycle of the app. We need to spend time on the enrichment and the indexing process itself.

Let me know when you're free to talk 🙌🏻