examples/getting_started/custom_retriever_optimizer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Redis](https://redis.io/wp-content/uploads/2024/04/Logotype.svg?auto=webp&quality=85,75&width=120)\n",
    "\n",
    "# Custom Retrievers\n",
    "\n",
    "This notebook will walk through how to get going optimizing your vector retrieval with a custom retriever. This will allow you to test how different queries can be used to get a better overall results.\n",
    "\n",
    "<a href=\"https://colab.research.google.com/github/redis-applied-ai/retrieval-optimizer/blob/main/examples/getting_started/custom_retriever_optimizer.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
    "\n",
    "## Grab data (if colab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!git clone https://github.com/redis-applied-ai/retrieval-optimizer.git temp_repo\n",
    "!mv temp_repo/examples/getting_started/data .\n",
    "!mv temp_repo/examples/getting_started/schema .\n",
    "!mv temp_repo/examples/getting_started/study_config.yaml .\n",
    "!mv temp_repo/optimize .\n",
    "!mv temp_repo/images/\n",
    "!rm -rf temp_repo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1 - Load dataset\n",
    "\n",
    "For the retrieval optimization framework to run it requires a set of raw chunks and a set of queries with labeled chunks. The labeling tool can be used to generate these artifacts from an existing index or they can be created manually. See [README.md](../../README.md) for more information or checkout [populate_index.ipynb](./populate_index.ipynb).\n",
    "\n",
    "In the previous example, we loaded one set of chunks that only included data from one pdf document - a mazda 3 owner's manual. For this example we will load an already generated set of chunks for 2 different documents - mazda 3 and volvo c30."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'text': \"Mazda3_8Y64-EA-08A_Edition1 Page1 Tuesday, November 27 2007 9:0 AM\\n\\nForm No.8Y64-EA-08A\\n\\nBlack plate (1,1)\\n\\nMazda3_8Y64-EA-08A_Edition1 Page2 Tuesday, November 27 2007 9:0 AM\\n\\nForm No.8Y64-EA-08A\\n\\nBlack plate (2,1)\\n\\nMazda3_8Y64-EA-08A_Edition1 Page3 Tuesday, November 27 2007 9:0 AM\\n\\nBlack plate (3,1)\\n\\nA Word to Mazda Owners\\n\\nThank you for choosing a Mazda. We at Mazda design and build vehicles with complete customer satisfaction in mind.\\n\\nTo help ensure enjoyable and trouble-free operation of your Mazda, read this manual carefully and follow its recommendations.\\n\\nAn Authorized Mazda Dealer knows your vehicle best. So when maintenance or service is necessary, that's the place to go.\\n\\nOur nationwide network of Mazda professionals is dedicated to providing you with the best possible service.\\n\\nWe assure you that all of us at Mazda have an ongoing interest in your motoring pleasure and in your full satisfaction with your Mazda product.\\n\\nMazda Motor Corporation HIROSHIMA, JAPAN\\n\\nImportant Notes About This Manual Keep this manual in the glove box as a handy reference for the safe and enjoyable use of your Mazda. Should you resell the vehicle, leave this manual with it for the next owner.\\n\\nAll specifications and descriptions are accurate at the time of printing. Because improvement is a constant goal at Mazda, we reserve the right to make changes in specifications at any time without notice and without obligation.\\n\\nEvent Data Recorder This vehicle is equipped with an event data recorder. In the event of a crash, this device records data related to vehicle dynamics and safety systems for a short period of time. These data can help provide a better understanding of the circumstances in which crashes and injuries occur and lead to the designing of safer vehicles.\\n\\nAir Conditioning and the Environment Your Mazda's genuine air conditioner is filled with HFC134a (R134a), a refrigerant that has been found not to damage the earth's ozone layer. If the air conditioner does not operate properly, consult an Authorized Mazda Dealer.\\n\\nPerchlorate Certain components of this vehicle such as [air bag modules, seat belt pretensioners, lithium batteries, ...] may contain Perchlorate Material– Special handling may apply for service or vehicle end of life disposal. See www.dtsc.ca.gov/hazardouswaste/perchlorate.\\n\\nPlease be aware that this manual applies to all models, equipment and options. As a result, you may find some explanations for equipment not installed on your vehicle.\",\n",
       " 'query_metadata': {'make': 'mazda', 'model': '3'},\n",
       " 'item_id': 'mazda_3:0'}"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "with open(\"data/combined_chunks.json\", \"r\") as f:\n",
    "    chunks = json.load(f)\n",
    "\n",
    "chunks[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'query': 'At what speed should I shift from 2 to 3 with a manual transmission?',\n",
       " 'query_metadata': {'make': 'mazda', 'model': '3'},\n",
       " 'relevant_item_ids': ['mazda_3:86']}"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open(\"data/combined_labeled.json\", \"r\") as f:\n",
    "    labeled = json.load(f)\n",
    "\n",
    "labeled[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Notes on the data\n",
    "\n",
    "Unlike the basic example we are now loading data from multiple sources and there is an additional attribute called `query_metadata`. This is a named dict within the retrieval optimizer which can be used to load additional fields into the index and/or create more meaningful queries that combine vector search with additional filters etc. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 - python dependencies & env setup\n",
    "\n",
    "This project utilizes poetry for dependency management. If you like, in the project root you can run `poetry install` then set the kernel version of this notebook to point to the path of the python interpreter poetry is referencing. This can be accomplish by running `poetry env info --path` after installing and setting as your kernel for the notebook.\n",
    "\n",
    "Alternatively you can run the following line to import in a notebook native way for convenience."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install numpy redisvl sentence-transformers redis optuna pandas nest-asyncio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Important!\n",
    "\n",
    "The following code allows for us to run async functions in jupyter and adds the optimize module to path for easy import."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import os\n",
    "\n",
    "# Add the parent directory to the Python path to get optimize module\n",
    "sys.path.append(os.path.abspath(os.path.join(os.getcwd(), '../..')))\n",
    "\n",
    "import nest_asyncio\n",
    "\n",
    "# Apply the nest_asyncio patch: let's us run async code in Jupyter\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 - Define the study config\n",
    "\n",
    "Under the hood, this framework is performing bayesian optimization utilizing the optuna library. This process determines the set of decision variables that best optimize the cost function. It accomplishes this by trying different combinations of the decision variables and intelligently traversing the solution space in the direction of best results.\n",
    "\n",
    "At current the available decision variables are:\n",
    "\n",
    "- algorithm (\"hnsw\" or \"flat\")\n",
    "  - if hnsw secondary hyper-parameters of ef_runtime, ef_construction, and m can also be optimized.\n",
    "- vector data type\n",
    "- ret k\n",
    "- embedding model\n",
    "\n",
    "Different configurations of these variables will be attempted in order to **maximize** the following equation: `obj = w0 * f1 + w1 * norm_embedding_latency + w2 * norm_indexing_time` expressed in vector notation this is the dot product of `weights * metrics`. If there are additional metrics you would like to optimize over feel free to open an issue and we can look into adding it!\n",
    "\n",
    "#### study_config.yaml\n",
    "```yaml\n",
    "input_data_type: \"json\" # currently only json but considering alternatives\n",
    "raw_data_path: \"data/mazda_chunks.json\" # path to data files for easy read\n",
    "labeled_data_path: \"data/mazda_labeled.json\" # labeled data\n",
    "\n",
    "# optimization config\n",
    "n_trials: 20\n",
    "n_jobs: 1\n",
    "\n",
    "# Metric weights to be used in objective function\n",
    "metric_weights:\n",
    "  f1_at_k: 1\n",
    "  embedding_latency: 1\n",
    "  total_indexing_time: 1\n",
    "\n",
    "# optimization decision variables\n",
    "algorithms: [\"flat\", \"hnsw\"] # indexing algorithms variables\n",
    "vector_data_types: [\"float32\", \"float16\"] # vector type of indexed vectors\n",
    "ret_k: [1, 10] # potential range of value to be sampled during study\n",
    "\n",
    "# ef_runtime, ef_construction, and m only relevant if HNSW indexing\n",
    "ef_runtime: [10, 20, 30, 50]\n",
    "ef_construction: [100, 150, 200, 250, 300]\n",
    "m: [8, 16, 64]\n",
    "\n",
    "# embedding models to be used\n",
    "embedding_models:\n",
    "  - provider: \"hf\"\n",
    "    model: \"sentence-transformers/all-MiniLM-L6-v2\"\n",
    "    dim: 384\n",
    "  - provider: \"hf\"\n",
    "    model: \"intfloat/e5-large-v2\"\n",
    "    dim: 1024\n",
    "```\n",
    "\n",
    "#### Load the study config\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "StudyConfig(study_id='358c4a71-e799-440c-9432-22515a0d4cfd', redis_url='redis://localhost:6379/0', algorithms=['flat', 'hnsw'], vector_data_types=['float32', 'float16'], raw_data_path='data/combined_chunks.json', input_data_type='json', labeled_data_path='data/combined_labeled.json', embedding_models=[EmbeddingModel(provider='hf', model='sentence-transformers/all-MiniLM-L6-v2', dim=384), EmbeddingModel(provider='hf', model='intfloat/e5-large-v2', dim=1024)], metrics=['f1_at_k', 'embedding_latency', 'total_indexing_time'], weights=[1.0, 1.0, 1.0], n_trials=20, n_jobs=1, ret_k=(1, 10), ef_runtime=[10, 20, 30, 50], ef_construction=[100, 150, 200, 250, 300], m=[8, 16, 64])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from optimize.study import load_config\n",
    "\n",
    "study_config = load_config(\"combined_study_config.yaml\") # Loads yaml into pydantic model\n",
    "\n",
    "study_config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4 - running a redis instance\n",
    "\n",
    "For the following code to work we need to have a running instance of redis. \n",
    "\n",
    "## Running notebook local use docker:\n",
    "\n",
    "`docker run -d --name redis -p 6379:6379 -p 8001:8001 redis/redis-stack:latest`\n",
    "\n",
    "## Running in colab use the following shell command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sh\n",
    "curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\n",
    "echo \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\n",
    "sudo apt-get update  > /dev/null 2>&1\n",
    "sudo apt-get install redis-stack-server  > /dev/null 2>&1\n",
    "redis-stack-server --daemonize yes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5 - Define custom retriever class\n",
    "\n",
    "Since we now have 2 distinct document sets that we are fetching from `mazda` and `volvo` respectfully a default knn query is not going to be as effective. This is where custom retrievers come in. In the following code we extend the `DefaultQueryRetriever` and override the `query_fn` method to utilize the `query_metadata` we have added to our data which will let us augment our vector search on `make` and `model`.\n",
    "\n",
    "**Note:** for any custom query that can be run against the Redis instance via `index.query()` all that is required is to extend the `DefaultQueryRetriever`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from optimize.retrievers import DefaultQueryRetriever\n",
    "from optimize.models import LabeledItem\n",
    "from redisvl.query import VectorQuery\n",
    "from redisvl.utils.vectorize import BaseVectorizer\n",
    "from redisvl.query.filter import Tag\n",
    "\n",
    "class MakeModelQueryRetriever(DefaultQueryRetriever):\n",
    "    def query_fn(\n",
    "        self, emb_model: BaseVectorizer, labeled_item: LabeledItem, dtype: str, k: int\n",
    "    ) -> VectorQuery:\n",
    "        make = labeled_item.query_metadata[\"make\"]\n",
    "        model = labeled_item.query_metadata[\"model\"]\n",
    "\n",
    "        make_filter = Tag(\"make\") == make\n",
    "        model_filter = Tag(\"model\") == model\n",
    "        make_model_filter = make_filter & model_filter\n",
    "\n",
    "        vec_query = VectorQuery(\n",
    "            vector=emb_model.embed(labeled_item.query, as_buffer=True, dtype=dtype),\n",
    "            vector_field_name=\"vector\",\n",
    "            return_score=True,\n",
    "            return_fields=[\"item_id\"],\n",
    "            num_results=k,\n",
    "            filter_expression=make_model_filter,\n",
    "        )\n",
    "\n",
    "        print(f\"{vec_query.query_string()=}\")\n",
    "        return vec_query\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Important must also specify the additional fields to be added to the schema\n",
    "\n",
    "The `Eval` class dynamically creates a default index however since we have fields beyond just the basic `text` and `item_id` we need to provide a list of fields to index that matches the fields provided in the `query_metadata`. These fields should be provided in the format:\n",
    "\n",
    "```json\n",
    "[\n",
    "    {\n",
    "        \"name\": \"field_name\",\n",
    "        \"type\": \"field_type\"\n",
    "    }\n",
    "]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "additional_fields = [{\"name\": \"make\", \"type\": \"tag\"}, {\"name\": \"model\", \"type\": \"tag\"}]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have defined the custom query retriever all we have to do is pass it to the `Eval` class along with our additional fields:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:07:40,735 - INFO - Load pretrained SentenceTransformer: intfloat/e5-large-v2\n",
      "2024-11-25 11:07:43,975 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1cdb350bcd6e4a529744f3382cb5d72d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "49fd9bc79bc244e892c1da76370d129e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:08:12,680 - INFO - Indexing data...\n",
      "2024-11-25 11:08:13,018 - INFO - Data indexed. self.total_indexing_time=145.51400756835938s\n"
     ]
    }
   ],
   "source": [
    "from optimize.eval import Eval\n",
    "\n",
    "e = Eval(\n",
    "    model_provider=\"hf\",\n",
    "    model_str=\"intfloat/e5-large-v2\",\n",
    "    embedding_dim=1024,\n",
    "    raw_data_path=\"data/combined_chunks.json\",\n",
    "    labeled_data_path=\"data/combined_labeled.json\",\n",
    "    input_data_type=study_config.input_data_type,\n",
    "    vector_data_type=study_config.vector_data_types[0],\n",
    "    algorithm=study_config.algorithms[0],\n",
    "    ret_k=4,\n",
    "    find_threshold=False,\n",
    "    retriever=MakeModelQueryRetriever, # pass in custom retriever\n",
    "    additional_schema_fields=additional_fields,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Now let's calculate metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:08:19,662 - INFO - Load pretrained SentenceTransformer: intfloat/e5-large-v2\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:08:21,256 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "65e68b6498054857ac73d930ea257da2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "490713b1831c4ca4b66ef13a2a86f455",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2aada8cd5ff04747b26439ac2c7b316e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e834c22b09b3467ca629bda52ed31a05",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4920f11b6e6f407da9194d93cbf9161f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "35574fdca4024f56829e364f5db98b4b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fae25134af80471a91098e978808f8e8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9f5ce39b28324ba6a717455bf80a6085",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fb573dbb763f4f9e92bc007a62561c77",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ff582bfa57ba41c093ec837f9f75c666",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8d90c65128114d44902a4a492f9a8692",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "33b0e0a3350642319ab5cd962c2b46b4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e4c562624f94457789ce2ba66131f070",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "75f8de5a09284b9eb501d7d4c76439d2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9dd70927b80142c1ba0edae8c85654cb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0f2723dcfaff4a1e8db8a1b9f410468b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:08:22,674 - INFO - Calculating retrieval metrics for test_id: 4e8a5e9c-56ec-443b-832c-921804812def\n",
      "2024-11-25 11:08:22,682 - INFO - Overall f1 at 4 for retrieval: 0.5095238095238095\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    }
   ],
   "source": [
    "e.calc_metrics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Note the overall retrieval \n",
    "\n",
    "Now let's see what would have happened had we done this with the default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 10:07:26,971 - INFO - Load pretrained SentenceTransformer: intfloat/e5-large-v2\n",
      "2024-11-25 10:07:28,079 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0c8bc8d16d0948d4bead1304b117c371",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b8b409d1333e44b3af49fcfff7e789a5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 10:07:56,036 - INFO - Indexing data...\n",
      "2024-11-25 10:07:56,427 - INFO - Data indexed. self.total_indexing_time=211.94900512695312s\n",
      "2024-11-25 10:07:56,439 - INFO - Load pretrained SentenceTransformer: intfloat/e5-large-v2\n",
      "2024-11-25 10:07:59,033 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ac40561ba1bd42f190c035ba729cf179",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d33f6eddd2d04ff283f2567133ba2f87",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d74f7d2551da4074993f412ff3166c5c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a8bc1f8bac8f45e2b561c45d9d210236",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bbd748adbd16412298a6555305729bf1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9d614eb0bce74513a609e37c360ee916",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "76f4e94db06d4ad4bf38160495505648",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "65a9b062db9f4ec4b3baae0dbff2f20b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "df589c3c15a64ac2a145d9e77f478a52",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "95dd5ac30a7d48c3a9ba5e367fd2f39b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "80a01b6658d243f58b2dc1a35d18e0a7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "489b524b588f4a70b375ffaa1c082e1d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dc93698aa18f44d2b4f2149726ef84f5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "844b0f0bff9c4691b43356df85d4fe8b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8a25131b162c42c198dd22a93f567c32",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "92ccc18a6f11408bb6b3547f5ea0cc44",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 10:08:00,830 - INFO - Calculating retrieval metrics for test_id: 4a2cb0b3-cbea-4c31-849c-d11a27308fa0\n",
      "2024-11-25 10:08:00,837 - INFO - Overall f1 at 4 for retrieval: 0.3273544973544974\n"
     ]
    }
   ],
   "source": [
    "e = Eval(\n",
    "    model_provider=\"hf\",\n",
    "    model_str=\"intfloat/e5-large-v2\",\n",
    "    embedding_dim=1024,\n",
    "    raw_data_path=study_config.raw_data_path,\n",
    "    labeled_data_path=study_config.labeled_data_path,\n",
    "    input_data_type=study_config.input_data_type,\n",
    "    vector_data_type=study_config.vector_data_types[0],\n",
    "    algorithm=study_config.algorithms[0],\n",
    "    ret_k=4,\n",
    "    find_threshold=False,\n",
    ")\n",
    "e.calc_metrics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Notice the reduction in f1 score\n",
    "\n",
    "Not surprisingly adding the metadata improves are search because it narrows the vector similarity query to the right set of data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6 - Run an optimization\n",
    "\n",
    "Now let's imagine a scenario where you would like to test a few retrieval strategies as part of the optimization. You can do this by passing a dictionary of the `Retrieval` classes you would like to evaluate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2024-11-25 11:48:09,471] A new study created in memory with name: test\n",
      "2024-11-25 11:48:09,472 - INFO - Running for Retriever: DefaultQueryRetriever with additional_schema_fields=None\n",
      "2024-11-25 11:48:09,492 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      " Running for: \n",
      " model_str: sentence-transformers/all-MiniLM-L6-v2 \n",
      " ef_runtime: 20 \n",
      " ef_construction: 300 \n",
      " m: 64 \n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:48:10,214 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9d74ae855c504d7d81f964891907fc0a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0ed6dc722c6145ebb8f7ae3596503b33",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:48:11,549 - INFO - Indexing data...\n",
      "2024-11-25 11:48:12,338 - INFO - Data indexed. self.total_indexing_time=315.2080078125s\n",
      "2024-11-25 11:48:12,351 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n",
      "2024-11-25 11:48:12,953 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "82df677c68d64b2fb86382a903b73148",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "159fff4bf97641328193888deaca4780",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "756d8d2096b041fea826c45517c05730",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7947f3793bdb4f82bb8d1a0cfedfbb9d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b7e6422f363849cfa39f65cd475840d7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "87b08ce9ea824579ba1ca2f9fcaf4152",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0bd275adeb8d4d99b8e19103b2a36808",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2b59cef4bd5144e69d8bc566cd9f0806",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ea90ce2718c84e4d863037ff1d85bf71",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "478a990356c54489bbb0828391cd59c6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ce015fc4b29e4d39a0c5464a33e08b6a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "98068062dedf4894a2825bfe5fe232c5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5eb966af83304df88480bf0d3e130096",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fb69607a547d45438c4d9b4355c8d685",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "de107daf6890400fafab6bffd92c3d3d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "996a750c71074170b6b4e49a8d451c3b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:48:13,466 - INFO - Calculating retrieval metrics for test_id: d480aac4-36d7-4eba-8c06-9e81026f6fcf\n",
      "2024-11-25 11:48:13,470 - INFO - Overall f1 at 3 for retrieval: 0.47624338624338625\n",
      "[I 2024-11-25 11:48:13,471] Trial 0 finished with value: 0.9274864922181295 and parameters: {'model_info': {'provider': 'hf', 'model': 'sentence-transformers/all-MiniLM-L6-v2', 'dim': 384}, 'retriever': 'default', 'algorithm': 'hnsw', 'var_dtype': 'float32', 'ret_k': 3, 'ef_runtime': 20, 'ef_construction': 300, 'm': 64}. Best is trial 0 with value: 0.9274864922181295.\n",
      "2024-11-25 11:48:13,471 - INFO - Running for Retriever: MakeModelQueryRetriever with additional_schema_fields=[{'name': 'make', 'type': 'tag'}, {'name': 'model', 'type': 'tag'}]\n",
      "2024-11-25 11:48:13,480 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      " Running for: \n",
      " model_str: sentence-transformers/all-MiniLM-L6-v2 \n",
      " algorithm: flat\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:48:14,058 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "684202fad4b241daa370849b8a1ca51c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7a9e07c488b84ad9b67118281ac98efc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:48:15,317 - INFO - Indexing data...\n",
      "2024-11-25 11:48:16,346 - INFO - Data indexed. self.total_indexing_time=223.36700439453125s\n",
      "2024-11-25 11:48:16,355 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n",
      "2024-11-25 11:48:16,956 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "32e44d518ece41b4baece38cd64f5203",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8badb0a28d6541409caa58f16920c86f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "44e241b58db64d328290fc1d8e42a764",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "531ce1bd479d4d6792de8964941b008e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "317b9d9679f3459391fa1aef80984b41",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a0edbcf71fce4c2bada6fec9d7f25f80",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cc54a2401edf40d1ae28ef4d86312f99",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d058a0c8817442e1988610e21cc4eea0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4c440eed52054dcdae5b293c0a626fb0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b6d02a62016f4843b5a8a2bd643634fb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7b7714582d614178b75d4863ec71e3c6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4d0ffb33568343ec91436824a314286b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "01706f4e91474a63924a3c0e1857ab7c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2c029693f4cf4b7a89bdaa46a2f73d8b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6f395beec66e409eaac7d60f595ee13e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8142ec5769134d168407276c38924696",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:48:17,469 - INFO - Calculating retrieval metrics for test_id: 99bc05d9-16db-4658-a278-39ab31ea5eea\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:48:17,474 - INFO - Overall f1 at 2 for retrieval: 0.6558730158730159\n",
      "[I 2024-11-25 11:48:17,475] Trial 1 finished with value: 1.111346093640145 and parameters: {'model_info': {'provider': 'hf', 'model': 'sentence-transformers/all-MiniLM-L6-v2', 'dim': 384}, 'retriever': 'make_model', 'algorithm': 'flat', 'var_dtype': 'float32', 'ret_k': 2}. Best is trial 1 with value: 1.111346093640145.\n",
      "2024-11-25 11:48:17,475 - INFO - Running for Retriever: DefaultQueryRetriever with additional_schema_fields=None\n",
      "2024-11-25 11:48:17,482 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      " Running for: \n",
      " model_str: sentence-transformers/all-MiniLM-L6-v2 \n",
      " algorithm: flat\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:48:18,079 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "101ef8fee731412fa1ed0f727afa9f3a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6f715fa3c363442ca69ddbaa4bbdb4d6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:48:19,339 - INFO - Indexing data...\n",
      "2024-11-25 11:48:20,209 - INFO - Data indexed. self.total_indexing_time=156.33200073242188s\n",
      "2024-11-25 11:48:20,218 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n",
      "2024-11-25 11:48:20,909 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "91d27863eab8495aa80a0571160f6771",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8e2118c361db4589944658440c85c523",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4bf844073f2e410b8923acea6a2c7b2d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "afd2e1e3b5ec48b9acd9c0c6b1c5f93b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "88d88436b3aa4e279f691c4eaa54ab65",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a7b7c51d37ff4cc098149592c2dafb0a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3ff5bd7fde874d4fa304cfb303b0fcc4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c1928fc2016f433abf8de1d678b77604",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "695bfc10472346d1aba8b72714ac66c0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d7e1f5e7ed164b319ea5d7a204174f97",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "327fceaa959b4717a956168f0a3b4b45",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0ea2a087059141c080157ffd5fa3faea",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0f51b8e7a8ae40a0bfce0f6531b131a2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "aa1fe764bb4343149b1b8fde1c127a12",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a863f39906084799a355e5a3febb9bda",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f53b99aca0b44ecfa8c337d40cfd9781",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:48:21,435 - INFO - Calculating retrieval metrics for test_id: cc6f4e2c-ba73-4716-a968-4152a0242229\n",
      "2024-11-25 11:48:21,440 - INFO - Overall f1 at 2 for retrieval: 0.4212698412698413\n",
      "[I 2024-11-25 11:48:21,441] Trial 2 finished with value: 0.8785967260953116 and parameters: {'model_info': {'provider': 'hf', 'model': 'sentence-transformers/all-MiniLM-L6-v2', 'dim': 384}, 'retriever': 'default', 'algorithm': 'flat', 'var_dtype': 'float16', 'ret_k': 2}. Best is trial 1 with value: 1.111346093640145.\n",
      "2024-11-25 11:48:21,442 - INFO - Running for Retriever: DefaultQueryRetriever with additional_schema_fields=None\n",
      "2024-11-25 11:48:21,449 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      " Running for: \n",
      " model_str: sentence-transformers/all-MiniLM-L6-v2 \n",
      " algorithm: flat\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:48:22,160 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bfc84d2fa251402faa02e932b53b6073",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4fea2d3eddfc4514b42d350b67be7da6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:48:23,423 - INFO - Indexing data...\n",
      "2024-11-25 11:48:24,436 - INFO - Data indexed. self.total_indexing_time=156.87100219726562s\n",
      "2024-11-25 11:48:24,449 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n",
      "2024-11-25 11:48:25,077 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "afda338c752d4315b733068777aec292",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b9ee850544fe484e944189aced17e1b6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2e078dbe57e64f3492343b01d6a35020",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "13b57385a88f41c9bf2b014ed234c030",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "978a3b03de744d758350c1e72db85478",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d3212c2be61941729a65f2ac82f5f18d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c1bf0939117542bbb17c6a7ac9d96392",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7d1525754157481cb98e8d64f98c72dd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "664c88cf489d45aab6acbc3161c71079",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b07a1f2022054c16b46c464e5f8fd619",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5e61cc0067cf4854b971955a9889e56f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a470c174bb3d412b8b1c31d62ad74b06",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "16371a26b49b4af8b8e7b8dee2d026d0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "20bafdb845244067a1740a16dc528014",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "569a4bdf9e9949bd86ce28b8ee3ceaa7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d0cc8c5c07c74295845bb229327df88e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:48:25,664 - INFO - Calculating retrieval metrics for test_id: 7a0b31ea-3904-4a48-9c4a-5dbca88c9679\n",
      "2024-11-25 11:48:25,668 - INFO - Overall f1 at 1 for retrieval: 0.3222222222222222\n",
      "[I 2024-11-25 11:48:25,670] Trial 3 finished with value: 0.7788785494630805 and parameters: {'model_info': {'provider': 'hf', 'model': 'sentence-transformers/all-MiniLM-L6-v2', 'dim': 384}, 'retriever': 'default', 'algorithm': 'flat', 'var_dtype': 'float32', 'ret_k': 1}. Best is trial 1 with value: 1.111346093640145.\n",
      "2024-11-25 11:48:25,670 - INFO - Running for Retriever: DefaultQueryRetriever with additional_schema_fields=None\n",
      "2024-11-25 11:48:25,679 - INFO - Load pretrained SentenceTransformer: intfloat/e5-large-v2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      " Running for: \n",
      " model_str: intfloat/e5-large-v2 \n",
      " ef_runtime: 10 \n",
      " ef_construction: 150 \n",
      " m: 8 \n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:48:28,252 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d08570d7ecc34a028cc07e5e347d5514",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "325e79343db448e89755b821edaf9645",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:48:56,345 - INFO - Indexing data...\n",
      "2024-11-25 11:48:57,437 - INFO - Data indexed. self.total_indexing_time=384.28399658203125s\n",
      "2024-11-25 11:48:57,447 - INFO - Load pretrained SentenceTransformer: intfloat/e5-large-v2\n",
      "2024-11-25 11:48:58,868 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "24998b6bb4e146a99b6cd88f213c254b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4b556994e3d34ecc9c56ac2e216a7354",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0b6b6ed062b645f7b4964605b02800a5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "16bb4747b37c47d681e890a54f644e05",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "48c77be682894adc8f375f7d62bdf100",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ef26036118794a69bfa15a9828f83a10",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0e3f436ae33a45abb48ac83935fd43a1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "42b10f8012564848a62021086ae28af9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d327948e68e14bc08f8bd14d07997832",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bf6ce1a7cc394591bc7514981eab4bb9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7c83bd14fc75412fb8650dfb3fd783f6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3ae775a8b6024e8693afa080b3d72a7f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5186c6a1270244b99ea94e3bd8d13c3c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "71ab79be828e413faf9f464cd7119fcf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bb006dc6393e48789ac14a8ebdf2855b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bac5f6ec552b478896a62a72d385280c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:49:00,196 - INFO - Calculating retrieval metrics for test_id: 90a949ac-ab99-486d-80ed-4800e462f4fb\n",
      "2024-11-25 11:49:00,202 - INFO - Overall f1 at 9 for retrieval: 0.20736374736374735\n",
      "[I 2024-11-25 11:49:00,204] Trial 4 finished with value: 0.244817747183801 and parameters: {'model_info': {'provider': 'hf', 'model': 'intfloat/e5-large-v2', 'dim': 1024}, 'retriever': 'default', 'algorithm': 'hnsw', 'var_dtype': 'float32', 'ret_k': 9, 'ef_runtime': 10, 'ef_construction': 150, 'm': 8}. Best is trial 1 with value: 1.111346093640145.\n",
      "2024-11-25 11:49:00,204 - INFO - Running for Retriever: MakeModelQueryRetriever with additional_schema_fields=[{'name': 'make', 'type': 'tag'}, {'name': 'model', 'type': 'tag'}]\n",
      "2024-11-25 11:49:00,218 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      " Running for: \n",
      " model_str: sentence-transformers/all-MiniLM-L6-v2 \n",
      " algorithm: flat\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:49:00,875 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "666c620ac9664bc080cc52754b839a53",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8f7aec9026764ac9a8cf1eff251b243d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:49:02,174 - INFO - Indexing data...\n",
      "2024-11-25 11:49:03,150 - INFO - Data indexed. self.total_indexing_time=180.7779998779297s\n",
      "2024-11-25 11:49:03,159 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n",
      "2024-11-25 11:49:03,787 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "43a67f1b3b624f97bc4fa04f3c9fc55b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a6e94b744218404a8b23d8841785c53f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "792fca4b882f47e189e58b80e658f11d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fc4dbb454e7d438c9d7fc65409823f1a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "535fe6d067fd4dbcb4752c76de58dd35",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "93fbd991e3ec4c5a9d0173dc5425877e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9bf185dd933e4ac08ea141304afdcd3b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "954622870cb24165a80446d5d21c15a8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fce43749180141bbb0296e5d73306d5b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "61f304d18e4e430993b6c56505757cc1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "48e8f1e396b942e9bcf1bd8c52a64357",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "43b5b76bbcc9469aac2c6c75d6ed18f4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4139eb00750841318ca0cad216ba73ee",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0c36125b2c524477845e00eef96d4aec",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5c0b1e91b05447949238e2119ce05b88",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5ab38d7d774442fb9736b4def0f73ee3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:49:04,392 - INFO - Calculating retrieval metrics for test_id: 6b86eeae-d8e0-4680-9d5d-8f04e53b6da8\n",
      "2024-11-25 11:49:04,397 - INFO - Overall f1 at 7 for retrieval: 0.502986272986273\n",
      "[I 2024-11-25 11:49:04,398] Trial 5 finished with value: 0.9517931338796259 and parameters: {'model_info': {'provider': 'hf', 'model': 'sentence-transformers/all-MiniLM-L6-v2', 'dim': 384}, 'retriever': 'make_model', 'algorithm': 'flat', 'var_dtype': 'float16', 'ret_k': 7}. Best is trial 1 with value: 1.111346093640145.\n",
      "2024-11-25 11:49:04,398 - INFO - Running for Retriever: MakeModelQueryRetriever with additional_schema_fields=[{'name': 'make', 'type': 'tag'}, {'name': 'model', 'type': 'tag'}]\n",
      "2024-11-25 11:49:04,406 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      " Running for: \n",
      " model_str: sentence-transformers/all-MiniLM-L6-v2 \n",
      " algorithm: flat\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:49:05,169 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1200697193b443b08fe17001703be944",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7ddf3b6ba2f742ab9293a75451b70cce",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:49:06,621 - INFO - Indexing data...\n",
      "2024-11-25 11:49:07,846 - INFO - Data indexed. self.total_indexing_time=160.6199951171875s\n",
      "2024-11-25 11:49:07,859 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n",
      "2024-11-25 11:49:08,365 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8500df52131b4b6b86c8bd4a739fefa1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a0861394b9f84dd39ddb7eab321cc6c7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "76c8d53b1f9a44da972dbe4ba72603fe",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bc51d4fdeb9f4cbd9335988e5d4ce45e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fade0a179dc2494e93d21efa7fa9b61c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "97b5b1d0df6a47b2b6bfb60be425e6f0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "149538c7c547426b9e7a11737e27ff96",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4169c968e092462993e0139f02e59c22",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "79400d4147f7475e9cf68587e0f38d11",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "be9c1694eb4847dfa651a89760283038",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0906781d40854dc5a8590b0508d140bc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1f2f0a16c45d4840a0c5d1f5f249abfa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f5e832652c904104a933e8a60cc56014",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f694eee2a47843e291740061c06d4892",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "29b8296b60594d84b1668b1a669a424e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e93e1eec17ae4c808be87e8733ae113a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 7 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:49:08,957 - INFO - Calculating retrieval metrics for test_id: 6923b947-2566-4fd9-863f-122a63ff8739\n",
      "2024-11-25 11:49:08,962 - INFO - Overall f1 at 7 for retrieval: 0.502986272986273\n",
      "[I 2024-11-25 11:49:08,963] Trial 6 finished with value: 0.9239119259435168 and parameters: {'model_info': {'provider': 'hf', 'model': 'sentence-transformers/all-MiniLM-L6-v2', 'dim': 384}, 'retriever': 'make_model', 'algorithm': 'flat', 'var_dtype': 'float32', 'ret_k': 7}. Best is trial 1 with value: 1.111346093640145.\n",
      "2024-11-25 11:49:08,964 - INFO - Running for Retriever: MakeModelQueryRetriever with additional_schema_fields=[{'name': 'make', 'type': 'tag'}, {'name': 'model', 'type': 'tag'}]\n",
      "2024-11-25 11:49:08,974 - INFO - Load pretrained SentenceTransformer: intfloat/e5-large-v2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      " Running for: \n",
      " model_str: intfloat/e5-large-v2 \n",
      " ef_runtime: 30 \n",
      " ef_construction: 250 \n",
      " m: 64 \n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:49:10,050 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f08856089640483bbdf485b258625f75",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7e338c710f3b4313b32bd1741ba70c72",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:49:37,690 - INFO - Indexing data...\n",
      "2024-11-25 11:49:39,363 - INFO - Data indexed. self.total_indexing_time=471.0669860839844s\n",
      "2024-11-25 11:49:39,374 - INFO - Load pretrained SentenceTransformer: intfloat/e5-large-v2\n",
      "2024-11-25 11:49:41,119 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "57d6ebe3d063468eb1292ad73b39734f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7c82c58fa34e4dc289fad71dfcd03544",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "630ccdec37054929ae2439320011bbf3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e39ad39bec34479a82b482dfce27a47a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "08d6b9081df74b6998573996089147b4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b275d954c3984f0d87f1765ccfb357c5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f90734a9dff64f818d50da97e2aa499c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2d92c85f849f46099cfe650f1aad5caa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2c87433b7b6e4f0194ff881c049e4c3a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5f05f6f3231646f799fa539afcd40a0e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "735fd090bf534e4da559462780fdc155",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b59d72e71423484badcc8706802f0126",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8f68eeb99f2a4a2796d73ad86f7e7616",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "02d583d2da7649da81fa859e82ff7e19",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cac0322615e649f5bc0c2131c0324ff1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8ad7893fbeb841b28a6764853c48a916",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:49:42,246 - INFO - Calculating retrieval metrics for test_id: ea58f858-52fe-4834-87a3-da0808660af5\n",
      "2024-11-25 11:49:42,251 - INFO - Overall f1 at 2 for retrieval: 0.45809523809523806\n",
      "[I 2024-11-25 11:49:42,252] Trial 7 finished with value: 0.4954820002750173 and parameters: {'model_info': {'provider': 'hf', 'model': 'intfloat/e5-large-v2', 'dim': 1024}, 'retriever': 'make_model', 'algorithm': 'hnsw', 'var_dtype': 'float32', 'ret_k': 2, 'ef_runtime': 30, 'ef_construction': 250, 'm': 64}. Best is trial 1 with value: 1.111346093640145.\n",
      "2024-11-25 11:49:42,252 - INFO - Running for Retriever: MakeModelQueryRetriever with additional_schema_fields=[{'name': 'make', 'type': 'tag'}, {'name': 'model', 'type': 'tag'}]\n",
      "2024-11-25 11:49:42,260 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      " Running for: \n",
      " model_str: sentence-transformers/all-MiniLM-L6-v2 \n",
      " algorithm: flat\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:49:42,917 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "119240814a714426b8889f5babae5b3a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6b76a8a453e94139a135a03731e3714e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:49:44,292 - INFO - Indexing data...\n",
      "2024-11-25 11:49:45,882 - INFO - Data indexed. self.total_indexing_time=187.4759979248047s\n",
      "2024-11-25 11:49:45,898 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n",
      "2024-11-25 11:49:46,615 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a975007d050d42859b0de6b8a9a4dbc3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "10d7c49e0347468fb882a8bb7775b779",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6dc93c94c4914d91aa87e2e029855b1b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9e11539f4f1b4f18ac54a7011a920871",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8021abe19ecf4324bc633e091a5932e9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f7e3606be0584c628b6e26f8c3a6e4d7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e8c64a73bc9144fca236af0d5c52d768",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "462ca8be447942d1b8e477e0faba3036",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8cfae5547077495b94b543caa081ae85",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "01e362173b7b4096ab5b1b67561daae5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cf58db44a29a41879bb24107a9a2ec14",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "42b0c2d25d7546b5813fc1d61d726910",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0a94a87e00d6412a8c3ee08627e85fb5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "69eb4291cf554601a162f05ad83fde95",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "acfce76c2b984de6b274acbff3615519",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1725f71a0fae4b7a972eb8543029693c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:49:47,252 - INFO - Calculating retrieval metrics for test_id: a847eb04-f2a4-4871-bb35-d33aeddad0db\n",
      "2024-11-25 11:49:47,257 - INFO - Overall f1 at 2 for retrieval: 0.6558730158730159\n",
      "[I 2024-11-25 11:49:47,258] Trial 8 finished with value: 1.090247035116796 and parameters: {'model_info': {'provider': 'hf', 'model': 'sentence-transformers/all-MiniLM-L6-v2', 'dim': 384}, 'retriever': 'make_model', 'algorithm': 'flat', 'var_dtype': 'float16', 'ret_k': 2}. Best is trial 1 with value: 1.111346093640145.\n",
      "2024-11-25 11:49:47,259 - INFO - Running for Retriever: MakeModelQueryRetriever with additional_schema_fields=[{'name': 'make', 'type': 'tag'}, {'name': 'model', 'type': 'tag'}]\n",
      "2024-11-25 11:49:47,270 - INFO - Load pretrained SentenceTransformer: intfloat/e5-large-v2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      " Running for: \n",
      " model_str: intfloat/e5-large-v2 \n",
      " ef_runtime: 30 \n",
      " ef_construction: 150 \n",
      " m: 8 \n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:49:48,967 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "59f8559941cf4126a055e1dc3532fc46",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "39c0063f26e54efba5b88bff8555d76f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:50:16,607 - INFO - Indexing data...\n",
      "2024-11-25 11:50:18,759 - INFO - Data indexed. self.total_indexing_time=384.8320007324219s\n",
      "2024-11-25 11:50:18,770 - INFO - Load pretrained SentenceTransformer: intfloat/e5-large-v2\n",
      "2024-11-25 11:50:20,173 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a5a77f34f8b54fe2b5ad956ef6cc2b8e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "af1232d7066e462fafe806941227367d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ac836b8c89a641b1bf26ab263de4eafd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "676745fe953d448ba704753bd6ddf8f7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3b6843cb6c5840c8a6efa72e88495b45",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "48da765c4a7f400b80abf1e596fc3d1a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7dd8ff6ad5f54ab0b9c63ad483267930",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e49c19e030fe467f824f6518bbf87566",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3bfa1347313c4d4e9da0455bef43289e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "350b6a3de5fa4ec2894f8db03f516370",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "68037aac5d73457b8519fb4a104c707b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e1c4e24c23c843599cd8a3ec3d252477",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f23100bdba324a7cabcf207842bbc9a6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1672f10570644076b20b71d15aae2190",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "eb1fc82e36294e0c88c446df01a5a069",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2b819d7e75904d70a63dd061acd05fbd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 2 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:50:21,347 - INFO - Calculating retrieval metrics for test_id: 84428eb4-c384-4efe-9040-1d3727432ee3\n",
      "2024-11-25 11:50:21,352 - INFO - Overall f1 at 2 for retrieval: 0.45809523809523806\n",
      "[I 2024-11-25 11:50:21,354] Trial 9 finished with value: 0.4959678527701849 and parameters: {'model_info': {'provider': 'hf', 'model': 'intfloat/e5-large-v2', 'dim': 1024}, 'retriever': 'make_model', 'algorithm': 'hnsw', 'var_dtype': 'float32', 'ret_k': 2, 'ef_runtime': 30, 'ef_construction': 150, 'm': 8}. Best is trial 1 with value: 1.111346093640145.\n",
      "2024-11-25 11:50:21,355 - INFO - Running for Retriever: MakeModelQueryRetriever with additional_schema_fields=[{'name': 'make', 'type': 'tag'}, {'name': 'model', 'type': 'tag'}]\n",
      "2024-11-25 11:50:21,368 - INFO - Load pretrained SentenceTransformer: intfloat/e5-large-v2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      " Running for: \n",
      " model_str: intfloat/e5-large-v2 \n",
      " algorithm: flat\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:50:22,416 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "684b4d50deeb48658749650f71026b24",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2ac22d473cc44cac966f580cec34cc80",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:50:50,056 - INFO - Indexing data...\n",
      "2024-11-25 11:50:51,483 - INFO - Data indexed. self.total_indexing_time=188.4459991455078s\n",
      "2024-11-25 11:50:51,496 - INFO - Load pretrained SentenceTransformer: intfloat/e5-large-v2\n",
      "2024-11-25 11:50:53,260 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6b66eda610dd4daeba76b7ea35871aec",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e9291cfbeb5a4c2fbed742a98029f25c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5c277152f02a43f9a45e76efe402309a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fbf64e7648a14fdb8cd75559ea78f288",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "513ef50fe7244e47bfb2e3433620bc74",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c5ae959cf03948f1aa5dbe0923016dd1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "553806c1a0db4ef8a8d3bc08192cd624",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "eab8e48d70674726b84a438ae2a70a46",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b532ab4a5d5645b29d3641664a0bf3b6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "86190706fb7e4910967e0273e5d4c24b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "128e0840fc684649bf1a18f70bbcc2d4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6c77fa0ce6364c54a0f9690518fbbdbd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c6a6640962e0412db5e916c45d9bf5f1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2050c8fc3b4e485aaabbf7b50d07ebb9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7bbec9da062644ddb7da0d3cf66875e0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "85b40355d6b64534a1d92fbf9bbac573",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:50:54,375 - INFO - Calculating retrieval metrics for test_id: d55a1e2f-7ba2-4221-bfc8-c55153518e34\n",
      "2024-11-25 11:50:54,380 - INFO - Overall f1 at 5 for retrieval: 0.47309283309283306\n",
      "[I 2024-11-25 11:50:54,381] Trial 10 finished with value: 0.5135533048433937 and parameters: {'model_info': {'provider': 'hf', 'model': 'intfloat/e5-large-v2', 'dim': 1024}, 'retriever': 'make_model', 'algorithm': 'flat', 'var_dtype': 'float16', 'ret_k': 5}. Best is trial 1 with value: 1.111346093640145.\n",
      "2024-11-25 11:50:54,382 - INFO - Running for Retriever: MakeModelQueryRetriever with additional_schema_fields=[{'name': 'make', 'type': 'tag'}, {'name': 'model', 'type': 'tag'}]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      " Running for: \n",
      " model_str: sentence-transformers/all-MiniLM-L6-v2 \n",
      " algorithm: flat\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:50:54,394 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n",
      "2024-11-25 11:50:54,964 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "34d139a3e6934b5aaca5b14dc79f31cc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7205912e60e6417e90c547d6e8a1cc0a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:50:56,330 - INFO - Indexing data...\n",
      "2024-11-25 11:50:57,654 - INFO - Data indexed. self.total_indexing_time=172.89300537109375s\n",
      "2024-11-25 11:50:57,666 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n",
      "2024-11-25 11:50:58,460 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5d15a356ef2549688b8fa2f3a8df4234",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9f151a0d950e43dfb9039294f056a462",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d16c38834eed4ad78abf460774ab3dad",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7f55d3faa01a406c8b7816fdc3a5cdc1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "75da74037a9344c0863aa14b672f1c4f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7fa7b52c71c24fa8970685b974b763f4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1609d2a7774e4aa18fcf21ae346957d3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1a064ad2018e4345a847c40b773adf69",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1cf5dd280769478ba6ed8eb98675681e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "845119e89543420b95a4d338e8c30ca7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2f3737923bd2483da5cda236811fc02a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "15ffae45974441fc979a7f974566a9f2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "63228c58414746eb992d1c8ad378e86a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7a8122622e5f4746aff7a08adfde0d1d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "32a3ae7649184714b5ac0f59c2618c49",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "32d380d883104508902ec660fed443d6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:50:59,022 - INFO - Calculating retrieval metrics for test_id: 1dd82286-09af-425b-8f31-4f8f50a9ee4e\n",
      "2024-11-25 11:50:59,028 - INFO - Overall f1 at 4 for retrieval: 0.6031746031746031\n",
      "[I 2024-11-25 11:50:59,029] Trial 11 finished with value: 1.0392912216637609 and parameters: {'model_info': {'provider': 'hf', 'model': 'sentence-transformers/all-MiniLM-L6-v2', 'dim': 384}, 'retriever': 'make_model', 'algorithm': 'flat', 'var_dtype': 'float16', 'ret_k': 4}. Best is trial 1 with value: 1.111346093640145.\n",
      "2024-11-25 11:50:59,030 - INFO - Running for Retriever: MakeModelQueryRetriever with additional_schema_fields=[{'name': 'make', 'type': 'tag'}, {'name': 'model', 'type': 'tag'}]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      " Running for: \n",
      " model_str: sentence-transformers/all-MiniLM-L6-v2 \n",
      " algorithm: flat\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:50:59,045 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n",
      "2024-11-25 11:50:59,561 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5c800675b88a46fc97c74686b84f9810",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "066a4fd2fcb74937b425ff6524260b0f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:51:00,870 - INFO - Indexing data...\n",
      "2024-11-25 11:51:03,088 - INFO - Data indexed. self.total_indexing_time=213.38299560546875s\n",
      "2024-11-25 11:51:03,099 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n",
      "2024-11-25 11:51:03,711 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e0ce08b9a60b4077aa79899d730e0e55",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f70ca5dfc387488a9210efa34ae17b2f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b21e5b40d6c34c93a864643e60a6a4d3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e85f7d24744f4853b98a68fc3508ed15",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3165b5b453c4461faf8e9a25a5eee5d4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "36218c5aae0241109431ab2a5318ce78",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "62ce9d7580544c9c9522f592429926ba",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "26047423c2594d538f7ad9999cedf77d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "385ecfade3af484da3417609b516e4a2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "986a5ff1c70d41eaab6f365c91300510",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bdef7484093e4a6d9709d643ff051ec9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3fb66ca167914277ac699ee9b00e01b4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4b181d6dde8641cf8b8ec80323a79027",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c51e78e7d6db47c9b6a062e3fc5469af",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3f3a36a01f4b4916b8f74d4820ece0c9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "99697db4dc9f487cbd78df3075dd948a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:51:04,374 - INFO - Calculating retrieval metrics for test_id: 49ddb590-8098-48c7-ba5e-6e2f7e076a69\n",
      "2024-11-25 11:51:04,378 - INFO - Overall f1 at 4 for retrieval: 0.6031746031746031\n",
      "[I 2024-11-25 11:51:04,379] Trial 12 finished with value: 1.0492050396048045 and parameters: {'model_info': {'provider': 'hf', 'model': 'sentence-transformers/all-MiniLM-L6-v2', 'dim': 384}, 'retriever': 'make_model', 'algorithm': 'flat', 'var_dtype': 'float16', 'ret_k': 4}. Best is trial 1 with value: 1.111346093640145.\n",
      "2024-11-25 11:51:04,380 - INFO - Running for Retriever: MakeModelQueryRetriever with additional_schema_fields=[{'name': 'make', 'type': 'tag'}, {'name': 'model', 'type': 'tag'}]\n",
      "2024-11-25 11:51:04,392 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      " Running for: \n",
      " model_str: sentence-transformers/all-MiniLM-L6-v2 \n",
      " algorithm: flat\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:51:04,947 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9799486c4d1c4522bd81ef0e3168286a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e6074926e44244798640a4b872d19ffa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:51:06,220 - INFO - Indexing data...\n",
      "2024-11-25 11:51:08,736 - INFO - Data indexed. self.total_indexing_time=285.9150085449219s\n",
      "2024-11-25 11:51:08,744 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n",
      "2024-11-25 11:51:09,283 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3140e990338f4a50a542f4ae5bb54b8f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b18b2e4fd9054130926a97f3783e4168",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 1 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "31a9c2fee98d4767b3119f3ab7b959d3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 1 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1a94b48d2d814deebf624f1dc0732782",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 1 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bcffa5cc54a443eb99e8837acf53fe56",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 1 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6dbaf87e346040b69eecb4569cf86209",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 1 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9f49899aee844d459a9efeff69ecbd91",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 1 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7ef58a251f494aaaa555624897394091",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 1 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c6efba382fae4c709896411af21bce83",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 1 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e86f5bdedd1e43788aff17ea01be04b8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 1 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f352827aa7c2423c9730a95b8e8ccbec",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 1 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7fb2b734d8734b819d28e9e6ea47a890",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 1 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a228f25d030241f5a9e3918c462ce776",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 1 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "68b8eb60b26f4f0b9b8ef60a4a2f2afc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 1 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "23454c5e5cbf4ca09bd11612799a7ba2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 1 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e1a95abfd5eb4e47bcef2266309abdb6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 1 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:51:09,875 - INFO - Calculating retrieval metrics for test_id: 86200018-23f1-4f70-822e-f5da78838a04\n",
      "2024-11-25 11:51:09,881 - INFO - Overall f1 at 1 for retrieval: 0.5568253968253969\n",
      "[I 2024-11-25 11:51:09,882] Trial 13 finished with value: 1.0087157344363582 and parameters: {'model_info': {'provider': 'hf', 'model': 'sentence-transformers/all-MiniLM-L6-v2', 'dim': 384}, 'retriever': 'make_model', 'algorithm': 'flat', 'var_dtype': 'float16', 'ret_k': 1}. Best is trial 1 with value: 1.111346093640145.\n",
      "2024-11-25 11:51:09,883 - INFO - Running for Retriever: MakeModelQueryRetriever with additional_schema_fields=[{'name': 'make', 'type': 'tag'}, {'name': 'model', 'type': 'tag'}]\n",
      "2024-11-25 11:51:09,893 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      " Running for: \n",
      " model_str: sentence-transformers/all-MiniLM-L6-v2 \n",
      " algorithm: flat\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:51:10,498 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9d8d93e239314d608311b996eaf18bd4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f44795988a8544ba9136dfb61f103410",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:51:11,776 - INFO - Indexing data...\n",
      "2024-11-25 11:51:13,548 - INFO - Data indexed. self.total_indexing_time=207.84500122070312s\n",
      "2024-11-25 11:51:13,560 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n",
      "2024-11-25 11:51:14,217 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "40d8a13b591a40048bac4b233c850bb3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ca5e61d7ef104e81b86fcd196ab6cc1d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 6 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ba24773e207e49db86ca3a7c93f1488e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 6 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c81de4c2b6a843dcb7e194bb58c0c5af",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 6 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9c9b54f9976749d6a375bcad8ff2f9e7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 6 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "91e712b3c71a4163b1dc59a44ec83e81",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 6 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3df6b29f964c4ed398153b1ff6e98b1c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 6 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "545b5a6c193443d9a969764c3317fef3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 6 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6c155ed3a317450b80d301023a19a702",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 6 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f94183fe21a34aecb84c66b3871c4315",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 6 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6e956240723d421484261a6123b076f7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 6 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "07b1d3c9cc674347a70b696930656ee9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 6 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d7f8c0d094134267bb1122cf725d0cd9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 6 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e407640d5c9544d7ae7c2149fa926cb1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 6 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0d8c9cc89c74433ab90c32c86b57fcb9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 6 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3e66888c4fa84dd096e83e77265efa41",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 6 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:51:14,831 - INFO - Calculating retrieval metrics for test_id: 1700ec65-e82f-4c2a-9ce1-d29f8cebe37a\n",
      "2024-11-25 11:51:14,837 - INFO - Overall f1 at 6 for retrieval: 0.5599711399711399\n",
      "[I 2024-11-25 11:51:14,838] Trial 14 finished with value: 1.0130845142567686 and parameters: {'model_info': {'provider': 'hf', 'model': 'sentence-transformers/all-MiniLM-L6-v2', 'dim': 384}, 'retriever': 'make_model', 'algorithm': 'flat', 'var_dtype': 'float32', 'ret_k': 6}. Best is trial 1 with value: 1.111346093640145.\n",
      "2024-11-25 11:51:14,839 - INFO - Running for Retriever: MakeModelQueryRetriever with additional_schema_fields=[{'name': 'make', 'type': 'tag'}, {'name': 'model', 'type': 'tag'}]\n",
      "2024-11-25 11:51:14,849 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      " Running for: \n",
      " model_str: sentence-transformers/all-MiniLM-L6-v2 \n",
      " algorithm: flat\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:51:15,498 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c8cb1cd3d25d4b099d2286b553aa8cfb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4fbca42e56834462b1ce338032507393",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:51:16,785 - INFO - Indexing data...\n",
      "2024-11-25 11:51:18,553 - INFO - Data indexed. self.total_indexing_time=186.3939971923828s\n",
      "2024-11-25 11:51:18,628 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n",
      "2024-11-25 11:51:19,280 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3bb98d8779af4f2283ca3ec021e3c654",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "89e8e6fbcb7545e9bf5477416be541a6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 10 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "12521d4526a745059ba73250a37ecee4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 10 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e6d86dc1011740a3b8e63cd0045ff277",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 10 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1f7bcdb36b494ba0bfcc4fa9193f345d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 10 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "45d20340a6c44ce9ae07766971542dd0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 10 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6f2876353ce44951836d3095a462920a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 10 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3908da72ac1b47658a8d0ebc7f6d3bd0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 10 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fb47e63eda314e6aba86aeea74e00e00",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 10 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ae058c0bde684f6d80b4db0ec2ceca98",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 10 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5b47181c688f4a21aaae02fce16f5a79",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 10 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b613805e89f246609d2c81e24b832a1d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 10 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "11cd8b5a27314dbdbc826f55c5af5bf4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 10 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7309cf49921b41968c2f963e0e6a333e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 10 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c71eacb91ce149aab17e0527c5611cf8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 10 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fdd202afbdde47709f2e4be54e0dcd11",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 10 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:51:19,825 - INFO - Calculating retrieval metrics for test_id: 0e9a3343-b268-4d51-ae2a-5bb3af73c8bd\n",
      "2024-11-25 11:51:19,830 - INFO - Overall f1 at 10 for retrieval: 0.38599733599733604\n",
      "[I 2024-11-25 11:51:19,832] Trial 15 finished with value: 0.8373868590582951 and parameters: {'model_info': {'provider': 'hf', 'model': 'sentence-transformers/all-MiniLM-L6-v2', 'dim': 384}, 'retriever': 'make_model', 'algorithm': 'flat', 'var_dtype': 'float16', 'ret_k': 10}. Best is trial 1 with value: 1.111346093640145.\n",
      "2024-11-25 11:51:19,833 - INFO - Running for Retriever: MakeModelQueryRetriever with additional_schema_fields=[{'name': 'make', 'type': 'tag'}, {'name': 'model', 'type': 'tag'}]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      " Running for: \n",
      " model_str: sentence-transformers/all-MiniLM-L6-v2 \n",
      " algorithm: flat\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:51:19,846 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n",
      "2024-11-25 11:51:20,476 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c6e5b7847a54475abd015101c2569706",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3d5b73b58fbd40aab1cd5e5efe7db155",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:51:21,762 - INFO - Indexing data...\n",
      "2024-11-25 11:51:24,530 - INFO - Data indexed. self.total_indexing_time=282.0580139160156s\n",
      "2024-11-25 11:51:24,542 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n",
      "2024-11-25 11:51:25,178 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b4180203761647158671c383677b8984",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f70a0904c36742ffac3bcf2f71a5edf2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "23f89bdd2065491cbbf367a5005aef23",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3826a3af217d4fc0aff00ea32f64942c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1cdc63cd80cc44289c49d1456f3f7f05",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a690c20f500b46a7b941b4fb6e526761",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fa697c58f2d748f8a62bce5e02a777df",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c7b20d9151b648128a2ea14693ed9d2f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9ad55655f29040b9b466dc0550aef57c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7f53e3552f3841e7836755e336d4fc16",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8c485f9e708f4b4c94e8386641e117e2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d00d8f2f54044486b73b1d999114e068",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f76444baa35549f5b5a91195e6fb7e2c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3fa9baa7621a4d53b1b1537626c9af8f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "25d8935bf06748ab96203bd148466b3f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4870aa9ce26f4983a16c067dbaf4e0bc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 4 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:51:25,741 - INFO - Calculating retrieval metrics for test_id: df511477-93ad-4447-9523-9188adc992fe\n",
      "2024-11-25 11:51:25,746 - INFO - Overall f1 at 4 for retrieval: 0.6031746031746031\n",
      "[I 2024-11-25 11:51:25,747] Trial 16 finished with value: 1.052341438450084 and parameters: {'model_info': {'provider': 'hf', 'model': 'sentence-transformers/all-MiniLM-L6-v2', 'dim': 384}, 'retriever': 'make_model', 'algorithm': 'flat', 'var_dtype': 'float16', 'ret_k': 4}. Best is trial 1 with value: 1.111346093640145.\n",
      "2024-11-25 11:51:25,748 - INFO - Running for Retriever: MakeModelQueryRetriever with additional_schema_fields=[{'name': 'make', 'type': 'tag'}, {'name': 'model', 'type': 'tag'}]\n",
      "2024-11-25 11:51:25,761 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      " Running for: \n",
      " model_str: sentence-transformers/all-MiniLM-L6-v2 \n",
      " algorithm: flat\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:51:26,423 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a22876e8ec3742df9311128daf707751",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "32d836b6e5e1459186518d8d78e64002",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:51:27,741 - INFO - Indexing data...\n",
      "2024-11-25 11:51:29,655 - INFO - Data indexed. self.total_indexing_time=222.25599670410156s\n",
      "2024-11-25 11:51:29,664 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n",
      "2024-11-25 11:51:30,173 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "79848673fe65476f8c1241fd5a366da3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b203d6b51fe34c89a54a08ff5369737a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 3 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c62286478d1646c7aed45b0de9a58e7a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 3 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d6edc53b8285456bb0223a556bc839b5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 3 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dc1e9c8dd83b4b6fbce2fc73348b8705",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 3 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b36b1b65d82b4fb292bff41d0553bdf2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 3 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b2e772db4efa40a5911f310a9b7898e5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 3 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dc4a2aeb0051470fbbba4435260270f1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 3 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5e534e2581594b6c8607125f315664a6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 3 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e6c95892c87242809ceedcdd508cad0c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 3 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0b63a661e9ce41b9ab621e50db2cab6b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 3 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cd07627ce3e042dfbc6143cfe9e84ec6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 3 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6df726a34e5e4be4b37feb51173d589b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 3 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6d4b628fa08e4325a30b5bfcdb177815",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 3 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "16f11cfd3a374f02826870e0366b6875",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 3 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "05cb07015a2a4e01874429b1ef2a7ffc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 3 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:51:30,757 - INFO - Calculating retrieval metrics for test_id: a30e651d-b83d-48d5-a41f-15e11b7803c5\n",
      "2024-11-25 11:51:30,763 - INFO - Overall f1 at 3 for retrieval: 0.6782539682539682\n",
      "[I 2024-11-25 11:51:30,764] Trial 17 finished with value: 1.1223205945917392 and parameters: {'model_info': {'provider': 'hf', 'model': 'sentence-transformers/all-MiniLM-L6-v2', 'dim': 384}, 'retriever': 'make_model', 'algorithm': 'flat', 'var_dtype': 'float32', 'ret_k': 3}. Best is trial 17 with value: 1.1223205945917392.\n",
      "2024-11-25 11:51:30,766 - INFO - Running for Retriever: DefaultQueryRetriever with additional_schema_fields=None\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      " Running for: \n",
      " model_str: intfloat/e5-large-v2 \n",
      " ef_runtime: 50 \n",
      " ef_construction: 200 \n",
      " m: 16 \n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:51:30,777 - INFO - Load pretrained SentenceTransformer: intfloat/e5-large-v2\n",
      "2024-11-25 11:51:32,678 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cb84950854444deba0b3f690c7e968b2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4e130b84182143909749b7efd67fc91e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:52:00,277 - INFO - Indexing data...\n",
      "2024-11-25 11:52:03,091 - INFO - Data indexed. self.total_indexing_time=506.6369934082031s\n",
      "2024-11-25 11:52:03,109 - INFO - Load pretrained SentenceTransformer: intfloat/e5-large-v2\n",
      "2024-11-25 11:52:04,511 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b2a7c681a02847c5883c5a0b57ad3858",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "48cbd35a21304237a55ce4dedb191caa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "eb7befc6c49f4bde8fdaa0c536fb3b83",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fa94b6dc10984a3e9d11b269ba2643da",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e6b652872cf341d6bdba00f20945fd3e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "41ef6254bc2e4fef9f0618d678eec97e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5fd255e8799747ad8518d4a58e7a9ab4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5806222e9bde423bae6523e1c4257eb7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "33e1e471f62d486daf31cd3906da3620",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ca519c0495af41b480cff051bc1311d4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e74e07883dd8445dab014851eccfc873",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "35ef1610cc3d42d38c7e4a6f9a4c72a6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ee309504fc414a9ea406b6c8f84ffa76",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ab38fe837b06484dbefe98e649f54ee0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "47d178f6991d49a095cfc08ceb0b7fd8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cb8a9cc92f4a4b6e86271759e1114928",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:52:05,579 - INFO - Calculating retrieval metrics for test_id: 6e1bd366-588f-4a4f-bad3-e66527c1152c\n",
      "2024-11-25 11:52:05,585 - INFO - Overall f1 at 3 for retrieval: 0.2994179894179894\n",
      "[I 2024-11-25 11:52:05,589] Trial 18 finished with value: 0.3367085657417276 and parameters: {'model_info': {'provider': 'hf', 'model': 'intfloat/e5-large-v2', 'dim': 1024}, 'retriever': 'default', 'algorithm': 'hnsw', 'var_dtype': 'float32', 'ret_k': 3, 'ef_runtime': 50, 'ef_construction': 200, 'm': 16}. Best is trial 17 with value: 1.1223205945917392.\n",
      "2024-11-25 11:52:05,591 - INFO - Running for Retriever: MakeModelQueryRetriever with additional_schema_fields=[{'name': 'make', 'type': 'tag'}, {'name': 'model', 'type': 'tag'}]\n",
      "2024-11-25 11:52:05,613 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      " Running for: \n",
      " model_str: sentence-transformers/all-MiniLM-L6-v2 \n",
      " algorithm: flat\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:52:06,264 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "294ceb213f364be1916945f5d2966870",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b8513d7f0abd4c4db92884bdc1f7e998",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:52:07,569 - INFO - Indexing data...\n",
      "2024-11-25 11:52:10,282 - INFO - Data indexed. self.total_indexing_time=264.5989990234375s\n",
      "2024-11-25 11:52:10,295 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n",
      "2024-11-25 11:52:10,928 - INFO - Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "32e372971bd1405786ffcd2005422372",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0fa9c6e5e0f548a7af63c2b9f9f92a6d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8d9f129b4d124746811221f1023498cb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2c3582f5a9ab4ce1beae6c0d6e5ea183",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "53cfaa7351f24f8b83f9dc54074d2494",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a1c1ec9a17db4b00acdefd2a7d22ae4f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "65d6c589e3a549b4b0f0f6f41031c600",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b1cc21e0819f472e8ecd2ec8158838c6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "12b8a5098e924029bf1b1d6215de590e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "377fb266a3d04cd8ab26d1d114e72a93",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ee2e5f8d29864e24b58c35da4d2d9c55",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3f6f2e1c78ca4a359e4598c9ee488bd4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ec9bacfb75664294a56cd1beb6d9cecb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{mazda} @model:{3})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "de5fe7dc774344418d34297065f52d7d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0c12751b7f684288adfb08b9051a2ff2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4f13b135619142d7a7a5cb65fe36f2c6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vec_query.query_string()='(@make:{volvo} @model:{c30})=>[KNN 5 @vector $vector AS vector_distance]'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-25 11:52:11,442 - INFO - Calculating retrieval metrics for test_id: 9f41ef0b-c80b-4dfa-ac9e-f9efbb03242a\n",
      "2024-11-25 11:52:11,448 - INFO - Overall f1 at 5 for retrieval: 0.5874843674843675\n",
      "[I 2024-11-25 11:52:11,449] Trial 19 finished with value: 1.0334613618358586 and parameters: {'model_info': {'provider': 'hf', 'model': 'sentence-transformers/all-MiniLM-L6-v2', 'dim': 384}, 'retriever': 'make_model', 'algorithm': 'flat', 'var_dtype': 'float32', 'ret_k': 5}. Best is trial 17 with value: 1.1223205945917392.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Completed Bayesian optimization... \n",
      "\n",
      "\n",
      "Best Configuration: 17: {'model_info': {'provider': 'hf', 'model': 'sentence-transformers/all-MiniLM-L6-v2', 'dim': 384}, 'retriever': 'make_model', 'algorithm': 'flat', 'var_dtype': 'float32', 'ret_k': 3}:\n",
      "\n",
      "\n",
      "Best Score: [1.1223205945917392]\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from optimize.study import run_study\n",
    "\n",
    "custom_retrievers = {\n",
    "    \"make_model\": {\n",
    "        \"retriever\": MakeModelQueryRetriever,\n",
    "        \"additional_schema_fields\": [\n",
    "            {\n",
    "                \"name\": \"make\", \"type\": \"tag\"\n",
    "            },\n",
    "            {\n",
    "                \"name\": \"model\", \"type\": \"tag\"\n",
    "            }\n",
    "        ]\n",
    "    },\n",
    "    \"default\": {\n",
    "        \"retriever\": DefaultQueryRetriever\n",
    "    }\n",
    "}\n",
    "\n",
    "run_study(study_config, custom_retrievers=custom_retrievers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Success!\n",
    "\n",
    "From the above we can see the overall best configuration and the settings used this is the config that best optimizes the cost function.\n",
    "\n",
    "For a more complete view of our results let's grab the metric outputs of our study and load into a pandas dataframe for our review."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from redis import Redis\n",
    "import pandas as pd\n",
    "study_key = f\"study:{study_config.study_id}\"\n",
    "\n",
    "redis_client = Redis.from_url(\"redis://localhost:6379/0\")\n",
    "\n",
    "res = redis_client.json().get(study_key)\n",
    "\n",
    "res_df = pd.DataFrame(res)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ret_k</th>\n",
       "      <th>algorithm</th>\n",
       "      <th>ef_construction</th>\n",
       "      <th>ef_runtime</th>\n",
       "      <th>m</th>\n",
       "      <th>distance_metric</th>\n",
       "      <th>vector_data_type</th>\n",
       "      <th>model</th>\n",
       "      <th>model_dim</th>\n",
       "      <th>f1@k</th>\n",
       "      <th>embedding_latency</th>\n",
       "      <th>indexing_time</th>\n",
       "      <th>avg_query_latency</th>\n",
       "      <th>obj_val</th>\n",
       "      <th>retriever</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>3</td>\n",
       "      <td>flat</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>cosine</td>\n",
       "      <td>float32</td>\n",
       "      <td>sentence-transformers/all-MiniLM-L6-v2</td>\n",
       "      <td>384</td>\n",
       "      <td>0.678254</td>\n",
       "      <td>1.274860</td>\n",
       "      <td>222.255997</td>\n",
       "      <td>0.037223</td>\n",
       "      <td>1.122321</td>\n",
       "      <td>MakeModelQueryRetriever</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2</td>\n",
       "      <td>flat</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>cosine</td>\n",
       "      <td>float16</td>\n",
       "      <td>sentence-transformers/all-MiniLM-L6-v2</td>\n",
       "      <td>384</td>\n",
       "      <td>0.655873</td>\n",
       "      <td>1.330631</td>\n",
       "      <td>187.475998</td>\n",
       "      <td>0.025454</td>\n",
       "      <td>1.090247</td>\n",
       "      <td>MakeModelQueryRetriever</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>flat</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>cosine</td>\n",
       "      <td>float32</td>\n",
       "      <td>sentence-transformers/all-MiniLM-L6-v2</td>\n",
       "      <td>384</td>\n",
       "      <td>0.655873</td>\n",
       "      <td>1.217216</td>\n",
       "      <td>223.367004</td>\n",
       "      <td>0.025062</td>\n",
       "      <td>1.111346</td>\n",
       "      <td>MakeModelQueryRetriever</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>4</td>\n",
       "      <td>flat</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>cosine</td>\n",
       "      <td>float16</td>\n",
       "      <td>sentence-transformers/all-MiniLM-L6-v2</td>\n",
       "      <td>384</td>\n",
       "      <td>0.603175</td>\n",
       "      <td>1.323604</td>\n",
       "      <td>172.893005</td>\n",
       "      <td>0.018080</td>\n",
       "      <td>1.039291</td>\n",
       "      <td>MakeModelQueryRetriever</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>4</td>\n",
       "      <td>flat</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>cosine</td>\n",
       "      <td>float16</td>\n",
       "      <td>sentence-transformers/all-MiniLM-L6-v2</td>\n",
       "      <td>384</td>\n",
       "      <td>0.603175</td>\n",
       "      <td>1.265694</td>\n",
       "      <td>213.382996</td>\n",
       "      <td>0.038584</td>\n",
       "      <td>1.049205</td>\n",
       "      <td>MakeModelQueryRetriever</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>4</td>\n",
       "      <td>flat</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>cosine</td>\n",
       "      <td>float16</td>\n",
       "      <td>sentence-transformers/all-MiniLM-L6-v2</td>\n",
       "      <td>384</td>\n",
       "      <td>0.603175</td>\n",
       "      <td>1.243994</td>\n",
       "      <td>282.058014</td>\n",
       "      <td>0.039360</td>\n",
       "      <td>1.052341</td>\n",
       "      <td>MakeModelQueryRetriever</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>5</td>\n",
       "      <td>flat</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>cosine</td>\n",
       "      <td>float32</td>\n",
       "      <td>sentence-transformers/all-MiniLM-L6-v2</td>\n",
       "      <td>384</td>\n",
       "      <td>0.587484</td>\n",
       "      <td>1.261359</td>\n",
       "      <td>264.598999</td>\n",
       "      <td>0.027733</td>\n",
       "      <td>1.033461</td>\n",
       "      <td>MakeModelQueryRetriever</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5</td>\n",
       "      <td>hnsw</td>\n",
       "      <td>100</td>\n",
       "      <td>50</td>\n",
       "      <td>64</td>\n",
       "      <td>cosine</td>\n",
       "      <td>float16</td>\n",
       "      <td>sentence-transformers/all-MiniLM-L6-v2</td>\n",
       "      <td>384</td>\n",
       "      <td>0.587484</td>\n",
       "      <td>1.250739</td>\n",
       "      <td>1353.703003</td>\n",
       "      <td>0.042778</td>\n",
       "      <td>1.032521</td>\n",
       "      <td>MakeModelQueryRetriever</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>6</td>\n",
       "      <td>flat</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>cosine</td>\n",
       "      <td>float32</td>\n",
       "      <td>sentence-transformers/all-MiniLM-L6-v2</td>\n",
       "      <td>384</td>\n",
       "      <td>0.559971</td>\n",
       "      <td>1.230524</td>\n",
       "      <td>207.845001</td>\n",
       "      <td>0.062395</td>\n",
       "      <td>1.013085</td>\n",
       "      <td>MakeModelQueryRetriever</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>1</td>\n",
       "      <td>flat</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>cosine</td>\n",
       "      <td>float16</td>\n",
       "      <td>sentence-transformers/all-MiniLM-L6-v2</td>\n",
       "      <td>384</td>\n",
       "      <td>0.556825</td>\n",
       "      <td>1.230127</td>\n",
       "      <td>285.915009</td>\n",
       "      <td>0.038707</td>\n",
       "      <td>1.008716</td>\n",
       "      <td>MakeModelQueryRetriever</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>7</td>\n",
       "      <td>flat</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>cosine</td>\n",
       "      <td>float32</td>\n",
       "      <td>sentence-transformers/all-MiniLM-L6-v2</td>\n",
       "      <td>384</td>\n",
       "      <td>0.502986</td>\n",
       "      <td>1.411159</td>\n",
       "      <td>160.619995</td>\n",
       "      <td>0.041576</td>\n",
       "      <td>0.923912</td>\n",
       "      <td>MakeModelQueryRetriever</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>flat</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>cosine</td>\n",
       "      <td>float16</td>\n",
       "      <td>sentence-transformers/all-MiniLM-L6-v2</td>\n",
       "      <td>384</td>\n",
       "      <td>0.502986</td>\n",
       "      <td>1.255780</td>\n",
       "      <td>180.778000</td>\n",
       "      <td>0.056699</td>\n",
       "      <td>0.951793</td>\n",
       "      <td>MakeModelQueryRetriever</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>hnsw</td>\n",
       "      <td>300</td>\n",
       "      <td>20</td>\n",
       "      <td>64</td>\n",
       "      <td>cosine</td>\n",
       "      <td>float32</td>\n",
       "      <td>sentence-transformers/all-MiniLM-L6-v2</td>\n",
       "      <td>384</td>\n",
       "      <td>0.476243</td>\n",
       "      <td>1.231741</td>\n",
       "      <td>315.208008</td>\n",
       "      <td>0.021955</td>\n",
       "      <td>0.927486</td>\n",
       "      <td>DefaultQueryRetriever</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>5</td>\n",
       "      <td>flat</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>cosine</td>\n",
       "      <td>float16</td>\n",
       "      <td>intfloat/e5-large-v2</td>\n",
       "      <td>1024</td>\n",
       "      <td>0.473093</td>\n",
       "      <td>27.423688</td>\n",
       "      <td>188.445999</td>\n",
       "      <td>0.030542</td>\n",
       "      <td>0.513553</td>\n",
       "      <td>MakeModelQueryRetriever</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2</td>\n",
       "      <td>hnsw</td>\n",
       "      <td>250</td>\n",
       "      <td>30</td>\n",
       "      <td>64</td>\n",
       "      <td>cosine</td>\n",
       "      <td>float32</td>\n",
       "      <td>intfloat/e5-large-v2</td>\n",
       "      <td>1024</td>\n",
       "      <td>0.458095</td>\n",
       "      <td>27.353979</td>\n",
       "      <td>471.066986</td>\n",
       "      <td>0.062210</td>\n",
       "      <td>0.495482</td>\n",
       "      <td>MakeModelQueryRetriever</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>2</td>\n",
       "      <td>hnsw</td>\n",
       "      <td>150</td>\n",
       "      <td>30</td>\n",
       "      <td>8</td>\n",
       "      <td>cosine</td>\n",
       "      <td>float32</td>\n",
       "      <td>intfloat/e5-large-v2</td>\n",
       "      <td>1024</td>\n",
       "      <td>0.458095</td>\n",
       "      <td>27.344018</td>\n",
       "      <td>384.832001</td>\n",
       "      <td>0.031524</td>\n",
       "      <td>0.495968</td>\n",
       "      <td>MakeModelQueryRetriever</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>flat</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>cosine</td>\n",
       "      <td>float16</td>\n",
       "      <td>sentence-transformers/all-MiniLM-L6-v2</td>\n",
       "      <td>384</td>\n",
       "      <td>0.421270</td>\n",
       "      <td>1.217438</td>\n",
       "      <td>156.332001</td>\n",
       "      <td>0.020650</td>\n",
       "      <td>0.878597</td>\n",
       "      <td>DefaultQueryRetriever</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>10</td>\n",
       "      <td>flat</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>cosine</td>\n",
       "      <td>float16</td>\n",
       "      <td>sentence-transformers/all-MiniLM-L6-v2</td>\n",
       "      <td>384</td>\n",
       "      <td>0.385997</td>\n",
       "      <td>1.241885</td>\n",
       "      <td>186.393997</td>\n",
       "      <td>0.032039</td>\n",
       "      <td>0.837387</td>\n",
       "      <td>MakeModelQueryRetriever</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>flat</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>cosine</td>\n",
       "      <td>float32</td>\n",
       "      <td>sentence-transformers/all-MiniLM-L6-v2</td>\n",
       "      <td>384</td>\n",
       "      <td>0.322222</td>\n",
       "      <td>1.220633</td>\n",
       "      <td>156.871002</td>\n",
       "      <td>0.035422</td>\n",
       "      <td>0.778879</td>\n",
       "      <td>DefaultQueryRetriever</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>3</td>\n",
       "      <td>hnsw</td>\n",
       "      <td>200</td>\n",
       "      <td>50</td>\n",
       "      <td>16</td>\n",
       "      <td>cosine</td>\n",
       "      <td>float32</td>\n",
       "      <td>intfloat/e5-large-v2</td>\n",
       "      <td>1024</td>\n",
       "      <td>0.299418</td>\n",
       "      <td>27.312038</td>\n",
       "      <td>506.636993</td>\n",
       "      <td>0.021990</td>\n",
       "      <td>0.336709</td>\n",
       "      <td>DefaultQueryRetriever</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>9</td>\n",
       "      <td>hnsw</td>\n",
       "      <td>150</td>\n",
       "      <td>10</td>\n",
       "      <td>8</td>\n",
       "      <td>cosine</td>\n",
       "      <td>float32</td>\n",
       "      <td>intfloat/e5-large-v2</td>\n",
       "      <td>1024</td>\n",
       "      <td>0.207364</td>\n",
       "      <td>27.687398</td>\n",
       "      <td>384.283997</td>\n",
       "      <td>0.022923</td>\n",
       "      <td>0.244818</td>\n",
       "      <td>DefaultQueryRetriever</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    ret_k algorithm  ef_construction  ef_runtime   m distance_metric  \\\n",
       "18      3      flat                0           0   0          cosine   \n",
       "9       2      flat                0           0   0          cosine   \n",
       "2       2      flat                0           0   0          cosine   \n",
       "12      4      flat                0           0   0          cosine   \n",
       "13      4      flat                0           0   0          cosine   \n",
       "17      4      flat                0           0   0          cosine   \n",
       "20      5      flat                0           0   0          cosine   \n",
       "0       5      hnsw              100          50  64          cosine   \n",
       "15      6      flat                0           0   0          cosine   \n",
       "14      1      flat                0           0   0          cosine   \n",
       "7       7      flat                0           0   0          cosine   \n",
       "6       7      flat                0           0   0          cosine   \n",
       "1       3      hnsw              300          20  64          cosine   \n",
       "11      5      flat                0           0   0          cosine   \n",
       "8       2      hnsw              250          30  64          cosine   \n",
       "10      2      hnsw              150          30   8          cosine   \n",
       "3       2      flat                0           0   0          cosine   \n",
       "16     10      flat                0           0   0          cosine   \n",
       "4       1      flat                0           0   0          cosine   \n",
       "19      3      hnsw              200          50  16          cosine   \n",
       "5       9      hnsw              150          10   8          cosine   \n",
       "\n",
       "   vector_data_type                                   model  model_dim  \\\n",
       "18          float32  sentence-transformers/all-MiniLM-L6-v2        384   \n",
       "9           float16  sentence-transformers/all-MiniLM-L6-v2        384   \n",
       "2           float32  sentence-transformers/all-MiniLM-L6-v2        384   \n",
       "12          float16  sentence-transformers/all-MiniLM-L6-v2        384   \n",
       "13          float16  sentence-transformers/all-MiniLM-L6-v2        384   \n",
       "17          float16  sentence-transformers/all-MiniLM-L6-v2        384   \n",
       "20          float32  sentence-transformers/all-MiniLM-L6-v2        384   \n",
       "0           float16  sentence-transformers/all-MiniLM-L6-v2        384   \n",
       "15          float32  sentence-transformers/all-MiniLM-L6-v2        384   \n",
       "14          float16  sentence-transformers/all-MiniLM-L6-v2        384   \n",
       "7           float32  sentence-transformers/all-MiniLM-L6-v2        384   \n",
       "6           float16  sentence-transformers/all-MiniLM-L6-v2        384   \n",
       "1           float32  sentence-transformers/all-MiniLM-L6-v2        384   \n",
       "11          float16                    intfloat/e5-large-v2       1024   \n",
       "8           float32                    intfloat/e5-large-v2       1024   \n",
       "10          float32                    intfloat/e5-large-v2       1024   \n",
       "3           float16  sentence-transformers/all-MiniLM-L6-v2        384   \n",
       "16          float16  sentence-transformers/all-MiniLM-L6-v2        384   \n",
       "4           float32  sentence-transformers/all-MiniLM-L6-v2        384   \n",
       "19          float32                    intfloat/e5-large-v2       1024   \n",
       "5           float32                    intfloat/e5-large-v2       1024   \n",
       "\n",
       "        f1@k  embedding_latency  indexing_time  avg_query_latency   obj_val  \\\n",
       "18  0.678254           1.274860     222.255997           0.037223  1.122321   \n",
       "9   0.655873           1.330631     187.475998           0.025454  1.090247   \n",
       "2   0.655873           1.217216     223.367004           0.025062  1.111346   \n",
       "12  0.603175           1.323604     172.893005           0.018080  1.039291   \n",
       "13  0.603175           1.265694     213.382996           0.038584  1.049205   \n",
       "17  0.603175           1.243994     282.058014           0.039360  1.052341   \n",
       "20  0.587484           1.261359     264.598999           0.027733  1.033461   \n",
       "0   0.587484           1.250739    1353.703003           0.042778  1.032521   \n",
       "15  0.559971           1.230524     207.845001           0.062395  1.013085   \n",
       "14  0.556825           1.230127     285.915009           0.038707  1.008716   \n",
       "7   0.502986           1.411159     160.619995           0.041576  0.923912   \n",
       "6   0.502986           1.255780     180.778000           0.056699  0.951793   \n",
       "1   0.476243           1.231741     315.208008           0.021955  0.927486   \n",
       "11  0.473093          27.423688     188.445999           0.030542  0.513553   \n",
       "8   0.458095          27.353979     471.066986           0.062210  0.495482   \n",
       "10  0.458095          27.344018     384.832001           0.031524  0.495968   \n",
       "3   0.421270           1.217438     156.332001           0.020650  0.878597   \n",
       "16  0.385997           1.241885     186.393997           0.032039  0.837387   \n",
       "4   0.322222           1.220633     156.871002           0.035422  0.778879   \n",
       "19  0.299418          27.312038     506.636993           0.021990  0.336709   \n",
       "5   0.207364          27.687398     384.283997           0.022923  0.244818   \n",
       "\n",
       "                  retriever  \n",
       "18  MakeModelQueryRetriever  \n",
       "9   MakeModelQueryRetriever  \n",
       "2   MakeModelQueryRetriever  \n",
       "12  MakeModelQueryRetriever  \n",
       "13  MakeModelQueryRetriever  \n",
       "17  MakeModelQueryRetriever  \n",
       "20  MakeModelQueryRetriever  \n",
       "0   MakeModelQueryRetriever  \n",
       "15  MakeModelQueryRetriever  \n",
       "14  MakeModelQueryRetriever  \n",
       "7   MakeModelQueryRetriever  \n",
       "6   MakeModelQueryRetriever  \n",
       "1     DefaultQueryRetriever  \n",
       "11  MakeModelQueryRetriever  \n",
       "8   MakeModelQueryRetriever  \n",
       "10  MakeModelQueryRetriever  \n",
       "3     DefaultQueryRetriever  \n",
       "16  MakeModelQueryRetriever  \n",
       "4     DefaultQueryRetriever  \n",
       "19    DefaultQueryRetriever  \n",
       "5     DefaultQueryRetriever  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res_df.sort_values([\"f1@k\", \"embedding_latency\", \"indexing_time\", \"avg_query_latency\"], ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>f1@k</th>\n",
       "      <th>avg_query_latency</th>\n",
       "      <th>indexing_time</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>retriever</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>DefaultQueryRetriever</th>\n",
       "      <td>0.345303</td>\n",
       "      <td>0.024588</td>\n",
       "      <td>303.8664</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MakeModelQueryRetriever</th>\n",
       "      <td>0.554534</td>\n",
       "      <td>0.038123</td>\n",
       "      <td>311.6020</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             f1@k  avg_query_latency  indexing_time\n",
       "retriever                                                          \n",
       "DefaultQueryRetriever    0.345303           0.024588       303.8664\n",
       "MakeModelQueryRetriever  0.554534           0.038123       311.6020"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res_df.groupby([\"retriever\"])[[\"f1@k\", \"avg_query_latency\", \"indexing_time\"]].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What did we accomplish?\n",
    "\n",
    "We have shown how a custom retriever can improve retrieval accuracy and now you can define your own custom retrievers to test against your data."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}