docs: Add documentation for search query flow. #1326
Conversation
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the ✨ Finishing touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
|
||
| Upon receiving the query, the server: | ||
|
|
||
| 1. **Creates two jobs in MySQL database**: |
There was a problem hiding this comment.
TODO: mention name of the query job table
| - Collection named after `searchJobId` | ||
| - Collection named after `aggregationJobId` |
There was a problem hiding this comment.
| - Collection named after `searchJobId` | |
| - Collection named after `aggregationJobId` | |
| - Collection named after `searchJobId`, for receiving log event results | |
| - Collection named after `aggregationJobId`, for receiving time-based aggregation results |
|
|
||
| ### 4. Result caching in MongoDB | ||
|
|
||
| Search results are stored in MongoDB with the following characteristics: |
There was a problem hiding this comment.
(Personally, I would like more description here about how the results are handled. Below is my attempt, and feel free to correct/refine it.)
MongoDB gathers the results from the distributed search and caches them for future use (e.g., subsequent user queries). After the workers finish their search tasks, they write the results into a MongoDB collection.
|
|
||
| Search results are stored in MongoDB with the following characteristics: | ||
|
|
||
| 1. **Collection structure**: Each job has its own collection named after its job ID |
There was a problem hiding this comment.
- Collection name: Each job has its own collection named after its job ID (see bullet 2 of 2. Server-side processing)
| The storage and query engine settings are configured in the package's `etc/clp-config.yml` and | ||
| they are passed to the WebUI client via `client/public/settings.json` and the server via | ||
| `server/settings.json` at startup. | ||
|
|
There was a problem hiding this comment.
Maybe list the major databases and table/collection structures here. They are mentioned several times throughout the document, so it's helpful to give readers a quick preview.
## Major Data Structures
1. A `query_jobs` table in MariaDB/MySQL tracks the search job status.
- `id`: The ID of the query job.
- `status`: The status of the query job.
- `job_config`: The configuration of the query job.
2. The `clp-query-results` database in MongoDB caches the search results.
...
3. The `results-metadata` database in MongoDB tracks ...
Description
Checklist
breaking change.
Validation performed
task docs:serveand viewed http://127.0.0.1:8080/dev-docs/design-search-query-flow.html to confirm the document was rendered as expected.