Improve use of tracing spans in query path

[hiltontj]

The purpose of this issue is to get a better understanding of how influxdb3 uses trace spans in the query path.

To better help diagnose and troubleshoot latency and performance issues in the query path, we may need to better leverage tracing spans in the query path, as what we have seen in results from the perf team, the information we are generating is sparse.

Objectives of this issue

1. analyze generated trace spans during query execution to see what information is being generated
2. determine which is being generated from iox_query and which is generated from this codebase
3. determine if there are any obvious gaps in that information
4. propose follow-up work

[MaduMitha-Ravi]

@hiltontj Here are the trace spans that are collected for IOx/InfluxDB 3 projects. These trace spans are customizable and they can be added/removed/updated/renamed per need. The IOx team at times for their works/projects create new or sub spans that can be analyzed for performance.

Note: Below is the exhaustive list, and most of these traces are not captured.

• IOx querier (ms)
• execute_stream_partitioned (ms)
• querier_table (ms)
• sql_to_logical_plan (ms)
• IOx ingester (ms)
• table_snapshot (ms)
• partition_snapshot (ms)
• initial_request (ms)
• stream_data_from_ingester (ms)
• querier_flight_request_to_one_ingester (ms)
• querier_table_chunks (ms)
• querier_ingesters (ms)
• prune_partitions (ms)
• get_namespace (ms)
• flight_planner (ms)
• planner (ms)
• get_metadata (ms)
• loader (ms)
• load_page_index (ms)
• load_metadata (ms)
• cache_metadata (ms)
• prune_files (ms)
• ParquetExec (ms)
• sql_to_physical_plan (ms)
• query_planning (ms)
• acquire (ms)
• permit (ms)
• create_physical_plan (ms)

[hiltontj]

Note: Below is the exhaustive list, and most of these traces are not captured.

Thanks @MaduMitha-Ravi - just to be clear, by "most of these traces are not captured", do you mean that monolith is not capturing them, while IOx is capturing them?

...or do you mean that they are not captured in general by both monolith and IOx?

[MaduMitha-Ravi]

@hiltontj Yes, most of those trace spans are captured in IOx but not with Monolith. I will try to get more details today.

[hiltontj]

Got it - that is very helpful. I can look into how IOx uses those spans and see how we can replicate. I think that would be a good starting point to help surface useful information in the tracing for your team.

Beyond that, we can start adding monolith-specific traces.

[MaduMitha-Ravi]

Here is the sample from IOx from a latest run that returns data,

Measurement	Query Name	IOx querier (ms)	execute_stream_partitioned (ms)	querier_table (ms)	sql_to_logical_plan (ms)	IOx ingester (ms)	table_snapshot (ms)	partition_snapshot (ms)	initial_request (ms)	stream_data_from_ingester (ms)	querier_flight_request_to_one_ingester (ms)	querier_table_chunks (ms)	querier_ingesters (ms)	prune_partitions (ms)	get_namespace (ms)	flight_planner (ms)	ParquetExec (ms)	sql_to_physical_plan (ms)	query_planning (ms)	acquire (ms)	permit (ms)	create_physical_plan (ms)
AVG	s-q1	85.183	60.245	1.058	1.414	7.559	1.011	0.790	5.252	11.278	16.584	18.905	16.640	0.101	1.407	21.808	58.870	21.953	82.721	0.005	60.197	20.454