Skip to content

Improve performance of segment metadata cache on Overlord #17785

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Mar 26, 2025

Conversation

kfaraz
Copy link
Contributor

@kfaraz kfaraz commented Mar 8, 2025

Description

#17653 introduces a cache for segment metadata on the Overlord.
This patch is a follow up to that to make the cache more robust, performant and debug-friendly.

Changes

  • Do not cache unused segments
    This significantly reduces sync time in cases where the cluster has a lot of unused segments.
    Unused segments are needed only during segment allocation to ensure that a duplicate ID is not allocated.
    This is a rare DB query which is supported by sufficient indexes and thus need not be cached at the moment.
  • Update cache directly when segments are marked as unused to avoid race conditions with DB sync.
  • Fix NPE when using segment metadata cache with concurrent locks.
  • Atomically update segment IDs and pending segments in a HeapMemoryDatasourceSegmentCache
    using methods syncSegmentIds() and syncPendingSegments() rather than updating one by one.
    This ensures that the locks are held for a shorter period and the update made to the cache is atomic.

Classes to review

  • IndexerMetadataStorageCoordinator
  • OverlordDataSourcesResource
  • HeapMemorySegmentMetadataCache
  • HeapMemoryDatasourceSegmentCache

Cleaner cache sync

In every sync, the following steps are performed for each datasource:

  • Retrieve ALL used segment IDs from metadata store
  • Atomically update segment IDs in cache and determine list of segment IDs which need to be refreshed.
  • Fetch payloads of segments that need to be refreshed
  • Atomically update fetched payloads into the cache
  • Fetch ALL pending segments
  • Atomically update pending segments into the cache
  • Clean up empty intervals from datasource caches

Testing

Verified the changes in this patch on the following clusters:

. Test cluster Staging cluster
Metadata store MySQL MySQL
Master node m5.large (2 vCPUs, 8 GB) m5.large (2 vCPUs, 8 GB)
Used segments 600k 120k
Unused segments 180k 1.5M
Pending segments 15k 1k
Old sync time 2500s 50-150s
New full sync time 850s 25s
New delta sync time 150s 5s

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@kfaraz kfaraz changed the title Use IndexerMetadataStorageCoordinator on Overlord to mark segments as unused Update cache on Overlord directly when marking segments as unused Mar 9, 2025
@kgyrtkirk kgyrtkirk closed this Mar 13, 2025
@kgyrtkirk kgyrtkirk reopened this Mar 13, 2025
@kfaraz kfaraz changed the title Update cache on Overlord directly when marking segments as unused Improve performance of segment metadata cache on Overlord Mar 17, 2025
@@ -66,19 +67,22 @@ public class OverlordDataSourcesResource
private static final Logger log = new Logger(OverlordDataSourcesResource.class);

private final SegmentsMetadataManager segmentsMetadataManager;
private final IndexerMetadataStorageCoordinator metadataStorageCoordinator;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just realized that the SegmentsMetadataManager is the Coordinator version of a metadata cache. Do you think we're moving towards getting rid of it completely on the Overlord side of things, in favor of using only IndexerMetadataStorageCoordinator?

It looks like after this patch, the SegmentsMetadataManager is still used in two places on the OL:

  • this class, for markAsUsedNonOvershadowedSegmentsInInterval, markAsUsedNonOvershadowedSegments, and markSegmentAsUsed.
  • in OverlordCompactionScheduler, for getSnapshotOfDataSourcesWithAllUsedSegments. to me it seems like this could already be replaced by IndexerSQLMetadataStorageCoordinator#retrieveAllUsedSegments. (although the SegmentsMetadataManager version would perform better, since it caches the snapshot, and the IndexerSQLMetadataStorageCoordinator version needs to build a timeline. I am not sure how much this matters.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, @gianm , I plan to get rid of SqlSegmentsMetadataManager from the Overlord completely.
Coordinator will continue to use it for the time being, but eventually we might just get rid of it altogether.

  • The usage of SqlSegmentsMetadataManager by OverlordCompactionScheduler does matter for performance because of the timeline and caching reasons as you guessed, but I should be able to resolve this soon.

@@ -167,4 +169,24 @@ public static SegmentId getValidSegmentId(String dataSource, String serializedSe
return parsedSegmentId;
}
}

/**
* Tries to parse the given serialized ID as {@link SegmentId}s of the given
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is suspicious to me, since it encourages ignoring seemingly-invalid segment IDs. Where are the places that we are encountering segment IDs that may not be valid? Is it ok to ignore them?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I guess it would be better to error out on invalid segment IDs.
The only place these can come from are REST APIs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.

}
}
catch (Exception e) {
log.noStackTrace().error(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When might this happen? Does it mean we will ignore segments that we can't read? That would be worrying, because Exception is very broad. Is there some kind of scenario you have in mind where a failure is possible?

Also, .noStackTrace().error is not a combination I expect to see. Typically error is reserved for serious problems, and for serious problems we should always have a stack trace.

Copy link
Contributor Author

@kfaraz kfaraz Mar 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When might this happen? Does it mean we will ignore segments that we can't read? That would be worrying, because Exception is very broad. Is there some kind of scenario you have in mind where a failure is possible?

This can happen in the rare case when the segment payload has been tampered or some other column was not parseable. It is not frequent but it can happen, as I only recently encountered this in a prod DB.

We are not throwing an error here so that the processing can continue with the rest of the segments.
Even though this segment is ignored here, it actually increments a skippedCount in the code where this class is used and we raise an alert if skippedCount > 0. The log here is just to allow the operator to go through Overlord logs and see which segment IDs failed and why.

Do you think it would be better to create an alert for each failing segment ID separately? (similar to SqlSegmentsMetadataManager
I felt it would be too noisy, so alerted at total count level instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, .noStackTrace().error is not a combination I expect to see. Typically error is reserved for serious problems, and for serious problems we should always have a stack trace.

Yeah, I see your point. I will include the stack trace here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think it would be better to create an alert for each failing segment ID separately? (similar to SqlSegmentsMetadataManager
I felt it would be too noisy, so alerted at total count level instead.

I think the way you did it here is OK. It's good that there is an alert, and IMO an alert with a number of bad segments is ok.

@kgyrtkirk kgyrtkirk added this to the 33.0.0 milestone Mar 26, 2025
@kfaraz
Copy link
Contributor Author

kfaraz commented Mar 26, 2025

Verified the changes in this patch on the following clusters:

. Test cluster Staging cluster
Metadata store MySQL MySQL
Master node m5.large (2 vCPUs, 8 GB) m5.large (2 vCPUs, 8 GB)
Used segments 600k 120k
Unused segments 180k 1.5M
Pending segments 15k 1k
Old sync time 2500s 50-150s
New full sync time 850s 25s
New delta sync time 150s 5s

@cryptoe
Copy link
Contributor

cryptoe commented Mar 26, 2025

@kfaraz
It looks like the new full sync time goes from
25 seconds for 120K used segments
to
850 seconds for 600K used segments .
Could you provide some clarifications around the non linear time increase.

Also could you provide some description around what is the delta sync?

@kfaraz
Copy link
Contributor Author

kfaraz commented Mar 26, 2025

@cryptoe , those are two separate clusters and the results are not really comparable between the two since the underlying metadata store is different.

The comparison is to be done between the old sync time and the new sync time for any given cluster.

Also, the 850s for 600k segments is a little pessimistic, since a real production cluster with say 1M segments would have a much beefier metadata store and master node.
Based on what I have seen with the metric segment/poll/time on production clusters with 2M segments or more,
a full sync would take like 3 minutes and a delta sync would be much less than that (exactly how much less could vary).

Full sync: fetch all used segment payloads
Delta sync: fetch all used segment IDs, fetch only updated segment payloads

Copy link
Contributor

@gianm gianm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kfaraz
Copy link
Contributor Author

kfaraz commented Mar 26, 2025

Thanks for the review, @gianm !

@kfaraz kfaraz merged commit c0cc27c into apache:master Mar 26, 2025
75 checks passed
@kfaraz kfaraz deleted the mark_unused_via_cache branch March 26, 2025 23:38
cecemei pushed a commit to cecemei/druid that referenced this pull request Mar 31, 2025
Description
-----------
apache#17653 introduces a cache for segment metadata on the Overlord.
This patch is a follow up to that to make the cache more robust, performant and debug-friendly.

Changes
---------
- Do not cache unused segments
This significantly reduces sync time in cases where the cluster has a lot of unused segments.
Unused segments are needed only during segment allocation to ensure that a duplicate ID is not allocated.
This is a rare DB query which is supported by sufficient indexes and thus need not be cached at the moment.
- Update cache directly when segments are marked as unused to avoid race conditions with DB sync.
- Fix NPE when using segment metadata cache with concurrent locks.
- Atomically update segment IDs and pending segments in a `HeapMemoryDatasourceSegmentCache`
using methods `syncSegmentIds()` and `syncPendingSegments()` rather than updating one by one.
This ensures that the locks are held for a shorter period and the update made to the cache is atomic.

Main updated classes
----------------------
- `IndexerMetadataStorageCoordinator`
- `OverlordDataSourcesResource`
- `HeapMemorySegmentMetadataCache`
- `HeapMemoryDatasourceSegmentCache`

Cleaner cache sync
--------------------
In every sync, the following steps are performed for each datasource:

- Retrieve ALL used segment IDs from metadata store
- Atomically update segment IDs in cache and determine list of segment IDs which need to be refreshed.
- Fetch payloads of segments that need to be refreshed
- Atomically update fetched payloads into the cache
- Fetch ALL pending segments
- Atomically update pending segments into the cache
- Clean up empty intervals from datasource caches
gianm added a commit that referenced this pull request Apr 25, 2025
)

* Some debug configs

* use postgresql as the default metadata store and set a few debug log

* Add s3 extension, update local storage directory, use emoji in website title

* Update favicon, easier to find the console tab

* Add indexer server, add some basic security config, updated historical and broker to use the common druid root directory

* Some policy config

* add checks for SegmentMetadataQuery

* Add thread.sleep for flaky.

* auth config

* format, and remove temp folder rules

* added NoopPolicyEnforcer and RestrictAllTablesPolicyEnforcer class

* Support pushing and streaming task payload for HDFS (#17742)

Implement pushTaskPayload/streamTaskPayload as introduced in #14887
for HDFS storage to allow larger mm-less ingestion payloads when using
HDFS as the deep storage location.

* Remove usages of deprecated API Files.write() (#17761)

* Add deprecated com.google.common.io.Files#write to forbiddenApis

* Replace deprecated Files.write()

* Doc: Fix description typo for sqlserver metadata store (#17771)

Mistakenly categories under deep storage instead of metadata store.

* Fix binding of segment metadata cache on CliOverlord (#17772)

Changes
---------
- Bind `SegmentMetadataCache` only once to
`HeapMemorySegmentMetadataCache` in `SQLMetadataStorageDruidModule`
- Invoke start and stop of the cache from `DruidOverlord` rather than on lifecycle start/stop
- Do not override the binding in `CliOverlord`

* Docs: Remove semicolon from example (#17759)

* Restrict segment metadata kill query till maxInterval from last kill task time (#17770)

Changes
---------
- Use `maxIntervalToKill` to determine search interval for killing unused segments.
- If no segment has been killed for the datasource yet, use durationToRetain

* Update the Supervisor endpoint to not restart the Supervisor if the spec was unmodified (#17707)

Add an optional query parameter called skipRestartIfUnmodified to the
/druid/indexer/v1/supervisor endpoint. Callers can set skipRestartIfUnmodified=true
to not restart the supervisor if the spec is unchanged.

Example:

curl -X POST --header "Content-Type: application/json" -d @supervisor.json
localhost:8888/druid/indexer/v1/supervisor?skipRestartIfUnmodified=true

* Reduce noisy coordinator logs (#17779)

* Emit time lag from Kafka supervisor (#17735)

Changes
---------
- Emit time lag from Kafka similar to Kinesis as metrics `ingest/kafka/lag/time`,
`ingest/kafka/maxLag/time`, `ingest/kafka/avgLag/time`
- Add new method in `KafkaSupervisor` to fetch timestamps of latest records in stream to compute time lag
- Add new field `emitTimeLagMetrics` in `KafkaSupervisorIOConfig` to toggle emission of new metrics

* fix processed row formatting (#17756)

* Web console: add suggestions for table status filtering. (#17765)

* suggest filter values when known

* update snapshots

* add more d

* fix load rule clamp

* better segment timeline init

* Remove all usages of skife config (#17776)


Changes
---------
- Usages of skife config had been deprecated in #14695 and
`LegacyBrokerParallelMergeConfig` is the last config class that still uses it.
- Remove `org.skife.config` from pom, licenses, log4j2.xml, etc.
- Add validation for deleted property paths in `StartupInjectorBuilder.PropertiesValidator`
- Use the replacement flattened configs (which remove the `.task` and `.pool` substring)

* Add field `taskLimits` to worker select strategies (#16889)

Changes
---------
- Add field `taskLimits` to the following worker select strategies
`equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacityWithCategorySpec`, `fillCapacity`
- Add sub-fields `maxSlotCountByType` and `maxSlotRatioByType` to `taskLimits`
- Apply these limits per worker when assigning new tasks

---------
Co-authored-by: sviatahorau <[email protected]>
Co-authored-by: Benedict Jin <[email protected]>
Co-authored-by: Kashif Faraz <[email protected]>

* remove NullValueHandlingConfig, NullHandlingModule, NullHandling (#17778)

* Docs: Add SQL query example (#17593)

* Docs: Add query example

* Update after review

* Update query

* Update docs/api-reference/sql-api.md

---------

Co-authored-by: Victoria Lim <[email protected]>

* More logging cleanup on Overlord (#17780)

* Remove maven.twttr repo from pom (#17797)

remove usage of dependency:go-offline from build scripts - as it tries to download excluded artifacts

---------

Co-authored-by: Zoltan Haindrich <[email protected]>

* fix bug (#17791)

* Log query stack traces for DEVELOPER and OPERATOR personas. (#17790)

Currently, query stack traces are logged only when "debug: true" is set
in the query context. This patch additionally logs stack traces targeted
at the DEVELOPER or OPERATOR personas, because for these personas, stack
traces are useful more often than not.

We continue to omit stack traces by default for USER and ADMIN, because
these personas are meant to interact with the API, not with code or logs.
Skipping stack traces minimizes clutter in the logs.

* Set useMaxMemoryEstimates=false for MSQ tasks (#17792)

* Web console: fix go to task selecting correct task type (#17788)

* fix go to task selecting correct task type

* support autocompact also

* support scheduled_batch, refactor

* one more state and update tests

* Enable ComponentSuppliers to run queries using Dart (#17787)



Enables Calcite*Test-s and quidem tests to run queries with Dart.

needed some minor tweaks:

    changed to use interfaces at some places
    renamed DartWorkerClient to DartWorkerClientImpl and made DartWorkerClient an interface
    reused existing parts of the MSQ test system to run the query

* Fix single container config creates failing peon tasks (#17794)

* Fix single container config creates failing peon tasks

* More obvious array error output

* Update `k8s-jobs.md` reference (#17805)

Signed-off-by: Emmanuel Ferdman <[email protected]>

* Footer Copyright Year Update (#17751)

* Update docusaurus.config.js

* Update docusaurus.config.js

* [Revert] Reduce number of metadata transaction retries (#17808)

* Revert "Run JDK 21 workflows with latest JDK. (#17694)" (#17806)

* Revert "Run JDK 21 workflows with latest JDK. (#17694)"

This reverts commit 31ede5c

* Review comments.

* Review comments.

* Revert "reject publishing actions with a retriable error code if a earlier task is still publishing (#17509)"

This reverts commit aca56d6.

* Fix unstable tests after #17787 and dart usage in quidem-ut (#17814)

* fixes

* fix cleanup

* Use "mix" shuffle spec for target size with nil clusterBy. (#17810)

When a nil clusterBy is used, we have no way of achieving a particular
target size, so we need to fall back to a "mix" spec (unsorted single
partition).

This comes up for queries like "SELECT COUNT(*) FROM FOO LIMIT 1" when
results use a target size, such as when we are inserting into another
table or when we are writing to durable storage.

* Docs: Recommend using runtime property javaOptsArray instead of javaOpts

* Add minor checks in jetty utils (#17817)

Add minor checks in jetty utils class

* CI improvement: Leverage cancelled() instead of always() for CI jobs (#17819)

* Make MSQ tests use the same datasets as other similar tests (#17818)

MSQ tests had their own way of creating the segments/etc - this have lead to that custom datasets didn't worked with them.
This patch alters a few things to make it possible to access CompleteSegment for the active segments - which fixed the issue and also enabled the removal of the extra loading codes.

* Add unnest tests to quidem (#17825)

This PR adds the sql-native unnest tests to quidem. This set of tests has 6392 queries in total, with 5247 positive tests and 1145 negative tests.

* Web console: show loader on aux queries (#17804)

* show loader on aux queries

* show supervisors if not on page 0

* refactor

* fix bug fetching data when columns are added or removed

* update test

* Use compaction dynamic config to enable compaction supervisors (#17782)

Changes
---------
- Remove runtime property object `CompactionSupervisorConfig`
- Add fields `useSupervisors` and `engine` to cluster-level compaction dynamic config
- Remove unused field `useAutoScaleSlots`

* Retry segment publish task actions without holding locks (#17816)

#17802 reverted a retry of failed segment publish actions.

This patch attempts to address the original issue by retrying the segment publish task actions
on the client (i.e. task) side without holding any locks so that other transactions are not blocked.
Changes

    Add retries to TransactionalSegmentPublisher
    Add field retryable to SegmentPublishResult
    Remove class DataStoreMetadataUpdateResult and use SegmentPublishResult instead

* Add the capability to turboload segments onto historicals (#17775)

Add the capability to set Historicals into a turbo loading mode,
to focus on loading segments at the cost of query performance.

Context
--------
Currently, when a new Historical is started, it initially starts out using a bootstrap thread pool.
It uses this thread pool to load any existing cached segments and broadcast segments.
Once it loads any segments from both these sources, the historical switches to a smaller thread-pool
and begins to serve queries.

In certain cases, it would be useful to have the historical switch back to this mode,
and focus on loading segments, either to continue loading the initial non-bootstrap segments,
or to catch up with assigned segments.

This PR adds a coordinator dynamic config that allows servers to be configured to use
the larger bootstrap threadpool to load segments faster.

Changes
---------
- Added a new dynamic coordinator configuration, `turboLoadingNodes`.
- Ignore  `druid.coordinator.loadqueuepeon.http.batchSize` for servers in `turboLoadingNodes`
- Add API on historical to return loading capabilities i.e. num loading threads in normal and turbo mode

* Fix resource leak for GroupBy query merge buffer when query matched result cache (#17823)

* Fix resource leak for GroupBy query merge buffer when match result cache

* Fix resource leak for GroupBy query merge buffer when match result cache

* Add test

* Add test

* Add comment

* Add test

* Add metric and simulation test for turbo loading mode (#17830)

Changes
---------
- Add field `loadingMode` to `SegmentChangeStatus`
- Including loading mode in `DataSegmentChangeResponse`
- Include loading mode in the `description` of metrics emitted from `HttpLoadQueuePeon`
- Add simulation test to verify loading mode metrics

* Update query example (#17811)

* String util upgrade for jdk9+ (#17795)

* Update StringUtils.replace() after fix in JDK9

* Upgrade optimized string replace algorithm

* Update methods by re-using declared StringUtils#replace method

* Replace hard-coded UTF-8 encodings with StandardCharsets

* Documentation Fix (#17826)

* Enable to run quidem tests against multiple configurations; add conditionals; cleanup framework init (#17829)

* cleans up `SqlTestFramework` initialization to leave the `OverrideModule` empty - so that tests could more easily take over parts
* remove the `QueryComponentSupplier#createEngine`  factory method - instead uses a `Class<SqlEngine>` and use the `injector` to initialize it
* enables the usage of `!disabled <supplier> <message>` - to mark cases which are not yet supported with a specific configuration for some reason
* fixes that `datasets` was not respecting the `rollup` specification of the ingest
* enables to use `MultiComponentSupplier` backed tests - these will turn into matrix tests over multiple componentsuppliers - enabling running the same testcase in different scenarios

* Fix failing test in DimensionSchemaUtilsTest (#17832)

* Improve performance of segment metadata cache on Overlord (#17785)

Description
-----------
#17653 introduces a cache for segment metadata on the Overlord.
This patch is a follow up to that to make the cache more robust, performant and debug-friendly.

Changes
---------
- Do not cache unused segments
This significantly reduces sync time in cases where the cluster has a lot of unused segments.
Unused segments are needed only during segment allocation to ensure that a duplicate ID is not allocated.
This is a rare DB query which is supported by sufficient indexes and thus need not be cached at the moment.
- Update cache directly when segments are marked as unused to avoid race conditions with DB sync.
- Fix NPE when using segment metadata cache with concurrent locks.
- Atomically update segment IDs and pending segments in a `HeapMemoryDatasourceSegmentCache`
using methods `syncSegmentIds()` and `syncPendingSegments()` rather than updating one by one.
This ensures that the locks are held for a shorter period and the update made to the cache is atomic.

Main updated classes
----------------------
- `IndexerMetadataStorageCoordinator`
- `OverlordDataSourcesResource`
- `HeapMemorySegmentMetadataCache`
- `HeapMemoryDatasourceSegmentCache`

Cleaner cache sync
--------------------
In every sync, the following steps are performed for each datasource:

- Retrieve ALL used segment IDs from metadata store
- Atomically update segment IDs in cache and determine list of segment IDs which need to be refreshed.
- Fetch payloads of segments that need to be refreshed
- Atomically update fetched payloads into the cache
- Fetch ALL pending segments
- Atomically update pending segments into the cache
- Clean up empty intervals from datasource caches

* GroupBy: Fix offsets on outer queries. (#17837)

Prior to this patch, an offset specified on a groupBy that itself has an
inner groupBy would lead to an error like "Cannot push down offsets". This
happened because of a violated assumption: the processing logic assumes that
offsets have been pushed into limits (so limit pushdown optimizations can
safely be used).

This patch adjusts processing to incorporate offsets into limits during
processing of subqueries. Later on, in post-processing, offsets are applied
as written.

* Enable build cache for web-console (#17831)

* run audit fix (#17836)

* Do not block task actions on Overlord if segment metadata cache is syncing (#17824)

* Do not use segment metadata cache until leader has synced

* Read from cache only when synced, but write even if sync is pending

* Fix compilation

* Fix checkstyle, test

* Revert some extra changes

* Add 3 modes of cache usage

* Move enum to SegmentMetadataCache

* Run tests in all 3 cache modes

* Fix docs and IT configs

* Fix config binding

* Remove forbidden api

* Fix typos, docs and enum casing

* Fix doc

* Add json, array, aggregation function tests to quidem (#17842)

This PR adds the sql-native portion of the json, array, and aggregation function tests to quidem.  It adds a total of 9965 queries, with 6752 positive tests and 3213 negative tests.

* Optionally include Content-Disposition header in statement results API response (#17840)

Adds support for an optional filename query parameter to the /druid/v2/sql/statements/{queryId}/results API. When provided, the response will include a header Content-Disposition: attachment; filename="{filename}", which will instruct a web browser to save the response as a file rather than displaying it inline.

This save-as-attachment behavior could be achieved by adding a "download" attribute to the results link, but this only works for same-origin URLs (as in the Web Console). If the UI origin is different from the Druid API origin, browsers will ignore the attribute and serve the results inline, which is poor UX for files that are potentially very large.

For the sake of consistency, all successful responses in SqlStatementResource.doGetResults may include this header, even if there are no results.
Release note

Improved: The "Get query results" statements API supports an optional filename query parameter. When provided, the response will instruct web browsers to save the results as a file instead of showing them inline (via the Content-Disposition header).

* Web console: download follow up (#17845)

* set filename

* update download button

* added markdown support

* add test

* better download

* fix TSV

* better download behaviour and tests

* always show download all button

* Fix flaky unit tests in SegmentBootstrapperTest and KinesisIndexTaskTest (#17841)

Changes:
- Fix flakiness in SegmentBootstrapperTest
- Make TestSegmentCacheManager thread safe by moving from ArrayList to CopyOnWriteArrayList
- Modify assertions to disregard list ordering since order of list modifications is not always deterministic
- Fix flaky KinesisIndexTask tests.

* Web console: responding to user feedback about the explore view and fixing bugs (#17844)

* better debounce

* better cumpose filter

* hook up preview filters

* better stack handling

* fix some props

* refactor stack to facet

* fix hover part 1

* line hover part 2

* start adding moduleWhere

* info popover

* add filter icon

* toggle button

* module filter bar

* update TestSegmentCacheManager

* revert some style changes

* validate datasource in CachingClusteredClient as well

* fix build failure and update style

* changes

* add inlineds test

* add sanity check on segment

* inject policy enforcer

* add PolicyEnforcer binding in MSQTestBase

* add check in SinkQuerySegmentWalker

* more tests in realtime server

* revert config change in examples

* revert config change in integration test config

* more tests in msq

* another test for unnest in msq

* add support for policy from extension

* more test

* refactor MSQTaskQueryMakerTest to use an instance of MSQTaskQueryMaker

* Add test for JoinDataSource

* add policyEnforcer to withPolicies, and validate segment after segment mapping

* fix binding and test

* add policy module

* mock planner toolbox

* revert some injection

* add test for stream appenderator

* update PolicyEnforcer to take ReferenceCountingSegment as param

* update to QueryLifecycleTest

* update to SqlTestFramework

* pass enforcer to BroadcastJoinSegmentMapFnProcessor and add test. PolicyEnforcer should also deal with multiple layer wrapped segments/

* ReferenceCountingSegment is not allowed to wrap with a SegmentReference, and PolicyEnforcer now validates all segments, remove test cases for inline/lookup.

* moving ReferenceCountingSegment to another pr

* Revert "Merge remote-tracking branch 'cecemei/debug' into policy"

This reverts commit 25ffb7c, reversing
changes made to 1e6632f.

---------

Signed-off-by: Emmanuel Ferdman <[email protected]>
Co-authored-by: Virushade <[email protected]>
Co-authored-by: Eyal Yurman <[email protected]>
Co-authored-by: Kashif Faraz <[email protected]>
Co-authored-by: Frank Chen <[email protected]>
Co-authored-by: Chetan Patidar <[email protected]>
Co-authored-by: aho135 <[email protected]>
Co-authored-by: Adithya Chakilam <[email protected]>
Co-authored-by: Vadim Ogievetsky <[email protected]>
Co-authored-by: Misha <[email protected]>
Co-authored-by: sviatahorau <[email protected]>
Co-authored-by: Benedict Jin <[email protected]>
Co-authored-by: Clint Wylie <[email protected]>
Co-authored-by: Katya Macedo <[email protected]>
Co-authored-by: Victoria Lim <[email protected]>
Co-authored-by: Zoltan Haindrich <[email protected]>
Co-authored-by: Gian Merlino <[email protected]>
Co-authored-by: Emmanuel Ferdman <[email protected]>
Co-authored-by: Om Kenge <[email protected]>
Co-authored-by: Karan Kumar <[email protected]>
Co-authored-by: Lars Francke <[email protected]>
Co-authored-by: Adarsh Sanjeev <[email protected]>
Co-authored-by: Akshat Jain <[email protected]>
Co-authored-by: Andy Tsai <[email protected]>
Co-authored-by: Maytas Monsereenusorn <[email protected]>
Co-authored-by: jtuglu-netflix <[email protected]>
Co-authored-by: Lucas Capistrant <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants