Repository navigation
[0.11.x] Stop rejecting queries by an estimated result window and fix stream scan budget exhaustion - #1449
Merged
hanahmily merged 10 commits intoOct 30, 2026
Conversation
Without a memory limit the query budget uses a 64 MiB fallback pool, a query is assigned 8 MiB, and admission rejected any limit+offset above ~1792 with RESOURCE_EXHAUSTED before reading data, which broke OAP's default LIMIT 10000. Remove the estimate-based window rejection; the absolute window ceiling, pool exhaustion, the fixed overhead charge and all execution charging remain. Also cap the sidx heap merge preallocation sized by the request limit.
… as memory is released A stream query with a small limit failed with "query memory pool exhausted" once its scan read more than the per-query allowance, although it returns at most limit rows. The vectorized scan decoded every block of a segment before handing out the first batch and charged each block to the query lease without ever refunding it, and the sorted merge charged every input batch it later pruned. The budget lease gains Refund and query.Refund. The scan pulls one scanner result batch at a time and gives back a block's lease charge and soft reservation when its cursor is released. The sorted merge refunds a batch when its last retained row is pruned and the rest at Close. The block scanner counts each block once against its quota and resets the total for every batch. Fixes apache/skywalking#14165
…est image) (apache#1378) (cherry picked from commit 655f54e)
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## v0.11.x #1449 +/- ##
==========================================
Coverage ? 56.85%
==========================================
Files ? 764
Lines ? 114877
Branches ? 0
==========================================
Hits ? 65314
Misses ? 43657
Partials ? 5906
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Backport the CI part of apache#1427: pin runners to ubuntu-24.04 instead of ubuntu-latest and upgrade apache/skywalking-infra-e2e to c45db27. On the current ubuntu-latest image the old harness restarts Docker after the testing image is loaded, so the E2E suites fail to find it.
hanahmily
force-pushed
the
fix-query-window-estimate-0.11.x
branch
from
October 30, 2026 02:34
81ee12e to
0f92ee3
Compare
Backport of apache#1406 and apache#1425 for the security fixes govulncheck reports on Go 1.25.13 and x/net v0.58.0 (GO-2026-6809, GO-2026-6610..6613, GO-2026-6617); no Go 1.25 release carries them. The plugin build image moves to golang:1.26-bookworm so plugins keep matching the server's toolchain. scripts/build/version.mk now notes the toolchain, which also changes the key of CI's tool cache: tools built by Go 1.25 cannot load Go 1.26 packages.
hanahmily
force-pushed
the
fix-query-window-estimate-0.11.x
branch
from
October 30, 2026 04:10
0f92ee3 to
07d1e16
Compare
(cherry picked from commit fad0afa)
…pache#1349) Keep unbounded schema-registry init retries, but let file/DNS ListNode re-fetch retry-queued peers before discovery Start so liaison can recover when data becomes reachable without restarting. (cherry picked from commit 0f49ae0)
Backport of the controller part of apache#1377. Linux CPU affinity is per thread and SchedSetaffinity(0) pinned only the calling thread, so the controller process stayed unpinned whenever the pinning goroutine was not on the main thread, and TestMergeExternalControllerSmoke failed with the whole CPU set reported. Pin every thread in /proc/self/task until no new one appears.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A patch for 0.11.x so OAP can query BanyanDB without a memory limit. OAP sends
LIMIT 10000by default on reads that have no limit of their own (apache/skywalking#13961), and 0.11.1 rejects it on any host without--allowed-bytesor a container memory limit.What changed
Stop rejecting queries by an estimated result window. Without a memory limit the query budget uses its 64 MiB fallback pool and assigns each query 8 MiB. Admission then assumed every requested row costs 4 KiB and rejected
limit + offsetabove (8 MiB − 1 MiB) / 4 KiB = 1792 withRESOURCE_EXHAUSTED, before reading any data, soLIMIT 1793failed even on a stream holding 50 rows (whilelimit = MaxUint32was admitted as a scan). The estimate came in with fix(query): bound Property/Stream/Trace query allocations #1334 next to the protections that actually close the preallocation problem that PR fixed, and those stay: preallocation is capped at 1,024 entries,limit + offsetabove 100,000 is rejected as an invalid argument, and execution charges the blocks it decodes and the results it retains, so a query that really holds more than its allowance still fails. The heap merge in sidx, which still preallocated by the request limit, gets the same 1,024 cap.Fix stream queries failing with
query memory pool exhausted([Bug] BanyanDB stream scans run out of query memory even with a small limit skywalking#14165). Without the estimate,LIMIT 10000on a stream of any real size still failed, because the vectorized stream scan decoded every block of a segment before returning its first batch and charged each block without refunding it, and the sorted merge charged every input batch it pruned. The scan now takes one scanner batch at a time and refunds a block's charge when its cursor is released, the merge refunds a batch once its last retained row is pruned, and the block scanner counts each block once against its quota. This is a minimal backport of the fix in feat(stream): stream query results between liaison and data nodes, acknowledge writes per batch, fix the scan budget #1433; the transport and status changes of that PR are not included, and there are no proto or wire changes.Charge tag and field values by their size. The merge charged every retained tag or field value
proto.Size × 8 + 256, so a 1.5 MB value was charged 12 MB. OAP's ai-agent storage keeps whole session files as onebodytag, and reading back a single file exhausted the 8 MiB per-query allowance. A value is now charged its payload once plus a fixed 64-byte object overhead; one such row peaks at about 2.5 MB of charge.Size the default query pool from host memory. Without
--allowed-bytesor a valid container memory limit, the query pool was a fixed 64 MiB, so every query got 8 MiB regardless of the machine. It is now host memory ×--allowed-percent(75% by default) ÷ 4, and each query an eighth of that; 64 MiB stays the fallback when host memory cannot be read. The memory protector itself stays disabled, and--allowed-bytesor a container limit still take precedence.Backport the liaison start-up fix from fix(metadata): re-fetch discovery nodes before Start during PreRun #1349 (with Update schema registry client init retry strategy #1344's changelog line). With file or DNS discovery, a liaison that starts before its data nodes failed its first metadata fetch from them and never re-fetched it before start-up finished, so it waited forever for a property schema node (
no property schema nodes found among 0 meta nodes) and never opened its ports. This is whyfodc-dashboard-kindfailed on this branch; it is also a deadlock 0.11.1 users can hit. Discovery now re-fetches retry-queued nodes during start-up, as main does.Backport the external merge controller's CPU pinning from refactor(test): move trace-pipeline dynamic-lifecycle coverage to e2e #1377.
PinToCPUsset the affinity of the calling thread only, so the controller stayed unpinned whenever its goroutine was not on the main thread, andTestMergeExternalControllerSmokefailed intermittently. It now pins every thread, as on main.Verification
The FODC dashboard E2E (
test/e2e-v2/cases/fodc/e2e.yaml) run locally in kind: the liaison never became ready before the fix(metadata): re-fetch discovery nodes before Start during PreRun #1349 backport and all 3 cases pass after it.SkyWalking main's ai-agent E2E (
test/e2e-v2/cases/ai-agent/banyandb/e2e.yaml, OAP5689aa8) run locally against this branch:sizefailed withquery memory pool exhaustedbefore the last two fixes and passes with them.The load spec "3 days with 1 day segment and 3 days ttl" fails on
v0.11.xwithquery memory pool exhaustedand passes with this branch.Protector tests cover the fallback pool admitting
limit + offsetof 1,792, 1,793, 10,000 and 100,000, the 100,000 ceiling, and pool exhaustion. A new data-driven stream case runsLIMIT 10000.go test -racepasses for the protector, liaison gRPC, query, stream, trace, sidx andpkg/querypackages; the standalone and distributed query integration suites pass;make lintandmake checkpass.The window fix, the value charge and the host-sized pool go to main in #1450; the stream fix is in #1433 there.
CI on
v0.11.xThe branch's CI had gone stale since 0.11.1, so this PR also carries what main already has:
plugin-sidecar-kindjobs fail before running.ubuntu-24.04andapache/skywalking-infra-e2eupgraded toc45db27. On the currentubuntu-latestimage the old harness restarts Docker after the testing image is loaded, so every E2E suite fails to find it.golang.org/x/netv0.60.0 (with the matchingx/crypto,x/sys,x/text,x/term,x/mod,x/sync), becausegovulncheckreports GO-2026-6809, GO-2026-6610..6613 and GO-2026-6617 on Go 1.25.13 and x/net v0.58.0, and no Go 1.25 release carries the fixes. The plugin build image moves togolang:1.26-bookwormto keep matching the server. On this branchmake vuln-check, the race unit tests, the standalone and distributed query suites,make lintandmake checkpass with Go 1.26.10.