Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(dataobj): backfill sparse rows using EncodeN #16485

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

ashwanthgoli
Copy link
Contributor

What this PR does / why we need it:

We currently backfill old rows with nulls if they have no value for a given column. We do this by calling Encode(value) N times, but for sparse columns we have to iterate over very large numbers to perform the backfill. This PR adds EncodeN(value, n) method to bitmapEncoder to allow appending a value N times - this runs in constant time unlike the existing approach which takes O(n)

goos: darwin
goarch: arm64
pkg: github.com/grafana/loki/v3/pkg/dataobj/internal/dataset
cpu: Apple M1 Pro
Benchmark_bitmap_EncodeN/Encode_1000_times-8              227778              5262 ns/op
Benchmark_bitmap_EncodeN/EncodeN_1000_times-8           83439080                14.29 ns/op
Benchmark_bitmap_EncodeN/Encode_10000_times-8              22904             52238 ns/op
Benchmark_bitmap_EncodeN/EncodeN_10000_times-8          74150934                16.06 ns/op
Benchmark_bitmap_EncodeN/Encode_100000_times-8              2307            526224 ns/op
Benchmark_bitmap_EncodeN/EncodeN_100000_times-8         72333274                16.34 ns/op
Benchmark_bitmap_EncodeN/Encode_1000000_times-8              226           5229343 ns/op
Benchmark_bitmap_EncodeN/EncodeN_1000000_times-8        77011526                15.92 ns/op
PASS

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

@ashwanthgoli ashwanthgoli force-pushed the backfill-optimisations branch from 9ec951b to 04fbb39 Compare February 27, 2025 08:16
@ashwanthgoli ashwanthgoli marked this pull request as ready for review February 27, 2025 10:08
@ashwanthgoli ashwanthgoli requested a review from a team as a code owner February 27, 2025 10:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant