Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to Query Data Beyond 8 Days in Thanos Query #8057

Open
anvar-bidgely opened this issue Jan 14, 2025 · 5 comments
Open

Unable to Query Data Beyond 8 Days in Thanos Query #8057

anvar-bidgely opened this issue Jan 14, 2025 · 5 comments
Labels
bug component: query

Comments

@anvar-bidgely
Copy link

Description:

We are encountering an issue where querying data beyond 8 days is not possible, whereas, in the past, we were able to retrieve data for up to 20 days. Recently, queries for longer time periods (e.g., 20 days) are returning no data.

Problem Description:
Issue: When querying data for periods longer than 8 days, the data is not returned. The logs show that certain store addresses do not have data for the requested time period.

Previously Working Period: We were able to query data for up to 20 days, but now we are unable to retrieve data beyond 8 days.

Expected Behavior:
When querying data beyond 8 days (e.g., 20 days), the data should be returned, as it was previously working without issues.
Data should be available from the Thanos Store for the configured retention periods, specifically --retention.resolution-raw=7d, --retention.resolution-5m=20d and --retention.resolution-1h=60d.

Logs:
Here is a sample log from when querying for a longer period:

ts=2025-01-14T07:09:48.457343893Z caller=proxy.go:274 level=debug msg="Tenant info in Series()" tenant=default-tenant
ts=2025-01-14T07:09:48.457624482Z caller=proxy.go:320 level=debug component=proxy request="min_time:1736838286201 max_time:1736838586201 matchers:<name:\"resource\" value:\"memory\" > matchers:<name:\"__name__\" value:\"kube_pod_container_resource_limits\" > aggregates:MAX without_replica_labels:\"replica\" " msg="Series: started fanout streams" status="Store Addr: 10.2.99.41:10901 LabelSets: {cluster=\"prodca-eks\", replica=\"prometheus-0\"} MinTime: 1736229600010 MaxTime: 9223372036854775807 queried;Store Addr: 172.20.178.26:10901 LabelSets: {@thanos_compatibility_store_type=\"store\"},{cluster=\"prodca\"},{cluster=\"prodca-eks\", replica=\"prometheus-0\"},{cluster=\"prodca-eks\"} MinTime: 1733356800029 MaxTime: 1736834400000 filtered out due to: does not have data within this time period: [1736838286201,1736838586201]. Store time ranges: [1733356800029,1736834400000]" ```

Thanos Configuration:

Compactor config
`spec:
  containers:
  - args:
    - compact
    - --wait
    - --log.level=debug
    - --log.format=logfmt
    - --http-address=0.0.0.0:10902
    - --data-dir=/data
    - --objstore.config-file=/conf/objstore.yml
    - --retention.resolution-raw=7d
    - --retention.resolution-5m=20d
    - --retention.resolution-1h=60d
    - --delete-delay=48h
    - --compact.concurrency=1`

Store config 
`spec:
  containers:
  - args:
    - store
    - --log.level=info
    - --log.format=logfmt
    - --data-dir=/data
    - --grpc-address=0.0.0.0:10901
    - --http-address=0.0.0.0:10902
    - --objstore.config-file=/conf/objstore.yml
    - --index-cache.config-file=/etc/thanos/index-cache.yml`

Query config
`spec:
  containers:
  - args:
    - query
    - --grpc-address=0.0.0.0:10901
    - --http-address=0.0.0.0:10902
    - --log.level=debug
    - --log.format=logfmt
    - --query.replica-label=replica
    - --endpoint=dnssrv+_grpc._tcp.thanos-storegateway.monitoring.svc.cluster.local
    - --endpoint=dnssrv+_grpc._tcp.prometheus-grpc.monitoring.svc.cluster.local`
    
  Thanos bucket inpect details 
    
    `|            ULID            |         FROM         |        UNTIL         |     RANGE      |  UNTIL-DOWN   |  #SERIES  |   #SAMPLES    |  #CHUNKS   | COMP-LEVEL | COMP-FAILED |                 LABELS                  | RESOLUTION |  SOURCE   |
|----------------------------|----------------------|----------------------|----------------|---------------|-----------|---------------|------------|------------|-------------|-----------------------------------------|------------|-----------|
| 01JG6W7PBYBQE29ZY2SV7BHPK1 | 2024-12-05T00:00:00Z | 2024-12-19T00:00:00Z | 335h59m59.971s | -             | 6,628,008 | 21,200,589    | 6,679,637  | 5          | false       | cluster=prodca                          | 1h0m0s     | compactor |
| 01JH7D6W3DM2W6HB45GJ0XXNCW | 2024-12-23T00:00:00Z | 2024-12-27T00:00:00Z | 95h59m59.978s  | 144h0m0.022s  | 1,914,841 | 54,755,104    | 2,190,150  | 4          | false       | cluster=prodca-eks,replica=prometheus-0 | 5m0s       | compactor |
| 01JG8EEBHWSHNVA03XV9G1BT7H | 2024-12-27T00:00:00Z | 2024-12-29T00:00:00Z | 47h59m59.955s  | 192h0m0.045s  | 624,646   | 26,329,675    | 743,194    | 3          | false       | cluster=prodca-eks                      | 5m0s       | compactor |
| 01JGDK71B8SXH8QBQEDZ9PPA0C | 2024-12-29T00:00:00Z | 2024-12-31T00:00:00Z | 47h59m59.999s  | 192h0m0.001s  | 596,436   | 28,905,058    | 735,419    | 3          | false       | cluster=prodca-eks                      | 5m0s       | compactor |
| 01JGJH60GXTPWPKJXYQNWF31QP | 2024-12-31T00:00:00Z | 2025-01-02T00:00:00Z | 48h0m0s        | 192h0m0s      | 644,618   | 30,100,008    | 800,849    | 3          | false       | cluster=prodca-eks                      | 5m0s       | compactor |
| 01JGQP0SZJQX5SRTGE96YXGB79 | 2025-01-02T00:00:00Z | 2025-01-04T00:00:00Z | 48h0m0s        | 192h0m0s      | 1,056,772 | 33,962,252    | 1,207,725  | 3          | false       | cluster=prodca-eks                      | 5m0s       | compactor |
| 01JGX1RGNS91NBJEC34RQYGW84 | 2025-01-04T08:00:00Z | 2025-01-06T00:00:00Z | 39h59m59.979s  | 21ms          | 321,658   | 456,889,444   | 3,847,327  | 3          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | compactor |
| 01JGVAHAYVP2Q61VJ6AP0QD28P | 2025-01-05T00:00:00Z | 2025-01-05T08:00:00Z | 7h59m59.935s   | 32h0m0.065s   | 172,030   | 104,261,693   | 923,796    | 2          | false       | cluster=prodca-eks                      | 0s         | compactor |
| 01JH26X6QWCRZM1WBY0KYPC5M4 | 2025-01-06T00:00:00Z | 2025-01-08T00:00:00Z | 47h59m59.935s  | -7h59m59.935s | 1,935,561 | 946,347,896   | 8,888,867  | 3          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | compactor |
| 01JH27GNW2HJ5J0KQQZSKZ7T0B | 2025-01-06T00:00:00Z | 2025-01-08T00:00:00Z | 47h59m59.935s  | 192h0m0.065s  | 1,935,561 | 49,570,973    | 2,099,678  | 3          | false       | cluster=prodca-eks,replica=prometheus-0 | 5m0s       | compactor |
| 01JH7BM70S8TRN08YTQ6AFMXSB | 2025-01-08T00:00:00Z | 2025-01-10T00:00:00Z | 47h59m59.99s   | -7h59m59.99s  | 2,126,668 | 1,147,285,134 | 10,692,150 | 3          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | compactor |
| 01JH7C9WTKQS3JJ2859HMT7CXQ | 2025-01-08T00:00:00Z | 2025-01-10T00:00:00Z | 47h59m59.99s   | 192h0m0.01s   | 2,126,666 | 59,951,343    | 2,335,245  | 3          | false       | cluster=prodca-eks,replica=prometheus-0 | 5m0s       | compactor |
| 01JHCG6B2BRF5WMTBA475XCNVR | 2025-01-10T00:00:00Z | 2025-01-12T00:00:00Z | 47h59m59.934s  | -7h59m59.934s | 1,219,486 | 715,545,665   | 6,567,471  | 3          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | compactor |
| 01JHCGJJKBYP6RG8W8R1Q5YS4F | 2025-01-10T00:00:00Z | 2025-01-12T00:00:00Z | 47h59m59.934s  | 192h0m0.066s  | 1,219,486 | 37,235,038    | 1,387,933  | 3          | false       | cluster=prodca-eks,replica=prometheus-0 | 5m0s       | compactor |
| 01JHC78P9YBEXS0PPQC2G5H4G5 | 2025-01-12T00:00:00Z | 2025-01-12T02:00:00Z | 1h59m59.985s   | 38h0m0.015s   | 162,292   | 41,390,053    | 383,180    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHDBHHMW0A16S1S9Y0BJ8HQ6 | 2025-01-12T00:00:00Z | 2025-01-12T08:00:00Z | 7h59m59.985s   | 32h0m0.015s   | 195,786   | 113,704,103   | 1,020,671  | 2          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | compactor |
| 01JHHQMF97NYNY978WWWQN0RX9 | 2025-01-12T00:00:00Z | 2025-01-14T00:00:00Z | 47h59m59.985s  | -7h59m59.985s | 571,376   | 614,706,655   | 5,382,963  | 3          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | compactor |
| 01JHHQVCR6N4ECM5E3AP83JNQ6 | 2025-01-12T00:00:00Z | 2025-01-14T00:00:00Z | 47h59m59.985s  | 192h0m0.015s  | 571,376   | 31,498,242    | 748,108    | 3          | false       | cluster=prodca-eks,replica=prometheus-0 | 5m0s       | compactor |
| 01JHCE4675ED5Z4YYZRY5GC6KS | 2025-01-12T02:00:00Z | 2025-01-12T04:00:00Z | 2h0m0s         | 38h0m0s       | 103,441   | 26,332,744    | 253,988    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHCMZXWAD52CHT858M0DV7WB | 2025-01-12T04:00:00Z | 2025-01-12T06:00:00Z | 1h59m59.934s   | 38h0m0.066s   | 59,136    | 22,936,788    | 194,832    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHCVVMQ5DBV5V8MDF09P4JJV | 2025-01-12T06:00:00Z | 2025-01-12T08:00:00Z | 1h59m59.995s   | 38h0m0.005s   | 49,069    | 23,044,518    | 188,671    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHD2QK9ZR0WPFA68ZFZT141C | 2025-01-12T08:00:00Z | 2025-01-12T10:00:00Z | 1h59m59.99s    | 38h0m0.01s    | 63,640    | 22,408,792    | 192,189    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHE706THK6DWM3HHT84H217P | 2025-01-12T08:00:00Z | 2025-01-12T16:00:00Z | 7h59m59.99s    | 32h0m0.01s    | 84,690    | 91,202,032    | 765,609    | 2          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | compactor |
| 01JHD9K374PXFRWKZYZX1TVG18 | 2025-01-12T10:00:00Z | 2025-01-12T12:00:00Z | 1h59m59.946s   | 38h0m0.054s   | 63,909    | 22,747,575    | 193,971    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHDGETT9WE17XT9GVCFFNTWK | 2025-01-12T12:00:00Z | 2025-01-12T14:00:00Z | 1h59m59.934s   | 38h0m0.066s   | 50,605    | 22,857,154    | 189,234    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHDQAHQ96P6PFZMWH4J2FD8Q | 2025-01-12T14:00:00Z | 2025-01-12T16:00:00Z | 1h59m59.976s   | 38h0m0.024s   | 49,090    | 23,188,511    | 190,215    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHDY68Z54GN40J4C694GWFH1 | 2025-01-12T16:00:00Z | 2025-01-12T18:00:00Z | 1h59m59.938s   | 38h0m0.062s   | 49,136    | 23,324,741    | 191,524    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHF2F5CRDKJKPT74Q4FVWTNM | 2025-01-12T16:00:00Z | 2025-01-13T00:00:00Z | 7h59m59.938s   | 32h0m0.062s   | 94,304    | 96,208,337    | 813,410    | 2          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | compactor |
| 01JHE520J9Y3EQ5X0RKPPAV5JF | 2025-01-12T18:00:00Z | 2025-01-12T20:00:00Z | 1h59m59.934s   | 38h0m0.066s   | 49,265    | 23,376,510    | 191,940    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHEBXQF70Z4QH187QMMREBFE | 2025-01-12T20:00:00Z | 2025-01-12T22:00:00Z | 1h59m59.963s   | 38h0m0.037s   | 49,265    | 23,395,975    | 192,154    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHEJSEQ4Y2AAQACAB6BSP6BA | 2025-01-12T22:00:00Z | 2025-01-13T00:00:00Z | 1h59m59.992s   | 38h0m0.008s   | 94,304    | 26,111,111    | 237,792    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHESN5Z6CS0W2NS21MSYDX0B | 2025-01-13T00:00:00Z | 2025-01-13T02:00:00Z | 1h59m59.963s   | 38h0m0.037s   | 98,176    | 26,259,593    | 239,225    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHHQD5RCPAV640N6B15FNVRA | 2025-01-13T00:00:00Z | 2025-01-13T08:00:00Z | 7h59m59.963s   | 32h0m0.037s   | 101,311   | 97,561,048    | 824,592    | 2          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | compactor |
| 01JHF0GX7VCB0RYKTHD5YW2H5G | 2025-01-13T02:00:00Z | 2025-01-13T04:00:00Z | 1h59m59.985s   | 38h0m0.015s   | 49,387    | 23,260,290    | 191,109    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHF7CMW4KAN7V265MDGBSS22 | 2025-01-13T04:00:00Z | 2025-01-13T06:00:00Z | 1h59m59.934s   | 38h0m0.066s   | 50,794    | 23,754,762    | 195,046    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHFE8BQ9WTV30EDT62Q9R215 | 2025-01-13T06:00:00Z | 2025-01-13T08:00:00Z | 1h59m59.959s   | 38h0m0.041s   | 51,578    | 24,286,403    | 199,212    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHFN42Z66HNJYHNBKVZ5TDVS | 2025-01-13T08:00:00Z | 2025-01-13T10:00:00Z | 1h59m59.991s   | 38h0m0.009s   | 51,658    | 24,512,180    | 201,457    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHHQEJVVPF147CKVCF9S7VG1 | 2025-01-13T08:00:00Z | 2025-01-13T16:00:00Z | 7h59m59.991s   | 32h0m0.009s   | 82,230    | 100,981,255   | 849,631    | 2          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | compactor |
| 01JHFVZT79YHJDKCP6KWEKWV11 | 2025-01-13T10:00:00Z | 2025-01-13T12:00:00Z | 1h59m59.971s   | 38h0m0.029s   | 53,919    | 24,760,957    | 204,162    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHG2VHF5QMM65E3HG6Z3K4GD | 2025-01-13T12:00:00Z | 2025-01-13T14:00:00Z | 1h59m59.976s   | 38h0m0.024s   | 72,730    | 26,166,513    | 227,013    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHG9Q8Q9R1T64MK2M49T7DVB | 2025-01-13T14:00:00Z | 2025-01-13T16:00:00Z | 1h59m59.997s   | 38h0m0.003s   | 63,268    | 25,541,605    | 216,999    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHGGK0C203S9GK1DS57FV81H | 2025-01-13T16:00:00Z | 2025-01-13T18:00:00Z | 1h59m59.934s   | 38h0m0.066s   | 154,383   | 27,867,000    | 291,219    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHHQG9ET0975CD89RN2F87H3 | 2025-01-13T16:00:00Z | 2025-01-14T00:00:00Z | 7h59m59.934s   | 32h0m0.066s   | 293,444   | 115,049,880   | 1,109,050  | 2          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | compactor |
| 01JHGQEQM111YC2TZ06TZA01HW | 2025-01-13T18:00:00Z | 2025-01-13T20:00:00Z | 1h59m59.934s   | 38h0m0.066s   | 97,546    | 26,183,738    | 248,052    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHGYAEFDV039CKHNHD6KHEKG | 2025-01-13T20:00:00Z | 2025-01-13T22:00:00Z | 1h59m59.984s   | 38h0m0.016s   | 88,232    | 26,662,161    | 242,459    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHH565QEWD1G0ME6XZ5B72XY | 2025-01-13T22:00:00Z | 2025-01-14T00:00:00Z | 1h59m59.997s   | 38h0m0.003s   | 139,167   | 34,336,981    | 327,320    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHHC1X5DBKNKA717JCZV8FK3 | 2025-01-14T00:00:00Z | 2025-01-14T02:00:00Z | 1h59m59.934s   | 38h0m0.066s   | 186,164   | 36,857,951    | 369,486    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHHJXV26CEVF6E2T86X7RWJF | 2025-01-14T02:00:00Z | 2025-01-14T04:00:00Z | 1h59m59.988s   | 38h0m0.012s   | 276,029   | 43,967,456    | 430,291    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHHSSBF94E7FJ98ZC34R0QBD | 2025-01-14T04:00:00Z | 2025-01-14T06:00:00Z | 2h0m0s         | 38h0m0s       | 207,251   | 44,879,457    | 459,685    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
ts=2025-01-14T07:29:25.503887965Z caller=main.go:174 level=info msg=exiting`
@dosubot DosuBot
Copy link

dosubot bot commented Jan 14, 2025

Hey @anvar-bidgely! 👋 I'm here to help you with bug fixes, questions, and becoming a contributor.

The issue you're experiencing with querying data beyond 8 days in Thanos Query could be related to the retention settings and the availability of data in the Thanos Store. Here are a few points to consider:

  1. Retention Settings: Your current retention settings are --retention.resolution-raw=7d, --retention.resolution-5m=20d, and --retention.resolution-1h=60d. This means that raw data is only retained for 7 days. If you are querying for data beyond 8 days, the raw data will not be available, and you will need to rely on the downsampled data (5m or 1h resolution) [1].

  2. Data Availability: The logs indicate that certain store addresses do not have data for the requested time period. This could be due to the retention policy deleting raw data after 7 days, and if the data has not been downsampled to 5m or 1h resolution, it will not be available for queries beyond 8 days.

  3. Compactor Role: Ensure that the Thanos Compactor is running correctly and performing downsampling as expected. The compactor is responsible for downsampling data, which allows you to query longer time ranges efficiently [2].

  4. Bucket Inspection: From the bucket inspection details, verify that there are blocks available for the time range you are querying. If blocks for the desired time range are missing, it could indicate an issue with data retention or compaction.

  5. Configuration Check: Double-check your Thanos Store and Compactor configurations to ensure they are set up correctly to handle the retention and downsampling processes.

By reviewing these aspects, you should be able to identify why data beyond 8 days is not being returned and make the necessary adjustments to your Thanos setup.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@PaintedShepherd
Copy link

Same issue here. We are using a query with a fixed step-size of 2 minutes. when we set a time range of 30 days its searching in the 1h-blocks but the 2 min steps are obviously not able to fetch data. This can be proofed in thanos-query when we force the query to only use 1h or 5m blocks. When we force it to use raw-data (30s) it is able to draw the graph for 30 days.

So it seems like there is a communication problem between the querier and the store where the store is not able to choose the right resolution based on the query.

@yeya24
Copy link
Contributor

yeya24 commented Feb 23, 2025

Hey I think it might be a known issue with downsampled data. What kind of query were you using? If it has rate it is known to have this issue

@yeya24
Copy link
Contributor

yeya24 commented Feb 25, 2025

I think #7012 could help with this issue

@anvar-bidgely
Copy link
Author

anvar-bidgely commented Feb 26, 2025 •
edited
Loading

@yeya24 Yes, i was using rate function ...Also, when can i expect the new change to be released

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug component: query
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@PaintedShepherd @yeya24 @anvar-bidgely