Reduce query parameter size for sqlite

[agoscinski]

Sqlite only accepts a limited number of parameters so we reduce the query into chunks
Fix #6876

TODO:

• determine max supported size for parameters pass to chunks

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.61%. Comparing base (5168032) to head (4dc2260).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6889      +/-   ##
==========================================
+ Coverage   78.59%   78.61%   +0.02%     
==========================================
  Files         567      567              
  Lines       43092    43117      +25     
==========================================
+ Hits        33865    33892      +27     
+ Misses       9227     9225       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[giovannipizzi]

I suggest to put a batch of ~950 (in any case < 999), see my discussion here (good to maybe copy the discussion also here):

https://github.com/aiidateam/disk-objectstore/blob/73f14e3953c276268b7f7395ad385e693798e247/disk_objectstore/container.py#L97-L107

[agoscinski]

I suggest to put a batch of ~950 (in any case < 999), see my discussion here (good to maybe copy the discussion also here):

This was quick-and-dirty to try out. My idea was to determine the limit beforehand and use this. This script I got from ChatGPT but did not verified it yet.

import sqlite3

conn = sqlite3.connect(":memory:")
cur = conn.cursor()

# Try increasing numbers until an error occurs
max_vars = 1
try:
    while True:
        placeholders = ",".join(["?"] * max_vars)
        query = f"SELECT {placeholders}"
        cur.execute(query, [None] * max_vars)
        max_vars += 1
except sqlite3.OperationalError as e:
    print(f"Maximum number of variables supported: {max_vars - 1}")

EDIT: The script seems not really nice. I think the global variable with the doc that @giovannipizzi posted is nicer solution.

EDIT2: There is the filter_size: int = 999, parameter in our code already implementing this kind of logic. Need to check if we can unify.