Skip to content

Update redisearch base client to include timeout. Minor improvements on general latency and exported metrics #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 222 commits into
base: master
Choose a base branch
from
Open

Update redisearch base client to include timeout. Minor improvements on general latency and exported metrics #1

wants to merge 222 commits into from

Conversation

filipecosta90
Copy link
Collaborator

No description provided.

filipecosta90 and others added 30 commits July 4, 2023 17:56
@filipecosta90
Update redisearch base client to include timeout. Extended latency me…
17ef49e 
…trics to include p50
@filipecosta90
Included batch_size and parallel info in upload results
1e8e719
@filipecosta90
Enable passing env variables for url and api_key qdrant parameters
084a68a
@filipecosta90
Print ef info if available in config
d890fa1
@filipecosta90
waiting on full index before return qdrant
dcdfe21
@filipecosta90
Revert "waiting on full index before return qdrant"
a1ba1b1 
This reverts commit dcdfe21.
@filipecosta90
Added REDIS_QUERY_TIMEOUT config parameter
87b0330
@filipecosta90
Introduced MILVUS_USER and MILVUS_PASS env vars
435038a
@filipecosta90
Introduce MILVUS_PORT env variable
f4e49be
@filipecosta90
Simplify milvus port settings
42133d0
@filipecosta90
Added missing milvus env variables on configure file
ee7cb9b
@filipecosta90
specify milvus uri when http is detected
dfefb49
@filipecosta90
wrap milvus upload_batch with backoff
9766207
@filipecosta90
Enabled using the API_KEY of weaviate and detecting secure connection…
9ad6022 
… setups
@filipecosta90
Only printing ef in search parameters if it's available
4e1408a
@filipecosta90
Simplified weviate client construct
824f17a
@filipecosta90
fix NameError on weviate search.py
31b2a18
@filipecosta90
Update weviate client to include multi-tenant schema check fix
909594d
@filipecosta90
Specify batch_size (and reduce from 64 to 32) on milvus configs.
42fc051
@filipecosta90
Updating milvus upload params from 16 to 4 concurrent clients doing b…
228b10d 
…atches of 32 vectors
@filipecosta90
Cleanup on milvus client
55f91c2
@alonre24
Change recall computation to be as ann-benchmarks'
9641080
@alonre24
Merge branch 'update.redisearch' of https://github.com/filipecosta90/…
666de49 
…vector-db-benchmark into update.redisearch
@filipecosta90
Reducing weaviate batch size due to recurrent ingest errors. ensuring…
e108874 
… cluster is ready/alive before benchmark
@filipecosta90
Reducing milvus batch size to ensure a steady ingestion
60360ad
@filipecosta90
Increasing timeout config for weaviate. Reduce parallel count on inge…
832fc86 
…stion to avoid bad state
@filipecosta90
Added single client weviate config
f8b4b7a
@filipecosta90
Added single client weviate config
c10df6d
@filipecosta90
Revert "Change recall computation to be as ann-benchmarks'"
32b4c6e 
This reverts commit 9641080.
@filipecosta90
Included single segment config for qdrant
0559dd4
filipecosta90 and others added 30 commits March 26, 2025 18:22
@filipecosta90
use uuids on ingest
3b91047
@filipecosta90
Allow specifying the hybrid policy
92b4c84
@filipecosta90
catching 2.10 error message. setting default to flushall to mimic com…
e4d9390 
…petiton
@filipecosta90
fixed missing } on hybrid policy
0e9f9ad
@filipecosta90
fixed missing } on hybrid policy
4aef017
@filipecosta90
fixed missing } on hybrid policy
92dee85
adding laion dataset'
312547d
adding laion dataset 1M
0c36709
chunk up the iterable before starting the processes
3b5e2c9
@slice4e
replace arbitrary 15 sec wait with barrier
92b4ddb
@slice4e
fixed chunk to correct size
deaf5ab
@slice4e
implemented custom process management , instead of using pool
4c1d080
@slice4e
measure time only during the critical work
0d513c2
@slice4e
Merge branch 'update.redisearch' into fix-sync
12868aa
@mpozniak95
Add itertools before islice function
2c592a0
@DvirDukhan
update vset tests
83e3f3e
@DvirDukhan
adding q8
2f1a6fb
@fcostaoliveira
Merge pull request #17 from redis-performance/dvirdu_vectorsets
ba175b1 
Redis Vector Sets support
@fcostaoliveira
Merge pull request #16 from mpozniak95/fix-sync
85a6bc7 
Fixing scalability issues with vector db benchmark
@filipecosta90
Revert "Merge pull request #16 from mpozniak95/fix-sync"
20cea00 
This reverts commit 85a6bc7, reversing
changes made to ba175b1.
@fcostaoliveira
Optimize client imports to only load engines specified in command line
8a21508
@fcostaoliveira
Add --queries option to specify number of queries to run
075c5c0
@fcostaoliveira
Add --ef-runtime option to filter search experiments by ef values
a8d26cd
@fcostaoliveira
cd /home/fco/redislabs/vector-db-benchmark && git status
0209d91 
cd /home/fco/redislabs/vector-db-benchmark && git add engine/base_client/search.py engine/base_client/client.py
chunk up the iterable before starting the processes
@fcostaoliveira
Restore performance optimizations from PR #16 (85a6bc7)
534de8c
@fcostaoliveira
Add real-time progress bar with throughput display for parallel execu…
71c1f3b 
…tion
@fcostaoliveira
Add test script for multiprocessing with progress bar
2eda5b7
@fcostaoliveira
Optimize query cycling for large query counts using generators
6127e84
@fcostaoliveira
Update test script to test query cycling optimization
343e906
@fcostaoliveira
Merge pull request #18 from redis-performance/restore-performance-opt…
25ca5ac 
…imizations

Restore performance optimizations
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@filipecosta90 @meiravgri @alonre24 @mpozniak95 @DvirDukhan @slice4e @fcostaoliveira