-
Notifications
You must be signed in to change notification settings - Fork 0
feat: add bulk dataset support with environment variables #16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds bulk dataset support with environment variables to the pre-compute system, enabling processing of multiple datasets in a single task execution. The implementation refactors dataset handling from individual fields to a unified Dataset structure.
- Creates a new Dataset structure to encapsulate dataset properties and operations
- Adds bulk processing environment variables (BULK_SIZE and BULK_DATASET_#_*)
- Refactors pre-compute arguments to use a vector of datasets instead of individual fields
Reviewed Changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 5 comments.
Show a summary per file
File | Description |
---|---|
pre-compute/src/compute/utils/env_utils.rs | Adds new environment variable enums for bulk dataset support |
pre-compute/src/compute/pre_compute_args.rs | Refactors to use Dataset vector and adds bulk processing logic |
pre-compute/src/compute/pre_compute_app.rs | Updates to process datasets from the new structure |
pre-compute/src/compute/dataset.rs | Creates new Dataset struct with download and decryption methods |
pre-compute/src/compute.rs | Adds dataset module export |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
No description provided.