Skip to content

Conversation

nabil-Tounarti
Copy link
Contributor

No description provided.

@nabil-Tounarti nabil-Tounarti self-assigned this Sep 25, 2025
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds bulk dataset support with environment variables to the pre-compute system, enabling processing of multiple datasets in a single task execution. The implementation refactors dataset handling from individual fields to a unified Dataset structure.

  • Creates a new Dataset structure to encapsulate dataset properties and operations
  • Adds bulk processing environment variables (BULK_SIZE and BULK_DATASET_#_*)
  • Refactors pre-compute arguments to use a vector of datasets instead of individual fields

Reviewed Changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
pre-compute/src/compute/utils/env_utils.rs Adds new environment variable enums for bulk dataset support
pre-compute/src/compute/pre_compute_args.rs Refactors to use Dataset vector and adds bulk processing logic
pre-compute/src/compute/pre_compute_app.rs Updates to process datasets from the new structure
pre-compute/src/compute/dataset.rs Creates new Dataset struct with download and decryption methods
pre-compute/src/compute.rs Adds dataset module export

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@nabil-Tounarti nabil-Tounarti merged commit c389c2e into main Sep 26, 2025
7 checks passed
@nabil-Tounarti nabil-Tounarti deleted the feature/bulk-processing branch September 26, 2025 13:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants