CTHUB - Monthly ICBC upload #469
Description
Title:
Enable IDIR users to upload monthly ICBC vehicle registration data to CTHUB.
Description:
- As an IDIR user, I want to upload monthly ICBC vehicle registration data files, so that the new data is automatically filtered, formatted, and appended to the historical ICBC dataset stored in CTHUB using a consistent schema and processing logic.
Wireframe: UI already exists.
Purpose and benefit to user:
- Enables users to maintain an up-to-date and comprehensive vehicle registration dataset for analysis and reporting.
- Supports policy planning, program evaluation, and reporting needs across multiple ministries and stakeholders.
- Automates data cleaning and formatting to ensure consistency with historical data, reducing manual effort and errors.
Acceptance Criteria:
- Given I am an IDIR user, when I upload a monthly ICBC data file, then the system validates the file format and schema before processing.
- Given I am an IDIR user, when the file is accepted, then the system runs a scripted process to clean, format, and transform the data to match the historical ICBC dataset schema.
- Given the data has been processed, then it is appended to the existing ICBC historical data table in CTHUB.
- Given the data has been appended, then I can query and visualize the updated dataset through PowerBI.
Development Checklist:
- Review and assess the existing script provided by the business area for formatting ICBC data
- Modify and integrate the script into the CTHUB backend as needed
- Implement frontend upload functionality for IDIR users
- Validate uploaded file format and schema
- Automate data cleaning, transformation, and appending process
- Ensure error handling and user feedback for failed uploads
Notes:
- The business area has provided a workign script for formatting the ICBC data. development team to review and adapt it for integration
- File size considerations (approximately 700MB per monthly dataset)