-
Notifications
You must be signed in to change notification settings - Fork 77
Efficient File I/O Plug-ins for Widely-used File Formats #985
Copy link
Copy link
Open
Labels
AMLS summer 2026Student project for the Architecture of ML Systems lecture at TU Berlin (summer 2026).Student project for the Architecture of ML Systems lecture at TU Berlin (summer 2026).LDE summer 2026Student project in the course Large-scale Data Engineering at TU Berlin (summer 2026).Student project in the course Large-scale Data Engineering at TU Berlin (summer 2026).LDE winter 2025/26Student project in the course Large-scale Data Engineering at TU Berlin (winter 2025/26).Student project in the course Large-scale Data Engineering at TU Berlin (winter 2025/26).student projectSuitable for a bachelor/master student's programming project.Suitable for a bachelor/master student's programming project.
Description
Metadata
Metadata
Assignees
Labels
AMLS summer 2026Student project for the Architecture of ML Systems lecture at TU Berlin (summer 2026).Student project for the Architecture of ML Systems lecture at TU Berlin (summer 2026).LDE summer 2026Student project in the course Large-scale Data Engineering at TU Berlin (summer 2026).Student project in the course Large-scale Data Engineering at TU Berlin (summer 2026).LDE winter 2025/26Student project in the course Large-scale Data Engineering at TU Berlin (winter 2025/26).Student project in the course Large-scale Data Engineering at TU Berlin (winter 2025/26).student projectSuitable for a bachelor/master student's programming project.Suitable for a bachelor/master student's programming project.
Type
Fields
Give feedbackNo fields configured for issues without a type.
Motivation: The input to integrated data analysis pipelines, that combine query processing, machine learning, and high-performance computing, could be provided in various general-purpose and domain-specific file formats. To embrace a large variety of such formats, DAPHNE is extensible w.r.t. to file readers/writers (pull request coming soon), i.e., expert users can add their own file I/O plug-ins without touching the source code of DAPHNE. However, so far there is only a limited number of plug-ins available.
Task: (in C++) Implement additional file I/O plug-ins for a range of widely used file formats for different data modalities, such as tabular data, (sparse) matrices/graphs, audio, images, and time series. These plug-ins may be based on existing open-source libraries (with compatible license). Apply format-specific and format-agnostic tricks to make the file I/O efficient, e.g., by exploiting parallelism or pushing down certain operations into the readers.
Hints:
coming soon