Add aesthetic filtering capabilities to video processing pipeline #847
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ding stages - Introduced `video_split_clip_example.py` to demonstrate video splitting functionality. - Added `ClipTranscodingStage` and `FixedStrideExtractorStage` for processing video clips. - Implemented command-line arguments for configuring video processing parameters. - Created utility functions for grouping iterables in `grouping.py`. - Added unit tests for the new stages in `test_clip_transcoding_stage.py` and `test_fixed_stride_extractor_stage.py`. Signed-off-by: Ao Tang <[email protected]>
Signed-off-by: Ao Tang <[email protected]>
…adStage to accept VideoTask. Enhance video reading capabilities with new tests for VideoReaderStage. Signed-off-by: Ao Tang <[email protected]>
…_read_example to include verbose argument. Signed-off-by: Ao Tang <[email protected]>
… additional metadata fields. Signed-off-by: Ao Tang <[email protected]>
Signed-off-by: Ao Tang <[email protected]>
- Introduced a new test package for tasks with an initial test suite for the video tasks module, including tests for the Clip, ClipStats, Video, VideoMetadata, and VideoTask classes. - Implemented various test cases to validate initialization, property calculations, metadata extraction, and size calculations. This enhances the testing coverage for video-related functionalities in the ray-curator project. Signed-off-by: Ao Tang <[email protected]>
- Expanded the test suite for the video tasks module by adding new test cases for the Clip, ClipStats, Video, VideoMetadata, and VideoTask classes. - Improved coverage for various functionalities including initialization, property calculations, and metadata extraction. This update strengthens the reliability of video-related features in the ray-curator project. Signed-off-by: Ao Tang <[email protected]>
…ay-video-clip-extraction Signed-off-by: Ao Tang <[email protected]>
Signed-off-by: Ao Tang <[email protected]>
…age integration Signed-off-by: Ao Tang <[email protected]>
Signed-off-by: Ao Tang <[email protected]>
- Introduced `ClipWriterStage` for writing clips and metadata during video processing. - Updated `video_split_clip_example.py` to include the new stage, allowing for clip writing functionality. - Enhanced command-line argument parsing for output clip path. - Added utility functions for managing storage paths and writing data in various formats. - Implemented unit tests for `ClipWriterStage` to ensure functionality and reliability. Signed-off-by: Ao Tang <[email protected]>
- Improved `ClipWriterStage` to support writing additional metadata during video processing. - Updated related utility functions to accommodate new metadata fields. - Refined unit tests to cover the new functionality and ensure reliability. Signed-off-by: Ao Tang <[email protected]>
- Integrated `MotionVectorDecodeStage` and `MotionFilterStage` into the video splitting pipeline for enhanced motion analysis. - Updated command-line arguments to configure motion filtering options, including GPU memory allocation and thresholds. - Modified `Clip` class to include a type for decoded motion data. Signed-off-by: Ao Tang <[email protected]>
Signed-off-by: Ao Tang <[email protected]>
- Integrated `ClipFrameExtractionStage` into the video splitting pipeline to support frame extraction at specified rates based on user-defined aesthetics and embeddings. - Updated command-line argument handling to accommodate new frame extraction configurations. - Added logic to determine target frame rates based on the presence of aesthetics and embeddings. Signed-off-by: Ao Tang <[email protected]>
…d add unit tests Signed-off-by: Ao Tang <[email protected]>
- Introduced `VideoFrameExtractionStage` to enhance the video splitting pipeline with additional frame extraction capabilities. - Updated the `create_video_splitting_pipeline` function to accommodate the new stage based on user-defined splitting algorithms. - Added logic for handling the "transnetv2" splitting algorithm. Signed-off-by: Ao Tang <[email protected]>
Signed-off-by: Ao Tang <[email protected]>
…ution timing - Added command-line arguments for TransNetV2 parameters, including thresholds, minimum and maximum clip lengths, cropping size, and GPU memory allocation. - Implemented execution time tracking in the video splitting pipeline to provide performance insights. - Updated `VideoFrameExtractionStage` to conditionally import `PyNvcFrameExtractor` based on GPU availability. Signed-off-by: Ao Tang <[email protected]>
…mple Signed-off-by: Ao Tang <[email protected]>
- Replaced the `FixedStrideExtractorStage` with `TransNetV2ClipExtractionStage` in the video split clip example to enhance clip extraction capabilities using the TransNetV2 model. - Updated command-line arguments to include parameters specific to the TransNetV2 stage, such as thresholds and clip length settings. - Introduced a new models package and added the TransNetV2 model implementation for shot transition detection. Signed-off-by: Ao Tang <[email protected]>
- Updated the video splitting pipeline to use `video_dir` instead of `video_folder` for input video paths, improving clarity in command-line arguments. - Added `model_dir` argument to support model directory specification for the TransNetV2 model. - Modified `TransNetV2ClipExtractionStage` to accept `model_dir` during initialization, ensuring proper model loading. - Introduced unit tests for the TransNetV2 model classes to validate functionality and integration. Signed-off-by: Ao Tang <[email protected]>
Signed-off-by: Ao Tang <[email protected]>
…tion' into aot/ray-video-transnet-clip-extraction Signed-off-by: Ao Tang <[email protected]>
Signed-off-by: Ao Tang <[email protected]>
Signed-off-by: Ao Tang <[email protected]>
…to streamline the model interface and improve code clarity. This change enhances maintainability by eliminating unnecessary properties that are no longer used. Signed-off-by: Ao Tang <[email protected]>
…-video-transnet-clip-extraction Signed-off-by: Ao Tang <[email protected]>
…VIDIA-NeMo/Curator into aot/ray-video-cosmos-embed Signed-off-by: Ao Tang <[email protected]>
…ration - Updated `create_video_splitting_pipeline` in `video_split_clip_example.py` to improve the handling of embedding algorithms, ensuring proper error messaging for unsupported algorithms. - Refactored the `CosmosEmbed1` model initialization in `cosmos_embed1.py` for improved readability and maintainability by restructuring the constructor. - Enhanced logging in `CosmosEmbed1FrameCreationStage` and `CosmosEmbed1EmbeddingStage` for better traceability during processing. - Cleaned up import statements in `cosmos_embed1.py` and `test_cosmos_embed1.py` for improved organization and clarity. These changes enhance the overall structure and usability of the video processing pipeline, ensuring better integration and error handling for embedding functionalities. Signed-off-by: Ao Tang <[email protected]>
- Removed the unused `AutoProcessor` mock from the `test_setup_model_load_failure` method in `test_cosmos_embed1.py` to streamline the test setup. - Improved clarity by adjusting the method signature to reflect the changes in mock parameters. These updates enhance the maintainability and readability of the test code for the CosmosEmbed1 model. Signed-off-by: Ao Tang <[email protected]>
…urator into aot/ray-video-aesthetic Signed-off-by: Ao Tang <[email protected]>
- Improved formatting in `video_split_clip_example.py` for better readability when adding `ClipAestheticFilterStage`. - Removed the `conda_env_name` property from `AestheticScorer` and `CLIPImageEmbeddings` classes to streamline the model interface. - Updated tests in `test_aesthetics.py` and `test_clip.py` to reflect the removal of the `conda_env_name` property and adjusted assertions for model ID names. - Enhanced `ClipAestheticFilterStage` with improved resource management and updated import statements for clarity. These changes enhance the maintainability and readability of the codebase, ensuring a cleaner interface for aesthetic scoring and clip processing. Signed-off-by: Ao Tang <[email protected]>
- Modified the initialization of `CosmosEmbed1EmbeddingStage` in `test_cosmos_embed1.py` to set `gpu_memory_gb` to 8, aligning the test with expected parameters. - Adjusted assertions to reflect the new default value for `gpu_memory_gb`, enhancing the accuracy of the test case. These changes improve the reliability of the test setup for the CosmosEmbed1 model. Signed-off-by: Ao Tang <[email protected]>
…urator into aot/ray-video-aesthetic
|
/ok to test 28b4747 |
- Adjusted the `gpu_memory_gb` parameter in the `CosmosEmbed1EmbeddingStage` initialization from 40 to 8 in `test_cosmos_embed1.py` to align with expected test parameters. - Updated assertions to reflect the new value, enhancing the accuracy and reliability of the test case. These changes ensure the test setup accurately reflects the intended configuration for the CosmosEmbed1 model. Signed-off-by: Ao Tang <[email protected]>
…urator into aot/ray-video-aesthetic
|
/ok to test b7d25ee |
…-video-aesthetic Signed-off-by: Ao Tang <[email protected]>
- Removed the conditional embedding generation logic from `create_video_splitting_pipeline` in `video_split_clip_example.py`, streamlining the pipeline setup. - Updated the target frame rate for `CosmosEmbed1FrameCreationStage` to use `FramePurpose.EMBEDDINGS.value` for improved clarity and consistency. - Cleaned up the test setup in `test_cosmos_embed1.py` by adjusting the method signature for better readability. These changes improve the maintainability and clarity of the video processing pipeline, ensuring better integration of embedding functionalities. Signed-off-by: Ao Tang <[email protected]>
|
/ok to test aab97f5 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Usage
# Add snippet demonstrating usageChecklist