v4.4.3

First C++ library release.

v4.4.1 🌈

Release Notes

🎉 New Features & Enhancements

Storage & API Improvements

• Storage list_dirs API - New API endpoint for listing directories by @khustup2
• Added mesh type support - Extended data type capabilities by @khustup2
• PLY visualization and local file listing - Enhanced node server with visualization tools by @azat-manukyan
• Simple visualizer - New visualization component for data inspection by @khustup2
• Credentials key property - Added property to expose creds key for read-only datasets by @activesoull

Data Management

• Efficient version-locked read-only view - Implemented optimized read-only dataset views by @ashotshakhkyan
• Library preloading - Improved performance with preloaded library support by @vahan-activeloop

⚡ Performance Improvements

• 30% LRU cache insertion runtime improvement - Significant speedup in cache operations by @ashotshakhkyan
• Cache clearing on dataset deletion - Minor performance optimization on cleanup by @ashotshakhkyan
• Refactored async_prefetcher - Simplified batch prefetch logic for better performance by @ashotshakhkyan

🐛 Bug Fixes

Core Functionality

• Fixed progress reporting of rebuild_branch - Corrected progress tracking during branch rebuilds by @khustup2
• Fixed deeplake behavior - Resolved critical deeplake module issues by @activesoull
• Fixed JS build - Resolved JavaScript compilation errors by @ashotshakhkyan

Data Handling

• Summary fix - Corrected summary generation logic by @activesoull
• MP3 header magic bytes - Fixed audio file format detection by @azat-manukyan

Indexing & Storage

• Corrected indexing mode of compactor - Fixed compactor indexing behavior by @ashotshakhkyan
• Duplicate index data handling - Resolved race condition in concurrent index builders by @ashotshakhkyan
• Fixed warning - Addressed compiler warnings by @activesoull

🏗️ Architecture & Refactoring

Code Organization

• Simplified rebuild_branch logic - Eliminated temporary branch creation for cleaner code by @khustup2
• Reduced public headers - Hidden implementation details from public API by @ashotshakhkyan
• Deeplog include cleanup - Organized include statements for better maintainability by @ashotshakhkyan

API Improvements

• Added merge commit message - Enhanced commit metadata by @activesoull

v4.3.4 🌈

🎉 New Features
pg_deeplake Enhancements

CHAR/BPCHAR data type support - pg_deeplake now supports CHAR and BPCHAR data types
Library preloading - Improved performance with preloaded libraries in pg_deeplake
Auto table reload - Tables are now automatically reloaded if not found in storage

Architecture Improvements

Query core module - Separated query engine from execution module for better modularity
Group by execution refactor - Separated group by execution logic for future enhancements

🐛 Bug Fixes

Fixed linked audio type data handling - Resolved issues with linked audio type data
Fixed compilation errors - Resolved compile errors in pg_deeplake

💬 User Experience

Better error messaging - Added clear error messages for missing credentials key

🧹 Code Quality

Cleanup and refactoring - Improved code organization with proper renames and cleanup

🧪 Testing

Enhanced test coverage - Added comprehensive tests for pg_deeplake

v3.9.52 🌈

🧭 What's Changed

• Allow numpy v2. (#3075) @khustup2

⚙️ Who Contributes

@activesoull and @khustup2

v4.4.0

Release Notes

New Features

Indexing Improvements

• Support for JSON data indexing - Added support for indexing JSON data types with proper handling and query optimization
• Generic column index specification - New ability to specify index type on generic columns for more granular control
• Progress bar for index creation - Improved user experience with visual progress indicator during index creation operations
• Index handling unification - Streamlined and unified index handling across the codebase for better consistency and maintainability

Documentation

• Added missing documentation sections - Expanded documentation coverage for previously undocumented features and APIs

Improvements

Performance & Reliability

• Fixed build errors on main branch - Resolved compilation issues affecting the main development branch
• Windows test fixes - Enhanced test reliability on Windows platform with targeted fixes
• Thread-safe cancellation - Improved cancellation mechanism with proper thread safety guarantees
• Rebuild branch internal API - Refactored internal API for better architecture and maintainability
• Fix rebuild indexes during compaction - Corrected index rebuilding behavior during database compaction operations

Platform & Build Support

• Windows Python build support - Added comprehensive support for building Python bindings on Windows
• Updated libtiff dependency - Upgraded libtiff library to latest version for improved compatibility and security

Bug Fixes

• Fixed browser crash - Resolved critical issue causing browser crashes in specific scenarios
• Invalid iterator fix - Fixed invalid iterator handling that could cause runtime errors
• General bug fixes - Various stability improvements and edge case handling

Contributors

Thanks to @activesoull, @vahan-activeloop, @khustup2, and @ashotshakhkyan for their contributions to this release.

v4.3.5 🌈

Data Type Support

Link to bytes support - Added support for link to bytes data type (DEE-299)
Flexible text column handling - Text columns now allow appending data with unknown dtype for better compatibility

🐛 Bug Fixes

deeplake.exists error handling - Fixed exception handling in deeplake.exists to gracefully handle errors instead of throwing exceptions (DEE-300)
Python PNG linking - Resolved Python PNG linking issues

📚 Documentation

Documentation updates - Improved and updated documentation (DEE-36)

🧹 Code Quality

General improvements - Various small fixes and code improvements

v4.3.4

🎉 New Features
PostgreSQL 18 Support

PostgreSQL 18 compatibility - pg_deeplake now supports PostgreSQL 18

Performance Optimizations

Adaptive backoff for worker threads - Implemented an intelligent backoff strategy for idle worker threads to reduce CPU usage
Cgroup-aware concurrency - Hardware concurrency now respects cgroup limits for better container performance

Azure Integration

Azure container name handling - Improved Azure getter with proper container name handling

🐛 Bug Fixes

Build errors resolved - Fixed compilation errors in pg_deeplake
Visualizer fixes - Resolved issues in the visualizer component

🧹 Code Quality

Code cleanup - Improved code organization with proper renames and refactoring
Modular architecture - Continued separation of concerns for better maintainability

🧪 Testing

Enhanced test coverage - Added comprehensive tests for pg_deeplake

v4.3.0 🌈

Deeplake 4.3.0

Deeplake 4.3.0 is a major update bringing many new features to the Deeplake ecosystem.

New Data and Index Types

• Complete revisit of Sequence types to support visual and structured data
• Video type support is now available in Deeplake, supporting MP4 and MKV videos with H264 codec and providing fast random access to video frames
• Indexing for numeric types, enabling fast queries for numeric comparisons in TQL, including IN and BETWEEN operations
• Significant improvements to textual index types, providing faster search without requiring index regeneration

Data Import/Export

• Fully rewritten from_csv function with support for large CSV files
• New to_csv API to export Deeplake datasets/views to CSV format

Python Typing

• Support for specifying Python builtin types when defining dataset schemas
• Support for using Pydantic Models as dataset schemas
• Enriched async operations typing, to support better integration with linters and IDEs.

Improvements and Bug Fixes

• Improved TQL data fetching and linear scan performance for non-indexed columns
• Better memory usage tracking to prevent out-of-memory errors
• Various stability improvements and bug fixes

Compatibility Notice

Deeplake 4.3.0 is backward compatible with datasets created in v4.2.x. However, datasets created or modified with v4.3.0 cannot be opened with v4.2.x versions due to internal format enhancements. We recommend upgrading all environments to v4.3.0 when working with shared datasets.

v3.9.51 🌈

🧭 What's Changed

• Update version for 3.9.51 release (#3073) @activesoull
• [DEE-180] fix dependency resolver issue (#3072) @activesoull

⚙️ Who Contributes

@activesoull

v3.9.50 🌈

🧭 What's Changed

• Updated versions for 3.9.50 release (#3069) @activesoull
• Refactor frame extraction logic and improve frame rate retrieval in L… (#3071) @azat-manukyan

⚙️ Who Contributes

@activesoull and @azat-manukyan