datalake

Here are 245 public repositories matching this topic...

Sinaptik-AI / pandas-ai

Chat with your database (SQL, CSV, pandas, polars, mongodb, noSQL, etc). PandasAI makes data analysis conversational using LLMs (GPT 3.5 / 4, Anthropic, VertexAI) and RAG.

data-science data csv sql database ai pandas data-analysis datalake gpt-3 gpt-4 llm

Updated Nov 22, 2024
Python

trinodb / trino

Star

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)

java distributed-systems data-science sql database big-data presto hive hadoop analytics jdbc databases distributed-database query-engine iceberg datalake prestodb trino delta-lake

Updated Nov 30, 2024
Java

The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.

Updated Nov 30, 2024
Java

activeloopai / deeplake

Star

Database for AI. Store Vectors, Images, Texts, Videos, etc. Use with LLMs/LangChain. Store, query, version, & visualize any AI data. Stream data in real-time to PyTorch/TensorFlow. https://activeloop.ai

Updated Nov 30, 2024
Python

paradedb / paradedb

Sponsor

Star

Postgres for Search and Analytics

Updated Nov 30, 2024
Rust

apache / hudi

Star

Upserts, Deletes And Incremental Processing on Big Data.

bigdata stream-processing data-integration datalake apachespark hudi apachehudi incremental-processing apacheflink

Updated Dec 1, 2024
Java

treeverse / lakeFS

Star

lakeFS - Data version control for your data lake | Git for data

go golang apache-spark aws-s3 google-cloud-storage data-engineering data-lake azure-storage data-version-control object-storage datalake hadoop-filesystem data-quality data-versioning azure-blob-storage apache-sparksql git-for-data lakefs datalakes

Updated Dec 1, 2024
Go

DataLinkDC / dinky

Star

Dinky is a real-time data development platform based on Apache Flink, enabling agile data development, deployment and operation.

sql olap flink datawarehouse datalake flinksql flinkcdc real-time-computing-platform

Updated Dec 1, 2024
Java

lakesoul-io / LakeSoul

Star

LakeSoul is an end-to-end, realtime and cloud native Lakehouse framework with fast data ingestion, concurrent update and incremental data analytics on cloud storages for both BI and AI applications.

python rust streaming sql big-data spark arrow postgresql pytorch flink datalake vectorized velox huggingface datafusion lakehouse lakesoul

Updated Nov 25, 2024
Java

leo-project / leofs

Star

The LeoFS Storage System

erlang s3 nfs s3-storage distributed-storage distributed-file-system leofs nfs-server datalake

Updated Jun 2, 2020
Erlang

apache / gravitino

Star

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.

metadata data-catalog datalake stratosphere federated-query lakehouse model-catalog metalake skycomputing ai-catalog opendatacatalog

Updated Nov 29, 2024
Java

zinggAI / zingg

Star

Scalable identity resolution, entity resolution, data mastering and deduplication using ML

Updated Nov 29, 2024
Java

apache / amoro

Star

Apache Amoro (incubating) is a Lakehouse management system built on open data lake formats.

bigdata datalake lakehouse

Updated Nov 29, 2024
Java

leesf / hudi-resources

Star

汇总Apache Hudi相关资料

bigdata apache stream-processing data-integration datalake hudi apachehudi incremental-processing hudi-resources

Updated Dec 1, 2024

Datavault-UK / automate-dv

Star

A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)