Data Engineering

A Python library that facilitates the comparison of two DataFrames in Pandas, Polars, Spark and more. The library goes beyond basic equality checks by providing detailed insights into discrepancies at both row and column levels.

Data Validation Tool compares data from source and target tables to ensure that they match. It provides column validation, row validation, schema validation, custom query validation, and ad hoc SQL exploration.

A high-performance Python library for comparing large datasets (CSV, Parquet) locally using Rust and Polars. It features zero-copy streaming to prevent OOM errors and generates interactive HTML data quality reports.

AI-powered data operations SDK for Python. Semantic deduplication, fuzzy table merging, and intelligent row ranking using LLM agents.

CLI tool to copy data between databases with a single command. Supports 50+ sources including PostgreSQL, MySQL, MongoDB, Salesforce, Shopify to any data warehouse.

Data Acquisition and Processing Made Easy. Deprecated.

Universal data ingestion framework for Hadoop from LinkedIn.

Utility belt to handle data on AWS.

Live import all your Google Sheets to your data warehouse.

Lightweight Node.js ETL framework for databases → data lakes/warehouses.

Polyglot document intelligence library with a Rust core and bindings for Python, TypeScript, Go, and more. Extracts text, tables, and metadata from 62+ document formats for data pipeline ingestion.

Conflict-free merge for DataFrames, JSON, ML models & distributed agents — powered by CRDTs.

A high-performance Cloud-Native file system driven by object storage for large-scale data storage.

A bite-sized, lightweight HDFS compatible file system built over Cassandra.

Seaweed-FS is a simple and highly scalable distributed file system. There are two objectives: to store billions of files! to serve the files fast! Instead of supporting full POSIX file system semantics, Seaweed-FS choose to implement only a key~file mapping. Similar to the word "NoSQL", you can call it as "NoFS".

A file system that stores all its data online using storage services like Google Storage, Amazon S3, or OpenStack.

Protocol Buffers - Google's data interchange format.

A fast and efficient object graph serialization framework for Java.

An open source ETL framework to build fresh index for AI.

The Streaming SQL Database.

Forever scalable event processing & in-memory durable K/V store as a library with asyncio & static typing.

The streaming database built for IoT data storage and real-time processing.

An edge lightweight IoT data analytics/streaming software implemented by Golang, and it can be run at all kinds of resource-constrained edge devices.

An API gateway built for event-driven architectures and streaming that supports standard protocols such as HTTP, SSE, gRPC, MQTT, and the native Kafka protocol.

A framework for building real-time streaming data processing applications that supports a wide range of ingestion sources.

Performant open-source Python ETL framework with Rust runtime, supporting 300+ data sources.

A light-weight engine for general-purpose data processing including both batch and stream analytics. It is based on a novel unique data model, which represents data via functions and processes data via columns operations as opposed to having only set operations in conventional approaches like MapReduce or SQL.

A cloud native data pipeline and transformation toolkit written in Go.

Personal genome analysis toolkit with Python scripts analyzing raw DNA data across 17 categories (health risks, ancestry, pharmacogenomics, nutrition, psychology, etc.) and generating a terminal-style single-page HTML visualization.

Python helpers for building dashboards using Flask and React.

Flask, JS, and CSS boilerplate for interactive, web-based visualization apps in Python.

A modern, enterprise-ready business intelligence web application.

The easy, open source way for everyone in your company to ask questions and learn from data.

Natural language database query interface with automatic chart generation, supporting Chinese and English queries.

End-to-end data pipeline tool that combines ingestion, transformation (SQL + Python), and data quality in a single CLI. Connects to BigQuery, Snowflake, PostgreSQL, Redshift, and more. Includes VS Code extension with live previews.

A Python module that helps you build complex pipelines of batch jobs.

An application cron-like system. Used w/Luigi. Deprecated.

A system to programmatically author, schedule, and monitor data pipelines.

DAG based workflow manager. Job flows are defined programmatically in Python. Support output passing between jobs.

An open-source Python library for building data applications.

A lightweight library to define data transformations as a directed-acyclic graph (DAG). If you like dbt for SQL transforms, you will like Hamilton for Python processing.

Scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.

A warehouse-first Customer Data Platform that enables you to collect data from every application, website and SaaS platform, and then activate it in your warehouse and business tools.

An open source framework that allows you to enforce agreements on how data should be accessed, used, and transformed, regardless of the data platform (Snowflake, BigQuery, DataBricks, etc.)

The open-source reverse ETL, data activation platform for modern data teams.

An open source platform that delivers resilience and manageability to object-storage based data lakes.

A Transactional Catalog for Data Lakes with Git-like semantics. Works with Apache Iceberg tables.

An open-source, unified metadata management for data lakes, data warehouses, and external catalogs.

A highly configurable Logstash (1.4.4) - Docker image running Elasticsearch (1.7.0) - and Kibana (3.1.2).

JDBC importer for Elasticsearch.

PostgreSQL Extension that allows creating an index backed by Elasticsearch.

Package golang service into minimal Docker containers.

Easily manage Docker containers & their data.

Weaving Docker containers into applications.

A lightweight tool for easy deployment and rollback of dockerized applications.

Analyzes resource usage and performance characteristics of running containers.

Docker microservice for saving/restoring volume data to S3.

Docker composition tool with idempotency features for deploying apps composed of multiple containers. Deprecated.

A cluster manager, designed for both long-lived services and short-lived batch processing workloads.

A data catalog tool that integrates into your CI system exposing downstream impact testing of data changes. These tests prevent data changes which might break data pipelines or BI dashboards from making it to production.

An open-source data quality platform for the whole data platform lifecycle from profiling new data sources to applying full automation of data quality monitoring.

Decorator-first DataFrame contracts/validation (columns/dtypes/constraints) at function boundaries. Supports Pandas/Polars/PyArrow/Modin.

A Snowflake-compatible emulator for local development and testing.

A vendor-neutral, declarative data quality engine. Define checks in YAML, run anywhere. Includes 16 built-in check types, SQL batch optimizer, anomaly detection, and data contracts.