Hadoop

Elasticsearch real-time search and analytics natively integrated with Hadoop. Supports Map/Reduce, Cascading, Apache Hive and Apache Pig.

Python MapReduce library written in Cython.

mrjob is a Python 2.5+ package that helps you write and run Hadoop Streaming jobs.

HDFS-DU is an interactive visualization of the Hadoop distributed file system.

Hadoop log aggregator and dashboard

Genie provides REST-ful APIs to run Hadoop, Hive and Pig jobs, and to manage multiple Hadoop resources and perform job submissions across them.

Go-based toolkit for ETL and feature extraction on Hadoop

Running MPICH2 on Yarn

A developer-friendly Python library to interact with Apache HBase.

Hannibal is tool to help monitor and maintain HBase-Clusters that are configured for manual splitting.

Haeinsa is linearly scalable multi-row, multi-table transaction library for HBase

Secondary Index for HBase

Schema Registry provides a serving layer for your metadata. It provides a RESTful interface for storing and retrieving Avro schemas.

Schema Registry is a framework to build metadata repositories.

Airflow is a workflow automation and scheduling system that can be used to author and manage data pipelines

Netflix's distributed Data Pipeline

Universal data ingestion framework for Hadoop

Machine learning and natural language processing with Apache Pig

Open Source Big Data Security Analytics

Mozilla's utility library for Hadoop, HBase, Pig, etc.

Pig workflow visualization tool. Introducing Lipstick on A(pache) Pig

PigPen is map-reduce for Clojure, or distributed Clojure. It compiles to Apache Pig, but you don't need to know much about Pig to use it.

Native go clients for Apache Hadoop YARN.

A native go client for HDFS

Web tool for the Confluent Schema Registry in order to create / view / search / evolve / view history & configure Avro schemas of your Kafka cluster.

A big data cluster management tool that creates and manages clusters of different technologies.

Inviso is a lightweight tool that provides the ability to search for Hadoop jobs, visualize the performance, and view cluster utilization.

Kibana port for Apache Solr

The Yahoo! Cloud Serving Benchmark (YCSB) is an open-source specification and program suite for evaluating retrieval and maintenance capabilities of computer programs. It is often used to compare relative performance of NoSQL database management systems.

The Yahoo! Cloud Serving Benchmark (YCSB) is an open-source specification and program suite for evaluating retrieval and maintenance capabilities of computer programs. It is often used to compare relative performance of NoSQL database management systems.

Lambda architecture on Spark, Kafka for real-time large scale machine learning

including RHDFS, RHBase, RMR2, plyrmr

and awesome list.