Beam
Unified programming model for defining batch and stream data processing pipelines that can run portably across multiple execution engines.
Overview
Apache Beam lets a team write a data pipeline once using its unified batch/stream API, then run that same pipeline on different underlying execution engines - including Flink and Spark (also in this catalog) - without rewriting it for each. This avoids locking a pipeline's logic to one specific processing engine.
Beam is a fit for teams that want pipeline portability across execution engines, or who are building on Google Cloud Dataflow (which uses Beam as its programming model), a different layer than Flink or Spark themselves, which are the engines Beam pipelines actually run on.
- Categories
- Analytics
- Languages
- Java, Python, Go
- License
- Apache-2.0
Related Projects
Spark
Apache-2.0Unified analytics engine for large-scale batch and stream data processing, one of the most widely adopted big data frameworks.
- Analytics
Flink
Apache-2.0Distributed stream processing engine for computing over continuous, unbounded data streams with exactly-once correctness guarantees.
- Analytics
ClickHouse
Apache-2.0Column-oriented database built for real-time analytical queries over massive datasets, generating aggregated reports in milliseconds rather than seconds or minutes.
- Analytics
Druid
Apache-2.0Real-time analytics database built for sub-second queries over continuously streaming and historical event data at scale.
- Analytics
Spotted an error? Suggest an edit on GitHub.