Spark
Unified analytics engine for large-scale batch and stream data processing, one of the most widely adopted big data frameworks.
Overview
Apache Spark processes very large datasets across a distributed cluster of machines, supporting batch processing, streaming, SQL queries, and machine learning workloads through one unified engine and API, in multiple languages including Python, Scala, and Java. It's been the standard big-data processing framework for over a decade.
Spark is a fit for teams running large-scale data transformation, ETL, or machine learning pipelines across distributed clusters, a broader and more general-purpose engine than the more specialized Flink (real-time streaming) or Trino (federated SQL) also in this catalog.
- Categories
- Analytics
- Languages
- Scala, Python, Java
- License
- Apache-2.0
Related Projects
Beam
Apache-2.0Unified programming model for defining batch and stream data processing pipelines that can run portably across multiple execution engines.
- Analytics
Flink
Apache-2.0Distributed stream processing engine for computing over continuous, unbounded data streams with exactly-once correctness guarantees.
- Analytics
ClickHouse
Apache-2.0Column-oriented database built for real-time analytical queries over massive datasets, generating aggregated reports in milliseconds rather than seconds or minutes.
- Analytics
Druid
Apache-2.0Real-time analytics database built for sub-second queries over continuously streaming and historical event data at scale.
- Analytics
Spotted an error? Suggest an edit on GitHub.