Prefect
Python-native workflow orchestration platform for building, scheduling, and monitoring data pipelines with ordinary Python code.
Overview
Prefect lets developers define workflows as plain Python functions decorated as tasks and flows, rather than learning a separate DSL or YAML format. It handles scheduling, retries, caching, and failure handling automatically, and its UI gives visibility into every run without extra instrumentation code.
Prefect is a fit for data and platform teams building ETL, ML, or data-pipeline workflows in Python who want orchestration that stays close to regular code - a different niche than Temporal or Argo Workflows (also in this catalog), which are aimed at general-purpose distributed application workflows rather than data-pipeline-specific scheduling.
- Categories
- Automation
- Languages
- Python
- License
- Apache-2.0
Related Projects
Kestra
Apache-2.0Declarative, YAML-based orchestration platform for scheduling and orchestrating data pipelines, ETL jobs, and business workflows.
- Automation
Argo Workflows
Apache-2.0Kubernetes-native workflow engine that runs each step of a pipeline as a container, orchestrating parallel jobs and multi-step DAGs directly on a cluster.
- Automation
Temporal
MITDurable execution platform that lets developers write reliable, long-running workflows as ordinary code, with retries, state, and failure recovery handled automatically.
- Automation
SeaTunnel
Apache-2.0Distributed data integration platform for moving and transforming data between databases, data lakes, and messaging systems at scale.
- Backend
Spotted an error? Suggest an edit on GitHub.