A data pipeline is like an assembly line with multiple connected stages. Collecting data, transforming it, and loading it into storage — each stage must run in the right order, under the right conditions. Orchestration is the process of automatically coordinating this flow. Like a conductor leading an orchestra, an orchestration service directs each performer (service) to play their part at exactly the right moment. In the AWS DEA-C01 exam, "which orchestration service do you use for a given scenario?" is a frequently tested topic.
Orchestration Service Comparison
| Service | Approach | Characteristics | Best situation | |---------|----------|----------------|----------------| | AWS Step Functions | Serverless state machine | Visual workflow, AWS-native, powerful error handling | Workflows connecting AWS services, complex branching logic | | Amazon MWAA | Managed Apache Airflow | Python DAGs, rich operators | Existing Airflow teams, complex DAGs, external system integration | | Amazon EventBridge | Event bus + Scheduler | Event-driven routing, cron schedules | Event triggers, time-based automation | | AWS Glue Workflow | Glue-only orchestration | Chain Glue jobs and crawlers | Orchestrating only Glue ETL jobs |
!4 pipeline orchestration services compared
AWS Step Functions — The Conductor of Your Workflow
AWS Step Functions is a serverless workflow service that connects multiple AWS services into a coordinated sequence. You define the workflow using JSON-based Amazon States Language (ASL), and Step Functions automatically executes and monitors each step. Think of it like connecting LEGO blocks — each "State" is a block you snap together to build complex pipelines.
Understanding State types
A Step Functions workflow is composed of different types of states:
Task state: Executes an actual action. Directly integrates with AWS services — calling a Lambda function, starting a Glue job, querying DynamoDB, and more. Choice state: Branches to different paths based on conditions. Implements logic like "if data size is over 1 GB, route to EMR; otherwise use Lambda." Parallel state: Runs multiple branches simultaneously. Useful for processing data from three regions at the same time. Map state: Applies the same processing to each item in an array. Like a loop that processes 100 files one by one. Wait state: Pauses execution for a set duration or until a specific timestamp. Succeed / Fail state: Terminates the workflow as successful or failed.
Standard vs Express Workflows
| Aspect | Standard | Express | |--------|---------|---------| | Max execution duration | 1 year | 5 minutes | | Execution guarantee | Exactly once | At least once | | Audit history | Full execution history | Limited | | Pricing | Per state transition | Per execution count and duration | | Best for | Long-running workflows, audit requirements | High-frequency short jobs (thousands per second) |
Error handling: Retry and Catch
In the real world, steps can fail. Step Functions lets you define error handling directly on each state:
Retry: Automatically retries on failure. You configure the number of retries, interval between retries, and backoff multiplier. Example: "If the Lambda call fails, retry up to 3 times with a 2-second interval." Catch: When all retries are exhausted, moves to a fallback path. Example: "If processing fails, invoke an error-notification Lambda and send an email to the administrator."
Amazon MWAA — Managed Apache Airflow on AWS
Apache Airflow is an open-source tool for defining and scheduling data pipelines using Python code. Amazon MWAA (Managed Workflows for Apache Airflow) is a managed service where AWS installs, maintains, and scales Airflow for you.
What is a DAG (Directed Acyclic Graph)?
In Airflow, a workflow is represented as a DAG. A DAG is a "directed graph with no cycles." In plain terms, tasks are connected by arrows, and if you follow the arrows you never loop back to where you started.
Operators
An operator defines how each task runs. Airflow provides hundreds of operators: AWS operators: GlueJobOperator, EMROperator, S3CopyObjectOperator, etc. External systems: PostgresOperator, SparkSubmitOperator, HttpOperator, etc.
MWAA vs Step Functions decision guide
| Situation | Choose | |-----------|--------| | Existing team already uses Airflow | MWAA | | Need to integrate with systems outside AWS (on-premises DB, etc.) | MWAA | | Using only AWS-native services | Step Functions | | Complex branching/parallel logic with visual management | Step Functions | | Quick start, low learning curve | Step Functions |
Amazon EventBridge — The Heart of Event-Driven Automation
EventBridge is a service that detects events and routes them to other services. Its role is to "automatically start something else when something happens."
Event-driven operation
AWS service events: Detects events like EC2 instance state changes, S3 file uploads, or CodePipeline completions. Rules: Invoke a target when an event matches a specific pattern. Example: "When a new file is created in S3, start a Step Functions workflow." Targets: Delivers events to Lambda, Step Functions, SQS, SNS, Kinesis, API Gateway, and many other services.
Scheduler
Runs tasks on a schedule using cron or rate expressions: → Run every day at 8:00 AM UTC → Run every hour → Run at 9:00 AM on weekdays only
EventBridge serves as the "start signal" for pipelines. For example: "Start the Glue ETL job every midnight" or "Start the Step Functions pipeline when a file lands in S3."
Exam Key Points Summary
| Keyword | Choose this service | |---------|-------------------| | Serverless workflow, AWS-native, visual | Step Functions | | Workflow with branching logic | Step Functions Choice state | | Parallel processing steps | Step Functions Parallel state | | Retry on workflow failure | Step Functions Retry/Catch | | Managed Apache Airflow | Amazon MWAA | | Migrate existing Airflow DAGs to AWS | Amazon MWAA | | Event-driven trigger, cron schedule | EventBridge | | Orchestrating only Glue ETL jobs | Glue Workflow | | High-frequency short workflows (thousands per second) | Step Functions Express workflow |
The core distinction between Step Functions and MWAA: if your workflow is centered on AWS-native services, use Step Functions. If you need Python DAGs and the Airflow ecosystem, use MWAA.