One of the hardest parts of running data pipelines is managing repetitive work. Processing yesterday's sales data every morning at 8 AM, running transformations every time a new file lands in S3, generating weekly reports — none of these can be done manually at scale. Automation means making systems do this work instead. DEA-C01 exam questions on automation ask which trigger type (time-based or event-based) to use and which services to combine.
Time-Based Automation — Run on a Schedule
The most intuitive form of automation is "run this at a specific time" — like a recurring alarm clock for your data pipeline.
EventBridge Scheduler: The scheduling feature of AWS EventBridge. Use cron expressions for precise timing or rate expressions for intervals. For example, runs every day at 8 AM. Targets include Lambda functions, Step Functions state machines, Glue jobs, SQS queues, and more.
MWAA (Amazon Managed Workflows for Apache Airflow): Apache Airflow is an open-source workflow orchestration tool. You write DAGs (Directed Acyclic Graphs) in Python to define dependencies between tasks. MWAA is AWS's fully managed version of Airflow — no server installation or maintenance required.
When MWAA is the right choice: Multi-step complex pipelines (extract → transform → validate → load) Tasks with intricate dependencies between them Pipelines that need retry logic on failure Teams that already use Airflow and want to migrate existing DAGs
EventBridge Scheduler vs MWAA:
| Aspect | EventBridge Scheduler | MWAA | |--------|----------------------|------| | Complexity | Simple | Complex DAGs supported | | Management | Fully serverless | Managed server environment | | Best for | Single job scheduling | Multi-step workflows | | Cost model | Per event | Per environment runtime |
Event-Based Automation — Run When Something Happens
Instead of a schedule, processing starts automatically when a specific event occurs.
S3 Event → Lambda: The moment a file is uploaded to an S3 bucket, a Lambda function fires automatically. For example, when a CSV file arrives, Lambda can read it, validate the data, and write only valid records to DynamoDB — all without human involvement. Configure this in the S3 bucket's Event Notifications settings by pointing to a Lambda target.
DynamoDB Streams → Lambda: Every time a DynamoDB table item is inserted, updated, or deleted, the change is written to DynamoDB Streams. Lambda polls the stream and processes each change event. Common uses: sending a notification when an order status changes to "Delivered", or updating a summary table whenever new records arrive.
Advantages of event-driven patterns: No idle waiting — processing starts the instant data arrives Zero cost when there is no data to process Throughput scales automatically with event volume
Athena Query Automation
Athena is a serverless service for analyzing S3 data with SQL. These queries can also be automated.
StartQueryExecution API: The AWS SDK method for Athena. From inside a Lambda function or Glue job, you can trigger an Athena query in code. After starting the query, use to poll for completion, then to retrieve the results.
Athena Notebooks: An interactive notebook environment powered by Apache Spark. Mix SQL and Python (PySpark) to do exploratory analysis and complex transformations interactively, without managing any infrastructure.
CTAS (Create Table As Select): Create a new table from the results of an Athena query. Pre-aggregating expensive queries and storing the results as a Parquet table dramatically cuts cost and latency for subsequent queries.
DataBrew — No-Code Data Preparation
AWS Glue DataBrew is a visual data preparation tool. Non-developers can clean and transform data by clicking through a spreadsheet-like interface — no code needed.
Key features:
250+ built-in transforms: Standardize date formats, fill missing values, remove duplicates, normalize text, split columns, merge columns, and more — all through point-and-click. Data profiling: DataBrew automatically analyzes a dataset and computes column-level statistics (mean, min/max, missing rate, unique value count). This gives you a fast overview of data quality issues before you start cleaning. Recipes: Every transformation step you apply is recorded as a recipe. Save the recipe and apply it to a different dataset, or schedule it to run automatically on a recurring basis. Quality rules: Define rules that data must satisfy. For example, "the age column must be between 0 and 150." DataBrew automatically flags or separates records that violate the rules.
When DataBrew is the right tool: Business analysts without SQL or Python skills need to clean data Repetitive data cleaning work needs to be standardized into a reusable recipe You want to visually understand data quality issues before writing transformation code
SDK-Based Automation — Boto3
Boto3 is the Python SDK for AWS. Nearly every AWS service can be controlled through Boto3 code. Use it when you want to automate a pipeline entirely in code.
Commonly used patterns: Start a Glue job: Copy an S3 file: Run an Athena query: Read a DynamoDB item:
Combining Boto3 with Lambda lets you respond to EventBridge schedules or S3 events and execute complex pipeline logic with full programmatic control.
Exam Key Points
"Run a single job at a scheduled time" → EventBridge Scheduler "Schedule a complex multi-step workflow" → MWAA (Apache Airflow) "Automatically process data when a file lands in S3" → S3 Event → Lambda "Automatically react to DynamoDB changes" → DynamoDB Streams → Lambda "Run Athena queries from code" → StartQueryExecution API "Save Athena query results as a new table" → CTAS "Clean data without writing code" → DataBrew "Save and reuse DataBrew transformation steps" → Recipe "Control AWS services with Python" → Boto3