Where to store data is one of the most critical design decisions in data engineering. Choosing the wrong storage can make queries tens of times slower, cause costs to explode, or make certain features impossible to implement. Choosing the right storage makes queries fast, keeps costs low, and keeps the system simple. In the AWS DEA-C01 exam, you are tested on your ability to analyze a given set of requirements and access patterns to select the optimal storage service.
Storage Selection Guide — What to Use in Which Situation?
| Requirement | Recommended service | |------------|-------------------| | Large-scale analytical queries (OLAP), SQL aggregations and joins | Amazon Redshift | | Key-value lookups, millisecond response, large-scale read/write | Amazon DynamoDB | | S3-based data lake, serverless SQL analysis | Lake Formation + Athena | | Relational data, OLTP (transaction processing) | Amazon RDS / Aurora | | Ultra-low-latency in-memory cache | Amazon ElastiCache / MemoryDB | | ACID transactions on S3 data, time travel | Apache Iceberg | | Search, log analysis, full-text search | Amazon OpenSearch Service |
More important than memorizing this table is understanding "why you would use each storage service." If you understand the design principles behind each service, you can reason to the correct answer even in scenarios you have never seen before.
!Choosing a data store: Redshift, Iceberg, Lake Formation
Amazon Redshift — The Data Warehouse Optimized for Analytics
Amazon Redshift is a fully managed data warehouse designed for large-scale analytical workloads (OLAP). It is built to run aggregation queries across tens of billions of rows quickly.
Columnar Storage is the key
Traditional databases store data row by row. "Customer 1's name, age, address, purchase amount, and loyalty points" are stored as one row. Redshift stores data column by column instead. All customers' "purchase amounts" are stored together in one column-oriented block.
Why does this help analytics? When you run a query like "calculate the average purchase amount over the last year," a row-based database reads every column for every customer. Redshift only reads the "purchase amount" column. Out of potentially hundreds of columns, only the few needed ones are read — dramatically reducing I/O.
Redshift Spectrum — Query S3 Data Directly
Spectrum lets you run SQL queries against data stored in S3 without loading it into Redshift first. It acts as a bridge between the data lake (S3) and the data warehouse (Redshift).
For example: keep the last 3 years of data loaded in Redshift for fast access, while keeping older historical data cheaply in S3, and query it with Spectrum only when needed. The strategy is "hot data in Redshift, cold data in S3."
Federated Query
Query data in Amazon RDS, Aurora, or S3 directly from within Redshift. You do not need to copy the data into Redshift first. You can JOIN data from multiple sources in a single SQL query.
Concurrency Scaling
When query load spikes at peak times, Redshift automatically adds additional cluster capacity. Even when dozens of analysts run queries simultaneously during month-end reporting season, performance is maintained.
RA3 Instances — Separating Compute and Storage
Traditional Redshift nodes had compute and storage tightly coupled. Growing your data meant growing compute too. RA3 instances separate compute from storage. If your data grows but query volume stays the same, scale only storage. If query volume grows but data size stays the same, scale only compute — each independently.
Apache Iceberg — Adding Data Warehouse Capabilities to the Data Lake
S3 is cheap and infinitely scalable storage. But by default, modifying or overwriting data in S3 is awkward, and concurrent writes from multiple processes can cause data inconsistencies. Apache Iceberg is an open table format that brings relational database-level capabilities to data lakes like S3.
A library analogy: S3 is a warehouse full of books stacked in piles. Iceberg is the sophisticated catalog system that tracks "which book is where, when it was added, and which version it is."
ACID Transactions
ACID describes four properties that database transactions should have: Atomicity: Operations either fully succeed or fully fail — there is no partial state. Consistency: Data always remains in a valid state. Isolation: Concurrent transactions do not interfere with each other. Durability: Completed transactions persist even if the system crashes.
With Iceberg, data stored in S3 gains these properties. For example, if your system crashes while updating 100 million records, you will never be left with half-updated data in an inconsistent state.
Time Travel
Iceberg tracks the change history of your data. You can query "show me the data as it was at 10 AM yesterday." This is invaluable for recovering from accidental data deletion, auditing what data looked like at a specific point in time, or running point-in-time analyses.
Schema Evolution
You can change a table's schema without rewriting existing data. Adding new columns, renaming existing ones, dropping columns — all of this is possible without touching the existing files in S3. Old data remains compatible with the new schema and continues to work correctly.
Iceberg is fully compatible with Amazon Athena, Amazon EMR, AWS Glue, and Redshift Spectrum.
AWS Lake Formation — The Security Manager for Your Data Lake
Lake Formation is a service for building and securing an S3-based data lake with fine-grained access policies. Using the analogy of a building access control system: Lake Formation manages "which employee can access which floor and which room."
Fine-Grained Access Control
Lake Formation controls access permissions for a data lake at very granular levels:
Database/Table level: Specific teams can only see specific tables. Column level: The HR team can see the "salary" column, but other teams cannot. Row level: The sales team can only see data for their own region. Cell level: A specific user can see only a specific column of a specific row.
This level of fine-grained control is difficult to implement with IAM policies alone. Lake Formation manages this complex permission system centrally from a single place.
Data Filters
Lake Formation's data filters let you show different users different views of the same physical table. For example, from a table containing global customer data, configure it so the US team sees only US customers and the EU team sees only EU customers. The physical data is a single table, but what each user sees is filtered according to their permissions.
Integration with Glue Data Catalog
Lake Formation uses the AWS Glue Data Catalog as its metadata store. Glue Crawlers scan data and register table metadata into the catalog, and Lake Formation controls who can access those tables. Athena, Redshift Spectrum, and EMR all query through the Glue Data Catalog and are subject to Lake Formation's access controls.
Exam Key Points Summary
| Keyword | Service/Concept | |---------|----------------| | Large-scale OLAP, SQL aggregation analysis | Amazon Redshift | | SQL query S3 data without loading into Redshift | Redshift Spectrum | | Single SQL query across multiple sources (RDS, S3, Redshift) | Federated Query | | Handle sudden spike in concurrent queries at peak times | Concurrency Scaling | | Scale compute and storage independently | Redshift RA3 instances | | ACID transactions on S3 data | Apache Iceberg | | Query data at a specific point in the past | Iceberg Time Travel | | Change schema without rewriting existing data | Iceberg Schema Evolution | | Column/row level access control for data lake | AWS Lake Formation | | Show different rows/columns to different users | Lake Formation Data Filters | | Key-value lookups, millisecond response | Amazon DynamoDB |
The core message of Iceberg: "Add data warehouse-level capabilities (ACID, time travel, schema evolution) to the data lake (S3's low cost and scalability)."