Domain 4 of the official DP-900 skills outline includes the item "Identify Microsoft cloud services for real-time analytics." The Microsoft Learn module that prepares you for it now centers on Microsoft Fabric Real-Time Intelligence and Apache Spark Structured Streaming. If you walk into the exam knowing only Event Hubs and Stream Analytics, questions that mention Eventstream, Eventhouse, or KQL can easily trip you up.
This post follows up on the Real-Time Data Analytics entry in our DP-900 study series. That post covered batch versus streaming, Event Hubs, IoT Hub, Stream Analytics, and window functions. Here we start with the common elements of a stream processing architecture, then cover the components of Fabric Real-Time Intelligence, Spark Structured Streaming with Delta Lake, and how to choose a service for each situation.
---
Common Elements of a Stream Processing Architecture
Picture an airport baggage conveyor. Bags come up from the check-in counter, a scanner inspects them nonstop as they move along the belt, and they are sorted into exits by destination. Processing each item as it arrives, instead of waiting for all of them to pile up, is what stream processing means.
Whatever technology you use, most streaming solutions follow the same four steps.
An event generates data. A sensor signal, a social media post, or a new line in a log file is something that happens and produces digital data. The data is captured in a streaming source. In simple cases that can be a folder in cloud storage or a database table; in more advanced solutions it is a queue that keeps events in order and makes sure each one is processed only once. The data is processed. A query that runs continuously (a perpetual query) selects certain events, projects values, or aggregates them over time windows such as one minute. The results are written to an output, or sink. That can be a file, a database table, a real-time dashboard, or another queue for further processing.
| Role | Commonly used Azure services | |------|------| | Source (ingestion) | Azure Event Hubs, Azure IoT Hub, Azure Data Lake Storage Gen2, Apache Kafka | | Processing | Fabric Real-Time Intelligence, Spark Structured Streaming, Azure Stream Analytics | | Sink (files) | Azure Data Lake Storage Gen2, Microsoft OneLake, Azure Blob Storage | | Sink (tables) | Azure SQL Database, Azure Databricks, Microsoft Fabric | | Sink (visuals or further processing) | Microsoft Power BI, Azure Event Hubs |
Event Hubs is an ingestion service for high volumes of events. Events within a partition are delivered in order, and delivery is at-least-once. IoT Hub is a similar ingestion service optimized for managing IoT devices.
Exam tip: "A continuously running query that aggregates incoming events over time windows" → the perpetual query of stream processing. "The place that receives the results (a file, table, or dashboard)" → the sink.
---
Architectures That Combine Batch and Streaming
Real analytics solutions often do not pick just one approach. With road traffic, for example, the number of cars passing right now goes to a real-time dashboard, while the same data is stored so that traffic trends over the past year can be analyzed in batches.
The classic pattern for combining the two is the lambda architecture. Streaming data is processed immediately in a real-time layer and sent to dashboards, while data from other sources lands in a data lake, is processed periodically in a batch layer, and is stored in a data warehouse. The real-time results are also saved to the warehouse for historical analysis.
| Pattern | Structure | Characteristics | |------|------|------| | Lambda architecture | Separate real-time and batch layers | Delivers both real-time results and accurate historical analysis; two layers make it harder to manage | | Kappa architecture | No batch layer; all data is processed as a stream | Simpler structure; historical reprocessing replays the stream |
Microsoft Learn also mentions the delta architecture, but DP-900 does not test its details. Even when no real-time analysis or visualization is needed, it is common to capture data with streaming technology, store it, and process it later in batches.
Exam tip: "A design with both a real-time path and a batch path to support live dashboards and historical analysis" → lambda architecture.
---
Microsoft Fabric Real-Time Intelligence
Fabric Real-Time Intelligence is a set of tools built into Microsoft Fabric for ingesting, processing, and analyzing streaming data. Connecting event sources, storing and querying data, building live dashboards, and triggering automated actions when conditions are met all happen inside the same Fabric workspace. Think of it as one SaaS experience for work that used to require stitching several Azure services together.
Despite the name, your data does not have to arrive fast or in huge volumes. The point is that you build solutions that react to events as they happen, rather than jobs that run on a schedule.
Real-Time Hub: A Catalog of Streaming Data
The Real-Time hub is a catalog that brings an organization's streaming data together in one place. You can discover running data streams, Microsoft sources you can connect, and Fabric events, and share them with other teams.
Eventstream: Ingest, Transform, and Route
An eventstream brings real-time events into Fabric without code, optionally transforms them with operations such as filtering or aggregation, and sends them to one or more destinations. Sources include Azure Event Hubs, Azure IoT Hub, change data capture (CDC) from Azure SQL Database or Azure Cosmos DB, Apache Kafka, Amazon Kinesis, and Google Pub/Sub. Destinations include an Eventhouse, a Lakehouse (stored in Delta Lake format), Activator, and a custom endpoint for external apps.
Eventhouse and KQL: A Database for Event Data
An Eventhouse is a database designed for storing and analyzing time-based event data, able to query billions of events in seconds. One Eventhouse can hold multiple KQL databases, and it handles semi-structured data such as JSON and free text as well as structured data. It suits telemetry, logs, time series, and IoT data.
The query language is KQL (Kusto Query Language), built for fast analysis of logs and telemetry, and Copilot can turn natural-language questions into KQL. The standalone Azure PaaS service for the same kind of work is Azure Data Explorer, which also uses KQL.
Real-Time Dashboards and Activator
Real-Time Dashboards visualize incoming data live. Activator lets you define conditions without code; when the data meets a condition, it sends an email or Teams notification or runs an action such as a Power Automate flow or a Fabric pipeline or notebook. It turns insight into action, as in "notify the on-call engineer when a sensor's temperature crosses the threshold."
!How data flows through Fabric Real-Time Intelligence
| Component | Role | Keywords to recall in the exam | |------|------|------| | Real-Time hub | Catalog of the organization's streaming data | Discover and share streams | | Eventstream | No-code ingestion, transformation, and routing | Bring events into Fabric | | Eventhouse | Database for time-based event data | KQL database, logs and telemetry | | KQL | Query language of the Eventhouse | Kusto Query Language | | Real-Time Dashboard | Live visualization | The state of things right now | | Activator | Automated actions when conditions are detected | Alerts, Power Automate, no code |
Exam tip: "Ingest streaming events into Fabric without code, transform them, and send them to multiple destinations" → Eventstream. "A Fabric store that queries logs and telemetry quickly with KQL" → Eventhouse. "Automatically send an alert or run an action when data meets a condition" → Activator.
---
Spark Structured Streaming and Delta Lake
Apache Spark is an engine that splits large data processing across many machines (a cluster) so the work runs in parallel. On Azure you can use it in Microsoft Fabric and Azure Databricks, write code in Python, Scala, Java, R, or SQL, and run both batch and stream processing.
How Structured Streaming Works
Spark Structured Streaming is the stream processing library built into Spark. The easiest way to understand it is to treat a live stream like a table in a spreadsheet, except that new rows keep appearing as events arrive.
Connect to a streaming source, such as a message queue like Azure Event Hubs, a folder of files, or a network feed. Read the incoming data into a dataframe, a table of rows and columns that keeps filling as events arrive. Write a query on the dataframe, for example to count events per minute or calculate a rolling average. Write the query results to a sink such as a file, a database, or a dashboard.
It is a good choice when you already process data with Spark and want to extend that work to real-time streams.
Unifying Batch and Streaming with Delta Lake
By default, a data lake is just a collection of files, with nothing to guarantee that the data is complete or consistent. Delta Lake is an open-source storage format that adds database-like guarantees.
Reliability: changes are tracked, so a write that fails partway does not corrupt the data. Schema enforcement: only data that matches the defined structure is accepted. Unified batch and streaming: the same Delta table can be the sink for streaming results and the source for batch queries, so you do not need separate storage for historical and live data.
The Spark runtimes in Fabric and Azure Databricks support Delta Lake out of the box.
Exam tip: "A team already using Spark wants to extend its processing to real-time streams" → Spark Structured Streaming. "Use the same table as a streaming sink and a batch query source" → a Delta Lake table.
---
Which Real-Time Analytics Service Should You Choose?
Microsoft Learn presents three real-time analytics services. Which one is the ri