CDL Real-Time Analytics and Looker: Pub/Sub, Dataflow, Managed Service for Apache Spark, and Data Democratization

Batch vs. streaming processing, why real-time analytics matters, the roles of Pub/Sub, Dataflow, Managed Service for Apache Spark (formerly Dataproc), and BigQuery continuous queries, how Looker democratizes data access, and how Looker differs from Data Studio (formerly Looker Studio).

In the August 2026 revision of the Cloud Digital Leader exam guide, the last item in Domain 2 (about 18%) is how smart analytics, business intelligence (BI) tools, and streaming analytics add value to a business. It asks how Looker democratizes access to data, the value of analyzing BigQuery data in Looker to build real-time reports and dashboards and integrate them into workflows, why real-time streaming analytics matters, and the products that modernize data pipelines: Pub/Sub, Dataflow, and Managed Service for Apache Spark. Dataflow and Pub/Sub come up often in the question bank too, and Dataproc was renamed Managed Service for Apache Spark in April 2026.

This post starts with the difference between batch and streaming processing, then covers why real-time analytics matters, the products that modernize pipelines, how Looker democratizes data, and real-time reports and workflow integration with BigQuery and Looker. The big picture of the pipeline is covered first in Data Transformation, and the data supply chain from genesis to activation in Data Fundamentals.

---

 

Batch Processing vs. Streaming Processing

Think of a post office. Sorting all of the day's mail at once in the evening is batch processing; sorting each parcel the moment it arrives and loading it straight onto a delivery truck is streaming processing.

| Aspect | Batch processing | Streaming processing | |------|------|------| | When it runs | Data is collected and processed all at once at a set time | Continuously, as soon as data arrives | | Time to results | Hours to a day | Seconds or so | | Examples | Month-end settlement, daily sales reports, bulk data migration | Fraud detection, real-time inventory, IoT alerts | | Google Cloud examples | Dataflow batch jobs, Managed Service for Apache Spark, BigQuery | Pub/Sub with Dataflow streaming, BigQuery continuous queries |

Exam tip: "Process data the moment it is generated and respond within seconds" is streaming; "collect a day's worth and process it overnight" is batch. A hallmark of Dataflow is that it handles both with the same programming model.

---

 

Why Real-Time Streaming Analytics Matters

Some data has a short shelf life, like milk. You can stop fraud at the moment a payment happens, but if you find out from a report the next morning, the damage is already done. That is why the exam guide describes real-time streaming analytics as essential to modern business.

| Industry | What real-time analytics does | What happens if it is late | |------|------|------| | Finance | Detects suspicious transactions at the moment of payment | Fraud losses grow | | Retail | Adjusts inventory and prices in real time and personalizes recommendations | Stockouts and lost sales | | Manufacturing | Detects signs of equipment trouble from sensor data | Equipment failure and production stoppages | | Logistics | Updates vehicle locations and estimated arrival times | Delivery delays and customer complaints | | Media and gaming | Changes content and events to match user behavior | Users leave |

Exam tip: Questions where "now" matters, such as "send a tailored coupon while the customer is still in the store" or "warn before the machine breaks down," are answered by streaming analytics.

---

 

The Products That Modernize Pipelines: Pub/Sub, Dataflow, and Managed Service for Apache Spark

A real-time analytics pipeline is usually made up of Pub/Sub to receive events, Dataflow to process them, BigQuery to store and analyze them, and Looker to present and connect them. If you already have jobs built on Spark or Hadoop, you can move them as they are to Managed Service for Apache Spark.

!A real-time analytics pipeline running from event sources through Pub/Sub, Dataflow, and BigQuery to Looker, plus the Managed Service for Apache Spark path

| Product | Role | Characteristics | When it fits | |------|------|------|------| | Pub/Sub | Receives and delivers messages (publishers and subscribers) | Asynchronous, latency typically around 100 milliseconds, decouples senders from receivers | Collecting events from many devices and apps, connecting systems | | Dataflow | Processes streams and batches together | Built on Apache Beam, fully managed (creates and removes worker VMs automatically), processes each record exactly once by default | Real-time transformation and aggregation, ETL | | Managed Service for Apache Spark | Open source processing such as Spark and Hadoop (formerly Dataproc) | Cluster and serverless options; clusters spin up quickly and shut down when not needed | Moving existing Spark and Hadoop jobs, data science | | BigQuery continuous queries | Runs SQL continuously inside BigQuery | Analyzes incoming data with SQL in real time and exports results to Pub/Sub, Bigtable, and more | Simple real-time processing with SQL alone |

You might wonder why Pub/Sub is needed. If the sending system calls the receiving system directly, the sender stalls whenever the receiver pauses even briefly. With Pub/Sub acting as a mailbox in the middle, the sender just drops messages in and the receiver takes them out at its own pace. Decoupling them this way makes the whole system more flexible and robust.

Exam tip: "Receive events from thousands of devices and deliver them to several systems" is Pub/Sub. "Process streams and batches with one codebase and no servers to manage" is Dataflow. "Move existing Hadoop and Spark jobs to the cloud with minimal changes" is Managed Service for Apache Spark.

---

 

Looker: Democratizing Access to Data

Data democratization means making data understandable and usable for anyone in the organization, not just data specialists. The trouble is that when everyone calculates numbers differently, meetings end in arguments about whose number is right. Looker solves this by getting everyone to use the same dictionary.

| How Looker does it | Description | Effect | |------|------|------| | Semantic layer (LookML) | Defines metrics such as "revenue" or "active customer" once, in one place | Ends the confusion of each department getting different numbers | | Self-service exploration | Explore data and build charts with clicks, no SQL required | Less time spent asking the analytics team and waiting | | Direct database connection | Queries databases such as BigQuery directly instead of copying the data | You always see the latest data | | Dashboards and sharing | Build dashboards and schedule or share them | Executives and front-line teams look at the same numbers | | Embedding and APIs | Put dashboards inside other apps and portals, and send data to other systems | Analytics moves into the screens where work happens |

Exam tip: "Let the whole organization explore data directly on consistent metric definitions" and "democratize access to data" point to Looker. When you see the word LookML, think Looker.

---

 

BigQuery and Looker: Real-Time Reports and Workflow Integration

The exam guide describes the value of analyzing and visualizing BigQuery data in Looker in three ways: real-time reports, dashboards, and integrating data into workflows.

| Use | Example | |------|------| | Real-time reports | Orders streamed into BigQuery show up in the report within seconds | | Dashboards | Executives and front-line teams check regional sales and inventory on the same numbers | | Workflow integration | Alert the owner when inventory drops below a threshold, embed customer dashboards inside the sales portal, pass analysis results to other business systems |

Data that does not stop at a dashboard but leads to someone's next action: that is activation, the final stage of the data supply chain.

Telling Looker and Data Studio Apart

Google has two BI tools with similar names. The free visualization tool that was renamed from Data Studio to Looker Studio in 2022 went back to the name Data Studio in April 2026. The reason is to avoid confusing the personal visualization tool with Looker, the enterprise BI platform.

| Aspect | Looker | Data Studio (formerly Looker Studio) | |------|------|------| | Nature | Governance-focused enterprise BI platform | Visualization and reporting tool for individuals and teams | | Metric management | Defined centrally in the LookML semantic layer | Each report connects to its data sources on its own | | When it fits | Company-wide standard metrics, embedded analytics, workflow integration | Quick ad-hoc reports, marketing dashboards | | Pricing | Paid | Free (the paid Data Studio Pro is separate) |

Exam tip: "Consistent metric definitions across the company" and "governance" point to Looker; "build and share reports quickly for free" points to Data Studio.

---

 

Exam Quick Reference

The key points to review one last time before you walk into the exam.

| Topic | What to remember | |------|------| | Batch vs. streaming | Collect and process all at once vs. continuously as data arrives | | Why streaming is needed | Decisions where "now" matters, such as fraud detection, real-time inventory, equipment anomalies | | Pub/Sub | Asynchronous messaging, decouples senders and receivers, event ingestion | | Dataflow | Unified stream and batch processing, Apache Beam, fully managed | | Managed Service for Apache Spark | Formerly Dataproc, existing Spark · Hadoop jobs, cluster or serverless | | BigQuery continuous queries | Real-time analysis of incoming data with SQL | | Typical pipeline | Pub/Sub → Dataflow → BigQuery → Looker | | Data democratization | People who are not specialists understand and use data directly | | Looker | LookML semantic layer, consistent metrics, self-service BI, embedding | | BigQuery + Looker | Real-time reports, dashboards, workflow integration | | Data Studio | Formerly Looker Studio, free visualization and reporting tool |

---

 

Next Steps

With the data domain done, the next step is how to protect that data and those systems. Zero trust, the three states of encryption, Cloud Armor, and IAP continue in Security

Back to blog list