💬 In plain words: Think of Data Streams as inbound pipelines into Salesforce Data Cloud. Streaming ingestion brings in real-time event micro-batches (such as web clicks or mobile interactions), while Batch ingestion loads scheduled, high-volume files (like nightly ERP dumps or CRM syncs). If no native connector exists, the Ingestion API acts as your custom push gateway. Your choice always boils down to one factor: data freshness vs. volume.
📌 Real-Life Example: A shopper adds an item to their cart on your website. That event needs sub-second personalization, so it routes via Streaming Ingestion using the Web SDK or Ingestion API. Meanwhile, your ERP processes a nightly file of 40 million settled transactions—this routes through a scheduled Batch Stream. Match your pipeline to your latency requirements.
Core Concepts: Understanding Ingestion Architecture
A Data Stream is a configured pipeline that continuously imports data from an external source into Salesforce Data Cloud. When you deploy a stream, Data Cloud automatically generates a corresponding Data Source Object (DSO).
Data Cloud organizes data ingestion into two primary modes:
- Batch (Bulk) Ingestion: Scheduled file uploads designed for massive volumes. It processes records via CSV upserts and deletes, pulling data via standard connectors (Salesforce CRM, AWS S3, Google Cloud Platform, Snowflake, etc.). Use Batch when near-instant updates are not required.
- Streaming Ingestion: Event-driven, low-latency micro-batches pushed over HTTP using JSON payloads. Use Streaming when updates must trigger downstream actions in near real time, such as in-app signals, live cart updates, and telemetry feeds.
⚙️ Standard Setup Sequence for Data Streams:
- Select the source connector or API endpoint.
- Define or generate the Data Source Object (
DSO). - Map the object to a standard data category (
Profile,Engagement, orOther). - Apply formula-based data transformations if required.
- Set the refresh schedule and look-back window (e.g., 90 days for commerce, longer for historical marketing data).
Feature Comparison: Streaming vs. Batch Ingestion
| Attribute | Streaming Ingestion | Batch (Bulk) Ingestion |
|---|---|---|
| Latency | Near real-time (event-driven micro-batches) | Scheduled batches (hourly, daily, or on-demand) |
| Data Format | JSON payloads | CSV files, Parquet, or native connector queries |
| Best Used For | Live signals, web & mobile behavior, urgent events | High-volume master datasets, nightly transactions, logs |
| Primary API Endpoint | Ingestion API — Streaming Endpoint |
Ingestion API — Bulk Endpoint / Connectors |
🧠 RULE OF THUMB: Default to Batch ingestion for bulk workloads because it is cost-effective and simpler to maintain. Upgrade to Streaming only when operational decisions or personalizations go stale within minutes.
🧭 360 Architecture Card — Data Streams & Ingestion
- Golden Rule: Stream what must be live; batch the volume that can wait.
- Architectural Gain: Both ingestion modes map downstream to the same unified Data Model (
DMO), eliminating duplicate data pipelines. - When to Reach for Ingestion API: Use the Ingestion API whenever pre-built native connectors do not exist for your external proprietary systems. Define the schema in OpenAPI (
YAML) first. - Cost & Limits: Streaming consumes more credits per row. Do not stream multi-million row files when nightly batches achieve the exact same operational goal.
- Downstream Impact: Ingestion mode dictates which downstream transform features (batch vs. streaming transformations) are supported.
Scenario-Based Architecture Q&A
Q: You need real-time website behavioral tracking in Data Cloud within seconds, plus a nightly ingest of a 40-million row ERP transaction CSV file. How should you design the ingestion architecture?
🎯 Interview / Architectural Answer: "Separate the sources into two distinct Data Streams: Route real-time web telemetry through Streaming Ingestion (Web SDK or Ingestion API Streaming endpoint), and process the nightly 40-million row file through a scheduled Batch Ingestion stream."
Detailed Implementation Steps:
- Real-time Behavioral Data: Configure a streaming Data Stream using the Salesforce Interactions SDK or Ingestion API. Push events in JSON micro-batches to update unified profiles and trigger calculated insights in near real time.
- Nightly 40M File: Configure a batch Data Stream via an automated connector (such as Amazon S3, Google Cloud Storage, or Azure Blob) or the Ingestion API Bulk endpoint. Set the refresh schedule to off-peak hours with an optimized look-back window.
⚠️ Common Architecture Pitfall: Avoid trying to stream large legacy datasets into Data Cloud. Streaming large historical datasets increases API consumption costs without adding functional business value.
Q: What is the Ingestion API, and when should you choose it over standard pre-built connectors?
The Ingestion API is a RESTful interface that allows developers to programmatically push external datasets into Salesforce Data Cloud. It requires uploading an OpenAPI specification schema (YAML format) that describes the object and field structure.
- Choose Out-of-the-Box Connectors: When integrating supported platforms (Salesforce CRM, Marketing Cloud, AWS S3, Google Cloud, Snowflake). Native connectors require no custom code and handle incremental syncs automatically.
- Choose Ingestion API: For custom bespoke software, legacy on-premise systems, mobile app telemetry, or microservice architectures where direct HTTP pushing is preferred.