Skip to main content

Mastering Salesforce Data Cloud: Ingestion Streams & Transformations

Updated Add a comment
💬 In plain words: Building a data pipeline in Salesforce Data Cloud takes two distinct steps: Ingestion (bringing raw data in) and Transformation (shaping and cleaning that data). You can perform both of these steps in real-time (Streaming) for instant updates, or on a scheduled basis (Batch) for massive, historical datasets. Matching the right ingestion method with the right transformation method is the key to a high-performance Data Cloud architecture.

📌 Key Points

  • Data Streams are your inbound pipelines. They create Data Source Objects (DSO).
  • Transformations clean and map that data from Data Lake Objects (DLO) to target Data Model Objects (DMO).
  • Streaming (both ingestion and transformation) is event-driven and happens instantly as new data arrives.
  • Batching operates on a schedule, processing large volumes of data at rest and allowing for complex multi-table joins.
  • If a native connector doesn't exist for your external system, you use the Ingestion API to push data into Data Cloud via HTTP.

📥 1. Step One: Data Streams & Ingestion Architecture

A Data Stream is the configured pipeline that continuously imports data from an external source into Data Cloud. Depending on your business needs, you will choose one of two ingestion modes:

  • Streaming Ingestion: Event-driven, low-latency micro-batches pushed over HTTP using JSON payloads. Ideal for live signals, mobile app telemetry, and urgent web events.
  • Batch (Bulk) Ingestion: Scheduled file uploads designed for massive volumes. It processes records via CSV upserts/deletes pulling from standard connectors (Salesforce CRM, AWS S3, Snowflake, etc.).
📌 Real-Life Example: A shopper adds a pair of shoes to their cart on your website. That event requires sub-second personalization, so it routes via Streaming Ingestion using the Web SDK. Meanwhile, your backend ERP system processes a nightly CSV file of 40 million settled historical transactions—this routes through a scheduled Batch Stream.
Data Streams and Ingestion in Data Cloud

🔄 2. Step Two: Data Transformations

Once data lands inside Data Cloud, it sits in a raw format. Transformations are applied to clean, restructure, and enrich this source data before it maps to target Data Model Objects (DMO), which ultimately fuel your unified customer profiles.

  • Streaming Transforms: Execute immediately in the live ingestion path. They evaluate only newly arriving data packets, running lightweight calculations on the fly without looking backward at old records.
  • Batch Transforms: Run on data at rest on an automated schedule. Because they have access to the complete historical dataset, they are used for heavy data cleansing, multi-table joins, and complex denormalization.
Salesforce Data Cloud Streaming vs Batch Transformations Diagram

⚖️ 3. Feature Comparison: Streaming vs. Batch Pipelines

Pipeline Feature Streaming (Ingestion & Transform) Batch (Ingestion & Transform)
Latency & Trigger Near real-time (Event-driven as data arrives) Scheduled intervals (Hourly, daily, or manual)
Processing Scope New, incoming records only Entire historical datasets in the source DLO
Primary Use Case Live web signals, simple on-the-fly formulas Massive nightly dumps, multi-source joins, aggregations
Data Format & APIs JSON payloads / Ingestion API — Streaming CSV, Parquet / Connectors, Ingestion API — Bulk
🧠 CORE ARCHITECTURAL RULE: Stream what must be live; batch the volume that can wait. Use streaming transforms for immediate, single-record enrichment, and batch transforms for complete historical processing and heavy multi-object joins.

🚨 4. Common Traps & Best Practices

⚠ INTERVIEW TRAP: Streaming Transforms Do Not Backfill
A streaming transform will never update historical records that were ingested prior to its deployment. This is intentional architecture. If historical data must be modified to match a new formula, you must run a one-time Batch Transform backfill first, then rely on the streaming transform for continuous future updates.
⚠ ARCHITECTURE PITFALL: Overusing Streaming Ingestion
Do not stream massive legacy datasets into Data Cloud just because you can. Streaming large historical datasets spikes API consumption costs without adding functional business value. Default to Batch ingestion for bulk workloads.
🧭 360 Architecture Card — Streams & Transformations
  • Standard Setup Sequence: 1. Connect Source → 2. Generate DSO → 3. Map to Category → 4. Transform → 5. Map to DMO.
  • Ingestion API Requirement: Use the Ingestion API whenever pre-built native connectors (like S3 or Snowflake) do not exist for your external systems. You must define the schema in OpenAPI (YAML) first.
  • Compute Limits: Full-table batch runs consume significant compute credits. Schedule them during off-peak hours to prevent pipeline congestion.

🎤 5. Core Interview Q&A

Q: You need real-time website behavioral tracking in Data Cloud within seconds, plus a nightly ingest of a 40-million row ERP transaction CSV file. How should you design this architecture?

🎯 Say this first: "I would separate the sources into two distinct Data Streams based on latency needs."

Detailed Answer:

  • Real-time Behavioral Data: Configure a streaming Data Stream using the Salesforce Interactions SDK or the Ingestion API Streaming endpoint. This pushes events in JSON micro-batches to update unified profiles instantly.
  • Nightly 40M File: Configure a batch Data Stream via an automated connector (like Amazon S3 or Google Cloud Storage). Set the refresh schedule to off-peak hours to efficiently handle the massive volume without impacting live system performance.

Q: A newly created streaming transform is working perfectly for incoming records, but users are complaining that the existing records in the DLO haven't updated. How do you resolve this?

Answer: Explain that this behavior is by design. Streaming transforms only evaluate incoming data at the moment of ingestion; they do not traverse historical data. To fix it, you should not alter the streaming transform. Instead, create and execute a one-time Batch Transform across the source DLO to backfill the historical records. Once complete, the streaming transform will seamlessly handle all new inbound records moving forward.

Q: What exactly is the Ingestion API, and when should you choose it over standard pre-built Data Cloud connectors?

Answer: The Ingestion API is a RESTful interface that allows developers to programmatically push custom datasets into Salesforce Data Cloud. You should choose Out-of-the-Box Connectors when integrating supported platforms (like Marketing Cloud, AWS, or Snowflake) because they require zero custom code. You should reach for the Ingestion API when dealing with bespoke in-house software, legacy on-premise systems, or custom mobile app telemetry where direct HTTP pushing is the only viable method.

❓ Follow-Up Question: Can you apply a Batch Transform to a Streaming Ingestion pipeline?

Yes! This is actually a very common hybrid architecture. You can ingest data continuously via a Streaming Data Stream to get raw data into a Data Lake Object (DLO) quickly. Then, you can run a scheduled Batch Transform on that same DLO later that night to perform heavy, multi-table joins (like merging today's live web clicks with historical CRM purchase data) that a lightweight streaming transform simply couldn't handle.

Was this article helpful?

Have a question or spotted a mistake? Ask here about this article

Found this guide useful? Buy me a coffee (opens in a new tab)

Comments

Questions, corrections and your own experience are all welcome.