Published November 14, 2026 · By Henry Palma

Databricks Data Engineer Associate Exam Prep, Lesson 2: Batch, Streaming and Incremental Ingestion

Databricks Data Engineer Associate exam prep, Section 2: compare batch, incremental and streaming ingestion with Auto Loader, and Lakeflow Connect connectors.

Data gets into Databricks in three ways: a batch load re-reads the whole source every run, an incremental load processes only what is new since the last run, and streaming runs the same incremental idea continuously. This post follows Lesson 2 of the Databricks Data Engineer Associate exam prep series. It opens Section 2 of the August 2026 exam guide, Data Ingestion and Loading, which carries 21% of the exam weight.

Watch on YouTube

The notebook and cheat sheet are in the A02 folder on GitHub. This is independent study material, not official Databricks training.

The exam objective

Section 2 opens with a broad objective: enable and detail data ingestion patterns, including batch, streaming and incremental loading, and import data from sources such as local files, Lakeflow Connect standard connectors and Lakeflow Connect managed connectors. This lesson is the map for the section.

Batch, incremental and streaming

Batch reads the whole source every time. It is simple and fine for small sources or ones that get fully replaced, but the cost grows with all the data you have ever received.
Incremental processes only data that is new since the last run. It remembers what it already loaded, so each run stays small.
Streaming is the same idea running all the time, picking data up within seconds or minutes of arrival.

On Databricks, incremental and streaming use the same Structured Streaming engine. The trigger decides how often it runs.

Set up a landing zone in a volume

In a Python notebook on Serverless, the first cell creates a new schema and a Unity Catalog volume called landing. A volume stores files instead of tables, and its path starts with /Volumes.

schema = f"bt_ingest_{uuid.uuid4().hex[:12]}"      # new every run
spark.sql(f"CREATE SCHEMA {catalog}.{schema} COMMENT 'Build and Teach A02'")
spark.sql(f"CREATE VOLUME {catalog}.{schema}.landing")
base = f"/Volumes/{catalog}/{schema}/landing"
src  = f"{base}/orders"                     # raw files land here
ckpt = f"{base}/_checkpoints/orders_bronze"  # Auto Loader progress

Cell two plays the source system and writes a CSV file, orders_day1.csv, with three fictional orders. You can see it in Catalog Explorer under the schema's Volumes > landing > orders.

Pattern 1: batch with overwrite

order_schema = "order_id INT, customer STRING, amount DOUBLE, order_date DATE"

def runbatch(): df = (spark.read.format("csv").option("header", "true") .schema(orderschema).load(src)) # reads EVERY file df.write.mode("overwrite").saveAsTable(f"{catalog}.{schema}.ordersbatch") return spark.table(f"{catalog}.{schema}.ordersbatch").count() ```

Overwrite matters. A batch read sees every file each time, so appending would load the same orders again on every run. After the first run, the table has three rows.

Pattern 2: incremental with Auto Loader

def run_auto_loader():
    stream = (spark.readStream.format("cloudFiles")
              .option("cloudFiles.format", "csv").option("header", "true")
              .schema(order_schema).load(src)
              .withColumn("source_file", col("_metadata.file_name")))
    (stream.writeStream.option("checkpointLocation", ckpt)
           .trigger(availableNow=True).toTable(bronze).awaitTermination())

Auto Loader is the cloudFiles source in spark.readStream. The source_file column comes from the hidden file metadata, so you can see where each row came from. On the write side:

The checkpoint location records which files Auto Loader has already processed.
availableNow processes everything new, then stops. That is a streaming query running as an incremental batch.

The first run also finds three rows.

Why availableNow on serverless

On serverless notebooks and jobs, which covers everything in Free Edition, Structured Streaming supports the availableNow trigger and the older trigger once, which is deprecated. Processing-time and continuous triggers are not supported there and raise an error. For always-on streaming, Databricks points you to a pipeline in continuous mode or an availableNow query inside a continuous job. Most incremental loads are availableNow runs on a schedule.

Day 2: same total, different work

A second file arrives with two more orders, and both loads run again:

The batch load reads both files, all five rows, and overwrites the table with five rows.
Auto Loader checks its checkpoint, skips the day 1 file, reads only the new file, and adds two rows for five in total.

Same final count, very different work. With five years of files, the batch job re-reads five years every night, and the incremental job reads one day.

Grouping the bronze table by source_file shows three rows from day 1 and two from day 2, each file loaded exactly once. Keeping the source file name in a bronze table is a cheap habit that makes reruns and audits easy to check.

Local file uploads

The exam names local files as a source. There are two common routes:

From New, choose Add or upload data, then Create or modify a table. Drop in a file, preview it, and Databricks creates a managed Delta table. That is a one-time batch load.
Upload the file into a volume and read it with code, which is the pattern in this lesson.

Uploads suit small reference files, not data that keeps arriving.

Lakeflow Connect: standard vs managed connectors

Lakeflow Connect is the umbrella name for Databricks ingestion.

Standard connectors use Spark and SQL APIs, such as Auto Loader and COPY INTO for cloud object storage, or Structured Streaming for message buses such as Kafka. You get more control and you maintain the logic. This lesson builds a standard connector pattern.
Managed connectors are configured, not coded. You create a connection and an ingestion pipeline for a source such as Salesforce, Workday, SharePoint or a SQL Server database, and Databricks handles authentication, incremental reads, retries and schema changes. They run on serverless Lakeflow pipelines and land in Unity Catalog tables.

In Free Edition, managed connectors need a real source system and its credentials, and database connectors that use change data capture (CDC) are paid workspace features. Free Edition also restricts outbound internet access and allows one active pipeline per type, so treat managed connectors as concepts there.

Reading scenario questions

| Scenario words | Pattern | |---|---| | Small source, fully replaced, daily snapshot, one-time upload | Batch, with overwrite | | New files keep arriving, process only new data, do not reprocess files | Incremental with Auto Loader and a checkpoint | | Latency of seconds or minutes | Streaming | | Popular software-as-a-service app or database with built-in support | Managed connector |

Practice challenge

A day 3 file lands with four new orders and you run both loads again. How many rows are in ordersbatch? How many rows does this Auto Loader run read, and what is the bronze total? And if all three batch runs had used append instead of overwrite, how many rows would ordersbatch hold?

Answers: orders_batch holds 9 rows, because the batch load re-reads all three files and overwrites. Auto Loader reads only the 4 new rows, and bronze also ends at 9. With append, the runs add 3, then 5, then 9, for 17 rows full of duplicates. The solution cell is commented out in the notebook.

Key takeaways

Batch re-reads the whole source, so overwrite to stay correct.
Incremental loading with Auto Loader and a checkpoint loads each file exactly once.
Streaming uses the same engine continuously. On serverless, run it with availableNow.
Lakeflow Connect has standard connectors you code and managed connectors you configure.

Previous: Lesson 1, platform, Unity Catalog and compute. Next: Lesson 3, COPY INTO.