Top 30 Data Engineer Interview Questions and Answers [2026]

TL;DR: Data engineer interviews in 2026 test modeling judgment, SQL written live (dedup, gaps and islands, safe upserts), and whether a candidate can rerun a pipeline without double counting. Candidates should also know the recent platform changes: Airflow 3 (now 3.3.2), Spark 4 with ANSI mode on by default (now 4.2.0), and Kafka 4 running without ZooKeeper.

Apache Spark 4.2.0 shipped on July 14, 2026 with a SQL CHANGES clause for reading row-level changes and declarative SCD Type 1 upserts. Change data capture now sits in the engine, not in hand-written merge jobs.

These questions cover modeling, SQL, pipelines, table formats, cost and privacy. They are written for US hiring managers running a senior loop and for candidates preparing for one.

Data Modeling

1. How do you choose the grain of a fact table, and why first?

Declare the grain before picking any dimension or measure, because every column must agree with it. The Kimball Group's grain guidance calls the grain "a binding contract on the design": it states exactly what one fact row represents.

It recommends starting at the atomic grain, the lowest level a business process captures.

In practice: "one row per order line, per shipment" is a grain. "Orders data" is not. A strong candidate states the grain out loud, then tests each proposed column against it.

A discount stored once per order cannot sit on a line-level fact without being allocated or double counted. Most wrong revenue numbers trace back to a fact table that mixed two grains.

2. Explain slowly changing dimension Types 1, 2 and 3. When do you use each?

Type 1 overwrites the old value, Type 2 adds a new row per change, and Type 3 adds a column for the prior value. Pick by what the business needs to report about the past.

• Type 1: fixing a misspelled name. History is lost, which is the point.
• Type 2: a customer moves from Texas to Ohio and sales must stay with the state at the time of sale. Per the Kimball Type 2 definition, each new row gets a new surrogate key and needs at least three extra columns: an effective date, an expiration date and a current-row flag.
• Type 3: a sales territory realignment where reports need "old region" and "new region" side by side for one transition.

Kimball lists Types 0 through 7. Interviewers rarely need more than 1, 2 and 3, plus Type 6, which combines Type 1 attributes with a Type 2 dimension. The follow-up that separates levels: how do facts loaded late find the right Type 2 row?

Join on the natural key and the event time between the effective and expiration dates, not on the current flag.

3. Why does a dimension need a surrogate key?

Because the natural key stops being unique once you track history, and because source systems do not agree on keys.

The Kimball surrogate key page gives both reasons: a Type 2 dimension holds several rows per natural key, and natural keys from several sources may be "incompatible or poorly administered."

In a modern warehouse the surrogate is often a hash of the natural key plus the effective timestamp, rather than a sequence. Hashes can be computed in parallel and stay stable across reloads.

A good answer also covers the unknown member: a fact row whose dimension has not arrived yet points at a placeholder row, and is re-keyed when the real row lands.

4. Star schema, data vault or one wide table: how do you choose?

Match the model to who changes the data and who reads it. A star schema serves analysts and BI tools well. A data vault serves teams integrating many changing sources with a full audit trail.

A wide denormalized table serves one dashboard or one model that needs speed over flexibility.

Data vault splits data into three parts, as the dbt Labs data vault overview describes. Hubs hold business keys. Links store relationships between hubs as data. Satellites hold the descriptive and historical attributes.

New sources add satellites without reworking existing tables, at the cost of many more joins.

The senior answer is usually layered: raw or vault tables for integration, star schemas for reporting, and wide tables generated from the stars for the few consumers that need them.

Candidates who pick one model for every layer have usually only worked at one company size.

5. How would you build an SCD Type 2 dimension in dbt?

Use a dbt snapshot. It compares the source to the snapshot table on each run and maintains dbt_valid_from and dbt_valid_to.

Since dbt 1.9, snapshots can be defined in YAML, and the snapshot docs list two change-detection strategies, timestamp and check.

snapshots:
  - name: customers_snapshot
    relation: source('crm', 'customers')
    config:
      unique_key: customer_id
      strategy: timestamp
      updated_at: updated_at
      hard_deletes: new_record

The detail to probe is deletes. The hard_deletes config accepts ignore (the default), invalidate and new_record. With the default, a customer deleted in the source stays "current" in the snapshot forever.

The new_record mode arrived in dbt-core 1.10.0 and writes an explicit deletion row, which keeps the history auditable.

SQL Questions

The six SQL answers below were run on SQLite 3.50.4 on October 16, 2026, and the outputs shown are the actual results. Warehouse dialects differ in small ways, noted where they matter.

6. Write a query that keeps only the latest version of each order.

Number the rows per key with ROW_NUMBER(), newest first, and keep row 1. Add a tiebreaker so the result is deterministic when two versions share a timestamp.

SELECT order_id, status, updated_at
FROM (
  SELECT r.*,
         ROW_NUMBER() OVER (
           PARTITION BY order_id
           ORDER BY updated_at DESC, _loaded_at DESC
         ) AS rn
  FROM raw_orders r
) AS t
WHERE rn = 1;

-- order 1 has three versions, two with the same updated_at
-- result: (1, 'shipped', '2026-10-02 14:00'), (2, 'pending', '2026-10-01 10:00')

Follow-ups worth asking. Why not GROUP BY order_id with MAX(updated_at)? Because it returns the timestamp, not the rest of the row, and a join back duplicates rows on ties. Why not DISTINCT?

Two versions differ in a load column, so both survive. Snowflake, BigQuery and Databricks also support QUALIFY rn = 1, which removes the subquery.

7. What is the difference between ROW_NUMBER, RANK and DENSE_RANK?

They differ only on ties. ROW_NUMBER breaks ties arbitrarily, RANK gives ties the same number and leaves a gap, and DENSE_RANK gives ties the same number with no gap.

SELECT rep, total,
  ROW_NUMBER() OVER (PARTITION BY region ORDER BY total DESC) AS rn,
  RANK()       OVER (PARTITION BY region ORDER BY total DESC) AS rnk,
  DENSE_RANK() OVER (PARTITION BY region ORDER BY total DESC) AS drnk
FROM sales;

-- ana  500  1  1  1
-- ben  500  2  1  1
-- cy   300  3  3  2
-- dee  200  4  4  3

The question behind the question is "top 2 reps per region." With ROW_NUMBER, Ben or Ana is dropped at random. With RANK() <= 2 both appear and Cy does not. With DENSE_RANK() <= 2 all three appear.

A senior candidate asks the interviewer which answer the business wants before writing it.

8. Find each user's streaks of consecutive login days (gaps and islands).

Subtract a row number from the date. Consecutive days share the same difference, so the difference becomes a group key for each island. Deduplicate first, or a user who logs in twice on one day breaks the streak.

WITH days AS (
  SELECT DISTINCT user_id, login_date FROM logins
),
grp AS (
  SELECT user_id, login_date,
         julianday(login_date)
           - ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY login_date) AS island
  FROM days
)
SELECT user_id,
       MIN(login_date) AS streak_start,
       MAX(login_date) AS streak_end,
       COUNT(*)        AS streak_days
FROM grp
GROUP BY user_id, island;

-- (7, '2026-10-01', '2026-10-03', 3)
-- (7, '2026-10-06', '2026-10-07', 2)
-- (9, '2026-10-05', '2026-10-05', 1)

julianday is SQLite. In Snowflake or PostgreSQL the same idea is login_date - ROW_NUMBER() ... cast to an interval of days.

The pattern also solves sessionization, outage windows and "longest run of failed jobs", so ask the candidate where else they have used it.

9. What is the difference between ROWS and RANGE in a window frame?

ROWS counts physical rows. RANGE groups rows with equal ORDER BY values as peers.

When a window has ORDER BY but no frame clause, the SQL default is RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, so tied rows all get the same running total.

SELECT day, amt,
  SUM(amt) OVER (ORDER BY day) AS range_default,
  SUM(amt) OVER (ORDER BY day
                 ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS rows_frame
FROM ev;

-- 2026-10-01  10  30  10
-- 2026-10-01  20  30  30
-- 2026-10-02   5  35  35

The second trap is a "7-day moving average" written as ROWS BETWEEN 6 PRECEDING AND CURRENT ROW. That is seven rows, not seven days. If a day has no data, the window reaches further back.

In the test data, a three-row average on October 4 included October 1 because October 3 was missing. The fix is a date spine: left join a calendar table so every day has a row, then apply the frame.

10. Write an incremental upsert that is safe to run twice and never lets stale data win.

Upsert on the business key, and only overwrite when the incoming row is newer. Without the newer-than check, a late-arriving old record overwrites a fresh one.

INSERT INTO dim_customer (customer_id, email, updated_at)
SELECT customer_id, email, updated_at FROM stg_customer WHERE true
ON CONFLICT (customer_id) DO UPDATE SET
  email      = excluded.email,
  updated_at = excluded.updated_at
WHERE excluded.updated_at > dim_customer.updated_at;

-- before: (1, a@old.com, 10-01), (2, b@x.com, 10-05)
-- stage:  (1, a@new.com, 10-03), (2, b@stale.com, 10-02), (3, c@x.com, 10-04)
-- after:  (1, a@new.com, 10-03), (2, b@x.com, 10-05), (3, c@x.com, 10-04)
-- second run: identical result

This is SQLite and PostgreSQL syntax (the WHERE true resolves a SQLite parsing ambiguity).

On Snowflake, BigQuery, Databricks and Iceberg tables the same logic is a MERGE INTO ... WHEN MATCHED AND s.updated_at > t.updated_at THEN UPDATE ... WHEN NOT MATCHED THEN INSERT; that version was not executed for this guide.

Two senior details: deduplicate the staging rows first (question 6), because MERGE fails or behaves unpredictably when two source rows match one target row, and filter the target scan to recent partitions so the merge does not read the whole table.

11. Why does this NOT IN query return no rows?

Because the subquery contains a NULL. x NOT IN (2, NULL) means x <> 2 AND x <> NULL, and any comparison with NULL is unknown, so no row qualifies.

-- churned contains (2) and (NULL)
SELECT id FROM customers
WHERE id NOT IN (SELECT customer_id FROM churned);
-- returns nothing

SELECT id FROM customers c
WHERE NOT EXISTS (
  SELECT 1 FROM churned x WHERE x.customer_id = c.id
);
-- returns 1 and 3

Use NOT EXISTS or a left anti join for "rows with no match." It is a short question, but it shows whether the candidate thinks in three-valued logic, which matters every time a pipeline joins on a nullable key.

Pipelines and Streaming

12. Batch or streaming: how do you decide?

Start from the freshness the consumer will act on, then price it. If the business looks at a dashboard each morning, an hourly or daily batch is cheaper to build, test and rerun.

Streaming earns its cost when a decision happens within seconds or minutes: fraud checks, inventory holds, alerting.

Candidates should name what streaming adds: state that must be checkpointed, late and duplicate events, schema changes mid-stream, and harder backfills.

Many teams settle on micro-batches every few minutes into a table format that supports upserts. That gives near-real-time data with batch-style reruns.

A good answer names the cheapest design that meets the latency target, not the most interesting one.

13. What does exactly-once mean in a pipeline, and where does it break?

It means each input affects the output once, even after failures and retries. Engines get there with replayable sources plus idempotent or transactional sinks, not with magic.

The Structured Streaming guide states it plainly: checkpointing and write-ahead logs record offset ranges, sinks are designed to be idempotent, and "using replayable sources and idempotent sinks, Structured Streaming can ensure end-to-end exactly-once semantics."

Where it breaks is the sink.

Per the sink table, the file sink is exactly-once, while the Kafka sink and foreach sink are at-least-once. foreachBatch is at-least-once by default; the docs say to use the batch ID it passes in to deduplicate output.

def upsert_batch(df, batch_id):
    # Skip a batch that already committed, then MERGE on the business key.
    if already_committed("orders_sink", batch_id):
        return
    merge_into_target(df.dropDuplicates(["order_id"]))
    record_commit("orders_sink", batch_id)

stream.writeStream.foreachBatch(upsert_batch) \
    .option("checkpointLocation", "s3://bucket/chk/orders") \
    .start()

Kafka's own transactional guarantees are a separate topic, covered in our Kafka interview guide. For a data engineer the practical rule is the same either way: design every write so a replay produces the same rows.

14. How do you handle late and duplicate events in a stream?

Use event time with a watermark, and deduplicate within that watermark. A watermark tells the engine how late data may arrive, so it can finalize windows and drop old state. Spark has supported watermarks since 2.1.

For duplicates whose event time differs between copies, for example when a non-idempotent producer stamps the time at send, the Spark docs describe dropDuplicatesWithinWatermark and advise setting the watermark delay longer than the largest gap between duplicate timestamps.

The docs guarantee that data inside the watermark delay is never dropped, but data later than that may or may not be processed, so a strong design also lands raw events in storage.

A daily batch can then reconcile anything the stream discarded. Ask the candidate how they would measure how much data arrives after the watermark, since that number sets the delay.

15. How do you make a backfill idempotent?

Scope each run to one logical partition, read only that partition's input, and replace or merge that partition's output, so a rerun produces the same rows.

The Airflow best practices say the same thing: tasks "should produce the same outcome on every re-run," do not use INSERT on a rerun, read and write a specific partition, and never read the latest available data in a task.

Ways to replace a partition: INSERT OVERWRITE on that partition, a delete-then-insert in one transaction, or a MERGE on the business key (question 10).

Run checks before the swap so a bad rerun fails while yesterday's correct data stays live. Two traps come up in senior loops.

Backfilling a Type 2 dimension out of order corrupts the effective dates, so history must be replayed in time order. And backfills compete with daily runs for warehouse capacity, so they need their own pool or warehouse.

16. Which dbt incremental strategy would you pick, and what is microbatch?

Pick by how rows change. The built-in strategies are append, merge, delete+insert, insert_overwrite and microbatch, and support varies by adapter.

• append: immutable events with no chance of duplicates. Rare in practice.
• merge: rows update in place and have a unique_key.
• delete+insert or insert_overwrite: replace whole partitions or key sets, often cheaper than a row-by-row merge on large tables.
• microbatch: large time series. Added in dbt-core 1.9.0, it splits the load into one query per time batch.

Per the microbatch docs, a model sets event_time, batch_size (hour, day, month or year) and begin, and lookback (default 1) reprocesses earlier batches to catch late records.

The trap: if upstream models do not set event_time, dbt cannot filter them and scans the full table for every batch. Failed batches can be retried on their own, which makes backfills much easier to reason about.

Spark and Lakehouse

17. One Spark task runs for an hour while the rest finish in minutes. What do you check?

Data skew on a join or aggregation key. Open the stage in the Spark UI and compare the max task duration and shuffle read size to the median; one huge partition is the signature.

Adaptive Query Execution handles some of this for you. Per the performance tuning docs, AQE has been on by default since Spark 3.2.0, and its skew join handling splits skewed partitions in sort-merge joins.

A partition counts as skewed when it is larger than 5 times the median partition and larger than 256 MB.

When AQE is not enough: check whether the hot key is NULL or a placeholder like "unknown" and filter it out or handle it separately; broadcast the smaller side if it fits in memory; or salt the key by adding a random suffix on the big side and exploding the small side to match.

A strong candidate finds the hot key with a simple GROUP BY key ORDER BY count DESC before changing any config.

18. Why are small files a problem, and how do you fix them?

Every file adds metadata and an open cost at read time, so thousands of tiny files slow down queries and planning. Streaming writes and frequent small batches create them.

The Iceberg maintenance docs describe the fix: the rewriteDataFiles action compacts small files into larger ones in parallel, with a target file size such as 500 MB in the docs' example.

Prevention matters as much as cleanup: repartition by the table's partition columns before writing, avoid partitioning on high-cardinality columns such as user ID, and schedule compaction for hot partitions.

In Hudi, Merge On Read tables compact their log files into base files as part of normal operation, per the table types docs.

19. Iceberg, Delta Lake or Hudi: how do you choose?

Choose by the engines and catalog you already run, then by update pattern. All three give ACID commits, time travel and row-level updates on files in object storage.

The differences that matter in production are engine support, defaults and maintenance.

Current releases as of October 16, 2026: Iceberg 1.12.0 (September 30, 2026), Delta Lake 4.4.x (4.4.0 added Spark 4.2 support; 4.4.1 followed on October 5, 2026) and Hudi 1.2.1 (September 24, 2026).

The Iceberg spec version 3 adds nanosecond timestamps, variant and geospatial types, default column values, row lineage and binary deletion vectors.

Delta Lake 4.0.0 (June 9, 2025) was built on Spark 4.0 and made the Variant type generally available.

A candidate who says "we used Delta because we were on Databricks" or "Iceberg because Snowflake, Trino and Spark all had to read it" is giving the right kind of answer.

20. Copy-on-write or merge-on-read: what is the trade-off?

Copy-on-write rewrites whole data files on update, so writes are slow and reads are fast. Merge-on-read writes small delete or log files, so writes are fast and reads pay to merge them until compaction runs.

In Iceberg it is a table property per operation. The configuration docs list write.delete.mode, write.update.mode and write.merge.mode, each defaulting to copy-on-write, with merge-on-read available on format v2 and above.

In Hudi it is the table type, chosen at creation. Use copy-on-write for tables read far more than they are written, such as reference data. Use merge-on-read for CDC streams with frequent small updates, and budget for compaction.

21. You ran DELETE on a lakehouse table. Why is storage unchanged, and is the data gone?

The data is not gone. A delete writes a new snapshot that no longer references the old files, but the files stay so time travel and concurrent readers keep working.

• Iceberg: snapshots accumulate until expireSnapshots runs, and data files are only deleted once no retained snapshot references them. Orphan file cleanup removes files no snapshot ever committed.
• Delta Lake: per the utility docs, VACUUM is not triggered automatically and the default retention threshold is 7 days. The docs warn against going shorter, since concurrent readers can fail and tables can be corrupted.
• Hudi: per the cleaning docs, automatic cleaning runs after each commit and retains 10 commits by default.

This is the question that connects storage cost to privacy (question 25). A candidate who says "we ran the delete, so we complied" has not operated a lakehouse.

Quality, Cost, Privacy

22. What is a data contract, and how do you enforce one?

A data contract is an agreed schema and set of guarantees between a producer and its consumers, checked by machines rather than by memory.

In dbt it is concrete: per the model contracts docs, setting enforced: true requires every column's name and data_type, and the build fails if the model's output does not match.

models:
  - name: dim_customers
    config:
      contract:
        enforced: true
    columns:
      - name: customer_id
        data_type: int
        constraints:
          - type: not_null
      - name: customer_name
        data_type: string

Limits worth knowing: contracts do not apply to Python models, ephemeral models or materialized views, and incremental models need on_schema_change set to append_new_columns or fail. Contracts cover shape.

Upstream of the warehouse, the contract is with the application team: schema registry rules for events, and a review step before a source column is renamed.

23. Which data quality checks do you add, and where?

Test at the boundaries: when data lands, when a model is built, and before it is published. dbt ships four generic data tests, per the data tests docs: unique, not_null, accepted_values and relationships.

They cover keys, required fields, enums and orphaned foreign keys.

Beyond those, senior candidates add checks that catch silent failures: source freshness (data stopped arriving), row-count change against the previous run, and reconciliation of a total, such as revenue, against the source system.

They also choose severity on purpose: a duplicate primary key blocks publishing; a 3% drop in row count warns and pages nobody at 2 a.m. Too many blocking tests train a team to ignore failures.

24. How do you cut cloud warehouse cost without slowing everyone down?

Find out what the bill is made of first, because the levers differ by pricing model.

• Snowflake bills compute per second while a warehouse runs, with a 60-second minimum each time it starts. Per the warehouse docs, an X-Small uses 1 credit per hour and each size up doubles that. Levers: right-size warehouses, set short auto-suspend times, and stop many tiny queries from waking a warehouse every minute.
• BigQuery on-demand bills bytes scanned. The pricing page lists $6.25 per TiB after the first free TiB each month. The cost best practices note that LIMIT does not reduce bytes read on non-clustered tables, and that a maximum bytes billed setting makes an oversized query fail without a charge.

Across platforms: partition and cluster on the columns queries filter by, select only needed columns, make large models incremental, and tag queries by team so cost has an owner.

Ask the candidate for one change they made and the before and after numbers.

25. How do you handle PII in a data platform?

Collect less, restrict access by column, and make deletion real. Start by tagging PII columns at ingestion, so later controls can find them.

For access, use the warehouse's column-level security instead of copying "safe" tables. Snowflake's Dynamic Data Masking applies masking policies at query time based on role; it requires Enterprise Edition or higher.

Hashing an email with a secret salt keeps it joinable for analytics without exposing it.

For deletion requests, the hard part is everything outside the main table: Type 2 history, snapshots and time travel (question 21), raw landing files, and backups.

A strong answer keeps a deletion log keyed by user ID and runs a scheduled job that deletes from every layer. Then it expires snapshots or vacuums within the retention window promised to users.

Interviewers should also ask whether PII ever lands in logs or dead-letter queues, because it usually does.

What Changed Recently

On dbt: dbt 2.0.0 shipped on September 14, 2026 and renamed its CLI branding to dbt (proprietary) and dbt-oss (open source). The 1.x line is still maintained: 1.12.6 and 1.11.16 were both released on October 14, 2026.

Ask candidates which line their project runs and why.

26. What changed in Airflow 3, and what matters most to a data engineer?

Airflow 3.0.0 shipped on April 22, 2025, and the current release is 3.3.2 (September 17, 2026), per the release notes. The changes that affect pipeline code:

• Task SDK and no direct metadata DB access. Task code can no longer query the metadata database; it goes through the REST API or the task context. DAGs import from the new airflow.sdk namespace.
• Scheduler-managed backfills. Backfills are scheduled and tracked like regular DAG runs, with UI and API support (AIP-78).
• DAG versioning. Airflow records structural changes to a DAG, so a run can be inspected against the code it actually ran (AIP-66).
• Assets. Datasets were renamed and redesigned as assets, with an @asset decorator for data-aware scheduling. Airflow 3.2.0 (April 7, 2026) added asset partitioning.
• Removals. SLAs, SubDAGs, DAG and XCom pickling and the SequentialExecutor are gone. schedule_interval and timetable were replaced by a single schedule argument.

27. What can break when a DAG moves from Airflow 2 to Airflow 3?

Scheduling defaults, more than syntax. Per the release notes, catchup_by_default is now False, so a DAG that relied on the old default no longer backfills missed intervals. The default schedule changed from @once to None.

The subtle one is create_cron_data_intervals, now False. A bare cron string is interpreted with CronTriggerTimetable instead of the old data-interval timetable.

The notes say teams that rely on data_interval_start, data_interval_end or ds should set it to True before upgrading. Flipping it after Airflow 3 runs exist skips one scheduled run on each affected DAG.

Asset-triggered and API-triggered runs now have logical_date=None, so reading data_interval_start in them raises a KeyError.

The upgrade guide requires Airflow 2.7 or later first and provides Ruff rules: ruff check dags/ --select AIR301 flags breaking changes, and AIR311 and AIR312 flag recommended ones.

28. Spark 4.0 turned on ANSI SQL mode. What breaks?

Jobs that relied on bad data silently becoming NULL. Spark 4.0.0 (May 23, 2025) made ANSI mode the default.

Per the ANSI compliance docs, Spark now throws runtime errors on invalid operations such as integer overflow and string parsing errors. An explicit CAST of a non-numeric string to an integer fails instead of returning NULL.

-- Spark 3.x default: NULL. Spark 4.x default: runtime error.
SELECT CAST('12a' AS INT);

-- Explicitly tolerant version that works under ANSI mode
SELECT TRY_CAST('12a' AS INT);   -- NULL

The right migration is to find where bad values come from and use try_cast or validation on purpose, not to set spark.sql.ansi.enabled=false everywhere.

Spark 4.0 also dropped JDK 8 and 11, making JDK 17 the default, and dropped Scala 2.12 for 2.13, which matters for anyone shipping custom JARs.

29. What did Spark 4.1 and 4.2 add for pipeline work?

Declarative pipelines and native change data capture. Spark 4.1.0 (December 16, 2025) added Spark Declarative Pipelines.

You define datasets and queries, and Spark handles the execution graph, dependency order, parallelism, checkpoints and retries.

It also added real-time mode for Structured Streaming, made SQL scripting GA, and made the VARIANT type GA with shredding for faster reads.

Spark 4.2.0 (July 14, 2026) added a SQL CHANGES clause and DataFrame APIs that read row-level changes in batch and streaming, plus Auto CDC in Declarative Pipelines for declarative SCD Type 1 upserts.

It also turned on Arrow-optimized Python UDFs by default and added metric views for semantic modeling. Good candidates will note that Auto CDC covers Type 1 only, so Type 2 history (question 2) still needs its own design.

30. Kafka 4.0 removed ZooKeeper. What does a data engineer need to know?

Mostly upgrade constraints, not new concepts. The Kafka 4.0 announcement (March 18, 2025) calls it the first major release to run entirely without ZooKeeper, in KRaft mode by default.

It made the new consumer group protocol from KIP-848 generally available and raised Java requirements: Java 11 for clients and Kafka Streams, and Java 17 for brokers, Connect and tools.

For pipeline owners the checklist is: confirm connector and client versions meet the new minimums (KIP-896), rebuild Connect workers on Java 17, and test consumer lag behavior under the new rebalance protocol.

The broker side, including KRaft controllers and migration from ZooKeeper, is covered in our Kafka guide. Kafka 4.3.0 shipped on May 22, 2026.

Signs of a Strong Answer

• They state the grain of a table before writing a join or a metric.
• They deduplicate staging data before a MERGE and add a newer-than condition so late records cannot overwrite fresh ones.
• They explain exactly-once as replayable source plus idempotent sink, and know which of their sinks is only at-least-once.
• They scope every backfill to a partition and can say what happens if it runs twice.
• They know that a lakehouse delete leaves files behind until snapshots expire or vacuum runs.
• They quote cost in the platform's unit (credits, bytes scanned) and name the query that drove it.

Hiring Data Engineers

Data engineers who have run backfills in production, owned a warehouse bill and passed a privacy review are hard to find through job boards.

The data engineer role guide covers the job itself, and for related stacks see our PostgreSQL and ClickHouse interview guides.

Second Talent helps US startups hire vetted AI-Native Engineers from Asia, at $29 to $39 USD an hour billed monthly, with matches in 24 hours and a 90-day one-time replacement guarantee.

See data engineers for hire, compare rates on the cost to hire a data engineer page, or tell us what you need.