Databricks Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 - Databricks Certified Associate Developer for Apache Spark 3.5 – Python

To compare a column value with a Python literal constant in a DataFrame expression, use F.lit() to convert it into a Spark literal.

Correct refactor:

from pyspark.sql import functions as F

min_price = 110.50

result_df = prices_df.filter(F.col("price") > F.lit(min_price)).agg(F.count("*"))

This avoids type mismatches and ensures Spark executes the filter expression on the cluster.

Why the other options are incorrect:

B: where() syntax is valid, but F.lit("price") is incorrect — wraps string literal, not a column.

C: withColumn adds a column, not needed for this aggregation.

D: Comparison logic reversed.

Spark automatically uses a broadcast hash join when one side of the join is small enough to fit within the broadcast threshold.

Default threshold:

spark.sql.autoBroadcastJoinThreshold = 10MB (as of Spark 3.5)

Since df2 is 8 MB, Spark automatically broadcasts it to all executors. This avoids a shuffle on the large dataset (df1) and speeds up the join.

Why the other options are incorrect:

A: 8 MB < 10 MB threshold → broadcast join is efficient.

B: AQE is not required for broadcast joins; it’s a static optimization.

C: Broadcast hints are optional — Spark infers automatically.

Both cache() and persist() are Spark DataFrame storage operations that store computed results in memory (and optionally on disk) to speed up subsequent actions on the same DataFrame.

Key difference:

cache() is a shorthand for persist(StorageLevel.MEMORY_AND_DISK).

persist() allows specifying different storage levels, such as MEMORY_ONLY, DISK_ONLY, or MEMORY_AND_DISK_SER.

Example:

df.cache() # Uses MEMORY_AND_DISK by default

df.persist(StorageLevel.MEMORY_ONLY) # Custom storage level

Both trigger caching upon an action (e.g., count(), collect()).

Why the other options are incorrect:

A: persist() default is not DISK_ONLY; default storage level is MEMORY_AND_DISK.

B/C: cache() cannot set arbitrary levels; only persist() can.

Exactly-once guarantees in Spark Structured Streaming require micro-batch mode (default), not continuous mode.

Continuous mode (.trigger(continuous=...)) only supports at-least-once semantics and lacks full fault-tolerance.

trigger(availableNow=True) is a batch-style trigger, not suited for low-latency streaming.

So:

Option A uses micro-batching with a tight trigger interval → minimal latency + exactly-once guarantee.

Final Answer: A

Broadcast joins work by sending the smaller DataFrame to all executors, eliminating the shuffle of the larger DataFrame.

From Spark documentation:

“Broadcast joins are efficient when one DataFrame is small enough to fit in memory. Spark avoids shuffling the larger table.”

DataFrame B (1 GB) fits within the default threshold and should be broadcasted.

It eliminates the need to shuffle the large DataFrame A.

Final Answer: B

To create a new calculated column in Spark, use the .withColumn() method.

To remove an unwanted column, use the .drop() method.

Correct syntax:

from pyspark.sql.functions import col

employees_df = employees_df.withColumn("tax", col("salary") * 0.1).drop("age")

.withColumn("tax", col("salary") * 0.1) → adds a new column where tax = 10% of salary.

.drop("age") → removes the age column from the DataFrame.

Why the other options are incorrect:

B: lit(0.1) creates a constant value, not a calculated tax.

C: .dropField() is not a DataFrame API method (used only in struct field manipulations).

D: Adds 0.1 to salary instead of calculating 10%.

Spark Connect introduces a decoupled client-server architecture. Its key feature is enabling Spark job submission and execution from remote clients — in Python, Java, etc.

From Databricks documentation:

“Spark Connect allows remote clients to connect to a Spark cluster and execute Spark jobs without being co-located with the Spark driver.”

A is close, but "any language" is overstated (currently supports Python, Java, etc., not literally all).

B refers to REST, which is not Spark Connect's mechanism.

D is incorrect; Spark Connect isn’t focused on ingestion.

Final Answer: C

In Apache Spark's cluster mode:

"The driver program runs on the cluster’s worker node instead of the client’s local machine. This allows the driver to be close to the data and other executors, reducing network overhead and improving fault tolerance for production jobs."

(Source: Apache Spark documentation – Cluster Mode Overview)

This deployment is ideal for production environments where the job is submitted from a gateway node, and Spark manages the driver lifecycle on the cluster itself.

Option A is partially true but less specific than D.

Option B is incorrect: the driver never executes all tasks; executors handle distributed tasks.

Option C describes client mode, not cluster mode.

In Spark, a shuffle occurs when data needs to be redistributed across partitions or nodes, typically due to operations that require grouping, joining, or sorting.

groupBy triggers a shuffle because it aggregates data based on key values, requiring data from multiple partitions to be moved across the cluster.

Operations like filter and select are narrow transformations (no shuffle).

coalesce can reduce the number of partitions without a full shuffle (unlike repartition).

Why the other options are incorrect:

A (filter) – Narrow transformation, no shuffle.

B (select) – Simple projection, no data movement.

D (coalesce) – Reduces partitions locally without shuffle.

Watermarking in Structured Streaming defines how late a record can arrive based on event time before Spark discards it.

Behavior:

withWatermark("event_time", "10 minutes")

This means Spark will keep state for 10 minutes beyond the maximum event time seen so far.

Any data arriving later than 10 minutes after the current watermark is ignored — it will not be included in the aggregation or output.

Why the other options are incorrect:

B: Late data beyond the watermark threshold is not included.

C: Late data is not moved to a new window; it’s simply dropped.

D: True for late data within the watermark threshold, not after it.