Databricks Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 - Databricks Certified Associate Developer for Apache Spark 3.5-Python

broadcast(df1) tells Spark to send the small DataFrame (df1) to all worker nodes.

This eliminates the need for shuffling df1 during the join.

Broadcast joins are optimized for scenarios with one large and one small table.

To define a micro-batch interval, the correct syntax is:

query = df.writeStream \

outputMode("append") \

trigger(processingTime='5 seconds') \

start()

This schedules the query to execute every 5 seconds.

Continuous mode (used in Option A) is experimental and has limited sink support.

Option D is incorrect because processingTime must be a string (not an integer).

Option B triggers as fast as possible without interval control.

Comprehensive and Detailed Explanation From Exact Extract:

In Apache Spark, broadcast variables are used to efficiently distribute large, read-only data to all worker nodes. However, broadcasting very large datasets can lead to memory issues on executors if the data does not fit into the available memory.

According to the Spark documentation:

"Broadcast variables allow the programmer to keep a read-only variable cached on each machine rather than shipping a copy of it with tasks. This can greatly reduce the amount of data sent over the network."

However, it also notes:

"Using the broadcast functionality available in SparkContext can greatly reduce the size of each serialized task, and the cost of launching a job over a cluster. If your tasks use any large object from the driver program inside of them (e.g., a static lookup table), consider turning it into a broadcast variable."

But caution is advised when broadcasting large datasets:

"Broadcasting large variables can cause out-of-memory errors if the data does not fit in the memory of each executor."

Therefore, if the broadcasted DataFrame containing millions of rows exceeds the memory capacity of the executors, the job may fail due to memory constraints.

Comprehensive and Detailed Explanation From Exact Extract:

To enable fault tolerance and ensure that Spark can resume from the last committed offset after failure, you must configure a checkpoint location using the correct option key:"checkpointLocation".

From the official Spark Structured Streaming guide:

“To make a streaming query fault-tolerant and recoverable, a checkpoint directory must be specified using.option("checkpointLocation", "/path/to/dir").”

Explanation of options:

Option A uses an invalid option name:"checkpoint"(should be"checkpointLocation")

Option B is correct: it setscheckpointLocationproperly

Option C lacks checkpointing and won’t resume after failure

Option D also lacks checkpointing configuration

Comprehensive Explanation:

To cover structured data processing, SQL querying, and machine learning in Apache Spark, the correct combination of components is:

Spark DataFrames: for structured data processing

Spark SQL: to execute SQL queries over structured data

MLlib: Spark's scalable machine learning library

This trio is designed for exactly this type of use case.

Why other options are incorrect:

A: GraphX is for graph processing — not needed here.

B: Pandas API on Spark is useful, but MLlib is essential for ML, which this option omits.

C: Spark Streaming is legacy; GraphX is irrelevant here.