Databricks Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 - Databricks Certified Associate Developer for Apache Spark 3.5 – Python

The number of tasks is controlled by the number of partitions. By default, spark.sql.shuffle.partitions is 200. If stages are showing very few tasks (less than total cores), you may not be leveraging full parallelism.

From the Spark tuning guide:

"To improve performance, especially for large clusters, increase spark.sql.shuffle.partitions to create more tasks and parallelism."

Thus:

A is correct: increasing shuffle partitions increases parallelism

B is wrong: it further reduces parallelism

C is invalid: increasing dataset size doesn’t guarantee more partitions

D is irrelevant to task count per stage

Final Answer: A

To persist a table and specify the save path, use:

users.write.option("path", "/some/path").saveAsTable("default_table")

The .option("path", ...) must be applied before calling saveAsTable.

Option A uses invalid syntax (write(path=...)).

Option B applies .option() after .saveAsTable()—which is too late.

Option D uses incorrect syntax (no path parameter in saveAsTable).

According to Spark’s watermarking rules:

“Records that are older than the watermark (event time < current watermark) are considered too late and are dropped.”

So, if a record’s event_time is earlier than (max event_time seen so far - 10 minutes), it is discarded.

The .mode("overwrite") ensures that existing files at the path will be replaced.

partitionBy("country") optimizes queries by writing data into partitioned folders.

Correct syntax:

df.write.mode("overwrite").partitionBy("country").parquet("/data/output")

— Source: Spark SQL, DataFrames and Datasets Guide

Temp views like createOrReplaceTempView are session-scoped.

They disappear once the Spark session ends.

To retain data across sessions, it must be persisted:

df.write.saveAsTable("users_vw")

Thus, the view needs to be persisted as a table to survive session termination.

By default, the spark.read.text() method reads a text file one line per record. This means that each line in a text file becomes one row in the resulting DataFrame.

To read each file as a single record, Apache Spark provides the option wholetext, which, when set to True, causes Spark to treat the entire file contents as one single string per row.

Correct usage:

df = spark.read.option("wholetext", True).text(txt_path)

This way, each record in the DataFrame will contain the full content of one file instead of one line per record.

To also include the file path, the function input_file_name() can be used to create an additional column that stores the complete path of the file being read:

from pyspark.sql.functions import input_file_name

df = spark.read.option("wholetext", True).text(txt_path) \

withColumn("file_path", input_file_name())

This approach satisfies both requirements from the question:

Each record holds the entire contents of a file.

Each record also contains the file path from which the text was read.

Why the other options are incorrect:

B or D (lineSep) – The lineSep option only defines the delimiter between lines. It does not combine the entire file content into a single record.

C (wholetext=False) – This is the default behavior, which still reads one record per line rather than per file.

References (Databricks Apache Spark 3.5 – Python / Study Guide):

PySpark API Reference: DataFrameReader.text — describes the wholetext option.

PySpark Functions: input_file_name() — adds a column with the source file path.

Databricks Certified Associate Developer for Apache Spark Exam Guide (June 2025): Section “Using Spark DataFrame APIs” — covers reading files and handling DataFrames.

When using the MEMORY_AND_DISK storage level, Spark attempts to cache as much of the DataFrame in memory as possible. If the DataFrame does not fit entirely in memory, Spark will store the remaining partitions on disk. This allows processing to continue, albeit with a performance overhead due to disk I/O.

As per the Spark documentation:

"MEMORY_AND_DISK: It stores partitions that do not fit in memory on disk and keeps the rest in memory. This can be useful when working with datasets that are larger than the available memory."

— Perficient Blogs: Spark - StorageLevel

This behavior ensures that Spark can handle datasets larger than the available memory by spilling excess data to disk, thus preventing job failures due to memory constraints.

spark.executor.cores determines how many concurrent tasks an executor can run.

For example, if set to 4, each executor can run up to 4 tasks in parallel.

Other settings:

spark.task.maxFailures controls task retry logic.

spark.driver.cores is for the driver, not executors.

spark.executor.memory sets memory limits, not task concurrency.

In Apache Spark, the execution hierarchy is structured as follows:

Application: The highest-level unit, representing the user program built on Spark.

Job: Triggered by an action (e.g., collect(), count()). Each action corresponds to a job.

Stage: A job is divided into stages based on shuffle boundaries. Each stage contains tasks that can be executed in parallel.

Task: The smallest unit of work, representing a single operation applied to a partition of the data.

When the collect() action is invoked, Spark initiates a job. This job is then divided into stages at points where data shuffling is required (i.e., wide transformations). Each stage comprises tasks that are distributed across the cluster's executors, operating on individual data partitions.

This hierarchical execution model allows Spark to efficiently process large-scale data by parallelizing tasks and optimizing resource utilization.

The correct answer is B because it uses the new function count_if, introduced in Spark 3.5.0, which simplifies conditional counting within aggregations.

F.count_if(condition) counts the number of rows that meet the specified boolean condition.

In this example, it directly counts how many times spot_price >= min_price evaluates to true, replacing the older verbose combination of when/otherwise and filtering or summing.

Official Spark 3.5.0 documentation notes the addition of count_if to simplify this kind of logic:

“Added count_if aggregate function to count only the rows where a boolean condition holds (SPARK-43773).”

Why other options are incorrect or outdated:

A uses a legacy-style method of adding a flag column (when().otherwise()), which is verbose compared to count_if.

C performs a simple min/max aggregation—useful but unrelated to conditional array operations or the updated functionality.

D incorrectly applies .filter() after .agg() which will cause an error, and misuses string "min_price" rather than the variable.

Therefore, B is the only option leveraging new functionality from Spark 3.5.0 correctly and efficiently.