Databricks Databricks-Certified-Data-Engineer-Associate - Databricks Certified Data Engineer Associate Exam

Lakehouse Federation is the Databricks feature built for querying external systems without moving all data into Databricks . Databricks documentation describes it as the platform for query federation , enabling users to run queries against multiple external data sources while keeping governance centralized through Unity Catalog . Databricks also documents support for external systems such as Amazon Redshift and other databases through connections and foreign catalogs, allowing read-only access to external data while managing permissions in Unity Catalog. This aligns directly with the requirement to minimize duplication and still maintain centralized governance. Databricks Connect is for local development against Databricks compute, not federated querying. MLflow is for machine learning lifecycle management. Delta Lake is a storage format and table layer, not a federation framework. Therefore, when the goal is unified governance across Databricks and external systems like MySQL, Redshift, and BigQuery without first ingesting the data, Lakehouse Federation is the documented answer. Databricks does note that for high-volume production ingestion, managed connectors may sometimes be preferred, but for direct querying without data movement, Lakehouse Federation is the correct feature.

dbfs:/user/hive/warehouse Thereby showing " dbfs:/user/hive/warehouse/customer360.db

 The location of the customer360 database depends on the value of the spark.sql.warehouse.dir configuration property, which specifies the default location for managed databases and tables. If the property is not set, the default value is dbfs:/user/hive/warehouse. Therefore, the customer360 database will be located in dbfs:/user/hive/warehouse/customer360.db. However, if the property is set to a different value, such as dbfs:/user/hive/database, then the customer360 database will be located in dbfs:/user/hive/database/customer360.db. Thus, more information is needed to determine the correct response.

Option A is not correct, as dbfs:/user/hive/database/customer360 is not the default location for managed databases and tables, unless the spark.sql.warehouse.dir property is explicitly set to dbfs:/user/hive/database.

Option B is not correct, as dbfs:/user/hive/warehouse is the default location for the root directory of managed databases and tables, not for a specific database. The database name should be appended with .db to the directory path, such as dbfs:/user/hive/warehouse/customer360.db.

Option C is not correct, as dbfs:/user/hive/customer360 is not a valid location for a managed database, as it does not follow the directory structure specified by the spark.sql.warehouse.dir property.

Databases and Tables

[Databricks Data Engineer Professional Exam Guide]

A table update trigger is designed for this dependency. Databricks Lakeflow Jobs can monitor one or more supported Unity Catalog tables and start a job when the monitored tables are updated. Supported changes include data updates, merges, and deletes, so the Silver pipeline can begin only after the Bronze table receives new or changed data. This event-driven approach avoids polling and does not require a continuously running cluster. A continuous trigger keeps the job running rather than reacting to a completed table update. A scheduled trigger runs according to time, regardless of whether the Bronze table changed. A file arrival trigger monitors new files in a Unity Catalog storage location, not updates to an upstream table. Therefore, option A directly implements the required table-to-pipeline orchestration.

================

A table is a data entity that is stored in a physical location and can be accessed by other data engineers in other sessions. A table can be created from one or more tables using the CREATE TABLE or CREATE TABLE AS SELECT commands. A table can also be registered from an existing DataFrame using the spark.catalog.createTable method. A table can be queried using SQL or DataFrame APIs. A table can also be updated, deleted, or appended using the MERGE INTO command or the DeltaTable API. References:

Create a table

Create a table from a query result

Register a table from a DataFrame

[Query a table]

[Update, delete, or merge into a table]

One of the features of DLT is that it provides data quality metrics for each dataset in the pipeline, such as the number of records that pass or fail expectations, the number of records that are dropped, and the number of records that are written to the target. These metrics can be accessed from the DLT pipeline page, where the data engineer can click on each table and view the data quality statistics for the latest update or any previous update. This way, they can identify which table is dropping the records and why. References:

Monitor Delta Live Tables pipelines

Manage data quality with Delta Live Tables

With schema evolution enabled, Auto Loader detects source fields that are not present in the current schema and evolves the destination schema to include them. The new userAgent field is therefore added to the Delta table as a column, and values from newly processed records are written to it. Existing rows cannot be retroactively populated because the earlier source records did not contain that field, so they receive NULL for the newly added column. The column is consequently nullable unless a later transformation explicitly enforces another constraint. Option A describes behavior without suitable schema evolution. Option B incorrectly claims that historical rows receive a default value and that the column becomes non-nullable. Option D contradicts schema-evolution behavior. Therefore, option C correctly describes how the new JSON field is handled.

 To grant full permissions on a database to a user, group, or service principal, the GRANT ALL PRIVILEGES ON DATABASE command can be used. This command grants all the applicable privileges on the database, such as CREATE, SELECT, MODIFY, and USAGE. The other options are either incorrect or incomplete, as they do not grant all the privileges or specify the wrong database or principal. References:

GRANT

Privileges

Option B reads the relational source through JDBC and writes the resulting Spark DataFrame directly as a registered Unity Catalog table using its three-level name: catalog, schema, and table. Once written with saveAsTable, the data is immediately discoverable and queryable through Databricks SQL, subject to Unity Catalog permissions. Because the read and write operations occur through supported Databricks workloads, Unity Catalog can capture lineage between the source operation, notebook or job, and destination table. Writing files only to DBFS or an external path does not by itself create the required governed table. Introducing a separate staging and loading job adds unnecessary processing and operational complexity. A Pandas extraction also reduces scalability and does not satisfy the direct managed-table requirement. Therefore, option B is correct.

================