Databricks-Certified-Data-Engineer-Associate Databricks Certified Data Engineer Associate Exam Questions and Answers

In the Medallion Architecture promoted by Databricks, the Bronze layer represents the raw ingestion layer where data is stored in its original form with minimal transformation. The primary goal is to ensure data fidelity and traceability , not to apply business logic or heavy transformations. Typical operations in the Bronze layer include schema inference (or enforcement), ingestion of raw data, and the addition of metadata columns such as ingestion timestamp, load date/time, and process identifiers. These metadata fields help track lineage, enable debugging, and support incremental processing downstream. Extensive data cleansing (A), aggregations (B), and business transformations (C) are intentionally deferred to the Silver and Gold layers , where data is refined and structured for analytics. By keeping Bronze data close to the source and append-only, organizations maintain a reliable audit trail and can reprocess data if needed. Therefore, adding operational metadata like load timestamps and process IDs is the correct characteristic of Bronze layer transformations.

=========

To grant full privileges on a table such as ' sales ' to a group like ' team ' , the correct SQL command in Databricks is:

GRANT ALL PRIVILEGES ON TABLE sales TO team;

This command assigns all available privileges, including SELECT, INSERT, UPDATE, DELETE, and any other data manipulation or definition actions, to the specified team. This is typically necessary when a team needs full control over a table to manage and manipulate it as part of a project or ongoing maintenance.

To read from a stream source, the data engineer needs to use the spark.readStream method instead of the spark.read method. The spark.readStream method returns a DataStreamReader object that can be used to specify the details of the input source, such as the format, the schema, the path, and the options. The spark.read method is only suitable for batch processing, not streaming processing. The other changes are not necessary or correct for reading from a stream source. References: Structured Streaming Programming Guide, Read a stream, Databricks Data Sources

The < = > operator performs null-safe equality in Databricks SQL. It returns TRUE when both operands contain the same non-NULL value and also when both operands are NULL. It returns FALSE when only one operand is NULL. By contrast, the standard equality operator returns UNKNOWN when either side is NULL, so it cannot match two NULL values in a WHERE condition. Option C implements inequality logic and does not provide null-safe equality. Option D can incorrectly treat a genuine empty string as equivalent to NULL, changing the meaning of the data. Therefore, option A is the only predicate that preserves SQL values while implementing the required NULL comparison behavior. This question was also checked against the existing Word question bank.

================

A materialized view is appropriate because it stores the results of a defining query and refreshes them as source data changes. Expensive joins and aggregations can therefore be computed before BI users execute their queries, improving response time and providing a consistent definition of business metrics. This aligns with the Gold layer, which normally contains highly refined, aggregated, and business-oriented datasets designed for reporting and analytics. A view over raw Bronze tables would expose unvalidated source data and repeatedly execute its underlying transformations. A streaming table is more suitable for incremental data processing than for presenting precomputed business metrics. CSV files would not provide the managed table semantics, centralized definitions, or efficient query experience required. Therefore, option A best satisfies the Gold-layer requirements.

================

Option A is correct because Lakeflow Connect provides a built-in, fully managed Salesforce connector. It ingests Salesforce data into Databricks, supports incremental processing, and publishes governed destination tables in Unity Catalog. Managed SaaS connectors handle source-specific authentication, incremental reads, schema evolution, and automated retries, reducing the custom engineering required for API pagination, state tracking, and failure recovery. Calling the Salesforce REST API from a notebook would require the team to develop and maintain those capabilities itself. Auto Loader is designed to discover and incrementally process files arriving in cloud object storage; it does not connect directly to Salesforce. Using an exported Amazon S3 path would also require a separate export system outside the requested managed Databricks ingestion experience. Therefore, option A is the low-code capability that directly satisfies the Salesforce ingestion requirement.

================

The processingTime option specifies a time-based trigger interval for fixed interval micro-batches. This means that the query will execute a micro-batch to process data every 5 seconds, regardless of how much data is available. This option is suitable for near-real time processing workloads that require low latency and consistent processing frequency. The other options are either invalid syntax (A, C), default behavior (B), or experimental feature (E). References: Databricks Documentation - Configure Structured Streaming trigger intervals, Databricks Documentation - Trigger.

A row-filter function must return a Boolean value indicating whether each row is visible to the current user. The function in option A returns true for members of the account-level all_regions group, allowing them to see every row. For other users, it returns the result of region = ' EU ' , limiting visibility to European records. is_account_group_member checks direct or indirect membership in an account-level group. The ALTER TABLE ... SET ROW FILTER ... ON (region) statement then associates the function with the table and passes the region column to it for every query. A column mask transforms displayed column values rather than removing rows. The object is a table, not a view, and USING COLUMNS is not the correct row-filter attachment syntax. Therefore, option A is correct.

================

The OPTIMIZE command in Delta Lake is designed to compact small files into larger files to improve query performance and reduce file system overhead. Importantly, OPTIMIZE is considered an idempotent operation when run on unchanged data. This means that once files have been compacted optimally, running OPTIMIZE again does not produce additional changes unless new small files have been added to the table. Databricks ensures that file compaction is efficient and avoids unnecessary rewrites, which helps control compute costs and maintain stable performance. While features like ZORDER can be applied during OPTIMIZE to improve data skipping, re-running the same command without new data will not further reorganize files. Options A, B, and C incorrectly imply additional restructuring or changes on repeated execution. According to Delta Lake best practices, OPTIMIZE should be run periodically after data ingestion events rather than repeatedly on unchanged datasets. This behavior reinforces predictable performance and efficient resource utilization in Databricks environments.

=========

With schema evolution enabled, Auto Loader detects source fields that are not present in the current schema and evolves the destination schema to include them. The new userAgent field is therefore added to the Delta table as a column, and values from newly processed records are written to it. Existing rows cannot be retroactively populated because the earlier source records did not contain that field, so they receive NULL for the newly added column. The column is consequently nullable unless a later transformation explicitly enforces another constraint. Option A describes behavior without suitable schema evolution. Option B incorrectly claims that historical rows receive a default value and that the column becomes non-nullable. Option D contradicts schema-evolution behavior. Therefore, option C correctly describes how the new JSON field is handled.

The SQL keyword that can be used to convert a table from a long format to a wide format is PIVOT. The PIVOT clause is used to rotate the rows of a table into columns of a new table1. The PIVOT clause can aggregate the values of a column based on the distinct values of another column, and use those values as the column names of the new table1. The PIVOT clause can be useful for transforming data from a long format, where each row represents an observation with multiple attributes, to a wide format, where each row represents an observation with a single attribute and multiple values2. For example, the PIVOT clause can be used to convert a table that contains the sales of different products by different regions into a table that contains the sales of each product by each region as separate columns1.

The other options are not suitable for converting a table from a long format to a wide format. CONVERT is a function that can be used to change the data type of an expression3. WHERE is a clause that can be used to filter the rows of a table based on a condition4. TRANSFORM is a keyword that can be used to apply a user-defined function to a group of rows in a table5. SUM is a function that can be used to calculate the total of a numeric column.

1: PIVOT | Databricks on AWS

2: Reshaping Data - Long vs Wide Format | Databricks on AWS

3: CONVERT | Databricks on AWS

4: WHERE | Databricks on AWS

5: TRANSFORM | Databricks on AWS

[SUM | Databricks on AWS]

In Databricks, when creating a table, you can add a comment to columns or the entire table to provide more information about the data it contains. In this case, since it’s organization policy to indicate that the new table includes personally identifiable information (PII), option D is correct. The line of code would be added after defining the table structure and before closing with a semicolon. References: Data Engineer Associate Exam Guide, CREATE TABLE USING (Databricks SQL)

https://www.w3schools.com/python/python_functions.asp

https://www.geeksforgeeks.org/python-functions/

A fan-in dependency pattern makes the dashboard task depend on all three parallel tasks. After the entry notebook completes, Databricks can execute the cleansing, feature-engineering, and model-update tasks concurrently because none depends on another. The dashboard refresh remains blocked until every configured upstream dependency has completed successfully. In Lakeflow Jobs, task dependencies form a directed acyclic graph, and Databricks runs as many eligible upstream tasks in parallel as possible. The default All succeeded Run if condition ensures that the downstream dashboard task runs only when all three dependencies succeed. Sequentially chaining the parallel tasks would unnecessarily increase the total execution time. Depending on only one task could refresh the dashboard before the other outputs are ready. Therefore, option B correctly implements the required fan-out and fan-in DAG.

================

https://spark.apache.org/docs/latest/api/python/reference/pyspark.ss/api/pyspark.sql.streaming.DataStreamWriter.trigger.html

Deploying a bundle creates or updates the resources defined in its configuration, but deployment does not automatically execute those resources. The databricks bundle run command starts a deployed job or pipeline by using its resource key from the bundle configuration. Because the job is keyed as etl_job, the correct command is databricks bundle run etl_job. The command bundle deploy run is not valid CLI syntax, and bundle execute is not the command used to run a bundle resource. databricks jobs submit creates and runs a one-time job from a supplied JSON definition rather than executing the job already managed by the bundle. Therefore, option D correctly triggers the deployed job while preserving the bundle-managed resource configuration.

================

 Data Explorer is a graphical interface that allows you to browse, create, and manage data objects such as databases, tables, and views in your workspace. You can also review and modify the permissions on these data objects using Data Explorer. To access Data Explorer, you can click on the Data icon in the sidebar, or use the %sql magic command in a notebook. You can then select a database and a table, and click on the Permissions tab to view and edit the access control lists (ACLs) for the table. You can also use SQL commands such as SHOW GRANT and GRANT to query and modify the permissions on a Delta table. References:

Data Explorer

Access control for Delta tables

SHOW GRANT

[GRANT]

For small-scale, ad hoc Python workloads that run frequently and should start quickly and stop without leaving idle infrastructure running, serverless compute is the best fit. Databricks documentation explains that serverless compute allows users to run workloads without provisioning compute resources manually , and Databricks automatically allocates and manages the required resources. This improves startup and scaling speed and minimizes idle time, which directly matches the requirement that compute should “wind down quickly” after the work completes. Databricks also recommends serverless compute broadly when the workload is supported because it is the simplest and most reliable option. An all-purpose cluster is better suited to interactive collaborative development and can remain running longer than needed. A job cluster is ephemeral and often a good batch choice, but it still requires more explicit configuration than serverless and is not as optimized for minimal operational overhead. A SQL warehouse is intended for SQL workloads, not general ad hoc Python script execution. Therefore, serverless compute is the most appropriate choice.

Delta Lake provides schema enforcement on write, preventing incompatible data from being silently added to a table. When schema evolution is explicitly enabled through the appropriate write or ingestion configuration, new optional columns can be added without dropping and recreating the existing table. Delta Lake also maintains a transaction log containing committed table versions, enabling table-history inspection and time-travel queries for debugging or auditing, subject to retention settings. A plain Parquet table does not provide Delta’s transactional history or equivalent write enforcement. Recreating a Delta table whenever columns change is unnecessary and can disrupt its history. Auto Loader can detect evolving source schemas, but an external table with schema inference alone does not provide the complete set of required Delta capabilities. Therefore, option D is correct.

================

Reading a Unity Catalog table requires privileges at all relevant levels of its three-part namespace. The group needs USE CATALOG on corp_marketing, USE SCHEMA on campaigns, and SELECT on email_stats. These privileges allow users to resolve the catalog and schema and read the table without granting permission to create or modify objects. Option A omits USE CATALOG, so table access is incomplete. Option B unnecessarily grants CREATE SCHEMA, violating the requirement that the group must not create objects. Option D omits USE CATALOG and grants MODIFY, which would permit data changes. Unity Catalog follows a hierarchical permissions model, but possession of SELECT alone is insufficient without the required usage privileges on the parent containers. Therefore, the least-privilege sequence in option C is correct.

================

The command databricks bundle deploy deploys the bundle’s local files, artifacts, and resource definitions to the configured remote Databricks workspace. Databricks documents this command as the deployment operation for Declarative Automation Bundles, and a specific target can be selected with the -t or --target option. The validate command checks whether the bundle configuration is valid but does not create or update workspace resources. The run command executes a resource that has already been deployed; it is not the deployment step. The init command creates a new bundle project from a template, while bundle deployment is not a valid command combination. Because the question asks for the single command that sends local bundle assets to the target workspace, option C is correct.

================

Unity Catalog attribute-based access control policies provide centralized, tag-driven governance. Administrators define policies at a catalog or schema scope and use governed tags to identify the tables or columns to which the policies apply. When matching tags are assigned to current or future objects, the applicable policy is evaluated automatically, avoiding repetitive manual attachment of filters and masks to individual tables. This approach improves scalability and consistency when many objects require the same protection rules. It is important that the policy applies to objects matching its tag conditions, rather than indiscriminately to every object. The other choices incorrectly describe supported object types, multiple-filter behavior, or implementation languages. Databricks specifically positions attribute-based access control as the centralized approach for consistently applying row filtering and column masking across many governed tables. Therefore, option A is correct.

================

Option D is directly associated with the failure because toPandas() collects the complete Spark DataFrame into a single pandas DataFrame in the driver process. With 50 million wide rows, the collected result can exceed the memory assigned to the Spark driver and produce a driver-side OutOfMemoryError. Databricks and PySpark documentation warn that toPandas() should be used only when the resulting dataset is expected to be small because all records are loaded into driver memory. spark.executor.memory controls the executor processes that perform distributed transformations, not the final local pandas object. spark.sql.shuffle.partitions controls the number of shuffle partitions, while spark.memory.fraction controls Spark execution and storage memory allocation. Therefore, the relevant configuration among the available choices is spark.driver.memory, making option D correct.

================

databricks bundle validate should be run first because it checks whether the bundle configuration files are syntactically valid and reports configuration warnings. After validation succeeds, databricks bundle deploy creates or updates the bundle-defined resources and uploads the required artifacts to the selected workspace target. The final command, databricks bundle run, starts a deployed job or Lakeflow pipeline by using its resource key. bundle init creates a new project from a template and is not the validation step for an already developed bundle. bundle sync synchronizes files for development workflows but does not replace a complete deployment. The commands bundle execute, bundle check, bundle upload, and bundle start are not the appropriate command sequence. Therefore, option B is correct.

Option D is the correct answer because it enables the Serverless feature for the SQL endpoint, which allows the endpoint to automatically scale up and down based on the query load. This way, the endpoint can handle more concurrent queries and reduce the time it takes to return results. The Serverless feature also reduces the cold start time of the endpoint, which is the time it takes to start the cluster when a query is submitted to a non-running endpoint. The Serverless feature is available for both AWS and Azure Databricks platforms.

 Databricks SQL Serverless, Serverless SQL endpoints, New Performance Improvements in Databricks SQL

 In Databricks, you can use different languages within the same notebook by using magic commands. Magic commands are special commands that start with a percentage sign (%) and allow you to change the behavior of the cell. To use SQL within a cell of a Python notebook, you can add %sql to the first line of the cell. This will tell Databricks to interpret the rest of the cell as SQL code and execute it against the default database. You can also specify a different database by using the USE statement. The result of the SQL query will be displayed as a table or a chart, depending on the output mode. You can also assign the result to a Python variable by using the -o option. For example, %sql -o df SELECT * FROM my_table will run the SQL query and store the result as a pandas DataFrame in the Python variable df. Option A is incorrect, as it is possible to use SQL in a Python notebook using magic commands. Option B is incorrect, as attaching the cell to a SQL endpoint is not necessary and will not change the language of the cell. Option C is incorrect, as simply writing SQL syntax in the cell will result in a syntax error, as the cell will still be interpreted as Python code. Option E is incorrect, as changing the default language of the notebook to SQL will affect all the cells, not just one. References: Use SQL in Notebooks - Knowledge Base - Noteable, [SQL magic commands - Databricks], [Databricks SQL Guide - Databricks]

 To return the number of null values in the member_id column, the best option is to use the count_if function, which counts the number of rows that satisfy a given condition. In this case, the condition is that the member_id column is null. The other options are either incorrect or not supported by Spark SQL. Option A will return the number of non-null values in the member_id column. Option B will not work because there is no count_null function in Spark SQL. Option D will not work because there is no null function in Spark SQL. Option E will not work because there is no count_null function in Spark SQL. References:

Built-in Functions - Spark SQL, Built-in Functions

count_if - Spark SQL, Built-in Functions

he approach that can be used to identify the owner of new_table is to review the Owner field in the table’s page in Data Explorer. Data Explorer is a web-based interface that allows users to browse, create, and manage data objects such as tables, views, and functions in Databricks1. The table’s page in Data Explorer provides various information about the table, such as its schema, partitions, statistics, history, and permissions2. The Owner field shows the name and email address of the user who created or owns the table3. The data engineer can use this information to contact the table owner and request for permission to access the table.

The other options are not correct or reliable for identifying the owner of new_table. Reviewing the Permissions tab in the table’s page in Data Explorer can show the users and groups who have access to the table, but not necessarily the owner4. Reviewing the Owner field in the table’s page in the cloud storage solution can be misleading, as the owner of the data files may not be the same as the owner of the table5. There is a way to identify the owner of the table, as explained above, so option E is false.

1: Data Explorer | Databricks on AWS

2: Table details | Databricks on AWS

3: Set owner when creating a view in databricks sql - Databricks - 9978

4: Table access control | Databricks on AWS

5: External tables | Databricks on AWS

A Gold table is a table that contains highly refined and aggregated data that powers analytics, machine learning, and production applications. It represents data that has been transformed into knowledge, rather than just information. A Gold table is typically the final output of a medallion lakehouse architecture, where data flows from Bronze to Silver to Gold tables, with each layer improving the structure and quality of data. A job that queries aggregated data designed to feed into a dashboard is an example of a data workload that will utilize a Gold table as its source, as it requires data that is ready for consumption and analysis. The other options are either data workloads that will use a Bronze or Silver table as their source, or data workloads that will produce a Gold table as their output. References: Databricks Documentation - What is the medallion lakehouse architecture?, Databricks Documentation - What is a Medallion Architecture?, K21Academy - Delta Lake Architecture & Azure Databricks Workspace.

In Production mode, the pipeline runs continuously and updates the output tables whenever new data is available in the input sources. The compute resources are allocated on demand and released when the pipeline is stopped. This mode is suitable for production workloads that require high availability and reliability. References: Configure pipeline settings for Delta Live Tables, Tutorial: Run your first Delta Live Tables pipeline, Building Reliable Data Pipelines Using DataBricks’ Delta Live Tables

 Create a table using SQL | Databricks on AWS, Create a table using SQL - Azure Databricks, Delta Lake Quickstart - Azure Databricks

Databricks Repos is a visual Git client and API in Databricks that supports common Git operations such as commit, pull, push, branch management, and visual comparison of diffs when committing1. However, merge is not supported in the Git dialog2. You need to use the Repos UI or your Git provider to merge branches3. Merge is a way to combine the commit history from one branch into another branch1. During a merge, a merge conflict is encountered when Git cannot automatically combine code from one branch into another. Merge conflicts require manual resolution before a merge can be completed1. References: 4: Run Git operations on Databricks Repos4, 1: CI/CD techniques with Git and Databricks Repos1, 3: Collaborate in Repos3, 2: Databricks Repos - What it is and how we can use it2.

Databricks Repos is a visual Git client and API in Databricks that supports common Git operations such as commit, pull, push, merge, and branch management. However, to clone a remote Git repository to a Databricks repo, you must use the Databricks UI or API. You cannot clone a Git repo using the CLI through a cluster’s web terminal, as the files won’t display in the Databricks UI1. References: 1: Run Git operations on Databricks Repos | Databricks on AWS2

Option B keeps environment differences entirely in the bundle configuration. Declarative Automation Bundles support custom variables that can be referenced by resource definitions, making the same job and notebook definitions reusable. Databricks targets represent deployment environments such as development, test, and production, and target-level settings take precedence over top-level settings, including variable overrides. This allows the team to deploy the same codebase while supplying different workspace paths, compute identifiers, catalog names, or other values for each environment. Editing deployed jobs in the workspace UI creates configuration drift and breaks the bundle as the authoritative source. Separate Git branches duplicate configuration and promotion logic, while notebook parameters move environment logic into application code. Therefore, defining variables and overriding them through targets is the correct configuration-only approach.

================

The workspace administrator is the only individual who can transfer ownership of the Delta tables in Data Explorer, assuming the original data engineer no longer has access. The workspace administrator has the highest level of permissions in the workspace and can manage all resources, users, and groups. The other options are either not possible or not sufficient to perform the ownership transfer. The Databricks account representative is not involved in the workspace management. The transfer is possible and not dependent on the original data engineer. The new lead data engineer may not have the necessary permissions to access or modify the Delta tables, unless granted by the workspace administrator or the original data engineer before leaving. References: Workspace access control, Manage Unity Catalog object ownership.

Databricks intentionally stops existing streaming reads and writes after an external Delta table is converted to a Unity Catalog managed table. This protects data consistency because the table’s managed storage location and access path have changed. The documented recovery action is to restart the stream with the same configuration. The existing checkpoint can then resume processing from the last committed offset, and supported path-based access is redirected to the converted managed table. REFRESH TABLE invalidates cached metadata but does not repair a stopped Structured Streaming query or modify its checkpoint. Deleting the checkpoint would discard progress information and could cause unnecessary reprocessing or duplicate output. Additional permissions are not the indicated solution when the failure is specifically caused by managed-table conversion. Therefore, option A is correct.

================

 An external table is a table that is defined in the metastore and points to an existing location in the storage system. When you drop an external table, only the metadata is deleted from the metastore, but the data files are not deleted from the storage system. This is because external tables are meant to be shared by multiple applications and users, and dropping them should not affect the data availability. On the other hand, a managed table is a table that is defined in the metastore and also managed by the metastore. When you drop a managed table, both the metadata and the data files are deleted from the metastore and the storage system, respectively. This is because managed tables are meant to be exclusive to the application or user that created them, and dropping them should free up the storage space. Therefore, the correct answer is C, because the table was external and only the metadata was deleted when the table was dropped. References: Databricks Documentation - Managed and External Tables, Databricks Documentation - Drop Table

Option A is the correct answer because it shows the result of an INNER JOIN between the two tables. An INNER JOIN returns only the rows that have matching values in both tables based on the join condition. In this case, the join condition is ON a.customer_id = c.customer_id, which means that only the rows that have the same customer ID in both tables will be included in the output. The output will have four columns: customer_id, name, account_id, and overdraft_amt. The output will have four rows, corresponding to the four customers who have accounts in the account table.

The use of INNER JOIN can be referenced from Databricks documentation on SQL JOIN or from other sources like W3Schools or GeeksforGeeks.

 The spark.sql method can be used to execute SQL queries programmatically and return the result as a DataFrame. The spark.sql method accepts a string argument that contains a valid SQL statement. The data engineer can use a formatted string literal (f-string) to insert the Python variable table_name into the SQL query. The other methods are either invalid or not suitable for running SQL queries. References: Running SQL Queries Programmatically, Formatted string literals, spark.sql

 Option E is the correct answer because it is the only mandatory requirement when creating a new Delta Live Tables pipeline. A pipeline is a data processing workflow that contains materialized views and streaming tables declared in Python or SQL source files. Delta Live Tables infers the dependencies between these tables and ensures updates occur in the correct order. To create a pipeline, you need to specify at least one notebook library to be executed, which contains the Delta Live Tables syntax. You can also specify multiple libraries of different languages within your pipeline. The other options are optional or not applicable for creating a pipeline. Option A is not required, but you can optionally provide a key-value pair configuration to customize the pipeline settings, such as the storage location, the target schema, the notifications, and the pipeline mode. Option B is not applicable, as the DBU/hour cost is determined by the cluster configuration, not the pipeline creation. Option C is not required, but you can optionally specify a storage location for the output data from the pipeline. If you leave it empty, the system uses a default location. Option D is not required, but you can optionally specify a location of a target database for the written data, either in the Hive metastore or the Unity Catalog.

 Tutorial: Run your first Delta Live Tables pipeline, What is Delta Live Tables?, Create a pipeline, Pipeline configuration.

Option A uses the supported ALTER TABLE ... SET MANAGED conversion for a Unity Catalog external Delta table. Databricks documents that this conversion retains the table name, permissions, dependent objects, settings, and table history while minimizing downtime for readers and writers. After conversion, Databricks manages the table’s storage lifecycle, satisfying the requirement for platform-controlled cleanup when the managed table is dropped. Dropping the original table with PURGE destroys the object instead of migrating it. SET LOCATION changes the storage path referenced by an external table but does not convert the table into a managed table. Creating a replacement table with CREATE TABLE AS SELECT introduces a new object and requires manual migration of permissions, dependencies, and workloads. Therefore, ALTER TABLE catalog.schema.sales SET MANAGED is the correct action.

================

Lakehouse Federation is the Databricks feature built for querying external systems without moving all data into Databricks . Databricks documentation describes it as the platform for query federation , enabling users to run queries against multiple external data sources while keeping governance centralized through Unity Catalog . Databricks also documents support for external systems such as Amazon Redshift and other databases through connections and foreign catalogs, allowing read-only access to external data while managing permissions in Unity Catalog. This aligns directly with the requirement to minimize duplication and still maintain centralized governance. Databricks Connect is for local development against Databricks compute, not federated querying. MLflow is for machine learning lifecycle management. Delta Lake is a storage format and table layer, not a federation framework. Therefore, when the goal is unified governance across Databricks and external systems like MySQL, Redshift, and BigQuery without first ingesting the data, Lakehouse Federation is the documented answer. Databricks does note that for high-volume production ingestion, managed connectors may sometimes be preferred, but for direct querying without data movement, Lakehouse Federation is the correct feature.

Databricks Asset Bundles provide a modern, declarative approach to defining, testing, and deploying Databricks workflows as version-controlled code artifacts . Using YAML configuration files, engineers can describe jobs, pipelines, dependencies, and environments in a structured and reproducible way. Asset Bundles integrate well with CI/CD pipelines and external schedulers, enabling teams to package workflows, run integration tests, and deploy consistently across environments such as development, staging, and production. This aligns directly with the requirement for version control and external orchestration. The Databricks CLI (A) is primarily used as a deployment interface but does not provide the structured packaging and workflow definition capabilities of Asset Bundles. Databricks Connect (C) is intended for local development and testing of Spark code, not workflow deployment. The Databricks SDK (D) provides programmatic access to APIs but lacks the declarative, packaged deployment model required here. Databricks documentation positions Asset Bundles as the recommended solution for production-grade workflow deployment and testing within modern data engineering practices.

Option D uses the business-key columns as the subset for duplicate comparison. Databricks documents that DataFrame.dropDuplicates can optionally accept a list of columns and then removes rows that are duplicates with respect to only those columns. Therefore, differing ingestion_timestamp values do not prevent records with the same customer_id and email from being treated as duplicates. Calling distinct() compares complete rows, so timestamp differences would preserve multiple records. Selecting only customer_id and email before distinct would remove every other column from the Silver result. The groupBy and max expression calculates a latest timestamp per key, but it changes the output shape and does not preserve a complete source record unless additional logic is added. For the stated requirement, option D is the direct and correct operation.

================

This approach would allow the data engineering team to use the existing SQL program and add some logic to control the execution of the final query based on the day of the week. They could use the datetime module in Python to get the current date and check if it is a Sunday. If so, they could run the final query, otherwise they could skip it. This way, they could schedule the program to run every day without changing the data model or the source table. References: PySpark SQL Module, Python datetime Module, Databricks Jobs

Unity Catalog organizes data objects through a three-level namespace consisting of catalog, schema, and object. To query sales_catalog.market_data.summary, the analyst requires USE CATALOG on sales_catalog, USE SCHEMA on market_data, and SELECT on the summary view. Granting SELECT on the view alone is insufficient when either parent usage privilege is missing. The analyst does not normally require access to the underlying tables queried by the view because the view owner’s privileges are used to resolve those objects. A workspace or cluster is not part of the Unity Catalog securable hierarchy for resolving the view name. Storage credentials and external locations govern cloud-storage access rather than ordinary view-query permissions. Therefore, the engineer should verify privileges in the order catalog, schema, and view, making option B correct.

================

The engineer should inspect the job’s maximum concurrent-runs configuration and its queueing setting. max_concurrent_runs controls how many active runs of the same job can execute simultaneously. When that limit has already been reached, Databricks can skip the newly triggered run if queueing is disabled. When queueing is enabled, the additional run waits until capacity becomes available instead of being skipped. Spot-instance availability might delay or prevent compute startup, but it would not normally produce a job-level concurrency message. SQL warehouse permissions affect SQL task execution after a run begins, while timeout_seconds determines how long a task may run before timing out. Therefore, option C directly addresses the reported run state. This candidate was also compared against the existing Word question bank.

================

The Auto Stop feature allows the SQL endpoint to automatically stop after a specified period of inactivity. This can help reduce the cost and resource consumption of the SQL endpoint, as it will only run when it is needed to refresh the dashboard or execute queries. The data engineer can configure the Auto Stop setting for the SQL endpoint from the SQL Endpoints UI, by selecting the desired idle time from the Auto Stop dropdown menu. The default idle time is 120 minutes, but it can be set to as low as 15 minutes or as high as 240 minutes. Alternatively, the data engineer can also use the SQL Endpoints REST API to set the Auto Stop setting programmatically. References: SQL Endpoints UI, SQL Endpoints REST API, Refreshing SQL Dashboard

Option D provides a practical balance between the 15-minute latency objective and compute cost. A scheduled Lakeflow Job running every 12 minutes leaves a small margin for startup and processing while avoiding an always-running stream. Trigger.AvailableNow processes all records available when the query starts in one or more incremental micro-batches and then terminates, allowing the associated compute to be released after the backlog is consumed. Running the job every minute would introduce unnecessary job-start and storage-check overhead. A continuous trigger would keep compute active even when little or no data is arriving. Triggering a separate run for every individual file can also create excessive runs and operational overhead. Therefore, the scheduled AvailableNow strategy in option D supports bounded latency while minimizing idle compute consumption.

================

Interactive compute is designed for exploratory development, iterative notebook execution, and debugging. Developers can attach notebooks to the running compute resource, execute individual cells, inspect intermediate results, and rapidly modify and retest code. After development is complete, the notebook can be configured as a scheduled Lakeflow Job using jobs compute or an appropriate serverless option. Classic jobs compute is optimized for non-interactive production runs and is normally created for a job execution rather than maintained for rapid manual iteration. A SQL warehouse is primarily intended for SQL analytics, dashboards, and queries, not general notebook development involving arbitrary Spark or Python code. An instance pool supplies prestarted virtual machines to other compute resources but is not itself an execution environment. Therefore, option B best supports the development-stage requirements.

================

Databricks Asset Bundles are a Databricks CLI feature that lets teams define Databricks projects “as code” using YAML configuration files . In a bundle, metadata is expressed in YAML (for example, a required databricks.yml at the root), and the configuration can declare resources (such as Databricks Jobs/workflows and other supported assets), artifacts , and deployment targets . After defining these YAML manifests, teams use the Databricks CLI to validate, deploy, and run the bundle into a Databricks workspace, which matches the scenario described (YAML + CLI deployment). This approach standardizes how workflows and related resources are packaged and promoted across environments (dev/test/prod) while keeping definitions version-controlled and reproducible. GitHub is only a hosting/version-control platform (not the YAML deployment feature itself), Terraform is a separate IaC tool, and “DataOps” is a methodology rather than a specific Databricks feature. ( Databricks Documentation )

 The VACUUM command is used to remove files that are no longer referenced by a Delta table and are older than the retention threshold1. The default retention period is 7 days2, but it can be changed by setting the delta.logRetentionDuration and delta.deletedFileRetentionDuration configurations3. If the VACUUM command was run on the table with a retention period shorter than 3 days, then the data files that were needed to restore the table to a 3-day-old version would have been deleted. The other commands do not delete data files from the table. The TIME TRAVEL command is used to query a historical version of the table4. The DELETE HISTORY command is not a valid command in Delta Lake. The OPTIMIZE command is used to improve the performance of the table by compacting small files into larger ones5. The HISTORY command is used to retrieve information about the operations performed on the table. References: 1: VACUUM | Databricks on AWS 2: Work with Delta Lake table history | Databricks on AWS 3: [Delta Lake configuration | Databricks on AWS]  4: Work with Delta Lake table history - Azure Databricks 5: [OPTIMIZE | Databricks on AWS] : [HISTORY | Databricks on AWS]

DataFrame.dropna() removes rows containing NULL or NaN values. Its default how= " any " behavior removes a row when any column included in the specified subset is NULL. By passing [ " customer_email " , " customer_phone " ] as the subset, the operation removes a record when either required field is missing, while ignoring NULL values in unrelated columns. Option B can produce the same filtering result, but it uses two chained where() calls and therefore does not meet the single-method-call requirement. Option C uses how= " all " and would remove a row only when all evaluated columns are NULL. Option D contains one filter() operation but requires multiple column-function calls. Consequently, option A is the direct PySpark method designed for this requirement.

================

One of the benefits of the Databricks Lakehouse Platform embracing open source technologies is that it avoids vendor lock-in. This means that customers can use the same open source tools and frameworks across different cloud providers, and migrate their data and workloads without being tied to a specific vendor. The Databricks Lakehouse Platform is built on open source projects such as Apache Spark™, Delta Lake, MLflow, and Redash, which are widely used and trusted by millions of developers. By supporting these open source technologies, the Databricks Lakehouse Platform enables customers to leverage the innovation and community of the open source ecosystem, and avoid the risk of being locked into proprietary or closed solutions. The other options are either not related to open source technologies (A, B, C, D), or not benefits of the Databricks Lakehouse Platform (A, B). References: Databricks Documentation - Built on open source, Databricks Documentation - What is the Lakehouse Platform?, Databricks Blog - Introducing the Databricks Lakehouse Platform.

 In Databricks SQL, you can use scheduled query executions to update your dashboards or enable routine alerts. By default, your queries do not have a schedule. To set the schedule, you can use the dropdown pickers to specify the frequency, period, starting time, and time zone. You can also choose to end the schedule on a certain date by selecting the End date checkbox and picking a date from the calendar. This way, you can ensure that the query does not run beyond the first week of the project’s release and does not incur any additional cost. Option A is incorrect, as setting a limit to the number of DBUs does not stop the query from running. Option B is incorrect, as there is no option to end the schedule after a certain number of refreshes. Option C is incorrect, as there is a way to ensure the query does not cost the organization money beyond the first week of the project’s release. Option D is incorrect, as setting a limit to the number of individuals who can manage the query’s refresh schedule does not affect the query’s execution or cost. References: Schedule a query, Schedule a query - Azure Databricks - Databricks SQL