Databricks-Generative-AI-Engineer-Associate Databricks Certified Generative AI Engineer Associate Questions and Answers

The Prompt Registry preserves identifiable prompt versions, allowing the team to recover the exact template associated with a historical execution. Linking inference activity to the prompt’s name and version establishes which registered template the application actually used. MLflow also supports lineage between prompt versions, application versions, and execution traces. However, satisfying the entire audit requires retaining the actual runtime variable values or the fully rendered prompt in the relevant execution records. Registry history alone preserves the template, not every value substituted into it. Therefore, D is the best option, provided the logging captures those inputs. Model webhooks, experiment tags, or table history alone do not provide the same explicit prompt-version lineage and complete historical reconstruction. Databricks documentation

The task is to benchmark LLMs for text translation using an evaluation set with known high-quality examples, requiring a performant metric. Let’s evaluate the options.

Option A: ROUGE metric

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) measures overlap between generated and reference texts, primarily for summarization. It’s less suited for translation, where precision and word order matter more.

Databricks Reference : " ROUGE is commonly used for summarization, not translation evaluation " ( " Generative AI Cookbook, " 2023).

Option B: BLEU metric

BLEU (Bilingual Evaluation Understudy) evaluates translation quality by comparing n-gram overlap with reference translations, accounting for precision and brevity. It’s widely used, performant, and appropriate for this task.

Databricks Reference : " BLEU is a standard metric for evaluating machine translation, balancing accuracy and efficiency " ( " Building LLM Applications with Databricks " ).

Option C: NDCG metric

NDCG (Normalized Discounted Cumulative Gain) assesses ranking quality, not text generation. It’s irrelevant for translation evaluation.

Databricks Reference : " NDCG is suited for ranking tasks, not generative output scoring " ( " Databricks Generative AI Engineer Guide " ).

Option D: RECALL metric

Recall measures retrieved relevant items but doesn’t evaluate translation quality (e.g., fluency, correctness). It’s incomplete for this use case.

Databricks Reference : No specific extract, but recall alone lacks the granularity of BLEU for text generation tasks.

Conclusion : Option B (BLEU) is the best metric for translation evaluation, offering a performant and standard approach, as endorsed by Databricks’ guidance on generative tasks.

Problem Context : The Generative AI Engineer needs a tool to build a multi-step LLM-based workflow . This type of workflow often involves chaining multiple steps together, such as query generation, retrieval of information, response generation, and post-processing, with LLMs integrated at several points.

Explanation of Options :

Option A: Pandas : Pandas is a powerful data manipulation library for structured data analysis, but it is not designed for managing or orchestrating multi-step workflows, especially those involving LLMs.

Option B: TensorFlow : TensorFlow is primarily used for training and deploying machine learning models, especially deep learning models. It is not designed for orchestrating multi-step tasks in LLM-based workflows.

Option C: PySpark : PySpark is a distributed computing framework used for large-scale data processing. While useful for handling big data, it is not specialized for chaining LLM-based operations.

Option D: LangChain : LangChain is a purpose-built framework designed specifically for orchestrating multi-step workflows with large language models (LLMs). It enables developers to easily chain different tasks, such as retrieving documents, summarizing information, and generating responses, all in a structured flow. This makes it the best tool for building complex LLM-based workflows.

Thus, LangChain is the most suitable library for creating multi-step LLM-based workflows.

To mitigate the issue of the LLM including explanations of how summaries are generated in its output, the best approach is to adjust the training or prompt structure. Here’s why Option D is effective:

Few-shot Learning : By providing specific examples of how the desired output should look (i.e., just the summary without explanation), the model learns the preferred format. This few-shot learning approach helps the model understand not only what content to generate but also how to format its responses.

Prompt Engineering : Adjusting the user prompt to specify the desired output format clearly can guide the LLM to produce summaries without additional explanatory text. Effective prompt design is crucial in controlling the behavior of generative models.

Why Other Options Are Less Suitable:

A : While technically feasible, splitting the output by newline and truncating could lead to loss of important content or create awkward breaks in the summary.

B : Tuning chunk sizes or changing embedding models does not directly address the issue of the model ' s tendency to generate explanations along with summaries.

C : Revisiting document ingestion logic ensures accurate source data but does not influence how the model formats its output.

By using few-shot examples and refining the prompt, the engineer directly influences the output format, making this approach the most targeted and effective solution.

In the context of developing a chatbot for a company ' s internal HelpDesk Call Center, the key is to select data sources that provide the most contextual and detailed information about the issues being addressed. This includes identifying the root cause and suggesting resolutions. The two most appropriate sources from the list are:

Call Detail (Option D):

Contents: This Delta table includes a snapshot of all call details updated hourly, featuring essential fields like root_cause and resolution.

Relevance: The inclusion of root_cause and resolution fields makes this source particularly valuable, as it directly contains the information necessary to understand and resolve the issues discussed in the calls. Even if some records are incomplete, the data provided is crucial for a chatbot aimed at speeding up resolution identification.

Transcript Volume (Option E):

Contents: This Unity Catalog Volume contains recordings in .wav format and text transcripts in .txt files.

Relevance: The text transcripts of call recordings can provide in-depth context that the chatbot can analyze to understand the nuances of each issue. The chatbot can use natural language processing techniques to extract themes, identify problems, and suggest resolutions based on previous similar interactions documented in the transcripts.

Why Other Options Are Less Suitable:

A (Call Cust History): While it provides insights into customer interactions with the HelpDesk, it focuses more on the usage metrics rather than the content of the calls or the issues discussed.

B (Maintenance Schedule): This data is useful for understanding when services may not be available but does not contribute directly to resolving user issues or identifying root causes.

C (Call Rep History): Though it offers data on call durations and start times, which could help in assessing performance, it lacks direct information on the issues being resolved.

Therefore, Call Detail and Transcript Volume are the most relevant data sources for a chatbot designed to assist with identifying and resolving issues in a HelpDesk Call Center setting, as they provide direct and contextual information related to customer issues.

In the context of legal document analysis where the " evolution of interpretation " is the primary goal, preserving narrative continuity is paramount. Windowed summarization with overlapping chunks is the most effective method for this use case. Overlapping (e.g., 10-15% of the chunk size) ensures that sentences or concepts split at the boundary of one chunk are preserved in the next, preventing the loss of critical context that often occurs in legal jargon. Furthermore, windowed summarization allows the system to condense long-form court opinions into manageable parts while maintaining the chronological " thread " of the argument. While sentence-level embeddings with metadata (D) are useful for filtering, they often lack the sufficient context required to understand the nuances of a legal ruling. A windowed approach provides the LLM with enough surrounding text to understand the " why " behind a legal evolution, rather than just the " when. "

Problem Context : The chatbot must handle various types of queries and intelligently route them to the appropriate responses or systems.

Explanation of Options :

Option A : Limiting the chatbot to only previous event information restricts its utility and does not meet the broader business requirements.

Option B : Having two separate chatbots could unnecessarily complicate user interaction and increase maintenance overhead.

Option C : Implementing a multi-step workflow where the chatbot first identifies the type of question and then routes it accordingly is the most efficient and scalable solution. This approach allows the chatbot to handle a variety of queries dynamically, improving user experience and operational efficiency.

Option D : Focusing solely on payments would not satisfy all the specified user interaction needs, such as inquiring about event details.

Option C offers a comprehensive workflow that maximizes the chatbot’s utility and responsiveness to different user needs, aligning perfectly with the business requirements.

The task involves building a question-answering chatbot for an automotive company using a database of vehicle-related documents. The chatbot must efficiently process customer inquiries and provide accurate responses. Let’s evaluate each component to determine which is not useful, per Databricks Generative AI Engineer principles.

Option A: Response-generating LLM

An LLM is essential for generating natural language responses to customer queries based on retrieved information. This is a core component of any chatbot.

Databricks Reference : " The response-generating LLM processes retrieved context to produce coherent answers " ( " Building LLM Applications with Databricks, " 2023).

Option B: Invite users to submit long, rather than concise, questions

Encouraging long questions is a user interaction design choice, not a technical component of the chatbot’s architecture. Moreover, long, verbose questions can complicate intent detection and retrieval, reducing efficiency and accuracy—counter to best practices for chatbot design. Concise questions are typically preferred for clarity and performance.

Databricks Reference : While not explicitly stated, Databricks’ " Generative AI Cookbook " emphasizes efficient query processing, implying that simpler, focused inputs improve LLM performance. Inviting long questions doesn’t align with this.

Option C: Vector database

A vector database stores embeddings of the vehicle documents, enabling fast retrieval of relevant information via semantic search. This is critical for a question-answering system with a large document corpus.

Databricks Reference : " Vector databases enable scalable retrieval of context from large datasets " ( " Databricks Generative AI Engineer Guide " ).

Option D: Embedding model

An embedding model converts text (documents and queries) into vector representations for similarity search. It’s a foundational component for retrieval-augmented generation (RAG) in chatbots.

Databricks Reference : " Embedding models transform text into vectors, facilitating efficient matching of queries to documents " ( " Building LLM-Powered Applications " ).

Conclusion : Option B is not a useful component in building the chatbot. It’s a user-facing suggestion rather than a technical building block, and it could even degrade performance by introducing unnecessary complexity. Options A, C, and D are all integral to a Databricks-aligned chatbot architecture.

Problem Context : The engineer needs an open-source LLM with a large context window to develop an application.

Explanation of Options :

Option A: DistilBERT : While an efficient and smaller version of BERT, DistilBERT does not provide a particularly large context window.

Option B: MPT-30B : This model, while large, is not specified as being particularly notable for its context window capabilities.

Option C: Llama2-70B : Known for its large model size and extensive capabilities, including a large context window. It is also available as an open-source model, making it ideal for applications requiring extensive contextual understanding.

Option D: DBRX : This is not a recognized standard model in the context of large language models with extensive context windows.

Thus, Option C (Llama2-70B) is the best fit as it meets the criteria of having a large context window and being available for open-source use, suitable for developing robust language understanding applications.

Microsoft Teams integration directly satisfies the requirement to use the agent within the analysts’ existing collaboration environment. Users can ask questions where they already discuss financial information, avoiding a separate portal or browser application. Databricks documents Teams integration that allows users to ask data questions through direct messages, channels, or group chats. Authentication and downstream authorization must still be configured correctly; selecting Teams does not automatically grant access to governed data. Slack conflicts with the company’s standardized communication platform. A custom frontend or Databricks App can provide an effective interface, but both introduce an additional destination when the requirement explicitly prioritizes staying in Teams. The deployment should connect the Teams interface to the appropriate governed agent backend. Databricks documentation

Separate Genie Spaces provide domain-specific access to structured business datasets through Databricks-native conversational querying. The Sales specialist calls the Sales Space, and the HR specialist calls the HR Space. Each Space should contain the appropriate datasets and business context, with Unity Catalog permissions enforcing access to its underlying resources. Separating Spaces makes domain boundaries explicit and avoids relying only on an LLM’s routing instructions. Knowledge Assistants primarily support document-based knowledge retrieval, making Genie the better fit for structured business questions. A shared Space with distinct permissions can enforce security, but separate Spaces better express the requested domain separation. The video’s phrase “HR Analyst for stock questions” appears inconsistent; “staff questions” is an inferred typing correction. Databricks documentation

Problem Context : The Generative AI Engineer needs a prompt that will enable an LLM trained on customer call transcripts to classify and respond correctly regarding product availability. The desired response should clearly indicate whether a product is " In Stock " or " Out of Stock, " and it should be formatted in a way that is structured and easy to parse programmatically, such as JSON.

Explanation of Options :

Option A : Respond with “In Stock” if the customer asks for a product. This prompt is too generic and does not specify how to handle the case when a product is not available, nor does it provide a structured output format.

Option B : This option is correctly formatted and explicit. It instructs the LLM to respond based on the availability mentioned in the customer call transcript and to format the response in JSON. This structure allows for easy integration into systems that may need to process this information automatically, such as customer service dashboards or databases.

Option C : Respond with “Out of Stock” if the customer asks for a product. Like option A, this prompt is also insufficient as it only covers the scenario where a product is unavailable and does not provide a structured output.

Option D : While this prompt correctly specifies how to respond based on product availability, it lacks the structured output format, making it less suitable for systems that require formatted data for further processing.

Given the requirements for clear, programmatically usable outputs, Option B is the optimal choice because it provides precise instructions on how to respond and includes a JSON format example for structuring the output, which is ideal for automated systems or further data handling.

To design an LLM-based application that can answer employee HR questions using HR PDF documentation, the most effective approach is option D. Here ' s why:

Chunking and Vector Store Embedding : HR documentation tends to be lengthy, so splitting it into smaller, manageable chunks helps optimize retrieval. These chunks are then embedded into a vector store (a database that stores vector representations of text). Each chunk of text is transformed into an embedding using a transformer-based model, which allows for efficient similarity-based retrieval.

Using Vector Search for Retrieval : When an employee asks a question, the system converts their query into an embedding as well. This embedding is then compared with the embeddings of the document chunks in the vector store. The most semantically similar chunks are retrieved, which ensures that the answer is based on the most relevant parts of the documentation.

LLM to Generate a Response : Once the relevant chunks are retrieved, these chunks are passed into the LLM, which uses them as context to generate a coherent and accurate response to the employee ' s question.

Why Other Options Are Less Suitable :

A (Calculate Averaged Embeddings) : Averaging embeddings might dilute important information. It doesn ' t provide enough granularity to focus on specific sections of documents.

B (Summarize HR Documentation) : Summarization loses the detail necessary for HR-related queries, which are often specific. It would likely miss the mark for more detailed inquiries.

C (Interaction Matrix and ALS) : This approach is better suited for recommendation systems and not for HR queries, as it’s focused on collaborative filtering rather than text-based retrieval.

Thus, option D is the most effective solution for providing precise and contextual answers based on HR documentation.

In the context of designing a chatbot to engage users on a gaming platform, diversity of responses (option B) is a key metric to increase user engagement and retention. Here’s why:

Diverse and Engaging Interactions : A chatbot that provides varied and interesting responses will keep users engaged, especially in an interactive environment like a gaming platform. Gamers typically enjoy dynamic and evolving conversations, and diversity of responses helps prevent monotony, encouraging users to interact more frequently with the bot.

Increasing Retention : By offering different types of responses to similar queries, the chatbot can create a sense of novelty and excitement, which enhances the user’s experience and makes them more likely to return to the platform.

Why Other Options Are Less Effective :

A (Randomness) : Random responses can be confusing or irrelevant, leading to frustration and reducing engagement.

C (Lack of Relevance) : If responses are not relevant to the user’s queries, this will degrade the user experience and lead to disengagement.

D (Repetition of Responses) : Repetitive responses can quickly bore users, making the chatbot feel uninteresting and reducing the likelihood of continued interaction.

Thus, diversity of responses (option B) is the most effective way to keep users engaged and retain them on the platform.

In this scenario, the Generative AI Engineer needs to design a system that can handle different types of queries about the monster truck team. The queries may involve text-based information, API lookups for event dates, or table queries for standings. The best solution is to implement a tool-based agent system .

Here’s how option B works, and why it’s the most appropriate answer:

System Design Using Agent-Based Model : In modern agent-based LLM systems, you can design a system where the LLM (Large Language Model) acts as a central orchestrator. The model can " decide " which tools to use based on the query. These tools can include API calls, table lookups, or natural language searches. The system should contain a system prompt that informs the LLM about the available tools.

System Prompt Listing Tools : By creating a well-crafted system prompt , the LLM knows which tools are at its disposal. For instance, one tool may query an external API for event dates, another might look up standings in a database, and a third may involve searching a vector database for general text-based information. The agent will be responsible for calling the appropriate tool depending on the query.

Agent Orchestration of Calls : The agent system is designed to execute a series of steps based on the incoming query. If a user asks for the next event date, the system will recognize this as a task that requires an API call. If the user asks about standings, the agent might query the appropriate table in the database. For text-based questions, it may call a search function over ingested data. The agent orchestrates this entire process, ensuring the LLM makes calls to the right resources dynamically.

Generative AI Tools and Context : This is a standard architecture for integrating multiple functionalities into a system where each query requires different actions. The core design in option B is efficient because it keeps the system modular and dynamic by leveraging tools rather than overloading the LLM with static information in a system prompt (like option D).

Why Other Options Are Less Suitable :

A (RAG Architecture) : While relevant, simply ingesting PDFs into a vector store only helps with text-based retrieval. It wouldn’t help with API lookups or table queries.

C (Conditional Logic with RAG/API/TABLE) : Although this approach works, it relies heavily on manual text parsing and might introduce complexity when scaling the system.

D (System Prompt with Event Dates and Standings) : Hardcoding dates and table information into a system prompt isn’t scalable. As the standings or events change, the system would need constant updating, making it inefficient.

By bundling multiple tools into a single agent-based system (as in option B), the Generative AI Engineer can best handle the diverse requirements of this system.

The task is to prevent toxic or unsafe responses in an LLM system using the Foundation Model API with minimal effort. Let’s assess the options.

Option A: Host Llama Guard on Foundation Model API and use it to detect unsafe responses

Llama Guard is a safety-focused model designed to detect toxic or unsafe content. Hosting it via the Foundation Model API (a Databricks service) integrates seamlessly with the existing system, requiring minimal setup (just deployment and a check step), and leverages provisioned throughput for performance.

Databricks Reference : " Foundation Model API supports hosting safety models like Llama Guard to filter outputs efficiently " ( " Foundation Model API Documentation, " 2023).

Option B: Add some LLM calls to their chain to detect unsafe content before returning text

Using additional LLM calls (e.g., prompting an LLM to classify toxicity) increases latency, complexity, and effort (crafting prompts, chaining logic), and lacks the specificity of a dedicated safety model.

Databricks Reference : " Ad-hoc LLM checks are less efficient than purpose-built safety solutions " ( " Building LLM Applications with Databricks " ).

Option C: Add a regex expression on inputs and outputs to detect unsafe responses

Regex can catch simple patterns (e.g., profanity) but fails for nuanced toxicity (e.g., sarcasm, context-dependent harm), requiring significant manual effort to maintain and update rules.

Databricks Reference : " Regex-based filtering is limited for complex safety needs " ( " Generative AI Cookbook " ).

Option D: Ask users to report unsafe responses

User reporting is reactive, not preventive, and places burden on users rather than the system. It doesn’t limit unsafe outputs proactively and requires additional effort for feedback handling.

Databricks Reference : " Proactive guardrails are preferred over user-driven monitoring " ( " Databricks Generative AI Engineer Guide " ).

Conclusion : Option A (Llama Guard on Foundation Model API) is the least-effort, most effective approach, leveraging Databricks’ infrastructure for seamless safety integration.

The Multi-Agent Supervisor is designed to coordinate specialized agents and tools behind a unified entry point. It interprets the incoming request, selects suitable capabilities, delegates work, and combines results when necessary. In this scenario, document questions can be handled by a knowledge-retrieval specialist, while event-code extraction can be assigned to a specialized extraction capability. The supervisor provides the coordination needed to select between those tasks and handle requests requiring multiple steps. A Knowledge Assistant’s conversational interface does not itself make it the appropriate general-purpose orchestrator. Adding vector indexes also does not establish specialized extraction and routing behavior. Although option A names the same component, its justification misses the central requirement: coordinated delegation across distinct capabilities. Databricks documentation

Addressing offensive or inflammatory outputs in a Retrieval-Augmented Generation (RAG) system is critical for improving user experience and ensuring ethical AI deployment. Here ' s why D is the most effective approach:

Manual data curation : The root cause of offensive outputs often comes from the underlying data used to train the model or populate the retrieval system. By manually curating the upstream data and conducting thorough reviews before the data is fed into the RAG system, the engineer can filter out harmful, offensive, or inappropriate content.

Improving data quality : Curating data ensures the system retrieves and generates responses from a high-quality, well-vetted dataset. This directly impacts the relevance and appropriateness of the outputs from the RAG system, preventing inflammatory content from being included in responses.

Effectiveness : This strategy directly tackles the problem at its source (the data) rather than just mitigating the consequences (such as informing users or restricting access). It ensures that the system consistently provides non-offensive, relevant information.

Other options, such as increasing the frequency of data updates or informing users about behavior expectations, may not directly mitigate the generation of inflammatory outputs.

The task is to deploy an LLM application using Foundation Model APIs in a production environment while adhering to security best practices. Authentication is critical for securing access to Databricks resources, such as the Foundation Model API. Let’s evaluate the options based on Databricks’ security guidelines for production scenarios.

Option A: Use an access token belonging to service principals

Service principals are non-human identities designed for automated workflows and applications in Databricks. Using an access token tied to a service principal ensures that the authentication is scoped to the application, follows least-privilege principles (via role-based access control), and avoids reliance on individual user credentials. This is a security best practice for production deployments.

Databricks Reference : " For production applications, use service principals with access tokens to authenticate securely, avoiding user-specific credentials " ( " Databricks Security Best Practices, " 2023). Additionally, the " Foundation Model API Documentation " states: " Service principal tokens are recommended for programmatic access to Foundation Model APIs. "

Option B: Use a frequently rotated access token belonging to either a workspace user or a service principal

Frequent rotation enhances security by limiting token exposure, but tying the token to a workspace user introduces risks (e.g., user account changes, broader permissions). Including both user and service principal options dilutes the focus on application-specific security, making this less ideal than a service-principal-only approach. It also adds operational overhead without clear benefits over Option A.

Databricks Reference : " While token rotation is a good practice, service principals are preferred over user accounts for application authentication " ( " Managing Tokens in Databricks, " 2023).

Option C: Use OAuth machine-to-machine authentication

OAuth M2M (e.g., client credentials flow) is a secure method for application-to-service communication, often using service principals under the hood. However, Databricks’ Foundation Model API primarily supports personal access tokens (PATs) or service principal tokens over full OAuth flows for simplicity in production setups. OAuth M2M adds complexity (e.g., managing refresh tokens) without a clear advantage in this context.

Databricks Reference : " OAuth is supported in Databricks, but service principal tokens are simpler and sufficient for most API-based workloads " ( " Databricks Authentication Guide, " 2023).

Option D: Use an access token belonging to any workspace user

Using a user’s access token ties the application to an individual’s identity, violating security best practices. It risks exposure if the user leaves, changes roles, or has overly broad permissions, and it’s not scalable or auditable for production.

Databricks Reference : " Avoid using personal user tokens for production applications due to security and governance concerns " ( " Databricks Security Best Practices, " 2023).

Conclusion : Option A is the best choice, as it uses a service principal’s access token, aligning with Databricks’ security best practices for production LLM applications. It ensures secure, application-specific authentication with minimal complexity, as explicitly recommended for Foundation Model API deployments.

To build an LLM-powered system that accesses up-to-date news articles and stock prices, the best approach is to create an agent that has access to specific tools (option D).

Agent with SQL and Web Search Capabilities : By using an agent-based architecture, the LLM can interact with external tools. The agent can query Delta tables (for up-to-date stock prices) via SQL and perform web searches to retrieve the latest news articles. This modular approach ensures the system can access both structured (stock prices) and unstructured (news) data sources dynamically.

Why This Approach Works :

SQL Queries for Stock Prices : Delta tables store stock prices, which the agent can query directly for the latest data.

Web Search for News : For news articles, the agent can generate search queries and retrieve the most relevant and recent articles, then pass them to the LLM for processing.

Why Other Options Are Less Suitable :

A (Summarizing News for Stock Prices) : This convoluted approach would not ensure accuracy when retrieving stock prices, which are already structured and stored in Delta tables.

B (Stock Price Volatility Queries) : While this could retrieve relevant information, it doesn ' t address how to obtain the most up-to-date news articles.

C (Vector Store) : Storing news articles and stock prices in a vector store might not capture the real-time nature of stock data and news updates, as it relies on pre-existing data rather than dynamic querying.

Thus, using an agent with access to both SQL for querying stock prices and web search for retrieving news articles is the best approach for ensuring up-to-date and accurate responses.

For agents deployed through Model Serving, Databricks supports automatic authentication passthrough for declared resources. The engineer records the agent’s dependencies through the resources parameter when logging the model. During deployment, Databricks checks the deployer’s access, creates a service principal for the agent model version, grants the required resource access, and supplies short-lived credentials. Credential rotation is handled automatically. Supported dependencies include Vector Search indexes, SQL warehouses, and Unity Catalog functions. This approach avoids embedding secrets in source code or recording authentication tokens as model artifacts. Permissions on the serving endpoint govern access to that endpoint; they do not independently establish all downstream resource permissions. All required dependencies, including relevant underlying resources, must be declared correctly. Databricks documentation

Option D best matches the small catalog, infrequent updates, and maintenance requirements. Databricks-managed embeddings remove the need to build and maintain a separate embedding-generation pipeline. A Delta Sync index connects the searchable content to its source Delta table. Triggered synchronization updates the index when explicitly invoked, making it suitable when inventory changes are occasional. Continuous synchronization maintains a running streaming pipeline and therefore introduces additional synchronization costs that this scenario does not justify. Self-managed embeddings add engineering responsibilities without a stated benefit. Storage-optimized endpoints primarily address much larger vector collections. Triggered synchronization reduces synchronization overhead; it does not eliminate the separate costs of serving the vector index or computing embeddings. Databricks documentation

The task involves improving a Retrieval-Augmented Generation (RAG) application’s performance by experimenting with embedding models. The choice of embedding model impacts retrieval accuracy, which is critical for RAG systems. Let’s evaluate the options based on Databricks Generative AI Engineer best practices.

Option A: Pick an embedding model trained on related domain knowledge

Embedding models trained on domain-specific data (e.g., industry-specific corpora) produce vectors that better capture the semantics of the application’s context, improving retrieval relevance. For RAG, this is a key strategy to enhance performance.

Databricks Reference : " For optimal retrieval in RAG systems, select embedding models aligned with the domain of your data " ( " Building LLM Applications with Databricks, " 2023).

Option B: Pick the most recent and most performant open LLM released at the time

LLMs are not embedding models; they generate text, not embeddings for retrieval. While recent LLMs may be performant for generation, this doesn’t address the embedding step in RAG. This option misunderstands the component being selected.

Databricks Reference : Embedding models and LLMs are distinct in RAG workflows: " Embedding models convert text to vectors, while LLMs generate responses " ( " Generative AI Cookbook " ).

Option C: Pick the embedding model ranked highest on the Massive Text Embedding Benchmark (MTEB) leaderboard hosted by HuggingFace

The MTEB leaderboard ranks models across general tasks, but high overall performance doesn’t guarantee suitability for a specific domain. A top-ranked model might excel in generic contexts but underperform on the engineer’s unique data.

Databricks Reference : General performance is less critical than domain fit: " Benchmark rankings provide a starting point, but domain-specific evaluation is recommended " ( " Databricks Generative AI Engineer Guide " ).

Option D: Pick an embedding model with multilingual support to support potential multilingual user questions

Multilingual support is useful only if the application explicitly requires it. Without evidence of multilingual needs, this adds complexity without guaranteed performance gains for the current use case.

Databricks Reference : " Choose features like multilingual support based on application requirements " ( " Building LLM-Powered Applications " ).

Conclusion : Option A is the best strategy because it prioritizes domain relevance, directly improving retrieval accuracy in a RAG system—aligning with Databricks’ emphasis on tailoring models to specific use cases.

Option B preserves the relationships between each user query, generated response, and expert assessment. That context is necessary to turn production feedback into reusable evaluation cases. The engineer can curate representative examples from the Delta table, map them into MLflow’s evaluation schema, and retain the dataset for consistent comparisons between agent versions. Where correctness scoring is required, experts should provide corrected answers or expected facts; a star rating alone is not a ground-truth answer. Keeping only written comments loses essential request context. Selecting only five-star interactions biases the dataset and excludes failure cases that future versions should improve. Deploying highly rated responses is not a substitute for evaluating an updated agent against a stable, representative benchmark. Databricks documentation