AIAI Agent Dev Labaiagentdev-lab.com
Sign in to save progressSign in
Menu

AI-300 · LESSON 5

Optimize generative AI systems and model performance10–15% · 2 topics

Learn optimizing RAG retrieval performance and accuracy, and model customization such as fine-tuning.

5.1Optimize retrieval-augmented generation (RAG) performance and accuracy

Concept

RAG answer quality depends on whether retrieval surfaces the right information. In Azure AI Search, accuracy improves with hybrid search combining keywords (BM25) and vectors, reranking with the semantic ranker, appropriate chunking and query rewriting.

How it works / steps

  1. Split documents into chunks by units of meaning, with overlap
  2. Create vectors with an embedding model and use hybrid vector-and-keyword search
  3. Rerank top results with the semantic ranker
  4. Tune top-k and relevance thresholds to reduce irrelevant chunks
  5. Measure retrieval evaluation (retrieval precision, recall) and answer evaluation (groundedness) separately

Workplace example

A problem where searching by product model number misses is solved by hybrid search also catching exact keyword matches, and the semantic ranker then reorders the top results.

Common confusions and exam tips

Mini quiz

Q1. In RAG, weakly related documents are mixed into the grounding for answers. Which retrieval-side items should be tuned?

  1. Maximum answer tokens
  2. Similarity threshold and number of results
  3. Tone of the system message
  4. Generation model temperature
Show answer and explanation

Answer: B Similarity threshold and number of results

Raising the threshold or reducing the number of results makes it easier to exclude weakly related documents. [Other options] Temperature, maximum tokens and tone are generation-side settings and do not change which documents retrieval returns.

Q2. In RAG over long policy documents, answers lack the surrounding context they need. What should be tuned?

  1. Chunk size (and overlap)
  2. Only the number of results (top-k)
  3. The generation model’s temperature value
  4. The embedding model’s number of dimensions
Show answer and explanation

Answer: A Chunk size (and overlap)

Chunks that are too small split context. Tune size and overlap while evaluating. [Other options] Retrieving more results does not restore context cut by splitting. Dimensions and temperature do not fix missing context.

Reference links

5.2Implement advanced fine-tuning and model customization

Concept

Fine-tuning further trains a model on your data to adapt it to format, style or specialized tasks. Foundry supports supervised fine-tuning (SFT), DPO (direct preference optimization) learning from preference pairs, reinforcement fine-tuning (RFT) for reasoning models and more. Distillation, training a small model on a large model’s outputs, is also effective for reducing cost.

How it works / steps

  1. First check whether prompts and RAG can meet the requirements
  2. Prepare training data as JSONL (conversation format) and split it into training and validation
  3. Choose the method (SFT, DPO, RFT) and base model suited to the goal
  4. Compare the trained model with the original on evaluation data
  5. Deploy the fine-tuned model, also considering hosting charges

Workplace example

To summarize insurance assessment memos in a fixed format, SFT is run on 500 inputs with ideal summaries, greatly reducing format breakage compared with prompting alone.

Common confusions and exam tips

Mini quiz

Q1. In which case is fine-tuning more suitable than RAG?

  1. When you want to narrow answers according to each user’s access rights
  2. When you want to stabilize output format, tone or behavior on a specific task
  3. When you want to show users citations of the documents that grounded the answer
  4. When you want answers to correctly reflect internal information updated daily
Show answer and explanation

Answer: B When you want to stabilize output format, tone or behavior on a specific task

RAG suits reflecting the latest knowledge; fine-tuning suits adjusting behavior. [Other options] Reflecting the latest information, showing citations and narrowing by permissions suit RAG and are hard to achieve with fine-tuning.

Q2. Which is an appropriate flow for putting a fine-tuned model into production?

  1. Deploy to production as soon as training ends and evaluate based on production usage
  2. Publish the old and new models on separate endpoints and let users choose
  3. Confirm the training loss dropped, replace the old model and switch everything at once
  4. Confirm it meets criteria in evaluation, version it and roll it out gradually
Show answer and explanation

Answer: D Confirm it meets criteria in evaluation, version it and roll it out gradually

Manage fine-tuned models from development to production the same as other models: evaluation, versioning and gradual rollout. [Other options] Training loss alone cannot judge business quality. Deploying before evaluation is risky. Letting users choose is not a gradual rollout.

Reference links

Lessons are written by this site based on public information and do not guarantee exam content. Product specifications change, so check the latest information at the reference links.

Notes (unofficial, original questions, no pass guarantee)