AIAI Agent Dev Labaiagentdev-lab.com
Sign in to save progressSign in
Menu

AIF-C01 · LESSON 3

Applications of Foundation Models28% · 4 topics

Learn application design with foundation models, prompt engineering, training and fine-tuning, and performance evaluation.

3.1Design considerations for applications that use foundation models

3.1 Describe design considerations for applications that use foundation models (FMs).

Concept

Choose models by cost, latency, modality, supported languages, context length and customizability. Inference parameters temperature and top-p control output randomness, and maximum tokens controls output length. To use internal knowledge, use RAG (Bedrock Knowledge Bases and vector stores: OpenSearch Serverless, Aurora, S3 Vectors and others); for multi-step work, use Bedrock Agents.

How it works / steps

  1. Choose a model from use-case requirements (accuracy, speed, cost, language)
  2. Decide inference parameters such as temperature for the use case
  3. To answer from internal data, use RAG (Knowledge Bases)
  4. If multiple steps such as API calls are needed, use agents
  5. Consider customization methods in order of increasing cost (prompting → RAG → fine-tuning → pre-training)

Workplace example

For internal policy Q&A, put the policies in S3, ingest them into a Bedrock knowledge base, set a low temperature, and have it answer with citations.

Common confusions and exam tips

Mini quiz

Q1. Which is the most appropriate description of RAG (retrieval-augmented generation)?

  1. A method for registering model answers in a search engine so people outside the company can search them
  2. A method for sending the same question to the model repeatedly and choosing the most frequent answer as correct
  3. A method for retraining a foundation model's parameters from scratch using internal documents
  4. A method that retrieves documents relevant to the question, adds them to the prompt, and has the model answer based on their content
Show answer and explanation

Answer: D A method that retrieves documents relevant to the question, adds them to the prompt, and has the model answer based on their content

RAG retrieves external knowledge and adds it to the prompt so the model can answer based on up-to-date or internal information it was not trained on.

Q2. Internal procedure manuals are updated frequently. Which is the most appropriate advantage of RAG compared with fine-tuning?

  1. Updates to documents apply without retraining, and source documents can be cited
  2. No search is needed, so latency is lower than fine-tuning
  3. The model learns the manuals' style, unifying answer tone
  4. Input tokens drop, lowering cost per call
Show answer and explanation

Answer: A Updates to documents apply without retraining, and source documents can be cited

RAG requires no retraining and handles information updates well. Because it can show its sources, answers are easier to verify. It reduces hallucinations but does not eliminate them. [Other options] RAG performs search and may add latency. Unifying style is a fine-tuning benefit. Retrieved results increase input tokens.

Reference links

3.2Effective prompt engineering

3.2 Choose effective prompt engineering techniques.

Concept

A prompt consists of instructions, context, input data and output format. Techniques include zero-shot (no examples), few-shot (show a few examples), chain-of-thought (have the model think step by step) and prompt templates. Negative prompts (what you do not want) can also be used. Risks include prompt injection, jailbreaks and prompt leaking.

How it works / steps

  1. Write the role and goal clearly
  2. Specify the output format (bullet points, JSON)
  3. Add few-shot examples for difficult tasks
  4. For problems that require reasoning, instruct it to think step by step
  5. Prevent adversarial prompts with guardrails

Workplace example

For inquiry classification, use few-shot by putting three examples (inquiry and category) in the prompt to improve classification accuracy.

Common confusions and exam tips

Mini quiz

Q1. A chat app sends the model the same long system prompt (internal rules and examples) every time. Which feature is the most suitable for lowering cost and response time?

  1. Send the system prompt twice each time so the rules are followed more strongly
  2. Send to a different model each time and use only the fastest answer
  3. Use prompt caching to reuse the identical leading portion sent repeatedly
  4. Set temperature to the maximum so the model finishes writing answers faster
Show answer and explanation

Answer: C Use prompt caching to reuse the identical leading portion sent repeatedly

Bedrock prompt caching caches the leading portion of prompts used repeatedly, reducing input processing cost and response time.

Q2. Which statement about how RAG works is not correct?

  1. Documents are split and stored in a vector store as embeddings
  2. With RAG, the model's parameters are updated every time with the retrieved internal documents
  3. The retrieved fragments are added to the prompt and the model generates the answer
  4. The user's question is also converted into an embedding, and semantically similar document fragments are retrieved from the vector store
Show answer and explanation

Answer: B With RAG, the model's parameters are updated every time with the retrieved internal documents

RAG does not change the model's parameters. It just adds retrieved information to the prompt.

Reference links

3.3Training and fine-tuning foundation models

3.3 Describe the training and fine-tuning process for FMs.

Concept

Elements of foundation model training are pre-training (learning general capabilities from large amounts of data), fine-tuning (adapting to specific tasks with labeled data) and continued pre-training (expanding knowledge with unlabeled domain data). There is also instruction tuning and reinforcement learning from human feedback (RLHF). In Bedrock, you can create custom models with fine-tuning, continued pre-training and distillation.

How it works / steps

  1. First check whether prompting or RAG can meet the requirements
  2. For fine-tuning, prepare input and output examples for the target task
  3. To expand knowledge in a domain with lots of jargon, consider continued pre-training
  4. Check data quality, quantity and bias
  5. Using a custom model may require Provisioned Throughput

Workplace example

To strengthen handling of medical terminology, run continued pre-training on unlabeled medical documents, then fine-tune on a summarization task.

Common confusions and exam tips

Mini quiz

Q1. In a RAG app, the correct answer is in the documents but often missing from answers. Looking at the retrieval results, long fragments with little relevance are returned. What is the most appropriate next step?

  1. Set temperature to the maximum so the model can recall more information
  2. Delete all documents and switch the model to answering with general knowledge only
  3. Revisit the chunk size and chunking method and verify with test questions whether retrieval results improved
  4. Pre-train the foundation model from scratch again to make it memorize all document contents
Show answer and explanation

Answer: C Revisit the chunk size and chunking method and verify with test questions whether retrieval results improved

RAG quality depends on retrieval quality. Adjust chunk size, overlap and splitting method, then evaluate and improve retrieval results.

Q2. Which is the most appropriate benefit of creating prompt versions in Amazon Bedrock Prompt Management?

  1. Once a version is created, that prompt incurs no charges at all
  2. Each time a version is created, the foundation model is automatically fine-tuned
  3. Creating a version publishes the prompt so anyone can use it
  4. Apps can reference a specific version, and you can roll back to a previous version if there is a problem
Show answer and explanation

Answer: D Apps can reference a specific version, and you can roll back to a previous version if there is a problem

Prompt Management saves prompt changes as versions, so you can pin the version an app uses or roll back.

Reference links

3.4Evaluating foundation model performance

3.4 Describe methods to evaluate FM performance.

Concept

Evaluation includes human evaluation, benchmark datasets, automatic metrics (ROUGE: summarization, BLEU: translation, BERTScore: semantic similarity) and using an LLM as a judge. Bedrock model evaluation provides automatic evaluation, human evaluation, LLM-based evaluation and RAG evaluation (knowledge base retrieval and generation). Ultimately, judge by whether it meets business goals (productivity, satisfaction).

How it works / steps

  1. Decide the purpose of evaluation (quality, safety, cost, speed)
  2. Choose metrics that fit the use case (ROUGE for summarization, BLEU for translation, etc.)
  3. Compare multiple models on a representative prompt dataset
  4. Supplement subjective quality with human or LLM-based evaluation
  5. Confirm impact with business KPIs

Workplace example

Compare three models for a summarization feature with Bedrock model evaluation, and choose the lowest-cost model that is best on both ROUGE and human ratings.

Common confusions and exam tips

Mini quiz

Q1. After revising a prompt and pushing it to production, reports of lower answer quality increased. What is the most appropriate next step?

  1. Delete the prompt change history and pretend it was rewritten from scratch
  2. Roll back to the previous prompt version, and redeploy the revision only after evaluating it with test questions
  3. Without investigating the cause, add even more instructions to the prompt and see what happens
  4. Hide the quality drop from users and wait until reports decrease
Show answer and explanation

Answer: B Roll back to the previous prompt version, and redeploy the revision only after evaluating it with test questions

First, roll back to the previous version to stop the impact. Deploy improvement ideas only after comparing them with evaluation questions.

Q2. When evaluating a RAG app, which is the most appropriate thing to check for retrieval and for generation respectively?

  1. Check only the number of documents stored in the vector store and not the retrieval results
  2. Check only the number of users and leave judging retrieval and answer quality to users
  3. Check only whether answers are long enough and not whether the content is correct
  4. Check whether retrieved documents are relevant to the question, and whether the answer is faithful to the retrieved content and correct
Show answer and explanation

Answer: D Check whether retrieved documents are relevant to the question, and whether the answer is faithful to the retrieved content and correct

In RAG, evaluate retrieval quality (relevance) and generation quality (faithfulness, correctness) separately. Bedrock can also run RAG evaluations.

Reference links

Lessons are written by this site based on public information and do not guarantee exam content. Product specifications change, so check the latest information at the reference links.

Notes (unofficial, original questions, no pass guarantee)