A GenAI application that runs is not the same as one you can trust. AIP-C01 Domain 5 (Testing, Validation, and Troubleshooting, 11%) covers how to close that gap. Lexical metrics like ROUGE and BLEU aren't enough: you need RAG evaluation, LLM-as-a-judge, agent evaluation, and a clear method for finding out where a bad answer came from.
How GenAI Evaluation Differs
A generated answer has no single correct form, so the exam guide asks for metrics such as relevance, factual accuracy, consistency, and fluency. Combine three methods:
| Method | Strength | Limitation | |--------|----------|------------| | Automatic metrics (ROUGE, BERTScore, accuracy) | Fast, cheap, repeatable | Miss meaning and factuality | | LLM-as-a-judge | Meaning-level scoring at scale, with explanations | Judge bias, cost | | Human evaluation | Nuance and business fit | Slow, expensive, limited scale |
Amazon Bedrock Evaluations
Upload a JSONL prompt dataset (optionally with ground truth) to S3 and create a job; results appear in the console and as an S3 report.
| Evaluation type | Representative metrics | |-----------------|------------------------| | Model, automatic | Accuracy, Robustness, Toxicity | | Model, LLM-as-a-judge | Correctness, Completeness, Faithfulness, Helpfulness, Relevance, Following instructions, Harmfulness, plus custom metrics | | Model, human | Ratings and preferences from your own work team | | RAG, retrieve only | Context relevance, Context coverage | | RAG, retrieve and generate | Correctness, Completeness, Faithfulness, Citation precision, Citation coverage, and more |
Know the definitions. Faithfulness checks whether the answer contains information that isn't in the provided context, which is the core hallucination measure. Context relevance checks whether retrieved chunks relate to the question. Context coverage checks whether they contain everything the ground truth needs, so it requires ground truth. Citation precision and coverage check whether citations are correct and whether they support the claims. You can also evaluate external models or your own RAG system by bringing your own inference responses.
Using LLM-as-a-Judge Well
A judge job uses a generator model and a separate evaluator model that returns a score and an explanation per metric. Add custom metrics for business rules such as brand tone or required disclaimers. Judges can favor similar models or longer answers, so use a different model as the judge and calibrate it against a human-scored sample. The same approach automates fairness checks: compare answer quality across prompt pairs that differ only in gender or region.
RAG Diagnosis: Retrieval or Generation?
First, look at what the model actually received: run the API on its own or inspect the citations returned by .
If the needed information is missing, fix retrieval:
Tune chunk size and strategy (fixed, hierarchical, semantic). Switch the parser to a foundation model parser or Bedrock Data Automation when tables come out broken. Add metadata and filter by department or version. Turn on hybrid search for product codes and proper nouns. Retrieve more results and rerank them. Re-embed every document after changing the embedding model, because old and new vectors aren't compatible.
If the information is there but the answer is wrong, fix generation: instruct the model to answer only from the provided documents and cite them, cut or summarize context that is too long or truncated, and use the Guardrails contextual grounding check in production.
!Is it a retrieval or a generation problem?
Agent Evaluation
Measure task completion rate, tool selection and parameter accuracy, reasoning quality across steps, and steps, latency, and tokens per task. Run test scenarios with expected tool-call sequences and compare them with the trace from . Fix wrong tool choices with better tool descriptions, loops with an iteration limit, and parameter errors with schema validation in Lambda.
Quality Gates Before and After Deployment
Golden dataset: fixed questions, expected answers, and source documents, rerun on every prompt, model, or chunking change. CI/CD gate: CodePipeline runs a Bedrock evaluation job and stops the release when faithfulness or correctness drops below the threshold. Canary and A/B tests: route part of the traffic to the new version, compare ratings and business metrics, and roll back through Prompt Management versions or AppConfig. Continuous evaluation: sample invocation logs, score them with a judge, and alarm on the CloudWatch metric.
Common Errors
| Symptom | Fix | |---------|-----| | ValidationException, input too long | Count tokens first, send fewer chunks, summarize history | | Answer cut off ( is ) | Raise or shrink the output format | | JSON parsing fails | Schema validation, tool use for structured output | | Odd answers for one question type | Search prompts and responses with CloudWatch Logs Insights | | Unknown slow segment | X-Ray across Lambda, Knowledge Base, and Bedrock |
Exam Quick Reference
"Did the answer invent content not in the retrieved documents?" -- Faithfulness "Are retrieved chunks related to the question?" -- Context relevance "Do retrieved chunks cover everything needed?" -- Context coverage (needs ground truth) "Are the citations correct?" -- Citation precision / Citation coverage "Meaning-level scoring at scale" -- Bedrock Evaluations with LLM-as-a-judge "Score against business-specific criteria" -- Custom metrics "Evaluate an external RAG system" -- Bring your own inference responses "Needed information isn't retrieved" -- Chunking, metadata filters, hybrid search, reranking "Retrieval degraded after changing the embedding model" -- Re-embed all documents "Agent calls the wrong tool" -- Check the trace, improve tool descriptions "Prevent regressions after prompt changes" -- Golden dataset + CI/CD quality gate