This guide to AI evals was written by Aakash Gupta, based on an episode with Ankit Shukla, published February 19, 2026.
Why AI Evals Matter for PMs
AI evaluations are a critical new skill for all Product Managers because AI functions differently from traditional deterministic code.
AI is Probabilistic
Unlike traditional code that is deterministic (e.g., 2+2 always equals 4), AI outputs are probabilistic and can vary.
Traditional Testing Insufficient
The probabilistic nature of AI means traditional feature testing methods are insufficient for AI features.
Three-Part Eval System
A comprehensive AI evaluation system requires three types of evaluations: offline, online, and human.
Offline Evals
Offline evals are used for testing AI features before they are launched.
Online Evals
Online evals are used for monitoring AI feature performance after launch in production.
Human Evals
Human evals involve spot-checking quality to determine if users actually like the AI feature.
Need All Three Evals
Teams commonly skip online and human evals, but all three types are necessary to prevent feature failure.
PMs Should Own Evals
AI product managers should own evaluations due to their unique position understanding business, customers, and technology outcomes.
PMs Understand Outcomes
Product Managers understand what success looks like, customer needs, and business value metrics.
Industry Leaders Agree
Product leaders like Todd Olson (CEO of Pendo) and Rachel Wolan (CPO of Webflow) agree that AI evals are the most important new skill for PMs.
Cost of Not Doing Evals
Skipping AI evaluations can lead to features failing in production, increased support tickets, and wasted development effort.
Failed Feature Scenario
A prototype works great in demo but fails after launch, causing user complaints, hallucinations, and a feature rollback.
Work Lost
Six months of work can be lost if a feature is rolled back due to lack of proper evaluations.
Fundamental Nature of LLMs
Understanding how Large Language Models (LLMs) fundamentally work is crucial before building effective evaluations.
LLMs Are Probabilistic
LLMs are statistical models that predict the next token based on probability distributions, not by 'knowing' facts.
Capital of France Query
When asked 'What’s the capital of France?', an LLM predicts 'The capital of France is Paris' based on probability, not a lookup.
Different Outputs Explanation
The same prompt can yield different results because the model samples from a probability distribution, leading to varied outputs.
The Temperature Problem
Temperature controls the randomness of LLM outputs, ranging from deterministic (Temperature = 0) to more random (Temperature = 1).
Typical Production Temperature
Most products use a temperature setting between 0.3 and 0.7 for LLMs.
Temperature Impact on Evals
Evaluations must be conducted at the same temperature setting used in production, or they become meaningless.
The Context Window Problem
LLMs have limited context windows, and response quality degrades as the window fills, especially for information in the middle.
GPT-4 Context Window
GPT-4 has a context window of 128K tokens.
Claude Context Window
Claude has a context window of 200K tokens.
Lost in the Middle Problem
Critical information placed in the middle of a long prompt might be missed by the LLM, even within the context window.
Context Length Impact on Evals
Evaluations must test different context lengths, as prompt performance can vary significantly.
The Prompt Sensitivity Problem
Small changes in prompts can lead to large and unpredictable changes in LLM outputs, making AI products fragile.
Prompt Variation Example
Changing 'Please summarize this document' to 'Summarize this document' or adding 'Be concise' yields different results.
Prompt Variation Impact on Evals
Evaluations must test multiple prompt variations that users might actually type, not just one canonical prompt.
The Hallucination Problem
LLMs can hallucinate, making up facts, citing non-existent sources, and inventing details because they predict plausible, not accurate, text.
Verification Required
It is essential to verify that LLM output is factually correct, not just that it 'looks good'.
Build Evaluation Rubric
Building an evaluation rubric is essential to define 'good' quality and measure the performance of AI features.
Start with User Scenarios
Begin rubric development by identifying the top 10 scenarios users will encounter with the AI feature.
Chatbot Scenarios
For a customer support chatbot, scenarios include questions about return policy, shipping times, product specs, account issues, and unclear queries.
Code Generation Tool Scenarios
For a code generation tool, scenarios include requests for simple functions, complex algorithms, refactoring, bug fixes, and tests.
User Scenarios as Test Cases
The identified user scenarios become the test cases for evaluating the AI feature.
Define Success Criteria
For each scenario, define specific, measurable, and unambiguous criteria for what constitutes a successful AI response.
Good vs Bad Criteria
Bad criteria: 'The response is helpful'; Good criteria: 'The response contains the correct return window (30 days) and includes the return portal link'.
Code Success Criteria
Good code success criteria: 'The code passes all test cases, follows project style guide, and includes error handling'.
Build Rubric Categories
A good rubric typically has 4-6 categories, each with a 1-5 scoring scale.
Rubric Categories
Key categories include Correctness, Completeness, Clarity, Tone, Safety, and Efficiency.
Scoring Scale Definition
The 1-5 scale ranges from 'Completely fails' (1) to 'Fully succeeds' (5), with clear definitions for each score.
Create Reference Examples
For each category and score level, create reference examples to establish ground truth for evaluators.
Correctness Example Scores
Examples for 'Correctness' in a support chatbot demonstrate scores of 5 (full success), 3 (partial success), and 1 (factual error).
Reference Examples Purpose
These examples show human or LLM evaluators what specific quality levels look like.
Test Your Rubric
Have 2-3 people independently grade the same 10 outputs using the rubric and calculate inter-rater reliability to refine it.
Reliable Evals Need Good Rubric
A bad rubric leads to bad evals, while a good rubric ensures reliable evaluations.
Evaluation Metrics Framework
Different AI use cases require specific metrics for effective evaluation, as there is no 'one size fits all' approach.
Retrieval Metrics
Metrics for AI systems that retrieve information, such as RAG systems, focus on relevance and surfacing the right documents.
Key Retrieval Metrics
Key metrics include Precision, Recall, F1 Score (harmonic mean), Mean Reciprocal Rank (MRR), and Normalized Discounted Cumulative Gain (NDCG).
Most Important Retrieval Metrics
For most AI Product Manager use cases, F1 and NDCG are the most important retrieval metrics.
Generation Metrics
Metrics for AI systems that generate text, focusing on similarity and overlap with reference texts.
Text Generation Metrics
Metrics include BLEU (translation), ROUGE (summarization), METEOR (synonyms/stemming), and BERTScore (semantic similarity).
Most Robust Generation Metric
BERTScore is recommended as the most robust generation metric for most AI products.
Task-Specific Metrics
Some AI tasks require custom metrics tailored to the specific requirements and desired outcomes of that task.
Code Generation Metrics
Metrics for code generation include compilation success, test passage, style guide adherence, and cyclomatic complexity.
Customer Support Metrics
Metrics for customer support include correct information, required links, brand tone matching, and issue resolution.
Summarization Metrics
Metrics for summarization include capturing key points, omitting irrelevant details, length, and coherence.
LLM-as-Judge Metrics
Using an LLM to grade outputs based on a rubric can correlate better with human judgment than traditional metrics.
LLM Judge Method
Ask an LLM (e.g., GPT-4 or Claude) to score a response on a 1-5 scale for helpfulness or other criteria.
Good Rubric is Key for LLM Judge
The effectiveness of an LLM judge relies on providing it with a well-defined rubric and reference examples.
Choosing the Right Metrics
The choice of metrics depends on the AI's primary function, often requiring a combination of multiple metrics.
Metrics Decision Tree
Use Precision/Recall/F1 for document retrieval, BERTScore for generating text similar to references, task-specific metrics for specific tasks, and LLM-as-judge for holistic quality.
Build LLM Judges Step-by-Step
Implementing LLM judges involves a step-by-step process from defining evaluation prompts to continuous evaluation.
Step 1: Define Evaluation Prompt
The evaluation prompt for an LLM judge needs four components: the rubric, reference examples, input query, and output to evaluate.
Evaluation Prompt Template
A template includes criteria like Correctness, Completeness, Clarity, and Tone with 1-5 scoring, reference examples, user query, and assistant response.
Step 2: Test on Known Examples
Manually test the LLM judge with 10 outputs that have known correct scores, comparing judge scores to ground truth and refining the prompt as needed.
Common Prompt Fixes
Common fixes include adding more reference examples, making criteria more specific, adding chain-of-thought reasoning, or using a better model (e.g., GPT-4 vs GPT-3.5).
Step 3: Implement with Claude Code
Automate the eval pipeline using tools like Claude Code by providing the evaluation prompt and test dataset to generate the necessary script.
Claude Code Prompt
A prompt for Claude Code can request an eval pipeline taking a CSV, running through Claude as a judge, parsing scores, and outputting a summary CSV with averages and flagged low scores.
PM Role in Implementation
The PM's role is defining the rubric, curating test cases, and interpreting results, while implementation is largely automated.
Step 4: Read Results and Find Gaps
Generate a summary dashboard from eval results, focusing on mean scores, distributions, and worst/best performing examples to identify patterns.
Claude Code Dashboard Request
Tell Claude Code to generate a dashboard showing mean score per criterion, score distribution, and top 10 worst/best performers.
Product Sense for Gaps
Product sense is crucial to interpret numbers, determine why failures occurred, and decide on corrective actions.
Step 5: Set Quality Thresholds
Define minimum acceptable scores for each criterion and add them as pass/fail gates in the evaluation pipeline.
Example Thresholds
Example thresholds include Correctness and Completeness ≥4.0 average, and Clarity and Tone ≥3.5 average.
Claude Code Threshold Integration
Instruct Claude Code to add pass/fail checks to the eval pipeline, flagging runs if any criterion average drops below defined thresholds.
Step 6: Run Evals Continuously
Implement continuous evaluation before every release, daily in production, after prompt changes, and after model updates to catch regressions.
Claude Code Continuous Eval Setup
Tell Claude Code to run the eval pipeline nightly against a 1% random sample of production traffic and send Slack notifications for threshold drops.
Common LLM Judge Pitfalls
Several common pitfalls can undermine the effectiveness of LLM judges if not addressed.
Pitfall 1: Same Model for Judge/Product
Using the same model as both the judge and the product can lead to biased evaluations.
Solution: Stronger Judge Model
Use a stronger model as the judge (e.g., GPT-4 to judge GPT-3.5 outputs).
Pitfall 2: Not Calibrating Judge
Failing to calibrate the LLM judge regularly can result in unreliable scores.
Solution: Calibrate Regularly
Regularly compare judge scores to human scores and adjust prompts to ensure accuracy.
Pitfall 3: Too Many Dimensions
Judging too many dimensions at once can overwhelm the LLM judge and reduce accuracy.
Solution: Break Down Rubrics
Break complex rubrics into multiple judge calls to evaluate fewer dimensions per call.
Pitfall 4: Non-Zero Temperature
Using a temperature greater than 0 for LLM judges introduces randomness, making scores inconsistent.
Solution: Use Temperature=0
Always use temperature=0 for LLM evaluations to ensure deterministic and consistent scores.
Production Monitoring That Works
Effective AI evaluation extends beyond launch with robust production monitoring to ensure ongoing performance and value.
Three Layers of Monitoring
Production monitoring involves three layers: System Metrics, Quality Metrics, and Business Metrics.
Layer 1: System Metrics
These are basic health metrics indicating infrastructure issues.
Examples of System Metrics
System metrics include Latency (p50, p95, p99), Error rate, Token usage, API costs, and Timeout rate.
Layer 2: Quality Metrics
These measure the performance and quality of the AI's output.
Examples of Quality Metrics
Quality metrics include average LLM judge scores, human feedback scores (thumbs up/down), task success rate, and hallucination rate.
Layer 3: Business Metrics
These measure if the AI is delivering value and achieving business objectives.
Examples of Business Metrics
Business metrics include feature adoption rate, user retention, customer satisfaction (CSAT/NPS), support ticket deflection, and revenue impact.
All Layers Are Critical
All three layers are necessary; ignoring any layer leaves gaps in understanding AI performance and value.
Setting Up Automatic Alerts
Establish alerts for critical performance deviations to enable immediate investigation.
Alert Criteria Examples
Alerts should be set for quality scores dropping below threshold, error rates spiking above 1%, p95 latency exceeding 3 seconds, or hallucination rate exceeding 5%.
The Human Review Queue
Sample 1% of production traffic daily for human review of 10-20 real user interactions using the rubric.
Human Review Catches Misses
Human review catches issues LLM judges miss and keeps the team connected to real user experiences.
Recalibrate Judge if Divergent
If human scores diverge from LLM judge scores, recalibrate the LLM judge.
The Feedback Loop
Production monitoring findings should feed back into the evaluation dataset to continuously improve evals.
Feedback Loop Steps
Add bad production outputs to the test dataset, label them with correct scores, and rerun evals to ensure the system catches them.
Virtuous Cycle of Evals
This feedback loop creates a virtuous cycle, continuously improving the evaluation system over time.
When to Rollback
Define clear rollback criteria beforehand to quickly reverse a feature if it fails in production.
Rollback Criteria Examples
Criteria include quality score drop >10% from baseline, error rate exceeding 5%, more than 3 critical bugs in 24 hours, or negative business impact.
Final Words: Key Actions
AI evals combine product sense, technical understanding, and statistical thinking, forming a new essential PM skill.
Build Eval Rubric
Build a comprehensive evaluation rubric specifically for your product.
Implement Right Metrics
Implement appropriate metrics, including retrieval, generation, task-specific, and LLM-as-judge metrics.
Create LLM Judge
Create an LLM judge to automate the evaluation process.
Set Quality Thresholds
Establish clear quality thresholds before launching any AI feature.
Monitor Quality Continuously
Monitor AI quality continuously once it is in production.
Feedback Production Learnings
Integrate learnings from production back into your evaluation dataset for ongoing improvement.
Do Not Ship Without Evals
It is crucial never to ship an AI product without a robust evaluation system in place.
Main idea
AI Evals Explained Simply
Ankit Shukla provides an intuitive walkthrough of AI evaluations, highlighting their importance for Product Managers.
This guide to AI evals was written by Aakash Gupta, based on an episode with Ankit Shukla, published February 19, 2026.
Concept
Why AI Evals Matter for PMs
AI evaluations are a critical new skill for all Product Managers because AI functions differently from traditional deterministic code.
Reason
AI is Probabilistic
Unlike traditional code that is deterministic (e.g., 2+2 always equals 4), AI outputs are probabilistic and can vary.
Insight
Traditional Testing Insufficient
The probabilistic nature of AI means traditional feature testing methods are insufficient for AI features.
Concept
Three-Part Eval System
A comprehensive AI evaluation system requires three types of evaluations: offline, online, and human.
Sub-concept
Offline Evals
Offline evals are used for testing AI features before they are launched.
Sub-concept
Online Evals
Online evals are used for monitoring AI feature performance after launch in production.
Sub-concept
Human Evals
Human evals involve spot-checking quality to determine if users actually like the AI feature.
Insight
Need All Three Evals
Teams commonly skip online and human evals, but all three types are necessary to prevent feature failure.
Concept
PMs Should Own Evals
AI product managers should own evaluations due to their unique position understanding business, customers, and technology outcomes.
Reason
PMs Understand Outcomes
Product Managers understand what success looks like, customer needs, and business value metrics.
Detail
Industry Leaders Agree
Product leaders like Todd Olson (CEO of Pendo) and Rachel Wolan (CPO of Webflow) agree that AI evals are the most important new skill for PMs.
Concept
Cost of Not Doing Evals
Skipping AI evaluations can lead to features failing in production, increased support tickets, and wasted development effort.
Example
Failed Feature Scenario
A prototype works great in demo but fails after launch, causing user complaints, hallucinations, and a feature rollback.
Data
Work Lost
Six months of work can be lost if a feature is rolled back due to lack of proper evaluations.
Concept
Fundamental Nature of LLMs
Understanding how Large Language Models (LLMs) fundamentally work is crucial before building effective evaluations.
Sub-concept
LLMs Are Probabilistic
LLMs are statistical models that predict the next token based on probability distributions, not by 'knowing' facts.
Example
Capital of France Query
When asked 'What’s the capital of France?', an LLM predicts 'The capital of France is Paris' based on probability, not a lookup.
Reason
Different Outputs Explanation
The same prompt can yield different results because the model samples from a probability distribution, leading to varied outputs.
Sub-concept
The Temperature Problem
Temperature controls the randomness of LLM outputs, ranging from deterministic (Temperature = 0) to more random (Temperature = 1).
Detail
Typical Production Temperature
Most products use a temperature setting between 0.3 and 0.7 for LLMs.
Insight
Temperature Impact on Evals
Evaluations must be conducted at the same temperature setting used in production, or they become meaningless.
Sub-concept
The Context Window Problem
LLMs have limited context windows, and response quality degrades as the window fills, especially for information in the middle.
Data
GPT-4 Context Window
GPT-4 has a context window of 128K tokens.
Data
Claude Context Window
Claude has a context window of 200K tokens.
Detail
Lost in the Middle Problem
Critical information placed in the middle of a long prompt might be missed by the LLM, even within the context window.
Insight
Context Length Impact on Evals
Evaluations must test different context lengths, as prompt performance can vary significantly.
Sub-concept
The Prompt Sensitivity Problem
Small changes in prompts can lead to large and unpredictable changes in LLM outputs, making AI products fragile.
Example
Prompt Variation Example
Changing 'Please summarize this document' to 'Summarize this document' or adding 'Be concise' yields different results.
Insight
Prompt Variation Impact on Evals
Evaluations must test multiple prompt variations that users might actually type, not just one canonical prompt.
Sub-concept
The Hallucination Problem
LLMs can hallucinate, making up facts, citing non-existent sources, and inventing details because they predict plausible, not accurate, text.
Insight
Verification Required
It is essential to verify that LLM output is factually correct, not just that it 'looks good'.
Concept
Build Evaluation Rubric
Building an evaluation rubric is essential to define 'good' quality and measure the performance of AI features.
Detail
Start with User Scenarios
Begin rubric development by identifying the top 10 scenarios users will encounter with the AI feature.
Example
Chatbot Scenarios
For a customer support chatbot, scenarios include questions about return policy, shipping times, product specs, account issues, and unclear queries.
Example
Code Generation Tool Scenarios
For a code generation tool, scenarios include requests for simple functions, complex algorithms, refactoring, bug fixes, and tests.
Insight
User Scenarios as Test Cases
The identified user scenarios become the test cases for evaluating the AI feature.
Detail
Define Success Criteria
For each scenario, define specific, measurable, and unambiguous criteria for what constitutes a successful AI response.
Example
Good vs Bad Criteria
Bad criteria: 'The response is helpful'; Good criteria: 'The response contains the correct return window (30 days) and includes the return portal link'.
Example
Code Success Criteria
Good code success criteria: 'The code passes all test cases, follows project style guide, and includes error handling'.
Detail
Build Rubric Categories
A good rubric typically has 4-6 categories, each with a 1-5 scoring scale.
Detail
Rubric Categories
Key categories include Correctness, Completeness, Clarity, Tone, Safety, and Efficiency.
Detail
Scoring Scale Definition
The 1-5 scale ranges from 'Completely fails' (1) to 'Fully succeeds' (5), with clear definitions for each score.
Detail
Create Reference Examples
For each category and score level, create reference examples to establish ground truth for evaluators.
Example
Correctness Example Scores
Examples for 'Correctness' in a support chatbot demonstrate scores of 5 (full success), 3 (partial success), and 1 (factual error).
Reason
Reference Examples Purpose
These examples show human or LLM evaluators what specific quality levels look like.
Detail
Test Your Rubric
Have 2-3 people independently grade the same 10 outputs using the rubric and calculate inter-rater reliability to refine it.
Reason
Reliable Evals Need Good Rubric
A bad rubric leads to bad evals, while a good rubric ensures reliable evaluations.
Concept
Evaluation Metrics Framework
Different AI use cases require specific metrics for effective evaluation, as there is no 'one size fits all' approach.
Sub-concept
Retrieval Metrics
Metrics for AI systems that retrieve information, such as RAG systems, focus on relevance and surfacing the right documents.
Detail
Key Retrieval Metrics
Key metrics include Precision, Recall, F1 Score (harmonic mean), Mean Reciprocal Rank (MRR), and Normalized Discounted Cumulative Gain (NDCG).
Insight
Most Important Retrieval Metrics
For most AI Product Manager use cases, F1 and NDCG are the most important retrieval metrics.
Sub-concept
Generation Metrics
Metrics for AI systems that generate text, focusing on similarity and overlap with reference texts.
Detail
Text Generation Metrics
Metrics include BLEU (translation), ROUGE (summarization), METEOR (synonyms/stemming), and BERTScore (semantic similarity).
Insight
Most Robust Generation Metric
BERTScore is recommended as the most robust generation metric for most AI products.
Sub-concept
Task-Specific Metrics
Some AI tasks require custom metrics tailored to the specific requirements and desired outcomes of that task.
Example
Code Generation Metrics
Metrics for code generation include compilation success, test passage, style guide adherence, and cyclomatic complexity.
Example
Customer Support Metrics
Metrics for customer support include correct information, required links, brand tone matching, and issue resolution.
Example
Summarization Metrics
Metrics for summarization include capturing key points, omitting irrelevant details, length, and coherence.
Sub-concept
LLM-as-Judge Metrics
Using an LLM to grade outputs based on a rubric can correlate better with human judgment than traditional metrics.
Detail
LLM Judge Method
Ask an LLM (e.g., GPT-4 or Claude) to score a response on a 1-5 scale for helpfulness or other criteria.
Reason
Good Rubric is Key for LLM Judge
The effectiveness of an LLM judge relies on providing it with a well-defined rubric and reference examples.
Recommendation
Choosing the Right Metrics
The choice of metrics depends on the AI's primary function, often requiring a combination of multiple metrics.
Detail
Metrics Decision Tree
Use Precision/Recall/F1 for document retrieval, BERTScore for generating text similar to references, task-specific metrics for specific tasks, and LLM-as-judge for holistic quality.
Concept
Build LLM Judges Step-by-Step
Implementing LLM judges involves a step-by-step process from defining evaluation prompts to continuous evaluation.
Detail
Step 1: Define Evaluation Prompt
The evaluation prompt for an LLM judge needs four components: the rubric, reference examples, input query, and output to evaluate.
Example
Evaluation Prompt Template
A template includes criteria like Correctness, Completeness, Clarity, and Tone with 1-5 scoring, reference examples, user query, and assistant response.
Detail
Step 2: Test on Known Examples
Manually test the LLM judge with 10 outputs that have known correct scores, comparing judge scores to ground truth and refining the prompt as needed.
Detail
Common Prompt Fixes
Common fixes include adding more reference examples, making criteria more specific, adding chain-of-thought reasoning, or using a better model (e.g., GPT-4 vs GPT-3.5).
Detail
Step 3: Implement with Claude Code
Automate the eval pipeline using tools like Claude Code by providing the evaluation prompt and test dataset to generate the necessary script.
Example
Claude Code Prompt
A prompt for Claude Code can request an eval pipeline taking a CSV, running through Claude as a judge, parsing scores, and outputting a summary CSV with averages and flagged low scores.
Insight
PM Role in Implementation
The PM's role is defining the rubric, curating test cases, and interpreting results, while implementation is largely automated.
Detail
Step 4: Read Results and Find Gaps
Generate a summary dashboard from eval results, focusing on mean scores, distributions, and worst/best performing examples to identify patterns.
Example
Claude Code Dashboard Request
Tell Claude Code to generate a dashboard showing mean score per criterion, score distribution, and top 10 worst/best performers.
Insight
Product Sense for Gaps
Product sense is crucial to interpret numbers, determine why failures occurred, and decide on corrective actions.
Recommendation
Step 5: Set Quality Thresholds
Define minimum acceptable scores for each criterion and add them as pass/fail gates in the evaluation pipeline.
Example
Example Thresholds
Example thresholds include Correctness and Completeness ≥4.0 average, and Clarity and Tone ≥3.5 average.
Example
Claude Code Threshold Integration
Instruct Claude Code to add pass/fail checks to the eval pipeline, flagging runs if any criterion average drops below defined thresholds.
Detail
Step 6: Run Evals Continuously
Implement continuous evaluation before every release, daily in production, after prompt changes, and after model updates to catch regressions.
Example
Claude Code Continuous Eval Setup
Tell Claude Code to run the eval pipeline nightly against a 1% random sample of production traffic and send Slack notifications for threshold drops.
Concept
Common LLM Judge Pitfalls
Several common pitfalls can undermine the effectiveness of LLM judges if not addressed.
Detail
Pitfall 1: Same Model for Judge/Product
Using the same model as both the judge and the product can lead to biased evaluations.
Recommendation
Solution: Stronger Judge Model
Use a stronger model as the judge (e.g., GPT-4 to judge GPT-3.5 outputs).
Detail
Pitfall 2: Not Calibrating Judge
Failing to calibrate the LLM judge regularly can result in unreliable scores.
Recommendation
Solution: Calibrate Regularly
Regularly compare judge scores to human scores and adjust prompts to ensure accuracy.
Detail
Pitfall 3: Too Many Dimensions
Judging too many dimensions at once can overwhelm the LLM judge and reduce accuracy.
Recommendation
Solution: Break Down Rubrics
Break complex rubrics into multiple judge calls to evaluate fewer dimensions per call.
Detail
Pitfall 4: Non-Zero Temperature
Using a temperature greater than 0 for LLM judges introduces randomness, making scores inconsistent.
Recommendation
Solution: Use Temperature=0
Always use temperature=0 for LLM evaluations to ensure deterministic and consistent scores.
Concept
Production Monitoring That Works
Effective AI evaluation extends beyond launch with robust production monitoring to ensure ongoing performance and value.
Sub-concept
Three Layers of Monitoring
Production monitoring involves three layers: System Metrics, Quality Metrics, and Business Metrics.
Detail
Layer 1: System Metrics
These are basic health metrics indicating infrastructure issues.
Detail
Examples of System Metrics
System metrics include Latency (p50, p95, p99), Error rate, Token usage, API costs, and Timeout rate.
Detail
Layer 2: Quality Metrics
These measure the performance and quality of the AI's output.
Detail
Examples of Quality Metrics
Quality metrics include average LLM judge scores, human feedback scores (thumbs up/down), task success rate, and hallucination rate.
Detail
Layer 3: Business Metrics
These measure if the AI is delivering value and achieving business objectives.
Detail
Examples of Business Metrics
Business metrics include feature adoption rate, user retention, customer satisfaction (CSAT/NPS), support ticket deflection, and revenue impact.
Insight
All Layers Are Critical
All three layers are necessary; ignoring any layer leaves gaps in understanding AI performance and value.
Detail
Setting Up Automatic Alerts
Establish alerts for critical performance deviations to enable immediate investigation.
Example
Alert Criteria Examples
Alerts should be set for quality scores dropping below threshold, error rates spiking above 1%, p95 latency exceeding 3 seconds, or hallucination rate exceeding 5%.
Detail
The Human Review Queue
Sample 1% of production traffic daily for human review of 10-20 real user interactions using the rubric.
Reason
Human Review Catches Misses
Human review catches issues LLM judges miss and keeps the team connected to real user experiences.
Recommendation
Recalibrate Judge if Divergent
If human scores diverge from LLM judge scores, recalibrate the LLM judge.
Detail
The Feedback Loop
Production monitoring findings should feed back into the evaluation dataset to continuously improve evals.
Detail
Feedback Loop Steps
Add bad production outputs to the test dataset, label them with correct scores, and rerun evals to ensure the system catches them.
Insight
Virtuous Cycle of Evals
This feedback loop creates a virtuous cycle, continuously improving the evaluation system over time.
Recommendation
When to Rollback
Define clear rollback criteria beforehand to quickly reverse a feature if it fails in production.
Example
Rollback Criteria Examples
Criteria include quality score drop >10% from baseline, error rate exceeding 5%, more than 3 critical bugs in 24 hours, or negative business impact.
Concept
Final Words: Key Actions
AI evals combine product sense, technical understanding, and statistical thinking, forming a new essential PM skill.
Recommendation
Build Eval Rubric
Build a comprehensive evaluation rubric specifically for your product.
Recommendation
Implement Right Metrics
Implement appropriate metrics, including retrieval, generation, task-specific, and LLM-as-judge metrics.
Recommendation
Create LLM Judge
Create an LLM judge to automate the evaluation process.
Recommendation
Set Quality Thresholds
Establish clear quality thresholds before launching any AI feature.
Recommendation
Monitor Quality Continuously
Monitor AI quality continuously once it is in production.
Recommendation
Feedback Production Learnings
Integrate learnings from production back into your evaluation dataset for ongoing improvement.
Insight
Do Not Ship Without Evals
It is crucial never to ship an AI product without a robust evaluation system in place.
Pressure-test your own plan before you defend it — free
Describe a plan, proposal or decision. Five experts with conflicting stakes argue it — you get a verdict, the risks ranked and the questions you’ll be asked.
Describe a plan, proposal or decision. Five experts with conflicting stakes argue it — you get a verdict, the risks ranked and the questions you’ll be asked. Free · 5 pressure tests/month.