Estimated reading time: 9 minutes

Before you share your LLM application with the world, you need to make sure that the system is capable of high-quality outputs.  

Moving from a proof of concept to production deployment of an LLM application requires finding a reliable way to evaluate its performance. In doing so, teams can make informed decisions on deployments and iterations. 

When making a decision on deploying an LLM application, the three key dimensions to consider are cost, latency, and quality. API or GPU providers will determine cost, and latency can be measured by running tests for your chosen infrastructure. Quality, however, has more subjective measurements, creating challenges for businesses. Finding the right evaluation process for measuring quality is, therefore, incredibly important.   

What is LLM application evaluation?

LLM evaluation is the systematic process of measuring how well an LLM performs its intended tasks, such as answering questions or summarizing text. 

Evaluation occurs both before and after deployment. 

Before deployment, the primary goal is to assess the system’s readiness and refine aspects to maximize the product’s functionality. Teams should use an evaluation dataset (a curated set of inputs paired with ideal outputs) to objectively measure the model’s readiness for production. The evaluation dataset is like an exam given to the model before it is put in production.

After deployment, evaluation is continued through capturing user feedback, selective human review, and automated monitoring, capturing real-world performance, and guiding ongoing improvements.

Why is LLM application evaluation important?

Designing evaluation datasets forces you to define what “good” looks like because it outlines clear, concrete examples of high-quality outputs to compare the model’s responses against. This requires teams to articulate success criteria in a structured, consistent way, which forms the foundation for meaningful LLM evaluation.

Evaluations shed light on whether your product/feature is ready, but they also ensure that all iterations improve the product while reducing the risk of making spot improvements to visible areas and letting less visible ones fall behind. 

LLM applications aren’t static. In other words, once you release a model into the ether, you will undoubtedly iterate on it when new technologies or techniques emerge. When this happens, and it’s time to add a new element to the system, a well-defined evaluation process allows you to quickly test how well suited it is for your application. By making informed data-driven decisions about adoption, you can better justify any associated investment or integration effort.

When done correctly, a well-designed evaluation future-proofs the entire system by generating useful artifacts such as a dataset of preferred responses that can directly improve your model. 

Steps for setting up an evaluation process

1. Clearly define the purpose of the LLM system

Before making any moves to build an LLM system, you must first clearly outline its purpose in writing. The document must be kept in a place that is visible to all engineers in the team. It should also be easily comprehensible to all. 

The outline should clearly define: 

1. The task that the LLM will be completing. 
2. The intended audience of the product/system. 
3. The form that the LLM will take. 
4. The content focus of the output. 

Example outline: The LLM system we’re building will be used for summarizing research papers for our lay audience. Each summary will be 2-3 sentences, communicating the most important insight from the research paper. 

2. Do a vibe check

A vibe check is often the first informal way teams assess whether an LLM system “feels” like it’s working. It typically involves asking the system a few example queries and verifying that the responses appear reasonable. Because it’s fast and instinct-driven, many treat it as a simple pass/fail gate; either the system looks good enough to move forward, or it doesn’t. But this limited approach is not enough.

Instead, treat the vibe check as a chance to collect valuable qualitative insights. Document what works, what doesn’t, examples that delight, and examples that fall flat. Capture both likes and dislikes; this early feedback becomes the foundation for building more structured evaluation datasets later on.

For example, say you’re part of a team that is helping build an enterprise search and answers system where a senior stakeholder frequently complains about poor calendar-related search results. The engineering team prioritizes improving this aspect through prompt tuning and by biasing retrieval. 

Later, you discover that most users actually care more about informational queries, and the quality of these responses has been inadvertently degraded by targeted optimizations elsewhere. 

Overoptimizing in one small area isn’t a rare occurrence, but it can be staved off with a more structured early evaluation approach that reveals broader user needs. In doing so, you help the team avoid misdirected effort.

3. Set up a golden dataset

A golden dataset is essentially a small set of curated examples that demonstrate what the best LLM outputs look like. It serves as a “single source of truth” for all teams and can help optimize decision-making, consistency, and efficiency. 

Best practices for a golden dataset

Make it representative: Include a small but diverse set of real and edge-case examples that reflect actual user needs.

Use multi-stakeholder review: Have domain experts and target users validate each example and “gold” output, and document why it’s good.

Keep it stable but versioned: Keep the dataset consistent for evaluations. Only update it after formal review when user needs evolve.

Upon completing it, use the opportunity to share this information with stakeholders. Letting key players contribute and sign off on the golden dataset does wonders for creating alignment.  

The golden dataset can look something like this:  

When building an LLM system to summarize research papers for non-experts, as shown above, create your golden dataset from dual perspectives: an expert, who can verify accuracy, and a non-expert from your target audience to assess comprehensibility. 

There is no standard magic number of “golden” examples, but often a hundred would be a great start. 

4. Setting up an evaluation dataset 

With the golden dataset in place, first extract guidelines from it. Look at the “good” examples and write down why they are good: structure, tone, length, required facts, and common failure modes. Turn this into a simple rubric (e.g., accuracy, clarity, conciseness). The aim is to get an evaluation dataset with examples that score “excellent” on each of the rubrics we care about. 

Next, collect a few thousand candidate inputs/prompts. For each input, draft a reference output, either written by non‑experts following the rubric or generated by a strong LLM (also prompted with the rubrics). Then have experts review. In high‑stakes domains like medicine, they may review everything; in lower‑stakes cases, they can spot‑check a random sample (e.g. 5–10%). If the sample meets quality thresholds (say ≥95% “good”), accept the dataset; if not, refine the guidelines, provide feedback, and regenerate. This keeps expert time focused while preserving dataset integrity.

Now that we have the evaluation dataset in place, we can evaluate the performance of our LLM system and make the go/no-go decision on launching. In all likelihood, there would be multiple iterations of LLM quality improvement before we consider our system ready for production. 

5. Set up a human annotation process 

Even with an evaluation dataset, you still need human review of LLM answers. The dataset gives you inputs and reference outputs, but evaluating performance is more than checking for an exact match, as there are many equally good ways to express the same content. 

Human annotators compare the system’s output to the reference (or the rubric) and judge dimensions like accuracy, clarity, and completeness. They can credit valid paraphrases, flag borderline cases, and surface new “good” examples to add back into the dataset. This produces trustworthy metrics and reusable artifacts for future improvement.

There are two scenarios here:

1. Comparison against a reference answer: Annotators compare the system output to the pre-defined “reference” answer in the evaluation dataset.
2. Assessment based on guidelines: Annotators should be able to assess outputs using guidelines alone when processing real user inputs after system deployment that don’t have available reference outputs. These examples can then be added back to the evaluation dataset.

Optional: Have evaluators give general feedback besides objective metrics.

Typically, the annotators would have a reference answer available when working on the evaluation dataset prior to system deployment, but guidelines can still come in handy to inform judgment. 

After the system is live, annotators will need to work from guidelines rather than reference answers because production inputs are new and varied – you can’t pre‑write “gold” outputs for every query. The evaluation dataset covers representative cases, but live traffic includes long‑tail queries and changing content (e.g., new documents, user‑specific context), so references typically don’t exist yet. 

6. Automate evaluation, with human oversight

After annotators have thoroughly evaluated the outputs and you trust your human scoring process, the next step is to automate parts of it for faster iteration and lower cost.

One common pattern is using an LLM-as-a-judge approach: you prompt a strong LLM with the input, the system’s output, and (if available) the reference output plus the rubric, and ask it to assign a rating. You then validate this automated scoring by comparing it against human annotations on a random sample; if the agreement is high, you can rely on the judge for most future evaluations and reserve humans for spot checks.

• With reference data: Provide the judge LLM the input, reference output, and candidate output; ask for a good/bad (or scaled) rating. 
• Without reference data: Give the judge LLM the input, the model output, and the scoring rubric; it rates based on guidelines alone.

Other automated signals include readability scores (e.g., Flesch-Kincaid, estimating grade level from sentence and word length) and factuality checks using textual entailment models (which test if the summary is logically supported by the source text).

Using evaluation results 

The numerical results from evaluation (e.g., the percentage of LLM output rated as good) are only the beginning. The real value lies in the artifacts the process produced – graded outputs, error notes, and preference pairs, which feed directly back into model improvement. 

Here are three practical ways to use them:

1. Error analysis and prompt tuning: Collect failing cases, look for patterns (e.g., missing citations, off-tone summaries), and refine prompts or instructions to address those gaps.
2. Supervised fine-tuning (SFT): This is a method where you “teach” AI by showing it “good” examples by feeding it prompt-and-answer pairs that are considered ideal. The model learns from this and eventually is able to reproduce high-quality answers. 
3. Preference fine-tuning: During preference fine-tuning, a human evaluator may be shown two example answers supplied by the LLM (or manually created). The evaluator chooses their preference of the two, where their choice becomes the “preferred answer” and the other the “non-preferred.” By learning from side-by-side choices, the model gets much better at understanding the subtleties that make a high-quality answer. 

The entire six-step process may not be implemented linearly in an organization. But it is important that, as the process matures, there is a clear path to setting up all stages. The evaluation datasets must also be continuously updated to reflect real-world usage and evolving user needs.

Key takeaways 

Evaluation is essential as LLMs and agents become ubiquitous. Not only does the process of evaluation bring long-term dividends for system performance, but it can also be continually iterated on to ensure quality is always front and centre.  

Get articles like this in your inbox

Choose your LeadDev newsletters to subscribe to.