AIHOT 于 2026-08-18 收录了“LangSmith 推出 Tuned Evaluators，为生产环境 trace 附加质量反馈”这一公开动态。以下先呈现从来源页面抓取的正文，再给出 AIHOT 摘要与 TopoReduce 编辑解读。
PUBLIC SOURCE CONTENT
已抓取公开正文公开原文内容
Introducing LangSmith Tuned Evaluators
LangSmith
Observability & Evals
Introducing LangSmith Tuned Evaluators, starting with Perceived Error
Jake Broekhuizen
Shamik Karkhanis
Vivek Trivedy
August 18, 2026
5
min
Go back to blog
Create agents
Share
Key Takeaways
- Automatically attach useful signal to production traces. Tuned Evaluators analyze agent interactions and attach feedback teams can use in their agent improvement workflows, starting with Perceived Error.
- Get a turnkey evaluator, managed end-to-end. Teams don’t need to write or maintain complex prompts, select or version LLM-as-judge models, manage credentials, or operate inference infrastructure. LangChain handles it end-to-end.
- Get frontier-level accuracy at a fraction of the cost. Perceived Error uses a specialized model trained by LangChain that exceeded frontier performance while reducing evaluation cost by up to 82%.
Every conversational agent in production needs evaluation for Perceived Error. It’s one of the clearest signals about whether your agent is giving users a helpful experience or not.
Until now, evaluating Perceived Error has been expensive and hard to scale. Teams had to rely on frontier model calls, sample only a small slice of traces, and spend time manually tuning the evaluator.
Today, we’re introducing Tuned Evaluators, which automatically attach quality feedback to production traces and threads. Teams can use that feedback to find behavior that needs attention, understand what went wrong, and take action to improve their agents.
Each Tuned Evaluator is a finished, versioned evaluator for a specific objective. LangChain has already done the work of turning that objective into a production-ready judge. Teams simply select the evaluator they need, and attach it to a tracing project.
High-Quality, Low-Cost Judge that Runs on Every Conversation
Applying high-quality evaluation across production traces has traditionally forced teams to choose between accuracy and coverage.
Frontier models can provide strong judgments, but applying them across many production conversations is expensive. Smaller models cost less, but their lower accuracy makes it harder to trust which traces deserve attention. As a result, many teams evaluate only a small sample of threads, which risks missing the critical signals that an agent is delivering a bad experience to customers.
Inference cost is only part of the work. Building a dependable evaluator also requires defining the behavior, resolving ambiguous cases, producing labels, selecting and benchmarking a judge, and maintaining it as models and production behavior change. Tuned Evaluators package that work into a product that teams can attach and use immediately.
Tuned Evaluators also address the cost/quality tradeoff with specialized models post-trained for narrowly defined evaluation objectives, exceeding frontier performance at a fraction of the cost. When you run Tuned Evaluators, you get more signal to steer agent improvements while saving significant engineering time and inference cost.
How Tuned Evaluators work
- Add a Tuned Evaluator to a LangSmith tracing project and save it.
- LangSmith identifies eligible traces or threads based on the selected evaluator’s requirements.
- A specialized, LangChain-managed judge evaluates each eligible trace or thread.
- The result and its explanation are attached to the evaluated trace or thread as feedback.
- Teams use the results to investigate traces, add useful examples to datasets, or build evals for their improvement workflow.
LangChain writes, tests, versions, and maintains the evaluator prompt. We also manage the judge model, benchmarking against frontier models, provider credentials, and inference infrastructure.
Use Tuned Evaluator results in your improvement workflow
Teams can use enriched traces in their own analytics, coding-agent, CI, human-review, and evaluation workflows. Tuned Evaluators help with:
- Finding failures that produced no system error or explicit user rating.
- Filter to conversations that deserve investigation and review the evaluator's explanation alongside the original interaction.
- Compare flagged traces to find repeated failure modes.
- Add tagged traces to evaluation datasets or route ambiguous conversations for human review.
- Use those examples to test and validate changes to the agent.
Starting with Perceived Error
Perceived Error detects conversations containing evidence that an agent made a mistake, misunderstood a request, or took the interaction in the wrong direction.
That evidence may be explicit, such as a user correction, repeated request, or rejected action. The model can also infer it from contradictory responses, acknowledged mistakes, persistent misunderstandings, or unresolved outcomes.
Perceived Error is a useful proxy for whether an agent is meeting a user’s needs. Most users never submit an explicit rating, so the evidence often has to be inferred from the conversation itself.
LangChain post-trained a specialized model on labeled traces of conversational agents. It outperformed every frontier model in our benchmark while reducing evaluation cost by 82%. In some early-partner workloads, savings reached 98%, with cost variance dependent on thread composition.
Vanta’s experience as an early partner demonstrates another benefit: teams can establish quality coverage immediately while developing their own business-specific evaluators.
This out-of-the-box evaluator gave us a safety net to catch failure modes from day one, while we built out our own business-specific evaluators. It’s the fastest way we’ve found to make a team operational on quality.
— Kevin Royer, Staff ML/AI Engineer, Vanta
Eligibility and timing
A thread becomes eligible for the Perceived Error evaluator after it contains at least two human-AI message pairs and reaches the configured idle period. The evaluation completes within 12 hours after the thread becomes eligible.
Availability and pricing
The Perceived Error Tuned Evaluator is available now for all Plus and Cloud Enterprise plans in the US. Each successful evaluation incurs a tuned evaluation charge. Skipped and failed evaluations are not billed. For general usage and billing information, see our pricing page.
Get started
You can try the Perceived Error evaluator today by logging in or signing up for LangSmith, and visit the docs for more detail.
If you have a Tuned Evaluator that you’d like us to build, tell us about your use case.
Related content
LangSmith
Managed Deep Agents is now in Public Beta
Victor Moreira
August 7, 2026
9
min
Observability & Evals
Evaluating code review agents with ReviewBench
Nick Hollon
July 31, 2026
6
min
Open Source
Observability & Evals
How We Benchmark Deep Agents
Nick Hollon
Harrison Chase
July 23, 2026
4
min
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
See what your agent is really doing
LangSmith, our agent engineering platform, helps developers debug every agent decision, eval changes, and deploy in one click.
Try LangSmith
Get a demo
Observability & Evals
Introducing LangSmith Tuned Evaluators, starting with Perceived Error
Jake Broekhuizen
Shamik Karkhanis
Vivek Trivedy
August 18, 2026
5
min
Go back to blog
Create agents
Share
Key Takeaways
- Automatically attach useful signal to production traces. Tuned Evaluators analyze agent interactions and attach feedback teams can use in their agent improvement workflows, starting with Perceived Error.
- Get a turnkey evaluator, managed end-to-end. Teams don’t need to write or maintain complex prompts, select or version LLM-as-judge models, manage credentials, or operate inference infrastructure. LangChain handles it end-to-end.
- Get frontier-level accuracy at a fraction of the cost. Perceived Error uses a specialized model trained by LangChain that exceeded frontier performance while reducing evaluation cost by up to 82%.
Every conversational agent in production needs evaluation for Perceived Error. It’s one of the clearest signals about whether your agent is giving users a helpful experience or not.
Until now, evaluating Perceived Error has been expensive and hard to scale. Teams had to rely on frontier model calls, sample only a small slice of traces, and spend time manually tuning the evaluator.
Today, we’re introducing Tuned Evaluators, which automatically attach quality feedback to production traces and threads. Teams can use that feedback to find behavior that needs attention, understand what went wrong, and take action to improve their agents.
Each Tuned Evaluator is a finished, versioned evaluator for a specific objective. LangChain has already done the work of turning that objective into a production-ready judge. Teams simply select the evaluator they need, and attach it to a tracing project.
High-Quality, Low-Cost Judge that Runs on Every Conversation
Applying high-quality evaluation across production traces has traditionally forced teams to choose between accuracy and coverage.
Frontier models can provide strong judgments, but applying them across many production conversations is expensive. Smaller models cost less, but their lower accuracy makes it harder to trust which traces deserve attention. As a result, many teams evaluate only a small sample of threads, which risks missing the critical signals that an agent is delivering a bad experience to customers.
Inference cost is only part of the work. Building a dependable evaluator also requires defining the behavior, resolving ambiguous cases, producing labels, selecting and benchmarking a judge, and maintaining it as models and production behavior change. Tuned Evaluators package that work into a product that teams can attach and use immediately.
Tuned Evaluators also address the cost/quality tradeoff with specialized models post-trained for narrowly defined evaluation objectives, exceeding frontier performance at a fraction of the cost. When you run Tuned Evaluators, you get more signal to steer agent improvements while saving significant engineering time and inference cost.
How Tuned Evaluators work
- Add a Tuned Evaluator to a LangSmith tracing project and save it.
- LangSmith identifies eligible traces or threads based on the selected evaluator’s requirements.
- A specialized, LangChain-managed judge evaluates each eligible trace or thread.
- The result and its explanation are attached to the evaluated trace or thread as feedback.
- Teams use the results to investigate traces, add useful examples to datasets, or build evals for their improvement workflow.
LangChain writes, tests, versions, and maintains the evaluator prompt. We also manage the judge model, benchmarking against frontier models, provider credentials, and inference infrastructure.
Use Tuned Evaluator results in your improvement workflow
Teams can use enriched traces in their own analytics, coding-agent, CI, human-review, and evaluation workflows. Tuned Evaluators help with:
- Finding failures that produced no system error or explicit user rating.
- Filter to conversations that deserve investigation and review the evaluator's explanation alongside the original interaction.
- Compare flagged traces to find repeated failure modes.
- Add tagged traces to evaluation datasets or route ambiguous conversations for human review.
- Use those examples to test and validate changes to the agent.
Starting with Perceived Error
Perceived Error detects conversations containing evidence that an agent made a mistake, misunderstood a request, or took the interaction in the wrong direction.
That evidence may be explicit, such as a user correction, repeated request, or rejected action. The model can also infer it from contradictory responses, acknowledged mistakes, persistent misunderstandings, or unresolved outcomes.
Perceived Error is a useful proxy for whether an agent is meeting a user’s needs. Most users never submit an explicit rating, so the evidence often has to be inferred from the conversation itself.
LangChain post-trained a specialized model on labeled traces of conversational agents. It outperformed every frontier model in our benchmark while reducing evaluation cost by 82%. In some early-partner workloads, savings reached 98%, with cost variance dependent on thread composition.
Vanta’s experience as an early partner demonstrates another benefit: teams can establish quality coverage immediately while developing their own business-specific evaluators.
This out-of-the-box evaluator gave us a safety net to catch failure modes from day one, while we built out our own business-specific evaluators. It’s the fastest way we’ve found to make a team operational on quality.
— Kevin Royer, Staff ML/AI Engineer, Vanta
Eligibility and timing
A thread becomes eligible for the Perceived Error evaluator after it contains at least two human-AI message pairs and reaches the configured idle period. The evaluation completes within 12 hours after the thread becomes eligible.
Availability and pricing
The Perceived Error Tuned Evaluator is available now for all Plus and Cloud Enterprise plans in the US. Each successful evaluation incurs a tuned evaluation charge. Skipped and failed evaluations are not billed. For general usage and billing information, see our pricing page.
Get started
You can try the Perceived Error evaluator today by logging in or signing up for LangSmith, and visit the docs for more detail.
If you have a Tuned Evaluator that you’d like us to build, tell us about your use case.
Related content
LangSmith
Managed Deep Agents is now in Public Beta
Victor Moreira
August 7, 2026
9
min
Observability & Evals
Evaluating code review agents with ReviewBench
Nick Hollon
July 31, 2026
6
min
Open Source
Observability & Evals
How We Benchmark Deep Agents
Nick Hollon
Harrison Chase
July 23, 2026
4
min
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
See what your agent is really doing
LangSmith, our agent engineering platform, helps developers debug every agent decision, eval changes, and deploy in one click.
Try LangSmith
Get a demo
AIHOT 摘要
LangSmith 发布 Tuned Evaluators，可为生产环境 trace 附加质量反馈，首个能力是 Perceived Error，帮助团队发现并修复智能体错误。该功能直接作用于线上运行数据，让质量评估不再局限于离线测试阶段。
为什么值得关注
该功能把质量反馈直接绑定到生产 trace，排查 Agent 错误时不用先从对话记录重建上下文，缩短定位问题链条。
工程化解读
从 TopoReduce 的工程视角看，这条信息属于“智能体与自动化”主题。它的价值不只在于一个新产品或新观点本身，还在于说明 AI 系统正在如何影响模型接入、智能体协作、研发流程、基础设施和团队决策。实际采用前，应结合原文确认版本、适用范围、价格和运行条件。
- 发布时间：2026-08-18；AIHOT 分类：智能体与自动化。
- AIHOT 标签：
- AIHOT 判断：该功能把质量反馈直接绑定到生产 trace，排查 Agent 错误时不用先从对话记录重建上下文，缩短定位问题链条。
- AIHOT 评分：38；评分用于站内排序，不等同于独立评测结论。
TopoReduce 编辑观察
当 AI 动态进入真实生产环境，团队需要同时关注能力边界、数据来源、调用成本、权限控制和可回滚性。把单条新闻放回完整工程链路中阅读，比只看标题更有助于判断它是否适合自己的产品和工作流。