这个正确吗？接下来该咋办 Step-by-step reference (exam format) What they're testing: Whether you can turn "trustworthy" into testable claims, and test each one with a metric, a benchmark and a decision. Step 0. Clarify (ask before answering) 1. What is being judged: factual correctness, task completion, preference, or safety? 2. Output type: binary pass/fail, a 1–5 score, or pairwise A vs B? 3. What decision uses it: filtering training data, ranking models, or gating a launch? 4. Which error costs more: accepting a bad response or rejecting a good one? Step 1. State what "valid" means as four testable claims - C1 Accuracy: the judge agrees with expert consensus about as well as experts agree with each other. - C2 Unbiasedness: its verdict does not change with things that should not matter (position, length, model identity). - C3 Reliability: the same input gets the same verdict on repeated runs. - C4 Fitness for the decision: its errors are small enough for the specific use (item-level for filtering, system-level for ranking). Each later step tests one claim. Step 2. Build the human reference set - Sample: a random draw from the target distribution (for unbiased rates) plus an oversample of hard and high-risk cases (for power on rare failures). Keep the sampling weights. - Raters: 3 qualified experts per item, blind to model identity and to the judge's output. - Gold label: majority or adjudicated consensus; keep the individual votes. - Human ceiling: compute human–human agreement (Krippendorff's alpha or kappa). This is the benchmark for C1. - Size: set by the precision needed on the rarest class. To estimate an 80% catch rate to within ±6.5 points you need about 150 truly bad items. Step 3. Split it - Development set: tune the judge prompt and rubric. - Test set: touched once, at the end. Every reported number comes from here. Step 4. Test C1, item-level agreement Judge output Metric Pass if Binary Cohen's kappa, sensitivity, specificity kappa within about 0.05 of the human ce

这份清单默认“评委”这个方案本身值得做，直接进入怎么测。立正主张，采用任何新技术前要先由自己的任务决定标准，并和不用它时的最佳合理方案比较。他举工单分类为例：质量、延迟、一致性、校准四项的权重由任务决定，实时交互在意几十毫秒，每天跑一次的批处理更在意错误怎么被发现和追回；厂商证明延迟下降，不等于延迟就是你此刻最该解决的问题。放到评委上：如果已有小模型换个读取方式就能完成判断，那“评委”要带来什么额外价值才值得接入？这一步没做，后面测得多严谨都可能是在优化一个不该选的方案⁠1,2。

用人类之间的分歧程度当上限、算 kappa 和敏感度特异度、按最稀有类别的精度定样本量、开发集调提示词而测试集只碰一次，这些和立正讲假设检验时的思路一致：先分清是在衡量证据还是在做长期决策，再谈 α、β、power 和样本量，不能一边用这些概念一边假装没进入决策框架。他讲 AB 实验时也强调，负向结果显著就可以提前终止，正向效果则要等实验跑完再做上线决定，因为早期探到正向没什么长期好处。评委评测同理：坏样本要够多才测得出漏放率，好结果不要急着宣布通过⁠3,4。

在补完 C1 到 C4 之前，先花时间回答一个问题——如果不用这个评委，针对同一个判断任务，你最合理的做法是什么？可能是已有小模型直接读 logits，可能是规则加人工抽检，也可能是换一个更简单的方案。把它实现出来，和评委在同一批样本上比。观察的反馈不是谁分数高，而是：评委相对这个替代方案，多解决了什么你真正在意的问题。如果答不上来，说明现在该做的是选型判断，不是继续加评测指标⁠1,2。