Snapshot 2026-08-04 16:17:00 UTC · version 1
The full procedure behind SKILL.md's "Judge Calibration Is a Prerequisite" section. Any grader routed to an LLM-judge follows this before its verdicts count toward a pass rate or a checkpoint promotion decision. Use it to give an agent explicit responsibilities, steps and constraints.
The full procedure behind SKILL.md's "Judge Calibration Is a Prerequisite" section. Any grader routed to an LLM-judge follows this before its verdicts count toward a pass rate or a checkpoint promotion decision. Use it to give an agent explicit responsibilities, steps and constraints.
Editorial note: curated source snapshot published by Collider.club under the MIT License. Source attribution is preserved in the front matter.
Last verified: 2026-07-14
The full procedure behind SKILL.md's "Judge
Calibration Is a Prerequisite" section. Any grader
routed to an LLM-judge follows this before its
verdicts count toward a pass rate or a checkpoint
promotion decision.
If error analysis produced zero buckets that route to
an LLM-judge — every grader is regex, schema, or
execution-based — this entire protocol is N/A for the
run, not an unsatisfiable checklist item. Phase 0
Exit Checklist item 4 in SKILL.md is satisfied by
stating this explicitly (e.g. "Judge calibration:
N/A — all N graded criteria are deterministic") rather
than leaving it blank or blocking Phase 0 completion
on a judge that was never going to exist. This is
common on a greenfield strict-schema or exact-match
task with synthetic goldens — don't invent a
subjective criterion just to have something to
calibrate.
Collect human labels for ≥100 items covering the
failure bucket the judge will grade. Use the same
labelers (or a labeling rubric tight enough to be
interchangeable) that produced the axial-coding
buckets in SKILL.md's Building Goldens section —
a judge calibrated against a different notion of
"pass" than the one used to build goldens will
silently diverge from what the harness is supposed
to measure.
Divide the labeled set three ways and keep the splits separate for the whole calibration cycle:
| Split | Purpose | Size |
|---|---|---|
| train | Few-shot examples embedded in the judge prompt | ~20-30% |
| dev | Iterate the prompt, catch obvious misses | ~30-40% |
| sealed test | Report TPR/TNR once; never re-touch after | ~30-40% |
The sealed-test split is sealed: if a dev-split iteration cycle causes the reported test-split number to move, that number is no longer a valid generalization estimate — re-seal a fresh test split instead of re-running against the same one.
Run the judge (with train-split few-shot examples in the prompt) against the sealed test split and compute:
def tpr_tnr(judge_verdicts: list[bool], human_labels: list[bool]) -> tuple[float, float]:
"""True Positive Rate (sensitivity) and True
Negative Rate (specificity) against human labels.
A single blended accuracy number hides which
direction the judge is biased toward — always
report both, never accuracy alone.
Raises ValueError on empty or mismatched-length
input, or if the sealed test split lacks either
class — a `zip()` over unequal lists silently
drops the extra items, which can produce
apparently valid metrics from a partial or
miscollected split.
"""
if not judge_verdicts or not human_labels:
raise ValueError("tpr_tnr: judge_verdicts and human_labels must be non-empty")
if len(judge_verdicts) != len(human_labels):
raise ValueError(
f"tpr_tnr: length mismatch ({len(judge_verdicts)} verdicts vs "
f"{len(human_labels)} labels) — check the split for a collection bug"
)
tp = sum(j and h for j, h in zip(judge_verdicts, human_labels))
fn = sum((not j) and h for j, h in zip(judge_verdicts, human_labels))
tn = sum((not j) and (not h) for j, h in zip(judge_verdicts, human_labels))
fp = sum(j and (not h) for j, h in zip(judge_verdicts, human_labels))
if (tp + fn) == 0:
raise ValueError("tpr_tnr: no positive-labeled items in this split — TPR undefined")
if (tn + fp) == 0:
raise ValueError("tpr_tnr: no negative-labeled items in this split — TNR undefined")
return tp / (tp + fn), tn / (tn + fp)
Agree on a TPR/TNR bar with whoever owns the eval harness before running this step — a common starting bar is ≥0.85 on both, tightened per bucket based on how costly a false pass or false fail is downstream.
A judge with unequal TPR/TNR does not report the true pass rate of the model under test — it reports a rate skewed by its own asymmetric error pattern. Correct the observed pass rate before publishing it:
def bias_corrected_pass_rate(observed_pass_rate: float, tpr: float, tnr: float) -> float:
"""Rogan-Gladen style correction: recovers the
true pass rate from the judge's observed rate and
its TPR/TNR, rather than reporting the judge's raw
output as if it were ground truth.
"""
denom = tpr + tnr - 1
if denom <= 0:
raise ValueError("Judge is at or below chance — do not correct, recalibrate")
return (observed_pass_rate + tnr - 1) / denom
If tpr + tnr - 1 is small (judge close to chance),
the correction blows up and becomes unreliable — that
is itself a signal the judge needs a prompt rewrite,
not a correction formula.
Record the exact judge model snapshot/version used for calibration alongside the TPR/TNR numbers. An unpinned judge (a "latest" alias that moves under you) invalidates the calibration the moment the underlying model changes — the TPR/TNR numbers stop describing the judge actually in use.
Two triggers, either one is sufficient:
Recalibration reruns steps 1-4 in full — it is not a partial refresh of just the test split.
If the judge cannot hit the agreed TPR/TNR bar after prompt iteration on the dev split:
SKILL.md's Graders section) over shipping a
permanently advisory-only judge.The judge model must come from a different model family than the model under test, for every calibration cycle and every recalibration — a judge evaluating outputs from its own family is a biased grader, and this cannot be corrected away with the TPR/TNR formula above since the bias is systematic rather than random.
This card belongs to the curated knowledge base of Collider.club — a closed business club for entrepreneurs, engineers, investors and domain experts building projects for international markets. Members work across DeFi, AI/ML, FinTech, Web3, banking, hardware and venture capital, and the club runs closed sessions on high-margin niches with anonymous speakers.
mdrss-card/v2)MIT License — Copyright (c) 2026 Collider.club. Full text: LICENSE · https://opensource.org/licenses/MIT
concept:training-fine-tuningorg:collider-club Sign in to join the discussion.
Snapshot 2026-08-04 16:17:00 UTC · version 1
The full procedure behind SKILL.md's "Judge Calibration Is a Prerequisite" section. Any grader routed to an LLM-judge follows this before its verdicts count toward a pass rate or a checkpoint promotion decision. Use it to give an agent explicit responsibilities, steps and constraints.
The full procedure behind SKILL.md's "Judge Calibration Is a Prerequisite" section. Any grader routed to an LLM-judge follows this before its verdicts count toward a pass rate or a checkpoint promotion decision. Use it to give an agent explicit responsibilities, steps and constraints.
Editorial note: curated source snapshot published by Collider.club under the MIT License. Source attribution is preserved in the front matter.
Last verified: 2026-07-14
The full procedure behind SKILL.md's "Judge
Calibration Is a Prerequisite" section. Any grader
routed to an LLM-judge follows this before its
verdicts count toward a pass rate or a checkpoint
promotion decision.
If error analysis produced zero buckets that route to
an LLM-judge — every grader is regex, schema, or
execution-based — this entire protocol is N/A for the
run, not an unsatisfiable checklist item. Phase 0
Exit Checklist item 4 in SKILL.md is satisfied by
stating this explicitly (e.g. "Judge calibration:
N/A — all N graded criteria are deterministic") rather
than leaving it blank or blocking Phase 0 completion
on a judge that was never going to exist. This is
common on a greenfield strict-schema or exact-match
task with synthetic goldens — don't invent a
subjective criterion just to have something to
calibrate.
Collect human labels for ≥100 items covering the
failure bucket the judge will grade. Use the same
labelers (or a labeling rubric tight enough to be
interchangeable) that produced the axial-coding
buckets in SKILL.md's Building Goldens section —
a judge calibrated against a different notion of
"pass" than the one used to build goldens will
silently diverge from what the harness is supposed
to measure.
Divide the labeled set three ways and keep the splits separate for the whole calibration cycle:
| Split | Purpose | Size |
|---|---|---|
| train | Few-shot examples embedded in the judge prompt | ~20-30% |
| dev | Iterate the prompt, catch obvious misses | ~30-40% |
| sealed test | Report TPR/TNR once; never re-touch after | ~30-40% |
The sealed-test split is sealed: if a dev-split iteration cycle causes the reported test-split number to move, that number is no longer a valid generalization estimate — re-seal a fresh test split instead of re-running against the same one.
Run the judge (with train-split few-shot examples in the prompt) against the sealed test split and compute:
def tpr_tnr(judge_verdicts: list[bool], human_labels: list[bool]) -> tuple[float, float]:
"""True Positive Rate (sensitivity) and True
Negative Rate (specificity) against human labels.
A single blended accuracy number hides which
direction the judge is biased toward — always
report both, never accuracy alone.
Raises ValueError on empty or mismatched-length
input, or if the sealed test split lacks either
class — a `zip()` over unequal lists silently
drops the extra items, which can produce
apparently valid metrics from a partial or
miscollected split.
"""
if not judge_verdicts or not human_labels:
raise ValueError("tpr_tnr: judge_verdicts and human_labels must be non-empty")
if len(judge_verdicts) != len(human_labels):
raise ValueError(
f"tpr_tnr: length mismatch ({len(judge_verdicts)} verdicts vs "
f"{len(human_labels)} labels) — check the split for a collection bug"
)
tp = sum(j and h for j, h in zip(judge_verdicts, human_labels))
fn = sum((not j) and h for j, h in zip(judge_verdicts, human_labels))
tn = sum((not j) and (not h) for j, h in zip(judge_verdicts, human_labels))
fp = sum(j and (not h) for j, h in zip(judge_verdicts, human_labels))
if (tp + fn) == 0:
raise ValueError("tpr_tnr: no positive-labeled items in this split — TPR undefined")
if (tn + fp) == 0:
raise ValueError("tpr_tnr: no negative-labeled items in this split — TNR undefined")
return tp / (tp + fn), tn / (tn + fp)
Agree on a TPR/TNR bar with whoever owns the eval harness before running this step — a common starting bar is ≥0.85 on both, tightened per bucket based on how costly a false pass or false fail is downstream.
A judge with unequal TPR/TNR does not report the true pass rate of the model under test — it reports a rate skewed by its own asymmetric error pattern. Correct the observed pass rate before publishing it:
def bias_corrected_pass_rate(observed_pass_rate: float, tpr: float, tnr: float) -> float:
"""Rogan-Gladen style correction: recovers the
true pass rate from the judge's observed rate and
its TPR/TNR, rather than reporting the judge's raw
output as if it were ground truth.
"""
denom = tpr + tnr - 1
if denom <= 0:
raise ValueError("Judge is at or below chance — do not correct, recalibrate")
return (observed_pass_rate + tnr - 1) / denom
If tpr + tnr - 1 is small (judge close to chance),
the correction blows up and becomes unreliable — that
is itself a signal the judge needs a prompt rewrite,
not a correction formula.
Record the exact judge model snapshot/version used for calibration alongside the TPR/TNR numbers. An unpinned judge (a "latest" alias that moves under you) invalidates the calibration the moment the underlying model changes — the TPR/TNR numbers stop describing the judge actually in use.
Two triggers, either one is sufficient:
Recalibration reruns steps 1-4 in full — it is not a partial refresh of just the test split.
If the judge cannot hit the agreed TPR/TNR bar after prompt iteration on the dev split:
SKILL.md's Graders section) over shipping a
permanently advisory-only judge.The judge model must come from a different model family than the model under test, for every calibration cycle and every recalibration — a judge evaluating outputs from its own family is a biased grader, and this cannot be corrected away with the TPR/TNR formula above since the bias is systematic rather than random.
This card belongs to the curated knowledge base of Collider.club — a closed business club for entrepreneurs, engineers, investors and domain experts building projects for international markets. Members work across DeFi, AI/ML, FinTech, Web3, banking, hardware and venture capital, and the club runs closed sessions on high-margin niches with anonymous speakers.
mdrss-card/v2)MIT License — Copyright (c) 2026 Collider.club. Full text: LICENSE · https://opensource.org/licenses/MIT
concept:training-fine-tuningorg:collider-club Sign in to join the discussion.