Prelaunch preview. Explore AROS products and skills. Checkout is in sandbox; no real payment.
Blog

Why five axioms from mathematical logic govern how we audit science

Many theorems rest on five levels of proof. The same five levels guide how our skills audit manuscripts and how an AI agent checks its own actions.

Five levels of proof from mathematical logic show what surface checks miss, in a manuscript and in an AI agent's own work.

Wang Ziyi43 min readEnglish

In an earlier essay we argued that the rules for auditing research (peer review, literature synthesis, meta-analysis, causal inference, figure checking and grant review) should run as software while the work is being done, rather than years after publication. A fair question follows at once. Across biology, clinical medicine and statistics there seem to be thousands of separate rules: check sample sizes, check unit conversions, check multiple-testing corrections, check whether an antibody binds its target, check whether a control group was run at every time point, check whether a confounder explains the difference between two cohorts. If every error needs its own hand-written rule, then any software that audits science is only a growing patchwork of checklists, always one step behind the next mistake.

Mathematics faced the same worry fifty years ago, and discovered that the patchwork is an illusion. When Harvey Friedman, and later Stephen Simpson and many colleagues, began asking which axioms are actually needed to prove the theorems of ordinary mathematics—a program called reverse mathematics—nothing guaranteed a tidy answer [1] [2]. Logicians can easily build artificial axiom systems that cannot even be compared with one another [18]. Yet when they measured theorems of real analysis, algebra and combinatorics, several hundred of them turned out to be provable in a weak base system or equivalent to one of four stronger ones: just five classes, ordered from weakest to strongest like rungs on a ladder [2] [18]:

RCA0⊊WKL0⊊ACA0⊊ATR0⊊Π11-CA0\text{RCA}_0 \subsetneq \text{WKL}_0 \subsetneq \text{ACA}_0 \subsetneq \text{ATR}_0 \subsetneq \Pi^1_1\text{-CA}_0

Logicians call those five systems the Big Five [18]. A few famous theorems, such as Ramsey's theorem for pairs, sit outside the ladder, and they are studied for exactly that reason [19]. Why would theorems invented across different centuries and cultures cluster into the same five boxes? Nobody yet knows. One open possibility, raised by Ludovic Patey, is that the pattern reflects a bias in which statements mathematicians choose to study [19]. We read it as a kind of anthropic principle: if all possible logical propositions form a vast high-dimensional space, human mathematicians explore only the tiny corner that a human mind can hold, check and pass on to a student.

This essay starts with the intuitive foundations of the Big Five—through the two-player games, monster functions, "lossy zip files" and mathematical plot twists that make reverse mathematics one of the most entertaining subjects in modern logic—and then shows how the five cognitive moves behind the Big Five guide the verification architecture across our seven research skills on nexitia.ai: the biomedical-literature-evidence skill, the Mendelian randomization skill, the meta-analysis skill, the scientific-figure-master skill, the grant-write-master skill, the scientific-writing skill and the academic-peer-review skill.

Why theorems get hard: quantifier alternation as a two-player game

Before climbing the Big Five ladder, ask a simpler question: what makes one mathematical statement harder for a human brain—or an AI agent—to verify than another?

In mathematical logic there are only two quantifiers: the universal quantifier ∀\forall ("for all…", "for every…") and the existential quantifier ∃\exists ("there exists at least one…"). Stacking the same quantifier over and over costs almost no mental effort, because identical quantifiers pack into a single draw:

∀x ∀y ∀z  ⟺  ∀(x,y,z)\forall x \, \forall y \, \forall z \iff \forall (x, y, z)

Picking three numbers (x,y,z)(x, y, z) in space at once is no harder than picking one. The trouble starts the moment the quantifiers alternate (∀\forall followed by ∃\exists, then ∀\forall, then ∃\exists). Every alternation turns a static statement into a two-player adversarial game:

  • Every ∀\forall is your opponent making a move, going to extremes to pick the worst, most awkward case against your claim.
  • Every ∃\exists is your side countering the move, choosing a response that resolves the crisis based on every move the opponent has made so far.

Count how fast the human brain heats up as the number of flips grows:

  • 0 flips (x+y=5x + y = 5): No game at all; you read the numbers as they sit on the page.
  • 1 flip (∀x ∃y,y>x\forall x \, \exists y, y > x): "For any number you name, there is a larger one." Even an elementary-school student can win this one-round game: whatever xx the opponent plays, you answer y=x+1y = x + 1.
  • 2 flips (∀ϵ>0 ∃δ>0 ∀x\forall \epsilon > 0 \, \exists \delta > 0 \, \forall x): This is the ϵ\epsilon-δ\delta definition of a limit, in the form Weierstrass gave it, the exact point where college freshmen get their first real headache in calculus: ∀ϵ>0 ∃δ>0 ∀x (0<∣x−x0∣<δ→∣f(x)−L∣<ϵ)\forall \epsilon > 0 \, \exists \delta > 0 \, \forall x \, \bigl(0 < |x - x_0| < \delta \to |f(x) - L| < \epsilon\bigr) Read it as a three-step rally: Round 1 (∀ϵ\forall \epsilon), the opponent demands an error tolerance ϵ\epsilon; Round 2 (∃δ\exists \delta), you calculate a safe neighbourhood width δ\delta tailored to that ϵ\epsilon; Round 3 (∀x\forall x), the opponent picks any point xx inside your δ\delta-window and checks whether ∣f(x)−L∣<ϵ|f(x) - L| < \epsilon holds. Two flips (∀→∃→∀\forall \to \exists \to \forall) already sit near the edge of untrained intuition.
  • 3 flips (∀→∃→∀→∃\forall \to \exists \to \forall \to \exists): The Weierstrass monster function. In the nineteenth century Karl Weierstrass shocked mathematicians by exhibiting a function f:R→Rf: \mathbb{R} \to \mathbb{R} that is continuous everywhere yet has a derivative nowhere—a jagged sawtooth at every scale. Writing "no point xx has any tangent slope LL" takes three quantifier flips: ∀(x,L)⟹∃ϵ>0⟹∀δ>0⟹∃h≠0 (∣h∣<δ∧∣f(x+h)−f(x)h−L∣≥ϵ)\forall (x, L) \Longrightarrow \exists \epsilon > 0 \Longrightarrow \forall \delta > 0 \Longrightarrow \exists h \neq 0 \, \left(|h| < \delta \land \left|\frac{f(x+h) - f(x)}{h} - L\right| \ge \epsilon\right) Watch the four-move match play out. Move 1 (Opponent): picks x=0.5x = 0.5 and insists the tangent slope there must be L=3L = 3. Move 2 (You): inspect the oscillation frequency near 0.50.5 and lock in an error floor, say ϵ=0.01\epsilon = 0.01, declaring that the secant slope will miss 33 by at least 0.010.01 no matter how closely the opponent zooms in. Move 3 (Opponent's counterattack): zooms in to an absurdly tiny microscope window δ=10−100\delta = 10^{-100}, betting the curve must smooth out locally. Move 4 (Your checkmate): inside ∣h∣<10−100|h| < 10^{-100}, you point to a specific step hh where the curve still wiggles steeply enough to knock the secant slope away from 33 by more than 0.010.01. Your winning move hh depends simultaneously on four nested choices (δ,ϵ,L,x)(\delta, \epsilon, L, x). At three flips, a picture stops being a reliable guide; intuition often guesses wrong, and step-by-step symbolic verification is what settles the question.
  • 4 flips (∀→∃→∀→∃→∀\forall \to \exists \to \forall \to \exists \to \forall): Egorov's theorem and conceptual zip files. Egorov's theorem in measure theory says that if a sequence of measurable functions fn(x)→f(x)f_n(x) \to f(x) converges at almost every point of [a,b][a, b], then after discarding an exceptional "bad" set EE of arbitrarily small measure δ\delta, the sequence converges uniformly on the rest: ∀δ>0⟹∃E⊂[a,b] (μ(E)<δ)⟹∀ϵ>0⟹∃N∈N⟹∀(n≥N, x∉E) (∣fn(x)−f(x)∣<ϵ)\forall \delta > 0 \Longrightarrow \exists E \subset [a, b]\,(\mu(E) < \delta) \Longrightarrow \forall \epsilon > 0 \Longrightarrow \exists N \in \mathbb{N} \Longrightarrow \forall (n \ge N,\, x \notin E)\,\bigl(|f_n(x) - f(x)| < \epsilon\bigr) Here the bad set E=E(δ)E = E(\delta) depends on δ\delta, and the cutoff index N=N(E,ϵ)N = N(E, \epsilon) depends on EE and on ϵ\epsilon. Human working memory holds roughly four chunks at once [20]; by our own loose analogy, four quantifier flips is where even a seasoned mathematician staring at scratch paper starts to feel their head spin.

How did mathematics keep moving forward past four flips? By encapsulating concepts—packing the back half of a quantifier chain into a named black box, like a zip file. Once you package the tail ∀ϵ>0 ∃N ∀n≥N ∀x∉E\forall \epsilon > 0 \, \exists N \, \forall n \ge N \, \forall x \notin E into a single phrase—"converges uniformly outside EE"—Egorov's four-flip monster collapses back into a polite one-flip sentence: ∀δ>0 ∃E (μ(E)<δ and fn converges uniformly outside E)\forall \delta > 0 \, \exists E \, (\mu(E) < \delta \text{ and } f_n \text{ converges uniformly outside } E). For two centuries mathematics climbed by stacking these zip files: Cauchy and Weierstrass packed infinitesimal arguments into limits; Poincaré, Fréchet and Hilbert packed limits into topology and spaces of functions; Alexander Grothendieck packed geometric spaces into schemes and topoi, and built a fresh 1-2-3-4 ladder on top.

Yet conceptual compression is lossy for the reader: the definitions keep every detail, but a name hides the dependencies underneath it. You can hand a colleague a sleek one-line theorem T(X)≅YT(X) \cong Y sitting atop seven nested definitions, and it looks like a zero-flip equation. The moment a referee tries to audit that theorem, however, they have to unzip all seven layers at once, and the hidden quantifier dependencies come crashing back like an avalanche. Worse, as Alfred Tarski proved in the 1930s with his Undefinability Theorem, no formula of arithmetic can define truth for all formulas of arithmetic. Truth can be defined for one restricted class of formulas at a time, level by level, but no single formula checks every level at once [21]. You cannot spiral into the sky by stepping on your right foot with your left foot: every check covers a stated scope and no more, and that is the intuition we carry onto the ladder below.

Why the escape routes keep circling back to the Big Five

What makes reverse mathematics funny—almost like a mathematical ghost story where every path through the woods leads back to the same cabin—is what happened each time twentieth-century mathematics was rebuilt on a new foundation. We tell it as three escape attempts:

  • Escape attempt 1: Infinite games and winning strategies. Logicians tried replacing an infinite chain of alternating moves ∀x0 ∃x1 ∀x2 ∃x3…\forall x_0 \, \exists x_1 \, \forall x_2 \, \exists x_3 \dots with Skolem functions x1=f0(x0)x_1 = f_0(x_0), x3=f1(x0,x2),…x_3 = f_1(x_0, x_2), \dots, which record how one player answers every earlier move. That turns the chain into an infinite game of perfect information, and once the game has a stated winning set it raises a single question: does one of the two players have a winning strategy τ\tau? Yet a strategy τ\tau that maps every finite history to a next move is itself an infinite object, one choice among 2ℵ02^{\aleph_0} possible strategies. When logicians measured the axioms needed to guarantee that a winning strategy exists, the Big Five were waiting at the finish line: open games played on a binary tree land on WKL0\text{WKL}_0 [22], while on the full tree of natural numbers, open or even clopen games land on ATR0\text{ATR}_0 and games whose winning set is an open set intersected with a closed set land on Π11-CA0\Pi^1_1\text{-CA}_0 [2].
  • Escape attempt 2: Abraham Robinson's hyperreal infinitesimals (∗R{}^*\mathbb{R}). In the 1960s nonstandard analysis promised to abolish the two-flip ϵ\epsilon-δ\delta headache of calculus by letting mathematicians divide by actual infinitesimals dxdx as if they were schoolroom fractions. When H. Jerome Keisler measured nonstandard arithmetic against the ladder, he built a nonstandard counterpart for each of the Big Five, and each counterpart proves no new standard theorems beyond its own rung; for example, his nonstandard version of WKL0\text{WKL}_0 is conservative over WKL0\text{WKL}_0 [23]. The same rungs, wearing a fresher coat.
  • Escape attempt 3: Type theory and Homotopy Type Theory (HoTT). Computer scientists replaced sets with types and programs ("propositions as types, proofs as programs"). When proof theorists measured the result, Martin-Löf type theory with its hierarchy of universes but without inductive WW-types, even with Voevodsky's univalence axiom added, turned out to have exactly the strength of ATR0\text{ATR}_0, with the same proof-theoretic ordinal Γ0\Gamma_0 [24]. Adding WW-types lifts it far above every rung of the Big Five, which shows that the ladder describes ordinary mathematics rather than every system one can build.

No matter how you dress the notation—sets, games, infinitesimals or types—human and algorithmic verification keeps circling back to the same five rungs, and we read them, in our own interpretation, as five cognitive moves:

1. RCA0\text{RCA}_0 (Execution): What a computer can calculate step by step

The base rung, Recursive Comprehension Axiom (RCA0\text{RCA}_0), guarantees only the sets that a Turing machine can compute from sets already in hand (Δ10\Delta^0_1 comprehension) [2] [3]. In mathematics, RCA0\text{RCA}_0 handles rational arithmetic, matrix algebra and the intermediate value theorem [2]. In scientific auditing, Execution is the domain of finite arithmetic, unit algebra and linear algebra: checking whether β^/SE=t\hat{\beta}/\text{SE} = t, whether spreadsheet cells reference the columns they claim to use, whether p>np > n makes a sample covariance matrix singular (rank(Σ^)≤n−1<p\text{rank}(\hat{\Sigma}) \le n - 1 < p), or what minimum detectable effect (MDE\text{MDE}) a sample size of n1,n2n_1, n_2 can support.

2. WKL0\text{WKL}_0 (Choice & Compactness): Threading a complete path through a branching maze

The second rung adds Weak König's Lemma (WKL0\text{WKL}_0): every infinite binary tree that has nodes at every depth contains at least one unbroken path from the root through every level [2] [4]. A finite algorithm looking one level at a time cannot always tell which branch hits a dead end fifty steps down; WKL0\text{WKL}_0 is the principle of compactness that bridges local step-by-step compatibility to a complete global path (and is equivalent over RCA0\text{RCA}_0 to the Heine–Borel theorem and Gödel's completeness theorem) [2]. In an experimental study, the design is a branching tree of choices (time points ×\times genotypes ×\times mRNA/protein readouts ×\times group labels). Claiming a continuous aging trajectory or a drug rescue asserts that the evidence can be followed through every level of that tree without a gap or a contradiction—not a path that silently drops the 6- and 12-month branches, reverses direction between mRNA and protein, or swaps group labels (YY-Y, YO-Y) between the figure legend and the Methods.

3. ACA0\text{ACA}_0 (Stabilization & The Halting Jump): Collecting the limit of an entire search family

The third rung, Arithmetical Comprehension Axiom (ACA0\text{ACA}_0), lets you form any set defined using quantifiers over the natural numbers; over RCA0\text{RCA}_0 this is equivalent to Σ10\Sigma^0_1 comprehension [2]. In computability theory, it amounts to granting, for any set, an oracle for its version of Alan Turing's Halting Problem (the Turing jump X↦X′X \mapsto X') [2] [3]; in analysis, it is equivalent to the Bolzano–Weierstrass theorem that every bounded sequence condenses onto a limit point [2]. An RCA0\text{RCA}_0 script can watch a researcher run test after test, but it takes a Stabilization rule, named by analogy with ACA0\text{ACA}_0, to step outside the loop and collect the entire family of MM tested hypotheses H={h1,…,hM}\mathcal{H} = \{h_1, \dots, h_M\} as a single object—checking how many of 40,236 gene models survive Holm family-wise error control, whether constant-sum normalisation across 61 metabolites forces spurious negative covariances, or whether selecting top features before leave-one-out cross-validation leaks the test fold into training.

4. ATR0\text{ATR}_0 (Stratification): Stacking hierarchies along a ruler without mixing levels

The fourth rung, Arithmetical Transfinite Recursion (ATR0\text{ATR}_0), allows comprehension jumps to be iterated along any countable well-ordering ≺\prec (ordinals climbing far past Gerhard Gentzen's ε0\varepsilon_0, the ordinal of ordinary arithmetic); its proof-theoretic ordinal is Γ0\Gamma_0, the Feferman–Schütte ordinal often called the limit of predicativity [5] [25]. Its cognitive move is Stratification: keeping ordered levels of construction strictly apart so a lower level cannot masquerade as a higher one. In a scientific manuscript, evidence lives on ordered strata:

  • Document sections: References (cited literature) ≺\prec Introduction ≺\prec Methods (what this study actually ran) ≺\prec Results ≺\prec Discussion.
  • Biological levels: In silico pilot ≺\prec Immortalised cell line ≺\prec Single-sex animal cohort ≺\prec Cross-species ecological mean ≺\prec Human clinical cohort.

The order records how far each level stands from the claim being made, not how strong its evidence is. When a naive keyword script searches a whole PDF for "male" and "female" to check whether an animal study used both sexes, a paper title in the References section that mentions "male" can silence the warning on a Methods section that used only female mice. Stratifying the document by section before checking rules stops lower-stratum text from polluting higher-stratum verification.

5. Π11-CA0\Pi^1_1\text{-CA}_0 (Self-Reflection): Surveying the whole space of competing explanations

The fifth rung, Π11\Pi^1_1 Comprehension (Π11-CA0\Pi^1_1\text{-CA}_0), adds impredicative comprehension, forming sets defined by universal quantification over all subsets (∀Y⊆N ψ(x,Y)\forall Y \subseteq \mathbb{N}\,\psi(x, Y)) [2]. Its cognitive move is Self-Reflection: testing whether a claim survives not just one chosen path, but every competing set YY in the surrounding space. In science, any causal claim asserts that no competing clinical confounder (age, BMI, surgical harvest route), co-eluting enantiomer (D-galactose versus L-galactose) or horizontal pleiotropy pathway explains the observation equally well.

Mapping the five moves onto our seven skills—and what an adversarial audit taught us

Of course, a Python verification script running on a finite manuscript does not literally solve the Halting Problem or quantify over uncountably many subsets of N\mathbb{N} [2]. Rather, the five moves of the Big Five (Execution, Choice, Stabilization, Stratification and Self-Reflection) expose why first-generation verification scripts miss so many scientific errors: almost every one of our baseline scripts sits on the syntactic surface of Rung 1 (RCA0\text{RCA}_0), checking whether a number appears in a source pool or whether a URL resolves, without checking tree completeness, hypothesis-family multiplicity, section stratification or competing confounders.

Here is how those five moves map onto each of the seven skills on nexitia.ai, and what we found when we subjected both our baseline scripts and our Big Five verification suite to a two-pass benchmark and an independent adversarial code audit across six real research repositories:

  1. biomedical-literature-evidence skill:
    • Existing public record: Logs PubMed/PMC queries, separates studies from overlapping cohort articles, holds imported RIS/CSL-JSON citations as unverified until checked (3 of 3 in its validation record), and allows a review ledger to conclude that evidence is insufficient.
    • Big Five mapping: Execution (RCA0\text{RCA}_0) parses physical units so particle counts (particles/g, MPs/g, filaments/individual) are never pooled as if they were pyrolysis-GC/MS mass concentrations (ug/g); Stratification (ATR0\text{ATR}_0) refuses silent 1:11:1 fallback substitutions when a conversion factor is missing; and Self-Reflection (Π11-CA0\Pi^1_1\text{-CA}_0) flags retracted sources before synthesis.
  2. Mendelian randomization skill:
    • Existing public record: Retains 157 of 163 archived association rows for estimation and matches all 19 headline comparison cells in Walker et al. 2019 (reproduction report), flagging a prose-versus-Table-4 discrepancy.
    • Big Five mapping: Execution (RCA0\text{RCA}_0) verifies finite standard errors, FF-statistics and outcome-scale-specific odds-ratio transforms; Choice (WKL0\text{WKL}_0) enforces deterministic estimator routing (K=1K = 1 Wald ratio, K=2K = 2 IVW, K≥3K \ge 3 IVW + weighted median + MR-Egger); Stabilization (ACA0\text{ACA}_0) recomputes Benjamini–Hochberg false-discovery-rate qq-values across both the full planned exposure family (M=8M = 8) and the core family (M=6M = 6) simultaneously (54 of 54 finite routing and multiplicity checks verified in our benchmark); and Stratification / Self-Reflection (ATR0\text{ATR}_0–Π11-CA0\Pi^1_1\text{-CA}_0) explicitly marks unexecuted Steiger directionality, IGX2I^2_{GX} dilution and multivariable conditioning as unavailable rather than letting a report imply they passed.
  3. meta-analysis skill:
    • Existing public record: Reproduces 1,060 of 1,061 pooled analyses and 10,797 of 11,351 checkable quantitative items (95%) across 28 Cochrane reviews (performance report).
    • Big Five mapping: Execution (RCA0\text{RCA}_0) rejects NaN/±Inf inputs before domain comparisons (a bug we found and patched in ma_validate.py), extracts .xlsx worksheet XML (xl/worksheets/sheet*.xml) while keeping cached values strictly separate from formula strings, and verifies signed_z == local_beta / local_se across all 35,130 testable rows of a 40,236-row cross-species gene table (the other 5,106 rows lack a value and are recorded as excluded, not passed); Stabilization (ACA0\text{ACA}_0) checks that stored Holm multiplicity pass counts (0 of 40,236) match the summary ledger.
  4. scientific-figure-master skill:
    • Existing public record: Parses 61 of 61 scored caption fields across 20 real captions (skill page) and refuses to draw a significance star when a recomputed pp-value is 0.20.2.
    • Big Five mapping: Execution & Choice (RCA0\text{RCA}_0–WKL0\text{WKL}_0) inspect embedded Office Open XML chart caches (word/charts/chart*.xml) directly inside submitted Word files and compare the complete set of plotted outliers against the figure caption. In our adversarial audit of one metabolomics supplement, this general chart-cache parser uncovered a remarkable double contradiction without needing OCR or human referee notes: in Fig. S2 (chart1.xml), the caption claimed samples OA-5, OA-7 and OA-10 exceeded the critical threshold (1.6225), yet the saved chart cache showed OA-10 sitting at 0.9108, well below the line; in Fig. S3 (chart2.xml), the caption named OA-1 and OA-8 above the critical threshold (2.2026), whereas the chart data actually exceeded the threshold at NON-OA2 and OA-1.
  5. grant-write-master skill:
    • Existing public record: Preserves null pilot results (p=0.20p = 0.20) and non-causal wording across revisions and languages (validation record, fact-preservation record).
    • Big Five mapping: Execution (RCA0\text{RCA}_0) measures character, UTF-8 byte and East-Asian display-width counts without inventing unspoken portal caps, and verifies SHA-256 package provenance against PROVENANCE.json; Choice & Self-Reflection (WKL0\text{WKL}_0–Π11-CA0\Pi^1_1\text{-CA}_0) structure the Hypothesis DAG so an immortalised cell line is never left as the sole branch supporting an in vivo claim and parallel fellowship/project grants keep distinct scopes.
  6. scientific-writing skill:
    • Existing public record: Screens reported test statistics against source files and keeps editorial notes out of the publication file (release notes).
    • Big Five mapping: Execution (RCA0\text{RCA}_0) binds prose differential-expression claims (563 DEGs: 209 up-regulated and 354 down-regulated) directly to the computed counts in the supplementary CSV table, and upgrades audit_numbers.py with a zero-dependency .xlsx XML reader that inspects all worksheets and shared strings without letting cell coordinates (U2, 1000) pollute the numeric evidence pool; Choice (WKL0\text{WKL}_0) pairs mRNA and protein readouts by analyte, contrast and time point so nonsignificant protein data are reported as inconclusive rather than a confirmed rescue.
  7. academic-peer-review skill:
    • Existing public record: Calibrated against 1,035 reviewer comments across 60 open reviews; caught 6 of 49 row-level reviewer points and demanded 5 of 9 v1-to-v2 changes on a blind wet-lab submission (skill page, benchmark, PeerJ 20995 blind review).
    • Big Five mapping: We backported three structural upgrades directly into screen.py: a zero-dependency .docx XML reader, a conservative margin line-number normaliser (requiring a sustained consecutive sequence before it removes apparent margin numbers, so an isolated number at a line break is kept), and ATR0\text{ATR}_0 section stratification that restricts animal-sex screening to the Methods (or pre-References body) and flags single-sex scope separately from missing sex reporting, preventing bibliography titles in References from masking single-sex experiments.

The checks catch the specific errors they were built for. Turning that structure into better decisions while an AI agent is still working is the question the next section takes up.

What an AI agent needs in order to check its own work

Consider an agent handling a product exchange. It can look up the order and still miss the question that decides the case: does the item the customer wants to exchange meet the conditions in the policy? The five rungs above were built to audit a finished document. Applied to that one exchange, they would read as follows: compute the order's dates and totals (Execution); follow the branch of the policy that applies to this kind of item (Choice); look at every item and earlier return on the order together (Stabilization); keep what the customer says apart from what the system records (Stratification); and ask whether any exception in the policy would change the answer (Self-Reflection). Here we asked whether the same structure helps an agent while it is still working, and what the evidence from outside our own studies says.

That evidence is clear on one point. Simply asking a model to reconsider its answer can fail badly: in one test, GPT-3.5 fell from 75.8% to 38.1% correct on CommonsenseQA after being asked to self-correct, with no new information to go on [7]. Self-correction becomes more reliable when the agent can consult something outside itself, such as a search engine, a code interpreter or a test that returns a verdict [8] [9]; training a model specifically to correct itself is another route [26]. An external check is useful because it brings in evidence the model does not already have.

The same evidence warns against assuming that more structure is always better. The authors of SOPBench, the benchmark we use below, gave four models an explicit checklist of the verifications each procedure requires; pass rates fell for two of them, and one model's unsafe actions rose from 21 cases to 74 [6]. Even when they handed the agent the true verdict for every constraint, a substantial share of cases still failed, 12% to 38% across the four models, because the final choice of whether to act or refuse remained hard [6]. In coding agents, repository instruction files written by a model raised inference cost by more than 20% without raising success [10]. Instructions are not free: every added rule is more text the agent must read and weigh.

Three ways to review an action

Generic reflection asks the agent to reconsider its work and leaves it to the model to decide what needs checking. Flat verification gives the agent an explicit checklist of requirements to review. Big5c, short for Big-5 compiler, uses the five rungs described above to organize the review. For a proposed action, the agent checks which facts it has established, which choices those facts support, and which prerequisites remain open. When it detects a gap, it asks for a correction and checks the result again. In the exchange example, that means connecting the policy to the particular item in the order.

We used one benchmark for this pilot, SOPBench, which tests whether an agent can complete a customer request while following the rules that govern its actions. Its authors provide 830 validated cases covering 70 procedures, with code that checks the final state, each step, and whether the required verifications came before the action [6].

We tested both models on the same 185 cases covering 20 procedures. The cases came from online shopping (49), healthcare (40), banking (27), motor vehicle services (27), hotels (25), universities (13), and libraries (4). The table shows the share of cases handled correctly, including correct refusals, and the average tokens used per case.

Review methodGPT-5.6 Luna successGemini 3.8 Flash low successGPT tokens per caseGemini tokens per case
Generic reflection63.8% (118/185)63.8% (118/185)58,178105,821
Flat verification64.9% (120/185)59.5% (110/185)57,198104,921
Big5c87.6% (162/185)88.1% (163/185)73,902116,664

Token averages use the same cases for all three methods within each model. One Gemini case lacked a token count for generic reflection, leaving 184 cases for the Gemini cost comparison.

On GPT-5.6 Luna, Big5c improved success by 23.8 percentage points over generic reflection and 22.7 points over flat verification. Each additional successful case cost about 66,100 extra tokens compared with generic reflection and 73,600 compared with flat verification. On Gemini 3.8 Flash low, the gains were 24.3 and 28.6 percentage points, at about 45,300 and 41,600 extra tokens per additional success, respectively.

These costs divide the extra tokens used across all cases with complete token records by the net increase in successful cases on that same set.

Pilot success rates and mean tokens per case across seven SOPBench domains, comparing generic reflection, flat verification and Big5c on GPT and Gemini.

Rows show service domains, with the number of cases in parentheses. Read across each row to compare the three review methods. Success rates use every case. Token means use cases with complete records for all three methods within each model.

What the pilot tells us

Giving an agent a checklist had different effects on the two models. Compared with generic reflection, flat verification yielded two more successes on GPT and eight fewer on Gemini. Big5c improved on both approaches on both models. One plausible explanation is that a checklist can tell an agent to verify eligibility, while a repair step gives it something to do when the evidence is missing. The pilot measured the whole workflow, so the contribution of each of the five rungs remains an open question.

Share of 185 SOPBench cases handled correctly with generic reflection, flat verification and Big5c, on GPT-5.6 Luna and Gemini 3.8 Flash low.

Each bar is the share of the same 185 cases that one review method handled correctly; the counts are in parentheses.

The extra review used tokens even on cases where the outcome stayed the same. For a researcher choosing where to use it, the relevant question is how much an additional correctly handled case is worth. That judgment will depend on the procedure. Online shopping contributed 49 cases and libraries contributed four, so the overall rates tell us more about this particular mix of requests than about any single service.

Where did structured review save tokens? Gemini’s healthcare runs give the clearest example. Big5c and generic reflection each passed 39 of 40 cases, but Big5c used about 3,760 fewer tokens per case (3.7%). The traces record 150 tool calls with Big5c and 159 with generic reflection. A shorter sequence of calls can save tokens because the agent has fewer tool results to read as it works. That is one plausible explanation here. Compared with flat verification, the saving was only about 170 tokens per case (0.2%), with one additional success.

Tool calls explain only part of the pattern. On the four library cases, Gemini passed all four and made 15 calls with either Big5c or generic reflection, yet Big5c used 6.9% fewer tokens. The amount of text the agent reads and generates can also vary within the same number of calls. In banking, Gemini’s Big5c runs cost slightly less than generic reflection and more than flat verification. For GPT, Big5c used more tokens on average than either alternative in every domain. These differences suggest a question for the full study. When does the work saved by catching a missing prerequisite outweigh the cost of checking it?

A second, smaller pilot points to one answer. We ran six synthetic research workflows twice, once with a working helper script and once with a faulty one, which gives 12 tasks per model. Big5c and generic reflection both produced correct final answers on all 12 tasks and repaired all six faulty helpers, but across the 12 tasks Big5c used 34% fewer tokens on GPT and 23% fewer on Gemini. The saving came from the workflows whose helper already worked: when a check can be executed and it passes, the agent can skip the critique altogether.

Mean tokens per task in the skill-repair pilot, generic reflection against Big5c, on GPT and Gemini.

Mean tokens per task across the 12 tasks per model, in thousands. Both methods produced correct answers on all 12.

Managing an AI agent the way we manage people

The pilot points to an idea that interests us more than any single number: the methods that organizations use to make human work reliable may make an AI agent's work reliable in a similar way. Others have built on the same idea. MetaGPT encodes the standard operating procedures and job roles of a software company into a team of AI agents [27]. A more recent system arranges its agents as planners, executors and critics with strict role boundaries, on the reasoning that AI agents, "just like all the humans we have ever hired", remain fallible [28].

Human quality management also offers a way to read our table. Asking the agent to reconsider its work, as generic reflection does, is the equivalent of telling a person to be more careful. A checklist alone helped one model and hurt the other. Surgery, too, has found that adopting a checklist does not bring the same improvement everywhere. When eight hospitals introduced the WHO surgical safety checklist as a program, with staff training and with team briefings before and after each operation, deaths after surgery fell from 1.5% of the 3,733 patients before the program to 0.8% of the 3,955 patients after it [29]. When 101 hospitals in Ontario adopted checklists as the province moved to require public reporting of their use, most of them with an educational program but with no formal team training required, the adjusted risk of death went from 0.71% to 0.65%, a change that was not significant, and the authors noted that, apart from the WHO study, the studies with large effects had either coupled the checklist with extensive team training or used a broader checklist covering the whole course of care [30]. Both are before-and-after comparisons rather than randomized trials, and neither isolates what made the difference, just as our pilot did not isolate the repair step. Together they raise the question we care about for AI agents: which practices around a checklist turn its checks into better decisions? Big5c is one answer under test, a checklist packaged with a repair step that follows a failed check.

Left: deaths after surgery before and after checklists in two studies. Right: change in correctly handled SOPBench cases with flat verification and with Big5c, against generic reflection.

Left: deaths after surgery before and after checklists were introduced in the two studies, which followed different populations; the Ontario rates are adjusted [29] [30]. Right: change in the number of correctly handled cases in our pilot compared with generic reflection, out of 185 per model.

The oldest management tool of all is the written procedure. Aviation solved a version of this problem decades ago. ASD-STE100, Simplified Technical English, restricts vocabulary and sentence structure so that maintenance technicians, many of them non-native readers, can read an instruction without ambiguity [11]. ISO 24495-1 sets out four principles for plain language: readers can find what they need, understand it and use it, and the content is relevant to them [12]. For human readers, Simplified English made complex procedures significantly easier to understand [13].

Many people now apply these standards to AI, mostly in one direction: they ask the model to write in Simplified Technical English so that a person can read its output more easily. Open tools already help apply the standard; one of them reports cutting measured violations by an average of 95.5% across eight writing tasks and twelve models [17]. Andrej Karpathy recommended the practice in early October 2026 [31]. Asking for the standard by name can also have a cost. In one developer's small test on four pieces of code, Claude's explanations lost 4 of 47 checked facts when asked for "Simple Technical English" and 22 of 47 when asked for ASD-STE100 [32].

The other direction, writing the procedure for the AI agent that must follow it, has stronger evidence. Giving an agent the workflow at all matters: on FlowBench, the share of sessions GPT-4o completed rose from 35.1% without the workflow to between 41.7% and 43.2% with it, whether written as text, code or a flowchart, across 51 scenarios [14]. On two ToolBench test sets, rewriting long tool documentation into concise instructions with parameter guidance and examples cut it to less than a third of its length and raised ChatGPT's answer-success score, as judged by GPT-4, from 15.0% to 52.8% [35]. Restructuring the procedure helps as well. On a screened set of 125 SOPBench banking cases, compiling the procedures' machine-readable constraints into structured pseudo-code with explicit verification steps improved success over the official prose by up to 16.0 percentage points across six models (from 70.4% to 86.4% for the largest gain) and lowered it for none of them [33]. A compiler that turns natural-language workflows into executable ones, with explicit inputs, outputs, checks and control flow for each step, raised task success from 56% to 84% on average across three models, on 488 cases from 11 workflows, while rewriting the same workflows as a clearer text document reached 63% [34]. Big5c takes the same route for review: it compiles what a procedure requires into checks on the five rungs, and adds a repair step for a check that fails.

Yet what helps a person does not always help an AI agent, and what helps one model can harm another. In the SOPBench study above, a runtime that showed the agent only the current step of the procedure helped two strong models and lowered GPT-4o-mini's success from 75.2% with the full compiled procedure to 48.8% [33]. Wording matters for obedience: when coding agents received one policy phrased four ways on a set of 86 coding tasks, the average compliance of the more capable of two Claude models was 78.0% with the direct wording and 64.2% with the conditional one ("if …, then …") [15], a tension to watch, because Simplified Technical English puts a condition that the reader must know first at the start of the sentence [11]. Simplification written for people does not automatically help models: when the input texts of benchmark tasks such as classification were simplified automatically, language-model performance fell by up to 20 points in English [16].

Before-and-after success or compliance rates from five studies of procedures written for AI agents.

Each row is one reported result: the circle is the rate before the change and the arrow points to the rate after it. The studies use different tasks and measures, so rows should be read one at a time.

So we expect the management system to carry over to AI agents (written procedures, checks that come with the means to repair, and separate roles), while the wording rules written for human readers have to be tested again on agents. We found no controlled study that tests whether agent procedures rewritten explicitly to ASD-STE100 or ISO 24495-1 help the agent complete more tasks.

We will report the full Big5c study as an academic article on arXiv and release the Big5c engine on GitHub.

References

  1. Friedman H (1975). Some systems of second order arithmetic and their use. Proceedings of the International Congress of Mathematicians (Vancouver, 1974), Vol. 1, pp. 235–242. Canadian Mathematical Congress. Introduces the program of reverse mathematics, showing that ordinary theorems of analysis are equivalent over a weak base theory to specific set-existence axioms of second-order arithmetic.
  2. Simpson SG (2009). Subsystems of Second Order Arithmetic, 2nd ed. Perspectives in Logic, Cambridge University Press. doi:10.1017/CBO9780511581007. Defines second-order arithmetic Z2Z_2 and the five principal subsystems (RCA0\text{RCA}_0, WKL0\text{WKL}_0, ACA0\text{ACA}_0, ATR0\text{ATR}_0 and Π11-CA0\Pi^1_1\text{-CA}_0), establishing their relationships to computable mathematics, Heine–Borel compactness, Bolzano–Weierstrass convergence, ordinal hierarchies up to Γ0\Gamma_0 and impredicative set comprehension.
  3. Turing AM (1936). On computable numbers, with an application to the Entscheidungsproblem. Proceedings of the London Mathematical Society, Series 2, 42(1): 230–265. doi:10.1112/plms/s2-42.1.230. Defines computable sequences via machines with a finite control and an unbounded tape, and proves the unsolvability of the halting/decision problem that separates computable checking (RCA0\text{RCA}_0) from arithmetical comprehension (ACA0\text{ACA}_0).
  4. König D (1927). Über eine Schlussweise aus dem Endlichen ins Unendliche. Acta Litterarum ac Scientiarum Regiae Universitatis Hungaricae Francisco-Josephinae, Sectio Scientiarum Mathematicarum 3: 121–130. Proves König's infinity lemma—that every infinite, finitely branching tree has an infinite path—whose binary-tree restriction forms the axiom of WKL0\text{WKL}_0.
  5. Gentzen G (1936). Die Widerspruchsfreiheit der reinen Zahlentheorie. Mathematische Annalen 112: 493–565. doi:10.1007/BF01565428. Establishes the consistency of first-order arithmetic by transfinite induction up to the ordinal ε0=ωωω⋯\varepsilon_0 = \omega^{\omega^{\omega^\cdots}}, foundational to the ordinal hierarchy analysis in proof theory and ATR0\text{ATR}_0.
  6. Li Z, Huang S, Wang J, Zhang N, Antoniades A, Hua W, Zhu K, Zeng S, Wang C, Wang WY, Yan X (2026). SOPBench: Evaluating language agents at following standard operating procedures and constraints. Proceedings of EMNLP 2026, pp. 21561–21616. https://aclanthology.org/2026.emnlp-main.936/. 830 validated cases across 70 procedures in seven customer-service domains, scored on final state, each action and the order of required verifications; in its ablation, an explicit checklist of required verifications lowered pass rates for two of four models, and supplying true constraint verdicts still left 12–38% of cases failing.
  7. Huang J, Chen X, Mishra S, Zheng HS, Yu AW, Song X, Zhou D (2024). Large language models cannot self-correct reasoning yet. ICLR 2024. arXiv:2310.01798. Without external feedback, prompting a model to self-correct lowered accuracy, for example GPT-3.5 on CommonsenseQA from 75.8% to 38.1%.
  8. Kamoi R, Zhang Y, Zhang N, Han J, Zhang R (2024). When can LLMs actually correct their own mistakes? A critical survey of self-correction of LLMs. Transactions of the Association for Computational Linguistics 12: 1417–1440. doi:10.1162/tacl_a_00713. Self-correction succeeds mainly in tasks with reliable external feedback.
  9. Gou Z, Shao Z, Gong Y, Shen Y, Yang Y, Duan N, Chen W (2024). CRITIC: Large language models can self-correct with tool-interactive critiquing. ICLR 2024. arXiv:2305.11738. Critiques grounded in tool outputs improved answers; without the tools, most of the gain disappeared in question answering and toxicity reduction, though not in mathematical program synthesis.
  10. Gloaguen T, Mündler-Sasahara N, Müller MN, Raychev V, Vechev M (2026). Evaluating AGENTS.md: are repository-level context files helpful for coding agents? Preprint. arXiv:2602.11988. Context files written by a model did not improve task success and raised inference cost by more than 20%; files written by developers helped slightly, at up to 19% more cost.
  11. ASD Simplified Technical English Maintenance Group (2025). ASD-STE100 Simplified Technical English, Issue 9. Aerospace, Security and Defence Industries Association of Europe. https://www.asd-ste100.org/. A controlled vocabulary and set of writing rules for technical documentation, first developed for aircraft maintenance.
  12. International Organization for Standardization (2023). ISO 24495-1:2023 Plain language — Part 1: Governing principles and guidelines. https://www.iso.org/standard/78907.html. Four principles (relevant, findable, understandable, usable), with success judged by how readers use the document.
  13. Shubert SK, Spyridakis JH, Holmback HK, Coney MB (1995). The comprehensibility of Simplified English in procedures. Journal of Technical Writing and Communication 25(4): 347–369. doi:10.2190/WG69-D74B-4DLL-2WBK. Simplified English significantly improved the comprehensibility of more complex procedures for human readers.
  14. Xiao R, Ma W, Wang K, Wu Y, Zhao J, Wang H, Huang F, Li Y (2024). FlowBench: Revisiting and benchmarking workflow-guided planning for LLM-based agents. Findings of EMNLP 2024, pp. 10883–10900. doi:10.18653/v1/2024.findings-emnlp.638. Across 51 scenarios, GPT-4o's session success was 35.1% without workflow knowledge and 41.7–43.2% with it as text, code or a flowchart.
  15. Tsuchida R, Fan Y, Shimari K, Kula RG, Matsumoto K (2026). Do coding agents follow AGENTS.md? The effect of policy wording on instruction compliance. Preprint. arXiv:2611.04367. One policy in four wordings on a set of 86 coding tasks; with the more capable of two Claude models, average compliance was 78.0% for the direct wording and 64.2% for the conditional one.
  16. Anschütz M, Damaratskaya A, Lee CJ, Schmalz A, Mosca E, Groh G (2025). (Dis)improved?! How simplified language affects large language model performance across languages. Proceedings of the GEM² Workshop, pp. 847–861. https://aclanthology.org/2025.gem-1.70/. Automatically simplified task texts mostly lowered model performance, by up to 20 points in English.
  17. abryfs (2026). agent-ste. GitHub repository, commit 7a2b81f (11 August 2026). https://github.com/abryfs/agent-ste. Author-reported benchmark: reductions in measured STE violations averaged 95.5% across twelve models on eight writing tasks; downstream task execution is not measured.
  18. Simpson SG (2010). The Gödel hierarchy and reverse mathematics. In Feferman S, Parsons C, Simpson SG (eds.), Kurt Gödel: Essays for his Centennial. Lecture Notes in Logic 33, Cambridge University Press, pp. 109–127. doi:10.1017/CBO9780511750762.008. Reports that several hundred theorems of core mathematics fall into the five classes, and that artificial theories can be incomparable while natural ones line up.
  19. Patey L (2016). The reverse mathematics of Ramsey-type theorems. arXiv:1601.04428. Studies theorems outside the Big Five, such as Ramsey's theorem for pairs, and asks (§2.3) whether the Big Five pattern reflects a bias in the statements mathematicians study.
  20. Cowan N (2001). The magical number 4 in short-term memory: a reconsideration of mental storage capacity. Behavioral and Brain Sciences 24(1): 87–114. doi:10.1017/S0140525X01003922. Estimates the capacity of short-term memory at about four chunks.
  21. Hájek P, Pudlák P (1993). Metamathematics of First-Order Arithmetic. Perspectives in Mathematical Logic, Springer. Shows that truth for formulas of a given quantifier complexity is definable at that same complexity, while Tarski's theorem rules out a single truth definition for all formulas.
  22. Nemoto T, Ould MedSalem M, Tanaka K (2007). Infinite games in the Cantor space and subsystems of second order arithmetic. Mathematical Logic Quarterly 53(3): 226–236. doi:10.1002/malq.200610041. Determinacy of open games on the binary tree (Cantor space) is equivalent to WKL₀ over RCA₀.
  23. Keisler HJ (2006). Nonstandard arithmetic and reverse mathematics. Bulletin of Symbolic Logic 12(1): 100–125. doi:10.2178/bsl/1140640945. Gives a nonstandard counterpart of each Big Five system that is conservative over the corresponding standard system.
  24. Rathjen M (2017). Proof theory of constructive systems: inductive types and univalence. In Feferman on Foundations, Springer, pp. 385–419. doi:10.1007/978-3-319-63334-3_14; arXiv:1610.02191. Martin-Löf type theory with universes but without W-types, with or without univalence, has the strength of ATR₀ (ordinal Γ₀); with W-types it reaches much stronger set theories.
  25. Eastaugh B (2024). Reverse mathematics. Stanford Encyclopedia of Philosophy, Metaphysics Research Lab, Stanford University. https://plato.stanford.edu/entries/reverse-mathematics/. Survey of the Big Five, their characteristic theorems and proof-theoretic ordinals (ε₀ for ACA₀, Γ₀ for ATR₀).
  26. Kumar A, Zhuang V, Agarwal R, Su Y, Co-Reyes JD, Singh A, Baumli K, Iqbal S, Bishop C, Roelofs R, Zhang LM, McKinney K, Shrivastava D, Paduraru C, Tucker G, Precup D, Behbahani F, Faust A (2024). Training language models to self-correct via reinforcement learning. ICLR 2025. arXiv:2409.12917. Trains a model to improve its own answer over two attempts, without external feedback at test time.
  27. Hong S, Zhuge M, Chen J, Zheng X, Cheng Y, Zhang C, Wang J, Wang Z, Yau SKS, Lin Z, Zhou L, Ran C, Xiao L, Wu C, Schmidhuber J (2023). MetaGPT: meta programming for a multi-agent collaborative framework. ICLR 2024. arXiv:2308.00352. Encodes the standard operating procedures and roles of a human software team into prompt sequences for a team of agents.
  28. Vijayaraghavan G, Jayachandran P, Murthy A, Govindan S, Subramanian V (2026). If you want coherence, orchestrate a team of rivals: multi-agent models of organizational intelligence. Preprint. arXiv:2601.14351. Describes a production system that reuses a corporate structure of planners, executors and critics with strict role boundaries.
  29. Haynes AB, Weiser TG, Berry WR, Lipsitz SR, Breizat AH, Dellinger EP, Herbosa T, Joseph S, Kibatala PL, Lapitan MC, Merry AF, Moorthy K, Reznick RK, Taylor B, Gawande AA; Safe Surgery Saves Lives Study Group (2009). A surgical safety checklist to reduce morbidity and mortality in a global population. New England Journal of Medicine 360(5): 491–499. doi:10.1056/NEJMsa0810119. In eight hospitals, 3,733 patients before and 3,955 after a checklist program: deaths fell from 1.5% to 0.8% and complications from 11.0% to 7.0%. The program included training and team briefings; the authors call the mechanism multifactorial and name the before-and-after design as a limit.
  30. Urbach DR, Govindarajan A, Saskin R, Wilton AS, Baxter NN (2014). Introduction of surgical safety checklists in Ontario, Canada. New England Journal of Medicine 370(11): 1029–1038. doi:10.1056/NEJMsa1308261. In 101 hospitals, adjusted mortality was 0.71% before and 0.65% after adoption (P = 0.13) and complications 3.86% and 3.82% (P = 0.29). The province required public reporting of checklist use; no formal team training was required.
  31. Southern MG (2026). OpenAI founding member: make LLMs write like an aircraft manual. Search Engine Journal, 2 October 2026. https://www.searchenginejournal.com/karpathy-llm-aircraft-manual-writing/591813/. Reports Andrej Karpathy's post recommending that a model explain things in ASD-STE100, or 80% of the way to it, so that people can read its output more easily.
  32. Ghinda L (2026). Explain to me in Simple Technical English. All About Coding, 14 August 2026. https://allaboutcoding.ghinda.com/explain-to-me-in-simple-technical-english. A developer's test on four pieces of code, which the author calls not rigorous: against 47 facts kept without a style instruction, Claude lost 4 with a loose instruction and 22 with full ASD-STE100.
  33. Yu C, Yin L, Fan Q, Yu Y, Zhong RR, Li M (2026). Compile, then page: executable SOP programs and a capability-gated runtime for procedural LLM agents. Preprint. arXiv:2607.11346. On a screened set of 125 SOPBench bank cases with six models, pseudo-code compiled from the procedures' machine-readable constraints, with added verification steps, raised success by 0 to 16.0 points over the official prose; showing only the active step helped the strongest models and harmed the two weakest.
  34. Xu X, Guo H, Shen G, Cheng S, Zhang X (2026). Natural-language workflows are not software yet: artifact-driven compilation for reliable agent execution. Preprint. arXiv:2608.21341. Compiling workflows so that each step declares the artifacts it reads and writes raised the average task resolve rate from 62%, 46% and 60% to 85%, 82% and 85% on three executor models (488 instances, 11 workflows); a clearer text rewrite reached 66%, 46% and 76%. The compiler also adds checks, explicit control flow, optimization and validation, so the gain belongs to that whole package.
  35. Yuan S, Song K, Chen J, Tan X, Shen Y, Ren K, Li D, Yang D (2024). EasyTool: enhancing LLM-based agents with concise tool instruction. NAACL 2025. arXiv:2401.06201. Rewriting tool documentation into concise instructions with parameter guidance and examples cut it from 2,530 to 748 tokens on ToolBench and raised ChatGPT's answer-success score, as judged by GPT-4 and averaged over two test sets, from 15.0% to 52.8%.

Our records:

  • Biomedical Literature Evidence 1.0.0-public.1 validation record: 19 unittest methods passed; 3 of 3 RIS and CSL-JSON records retained as unverified; synthetic completed-insufficient ledger check (completed: true, support_ready: false).
  • Mendelian randomization 1.0.0-public.2 reproduction report: Walker et al. 2019 v2, case 2 (157 of 163 archived rows retained, 19 of 19 headline cells matched); 54 of 54 finite routing, SE, odds-ratio and Benjamini–Hochberg family checks verified in the audited Big Five benchmark (internal benchmark record, no public record yet).
  • Meta-analysis 1.0.0 performance report: 28 open-access Cochrane reviews, 1,060 of 1,061 pooled analyses and 10,797 of 11,351 checkable quantitative items reproduced (95%); ma_validate.py patched to reject NaN/±Inf and verified on 32 stdlib unit tests plus 35,130 testable signed_z rows (5,106 excluded for missing values) and stored Holm counts (internal benchmark record, no public record yet).
  • Scientific Figure Master 1.1.2 skill page: 61 of 61 scored caption fields correct across 20 real captions; general OOXML word/charts/chart1.xml and chart2.xml verifier caught both Fig. S2 (OA-10 = 0.9108 < 1.6225) and Fig. S3 (NON-OA2 and OA-1 vs. OA-1 and OA-8) caption-to-chart outlier contradictions (internal benchmark record, no public record yet).
  • Grant Writing Master 7.0.0-public.1 validation record and fact-preservation record: null pilot results and non-causal language preserved across revisions and languages; SHA-256 package provenance and character/byte measurements verified (internal benchmark record, no public record yet).
  • Scientific Writing 1.1.0 release notes: complete manuscript revision comparison with separate editorial decision record; audit_numbers.py upgraded with stdlib .xlsx XML value-versus-formula separation and explicit 563 = 209 + 354 DEG prose-to-CSV binding (internal benchmark record, no public record yet).
  • Academic Peer Review 1.0.2 skill page, benchmark and PeerJ 20995 blind review: calibrated against 1,035 comments from 60 open reviews; screen.py upgraded with stdlib .docx extraction, sustained monotonic margin-number removal and Methods-stratified animal-sex screening (internal benchmark record, no public record yet).
  • Big5c SOPBench pilot (internal record; the full study will appear on arXiv): 185 cases across 20 procedures in seven SOPBench domains, on GPT-5.6 Luna and Gemini 3.8 Flash low, comparing generic reflection, flat verification and Big5c on cases handled correctly and tokens per case (184 Gemini cases with complete token records).
  • Big5c skill-repair pilot (internal record; the full study will appear on arXiv): six synthetic Python-backed research workflows, each run with a working and a faulty helper script (12 tasks per model), on GPT and Gemini; Big5c and generic reflection each completed 12 of 12 tasks and repaired 6 of 6 faulty helpers, and Big5c used 34.0% fewer tokens on GPT and 22.7% fewer on Gemini across the 12 tasks. The saving came from the six runs with a working helper, where a passing check skips the critique.