In an earlier essay we argued that the rules for auditing research (peer review, literature synthesis, meta-analysis, causal inference, figure checking and grant review) should run as software while the work is being done, rather than years after publication. A fair question follows at once. Across biology, clinical medicine and statistics there seem to be thousands of separate rules: check sample sizes, check unit conversions, check multiple-testing corrections, check whether an antibody binds its target, check whether a control group was run at every time point, check whether a confounder explains the difference between two cohorts. If every error needs its own hand-written rule, then any software that audits science is only a growing patchwork of checklists, always one step behind the next mistake.
Mathematics faced the same worry fifty years ago, and discovered that the patchwork is an illusion. When Harvey Friedman, and later Stephen Simpson and many colleagues, began asking which axioms are actually needed to prove the theorems of ordinary mathematics—a program called reverse mathematics—nothing guaranteed a tidy answer [1] [2]. Logicians can easily build artificial axiom systems that cannot even be compared with one another [18]. Yet when they measured theorems of real analysis, algebra and combinatorics, several hundred of them turned out to be provable in a weak base system or equivalent to one of four stronger ones: just five classes, ordered from weakest to strongest like rungs on a ladder [2] [18]:
Logicians call those five systems the Big Five [18]. A few famous theorems, such as Ramsey's theorem for pairs, sit outside the ladder, and they are studied for exactly that reason [19]. Why would theorems invented across different centuries and cultures cluster into the same five boxes? Nobody yet knows. One open possibility, raised by Ludovic Patey, is that the pattern reflects a bias in which statements mathematicians choose to study [19]. We read it as a kind of anthropic principle: if all possible logical propositions form a vast high-dimensional space, human mathematicians explore only the tiny corner that a human mind can hold, check and pass on to a student.
This essay starts with the intuitive foundations of the Big Five—through the two-player games, monster functions, "lossy zip files" and mathematical plot twists that make reverse mathematics one of the most entertaining subjects in modern logic—and then shows how the five cognitive moves behind the Big Five guide the verification architecture across our seven research skills on nexitia.ai: the biomedical-literature-evidence skill, the Mendelian randomization skill, the meta-analysis skill, the scientific-figure-master skill, the grant-write-master skill, the scientific-writing skill and the academic-peer-review skill.
Why theorems get hard: quantifier alternation as a two-player game
Before climbing the Big Five ladder, ask a simpler question: what makes one mathematical statement harder for a human brain—or an AI agent—to verify than another?
In mathematical logic there are only two quantifiers: the universal quantifier ("for all…", "for every…") and the existential quantifier ("there exists at least one…"). Stacking the same quantifier over and over costs almost no mental effort, because identical quantifiers pack into a single draw:
Picking three numbers in space at once is no harder than picking one. The trouble starts the moment the quantifiers alternate ( followed by , then , then ). Every alternation turns a static statement into a two-player adversarial game:
- Every is your opponent making a move, going to extremes to pick the worst, most awkward case against your claim.
- Every is your side countering the move, choosing a response that resolves the crisis based on every move the opponent has made so far.
Count how fast the human brain heats up as the number of flips grows:
- 0 flips (): No game at all; you read the numbers as they sit on the page.
- 1 flip (): "For any number you name, there is a larger one." Even an elementary-school student can win this one-round game: whatever the opponent plays, you answer .
- 2 flips (): This is the - definition of a limit, in the form Weierstrass gave it, the exact point where college freshmen get their first real headache in calculus: Read it as a three-step rally: Round 1 (), the opponent demands an error tolerance ; Round 2 (), you calculate a safe neighbourhood width tailored to that ; Round 3 (), the opponent picks any point inside your -window and checks whether holds. Two flips () already sit near the edge of untrained intuition.
- 3 flips (): The Weierstrass monster function. In the nineteenth century Karl Weierstrass shocked mathematicians by exhibiting a function that is continuous everywhere yet has a derivative nowhere—a jagged sawtooth at every scale. Writing "no point has any tangent slope " takes three quantifier flips: Watch the four-move match play out. Move 1 (Opponent): picks and insists the tangent slope there must be . Move 2 (You): inspect the oscillation frequency near and lock in an error floor, say , declaring that the secant slope will miss by at least no matter how closely the opponent zooms in. Move 3 (Opponent's counterattack): zooms in to an absurdly tiny microscope window , betting the curve must smooth out locally. Move 4 (Your checkmate): inside , you point to a specific step where the curve still wiggles steeply enough to knock the secant slope away from by more than . Your winning move depends simultaneously on four nested choices . At three flips, a picture stops being a reliable guide; intuition often guesses wrong, and step-by-step symbolic verification is what settles the question.
- 4 flips (): Egorov's theorem and conceptual zip files. Egorov's theorem in measure theory says that if a sequence of measurable functions converges at almost every point of , then after discarding an exceptional "bad" set of arbitrarily small measure , the sequence converges uniformly on the rest: Here the bad set depends on , and the cutoff index depends on and on . Human working memory holds roughly four chunks at once [20]; by our own loose analogy, four quantifier flips is where even a seasoned mathematician staring at scratch paper starts to feel their head spin.
How did mathematics keep moving forward past four flips? By encapsulating concepts—packing the back half of a quantifier chain into a named black box, like a zip file. Once you package the tail into a single phrase—"converges uniformly outside "—Egorov's four-flip monster collapses back into a polite one-flip sentence: . For two centuries mathematics climbed by stacking these zip files: Cauchy and Weierstrass packed infinitesimal arguments into limits; Poincaré, Fréchet and Hilbert packed limits into topology and spaces of functions; Alexander Grothendieck packed geometric spaces into schemes and topoi, and built a fresh 1-2-3-4 ladder on top.
Yet conceptual compression is lossy for the reader: the definitions keep every detail, but a name hides the dependencies underneath it. You can hand a colleague a sleek one-line theorem sitting atop seven nested definitions, and it looks like a zero-flip equation. The moment a referee tries to audit that theorem, however, they have to unzip all seven layers at once, and the hidden quantifier dependencies come crashing back like an avalanche. Worse, as Alfred Tarski proved in the 1930s with his Undefinability Theorem, no formula of arithmetic can define truth for all formulas of arithmetic. Truth can be defined for one restricted class of formulas at a time, level by level, but no single formula checks every level at once [21]. You cannot spiral into the sky by stepping on your right foot with your left foot: every check covers a stated scope and no more, and that is the intuition we carry onto the ladder below.
Why the escape routes keep circling back to the Big Five
What makes reverse mathematics funny—almost like a mathematical ghost story where every path through the woods leads back to the same cabin—is what happened each time twentieth-century mathematics was rebuilt on a new foundation. We tell it as three escape attempts:
- Escape attempt 1: Infinite games and winning strategies. Logicians tried replacing an infinite chain of alternating moves with Skolem functions , , which record how one player answers every earlier move. That turns the chain into an infinite game of perfect information, and once the game has a stated winning set it raises a single question: does one of the two players have a winning strategy ? Yet a strategy that maps every finite history to a next move is itself an infinite object, one choice among possible strategies. When logicians measured the axioms needed to guarantee that a winning strategy exists, the Big Five were waiting at the finish line: open games played on a binary tree land on [22], while on the full tree of natural numbers, open or even clopen games land on and games whose winning set is an open set intersected with a closed set land on [2].
- Escape attempt 2: Abraham Robinson's hyperreal infinitesimals (). In the 1960s nonstandard analysis promised to abolish the two-flip - headache of calculus by letting mathematicians divide by actual infinitesimals as if they were schoolroom fractions. When H. Jerome Keisler measured nonstandard arithmetic against the ladder, he built a nonstandard counterpart for each of the Big Five, and each counterpart proves no new standard theorems beyond its own rung; for example, his nonstandard version of is conservative over [23]. The same rungs, wearing a fresher coat.
- Escape attempt 3: Type theory and Homotopy Type Theory (HoTT). Computer scientists replaced sets with types and programs ("propositions as types, proofs as programs"). When proof theorists measured the result, Martin-Löf type theory with its hierarchy of universes but without inductive -types, even with Voevodsky's univalence axiom added, turned out to have exactly the strength of , with the same proof-theoretic ordinal [24]. Adding -types lifts it far above every rung of the Big Five, which shows that the ladder describes ordinary mathematics rather than every system one can build.
No matter how you dress the notation—sets, games, infinitesimals or types—human and algorithmic verification keeps circling back to the same five rungs, and we read them, in our own interpretation, as five cognitive moves:
1. (Execution): What a computer can calculate step by step
The base rung, Recursive Comprehension Axiom (), guarantees only the sets that a Turing machine can compute from sets already in hand ( comprehension) [2] [3]. In mathematics, handles rational arithmetic, matrix algebra and the intermediate value theorem [2]. In scientific auditing, Execution is the domain of finite arithmetic, unit algebra and linear algebra: checking whether , whether spreadsheet cells reference the columns they claim to use, whether makes a sample covariance matrix singular (), or what minimum detectable effect () a sample size of can support.
2. (Choice & Compactness): Threading a complete path through a branching maze
The second rung adds Weak König's Lemma (): every infinite binary tree that has nodes at every depth contains at least one unbroken path from the root through every level [2] [4]. A finite algorithm looking one level at a time cannot always tell which branch hits a dead end fifty steps down; is the principle of compactness that bridges local step-by-step compatibility to a complete global path (and is equivalent over to the Heine–Borel theorem and Gödel's completeness theorem) [2]. In an experimental study, the design is a branching tree of choices (time points genotypes mRNA/protein readouts group labels). Claiming a continuous aging trajectory or a drug rescue asserts that the evidence can be followed through every level of that tree without a gap or a contradiction—not a path that silently drops the 6- and 12-month branches, reverses direction between mRNA and protein, or swaps group labels (YY-Y, YO-Y) between the figure legend and the Methods.
3. (Stabilization & The Halting Jump): Collecting the limit of an entire search family
The third rung, Arithmetical Comprehension Axiom (), lets you form any set defined using quantifiers over the natural numbers; over this is equivalent to comprehension [2]. In computability theory, it amounts to granting, for any set, an oracle for its version of Alan Turing's Halting Problem (the Turing jump ) [2] [3]; in analysis, it is equivalent to the Bolzano–Weierstrass theorem that every bounded sequence condenses onto a limit point [2]. An script can watch a researcher run test after test, but it takes a Stabilization rule, named by analogy with , to step outside the loop and collect the entire family of tested hypotheses as a single object—checking how many of 40,236 gene models survive Holm family-wise error control, whether constant-sum normalisation across 61 metabolites forces spurious negative covariances, or whether selecting top features before leave-one-out cross-validation leaks the test fold into training.
4. (Stratification): Stacking hierarchies along a ruler without mixing levels
The fourth rung, Arithmetical Transfinite Recursion (), allows comprehension jumps to be iterated along any countable well-ordering (ordinals climbing far past Gerhard Gentzen's , the ordinal of ordinary arithmetic); its proof-theoretic ordinal is , the Feferman–Schütte ordinal often called the limit of predicativity [5] [25]. Its cognitive move is Stratification: keeping ordered levels of construction strictly apart so a lower level cannot masquerade as a higher one. In a scientific manuscript, evidence lives on ordered strata:
- Document sections:
References (cited literature)IntroductionMethods (what this study actually ran)ResultsDiscussion. - Biological levels:
In silico pilotImmortalised cell lineSingle-sex animal cohortCross-species ecological meanHuman clinical cohort.
The order records how far each level stands from the claim being made, not how strong its evidence is. When a naive keyword script searches a whole PDF for "male" and "female" to check whether an animal study used both sexes, a paper title in the References section that mentions "male" can silence the warning on a Methods section that used only female mice. Stratifying the document by section before checking rules stops lower-stratum text from polluting higher-stratum verification.
5. (Self-Reflection): Surveying the whole space of competing explanations
The fifth rung, Comprehension (), adds impredicative comprehension, forming sets defined by universal quantification over all subsets () [2]. Its cognitive move is Self-Reflection: testing whether a claim survives not just one chosen path, but every competing set in the surrounding space. In science, any causal claim asserts that no competing clinical confounder (age, BMI, surgical harvest route), co-eluting enantiomer (D-galactose versus L-galactose) or horizontal pleiotropy pathway explains the observation equally well.
Mapping the five moves onto our seven skills—and what an adversarial audit taught us
Of course, a Python verification script running on a finite manuscript does not literally solve the Halting Problem or quantify over uncountably many subsets of [2]. Rather, the five moves of the Big Five (Execution, Choice, Stabilization, Stratification and Self-Reflection) expose why first-generation verification scripts miss so many scientific errors: almost every one of our baseline scripts sits on the syntactic surface of Rung 1 (), checking whether a number appears in a source pool or whether a URL resolves, without checking tree completeness, hypothesis-family multiplicity, section stratification or competing confounders.
Here is how those five moves map onto each of the seven skills on nexitia.ai, and what we found when we subjected both our baseline scripts and our Big Five verification suite to a two-pass benchmark and an independent adversarial code audit across six real research repositories:
- biomedical-literature-evidence skill:
- Existing public record: Logs PubMed/PMC queries, separates studies from overlapping cohort articles, holds imported RIS/CSL-JSON citations as unverified until checked (3 of 3 in its validation record), and allows a review ledger to conclude that evidence is insufficient.
- Big Five mapping: Execution () parses physical units so particle counts (
particles/g,MPs/g,filaments/individual) are never pooled as if they were pyrolysis-GC/MS mass concentrations (ug/g); Stratification () refuses silent fallback substitutions when a conversion factor is missing; and Self-Reflection () flags retracted sources before synthesis.
- Mendelian randomization skill:
- Existing public record: Retains 157 of 163 archived association rows for estimation and matches all 19 headline comparison cells in Walker et al. 2019 (reproduction report), flagging a prose-versus-Table-4 discrepancy.
- Big Five mapping: Execution () verifies finite standard errors, -statistics and outcome-scale-specific odds-ratio transforms; Choice () enforces deterministic estimator routing ( Wald ratio, IVW, IVW + weighted median + MR-Egger); Stabilization () recomputes Benjamini–Hochberg false-discovery-rate -values across both the full planned exposure family () and the core family () simultaneously (
54 of 54finite routing and multiplicity checks verified in our benchmark); and Stratification / Self-Reflection (–) explicitly marks unexecuted Steiger directionality, dilution and multivariable conditioning asunavailablerather than letting a report imply they passed.
- meta-analysis skill:
- Existing public record: Reproduces 1,060 of 1,061 pooled analyses and 10,797 of 11,351 checkable quantitative items (95%) across 28 Cochrane reviews (performance report).
- Big Five mapping: Execution () rejects
NaN/±Infinputs before domain comparisons (a bug we found and patched inma_validate.py), extracts.xlsxworksheet XML (xl/worksheets/sheet*.xml) while keeping cached values strictly separate from formula strings, and verifiessigned_z == local_beta / local_seacross all 35,130 testable rows of a 40,236-row cross-species gene table (the other 5,106 rows lack a value and are recorded as excluded, not passed); Stabilization () checks that stored Holm multiplicity pass counts (0 of 40,236) match the summary ledger.
- scientific-figure-master skill:
- Existing public record: Parses 61 of 61 scored caption fields across 20 real captions (skill page) and refuses to draw a significance star when a recomputed -value is .
- Big Five mapping: Execution & Choice (–) inspect embedded Office Open XML chart caches (
word/charts/chart*.xml) directly inside submitted Word files and compare the complete set of plotted outliers against the figure caption. In our adversarial audit of one metabolomics supplement, this general chart-cache parser uncovered a remarkable double contradiction without needing OCR or human referee notes: inFig. S2(chart1.xml), the caption claimed samplesOA-5,OA-7andOA-10exceeded the critical threshold (1.6225), yet the saved chart cache showedOA-10sitting at0.9108, well below the line; inFig. S3(chart2.xml), the caption namedOA-1andOA-8above the critical threshold (2.2026), whereas the chart data actually exceeded the threshold atNON-OA2andOA-1.
- grant-write-master skill:
- Existing public record: Preserves null pilot results () and non-causal wording across revisions and languages (validation record, fact-preservation record).
- Big Five mapping: Execution () measures character, UTF-8 byte and East-Asian display-width counts without inventing unspoken portal caps, and verifies SHA-256 package provenance against
PROVENANCE.json; Choice & Self-Reflection (–) structure the Hypothesis DAG so an immortalised cell line is never left as the sole branch supporting an in vivo claim and parallel fellowship/project grants keep distinct scopes.
- scientific-writing skill:
- Existing public record: Screens reported test statistics against source files and keeps editorial notes out of the publication file (release notes).
- Big Five mapping: Execution () binds prose differential-expression claims (
563 DEGs: 209 up-regulated and 354 down-regulated) directly to the computed counts in the supplementary CSV table, and upgradesaudit_numbers.pywith a zero-dependency.xlsxXML reader that inspects all worksheets and shared strings without letting cell coordinates (U2,1000) pollute the numeric evidence pool; Choice () pairs mRNA and protein readouts by analyte, contrast and time point so nonsignificant protein data are reported as inconclusive rather than a confirmed rescue.
- academic-peer-review skill:
- Existing public record: Calibrated against 1,035 reviewer comments across 60 open reviews; caught 6 of 49 row-level reviewer points and demanded 5 of 9 v1-to-v2 changes on a blind wet-lab submission (skill page, benchmark, PeerJ 20995 blind review).
- Big Five mapping: We backported three structural upgrades directly into
screen.py: a zero-dependency.docxXML reader, a conservative margin line-number normaliser (requiring a sustained consecutive sequence before it removes apparent margin numbers, so an isolated number at a line break is kept), and section stratification that restricts animal-sex screening to theMethods(or pre-Referencesbody) and flags single-sex scope separately from missing sex reporting, preventing bibliography titles inReferencesfrom masking single-sex experiments.
The checks catch the specific errors they were built for. Turning that structure into better decisions while an AI agent is still working is the question the next section takes up.
What an AI agent needs in order to check its own work
Consider an agent handling a product exchange. It can look up the order and still miss the question that decides the case: does the item the customer wants to exchange meet the conditions in the policy? The five rungs above were built to audit a finished document. Applied to that one exchange, they would read as follows: compute the order's dates and totals (Execution); follow the branch of the policy that applies to this kind of item (Choice); look at every item and earlier return on the order together (Stabilization); keep what the customer says apart from what the system records (Stratification); and ask whether any exception in the policy would change the answer (Self-Reflection). Here we asked whether the same structure helps an agent while it is still working, and what the evidence from outside our own studies says.
That evidence is clear on one point. Simply asking a model to reconsider its answer can fail badly: in one test, GPT-3.5 fell from 75.8% to 38.1% correct on CommonsenseQA after being asked to self-correct, with no new information to go on [7]. Self-correction becomes more reliable when the agent can consult something outside itself, such as a search engine, a code interpreter or a test that returns a verdict [8] [9]; training a model specifically to correct itself is another route [26]. An external check is useful because it brings in evidence the model does not already have.
The same evidence warns against assuming that more structure is always better. The authors of SOPBench, the benchmark we use below, gave four models an explicit checklist of the verifications each procedure requires; pass rates fell for two of them, and one model's unsafe actions rose from 21 cases to 74 [6]. Even when they handed the agent the true verdict for every constraint, a substantial share of cases still failed, 12% to 38% across the four models, because the final choice of whether to act or refuse remained hard [6]. In coding agents, repository instruction files written by a model raised inference cost by more than 20% without raising success [10]. Instructions are not free: every added rule is more text the agent must read and weigh.
Three ways to review an action
Generic reflection asks the agent to reconsider its work and leaves it to the model to decide what needs checking. Flat verification gives the agent an explicit checklist of requirements to review. Big5c, short for Big-5 compiler, uses the five rungs described above to organize the review. For a proposed action, the agent checks which facts it has established, which choices those facts support, and which prerequisites remain open. When it detects a gap, it asks for a correction and checks the result again. In the exchange example, that means connecting the policy to the particular item in the order.
We used one benchmark for this pilot, SOPBench, which tests whether an agent can complete a customer request while following the rules that govern its actions. Its authors provide 830 validated cases covering 70 procedures, with code that checks the final state, each step, and whether the required verifications came before the action [6].
We tested both models on the same 185 cases covering 20 procedures. The cases came from online shopping (49), healthcare (40), banking (27), motor vehicle services (27), hotels (25), universities (13), and libraries (4). The table shows the share of cases handled correctly, including correct refusals, and the average tokens used per case.
| Review method | GPT-5.6 Luna success | Gemini 3.8 Flash low success | GPT tokens per case | Gemini tokens per case |
|---|---|---|---|---|
| Generic reflection | 63.8% (118/185) | 63.8% (118/185) | 58,178 | 105,821 |
| Flat verification | 64.9% (120/185) | 59.5% (110/185) | 57,198 | 104,921 |
| Big5c | 87.6% (162/185) | 88.1% (163/185) | 73,902 | 116,664 |
Token averages use the same cases for all three methods within each model. One Gemini case lacked a token count for generic reflection, leaving 184 cases for the Gemini cost comparison.
On GPT-5.6 Luna, Big5c improved success by 23.8 percentage points over generic reflection and 22.7 points over flat verification. Each additional successful case cost about 66,100 extra tokens compared with generic reflection and 73,600 compared with flat verification. On Gemini 3.8 Flash low, the gains were 24.3 and 28.6 percentage points, at about 45,300 and 41,600 extra tokens per additional success, respectively.
These costs divide the extra tokens used across all cases with complete token records by the net increase in successful cases on that same set.
Rows show service domains, with the number of cases in parentheses. Read across each row to compare the three review methods. Success rates use every case. Token means use cases with complete records for all three methods within each model.
What the pilot tells us
Giving an agent a checklist had different effects on the two models. Compared with generic reflection, flat verification yielded two more successes on GPT and eight fewer on Gemini. Big5c improved on both approaches on both models. One plausible explanation is that a checklist can tell an agent to verify eligibility, while a repair step gives it something to do when the evidence is missing. The pilot measured the whole workflow, so the contribution of each of the five rungs remains an open question.
Each bar is the share of the same 185 cases that one review method handled correctly; the counts are in parentheses.
The extra review used tokens even on cases where the outcome stayed the same. For a researcher choosing where to use it, the relevant question is how much an additional correctly handled case is worth. That judgment will depend on the procedure. Online shopping contributed 49 cases and libraries contributed four, so the overall rates tell us more about this particular mix of requests than about any single service.
Where did structured review save tokens? Gemini’s healthcare runs give the clearest example. Big5c and generic reflection each passed 39 of 40 cases, but Big5c used about 3,760 fewer tokens per case (3.7%). The traces record 150 tool calls with Big5c and 159 with generic reflection. A shorter sequence of calls can save tokens because the agent has fewer tool results to read as it works. That is one plausible explanation here. Compared with flat verification, the saving was only about 170 tokens per case (0.2%), with one additional success.
Tool calls explain only part of the pattern. On the four library cases, Gemini passed all four and made 15 calls with either Big5c or generic reflection, yet Big5c used 6.9% fewer tokens. The amount of text the agent reads and generates can also vary within the same number of calls. In banking, Gemini’s Big5c runs cost slightly less than generic reflection and more than flat verification. For GPT, Big5c used more tokens on average than either alternative in every domain. These differences suggest a question for the full study. When does the work saved by catching a missing prerequisite outweigh the cost of checking it?
A second, smaller pilot points to one answer. We ran six synthetic research workflows twice, once with a working helper script and once with a faulty one, which gives 12 tasks per model. Big5c and generic reflection both produced correct final answers on all 12 tasks and repaired all six faulty helpers, but across the 12 tasks Big5c used 34% fewer tokens on GPT and 23% fewer on Gemini. The saving came from the workflows whose helper already worked: when a check can be executed and it passes, the agent can skip the critique altogether.
Mean tokens per task across the 12 tasks per model, in thousands. Both methods produced correct answers on all 12.
Managing an AI agent the way we manage people
The pilot points to an idea that interests us more than any single number: the methods that organizations use to make human work reliable may make an AI agent's work reliable in a similar way. Others have built on the same idea. MetaGPT encodes the standard operating procedures and job roles of a software company into a team of AI agents [27]. A more recent system arranges its agents as planners, executors and critics with strict role boundaries, on the reasoning that AI agents, "just like all the humans we have ever hired", remain fallible [28].
Human quality management also offers a way to read our table. Asking the agent to reconsider its work, as generic reflection does, is the equivalent of telling a person to be more careful. A checklist alone helped one model and hurt the other. Surgery, too, has found that adopting a checklist does not bring the same improvement everywhere. When eight hospitals introduced the WHO surgical safety checklist as a program, with staff training and with team briefings before and after each operation, deaths after surgery fell from 1.5% of the 3,733 patients before the program to 0.8% of the 3,955 patients after it [29]. When 101 hospitals in Ontario adopted checklists as the province moved to require public reporting of their use, most of them with an educational program but with no formal team training required, the adjusted risk of death went from 0.71% to 0.65%, a change that was not significant, and the authors noted that, apart from the WHO study, the studies with large effects had either coupled the checklist with extensive team training or used a broader checklist covering the whole course of care [30]. Both are before-and-after comparisons rather than randomized trials, and neither isolates what made the difference, just as our pilot did not isolate the repair step. Together they raise the question we care about for AI agents: which practices around a checklist turn its checks into better decisions? Big5c is one answer under test, a checklist packaged with a repair step that follows a failed check.
Left: deaths after surgery before and after checklists were introduced in the two studies, which followed different populations; the Ontario rates are adjusted [29] [30]. Right: change in the number of correctly handled cases in our pilot compared with generic reflection, out of 185 per model.
The oldest management tool of all is the written procedure. Aviation solved a version of this problem decades ago. ASD-STE100, Simplified Technical English, restricts vocabulary and sentence structure so that maintenance technicians, many of them non-native readers, can read an instruction without ambiguity [11]. ISO 24495-1 sets out four principles for plain language: readers can find what they need, understand it and use it, and the content is relevant to them [12]. For human readers, Simplified English made complex procedures significantly easier to understand [13].
Many people now apply these standards to AI, mostly in one direction: they ask the model to write in Simplified Technical English so that a person can read its output more easily. Open tools already help apply the standard; one of them reports cutting measured violations by an average of 95.5% across eight writing tasks and twelve models [17]. Andrej Karpathy recommended the practice in early October 2026 [31]. Asking for the standard by name can also have a cost. In one developer's small test on four pieces of code, Claude's explanations lost 4 of 47 checked facts when asked for "Simple Technical English" and 22 of 47 when asked for ASD-STE100 [32].
The other direction, writing the procedure for the AI agent that must follow it, has stronger evidence. Giving an agent the workflow at all matters: on FlowBench, the share of sessions GPT-4o completed rose from 35.1% without the workflow to between 41.7% and 43.2% with it, whether written as text, code or a flowchart, across 51 scenarios [14]. On two ToolBench test sets, rewriting long tool documentation into concise instructions with parameter guidance and examples cut it to less than a third of its length and raised ChatGPT's answer-success score, as judged by GPT-4, from 15.0% to 52.8% [35]. Restructuring the procedure helps as well. On a screened set of 125 SOPBench banking cases, compiling the procedures' machine-readable constraints into structured pseudo-code with explicit verification steps improved success over the official prose by up to 16.0 percentage points across six models (from 70.4% to 86.4% for the largest gain) and lowered it for none of them [33]. A compiler that turns natural-language workflows into executable ones, with explicit inputs, outputs, checks and control flow for each step, raised task success from 56% to 84% on average across three models, on 488 cases from 11 workflows, while rewriting the same workflows as a clearer text document reached 63% [34]. Big5c takes the same route for review: it compiles what a procedure requires into checks on the five rungs, and adds a repair step for a check that fails.
Yet what helps a person does not always help an AI agent, and what helps one model can harm another. In the SOPBench study above, a runtime that showed the agent only the current step of the procedure helped two strong models and lowered GPT-4o-mini's success from 75.2% with the full compiled procedure to 48.8% [33]. Wording matters for obedience: when coding agents received one policy phrased four ways on a set of 86 coding tasks, the average compliance of the more capable of two Claude models was 78.0% with the direct wording and 64.2% with the conditional one ("if …, then …") [15], a tension to watch, because Simplified Technical English puts a condition that the reader must know first at the start of the sentence [11]. Simplification written for people does not automatically help models: when the input texts of benchmark tasks such as classification were simplified automatically, language-model performance fell by up to 20 points in English [16].
Each row is one reported result: the circle is the rate before the change and the arrow points to the rate after it. The studies use different tasks and measures, so rows should be read one at a time.
So we expect the management system to carry over to AI agents (written procedures, checks that come with the means to repair, and separate roles), while the wording rules written for human readers have to be tested again on agents. We found no controlled study that tests whether agent procedures rewritten explicitly to ASD-STE100 or ISO 24495-1 help the agent complete more tasks.
We will report the full Big5c study as an academic article on arXiv and release the Big5c engine on GitHub.
References
- Friedman H (1975). Some systems of second order arithmetic and their use. Proceedings of the International Congress of Mathematicians (Vancouver, 1974), Vol. 1, pp. 235–242. Canadian Mathematical Congress. Introduces the program of reverse mathematics, showing that ordinary theorems of analysis are equivalent over a weak base theory to specific set-existence axioms of second-order arithmetic.
- Simpson SG (2009). Subsystems of Second Order Arithmetic, 2nd ed. Perspectives in Logic, Cambridge University Press. doi:10.1017/CBO9780511581007. Defines second-order arithmetic and the five principal subsystems (, , , and ), establishing their relationships to computable mathematics, Heine–Borel compactness, Bolzano–Weierstrass convergence, ordinal hierarchies up to and impredicative set comprehension.
- Turing AM (1936). On computable numbers, with an application to the Entscheidungsproblem. Proceedings of the London Mathematical Society, Series 2, 42(1): 230–265. doi:10.1112/plms/s2-42.1.230. Defines computable sequences via machines with a finite control and an unbounded tape, and proves the unsolvability of the halting/decision problem that separates computable checking () from arithmetical comprehension ().
- König D (1927). Über eine Schlussweise aus dem Endlichen ins Unendliche. Acta Litterarum ac Scientiarum Regiae Universitatis Hungaricae Francisco-Josephinae, Sectio Scientiarum Mathematicarum 3: 121–130. Proves König's infinity lemma—that every infinite, finitely branching tree has an infinite path—whose binary-tree restriction forms the axiom of .
- Gentzen G (1936). Die Widerspruchsfreiheit der reinen Zahlentheorie. Mathematische Annalen 112: 493–565. doi:10.1007/BF01565428. Establishes the consistency of first-order arithmetic by transfinite induction up to the ordinal , foundational to the ordinal hierarchy analysis in proof theory and .
- Li Z, Huang S, Wang J, Zhang N, Antoniades A, Hua W, Zhu K, Zeng S, Wang C, Wang WY, Yan X (2026). SOPBench: Evaluating language agents at following standard operating procedures and constraints. Proceedings of EMNLP 2026, pp. 21561–21616. https://aclanthology.org/2026.emnlp-main.936/. 830 validated cases across 70 procedures in seven customer-service domains, scored on final state, each action and the order of required verifications; in its ablation, an explicit checklist of required verifications lowered pass rates for two of four models, and supplying true constraint verdicts still left 12–38% of cases failing.
- Huang J, Chen X, Mishra S, Zheng HS, Yu AW, Song X, Zhou D (2024). Large language models cannot self-correct reasoning yet. ICLR 2024. arXiv:2310.01798. Without external feedback, prompting a model to self-correct lowered accuracy, for example GPT-3.5 on CommonsenseQA from 75.8% to 38.1%.
- Kamoi R, Zhang Y, Zhang N, Han J, Zhang R (2024). When can LLMs actually correct their own mistakes? A critical survey of self-correction of LLMs. Transactions of the Association for Computational Linguistics 12: 1417–1440. doi:10.1162/tacl_a_00713. Self-correction succeeds mainly in tasks with reliable external feedback.
- Gou Z, Shao Z, Gong Y, Shen Y, Yang Y, Duan N, Chen W (2024). CRITIC: Large language models can self-correct with tool-interactive critiquing. ICLR 2024. arXiv:2305.11738. Critiques grounded in tool outputs improved answers; without the tools, most of the gain disappeared in question answering and toxicity reduction, though not in mathematical program synthesis.
- Gloaguen T, Mündler-Sasahara N, Müller MN, Raychev V, Vechev M (2026). Evaluating AGENTS.md: are repository-level context files helpful for coding agents? Preprint. arXiv:2602.11988. Context files written by a model did not improve task success and raised inference cost by more than 20%; files written by developers helped slightly, at up to 19% more cost.
- ASD Simplified Technical English Maintenance Group (2025). ASD-STE100 Simplified Technical English, Issue 9. Aerospace, Security and Defence Industries Association of Europe. https://www.asd-ste100.org/. A controlled vocabulary and set of writing rules for technical documentation, first developed for aircraft maintenance.
- International Organization for Standardization (2023). ISO 24495-1:2023 Plain language — Part 1: Governing principles and guidelines. https://www.iso.org/standard/78907.html. Four principles (relevant, findable, understandable, usable), with success judged by how readers use the document.
- Shubert SK, Spyridakis JH, Holmback HK, Coney MB (1995). The comprehensibility of Simplified English in procedures. Journal of Technical Writing and Communication 25(4): 347–369. doi:10.2190/WG69-D74B-4DLL-2WBK. Simplified English significantly improved the comprehensibility of more complex procedures for human readers.
- Xiao R, Ma W, Wang K, Wu Y, Zhao J, Wang H, Huang F, Li Y (2024). FlowBench: Revisiting and benchmarking workflow-guided planning for LLM-based agents. Findings of EMNLP 2024, pp. 10883–10900. doi:10.18653/v1/2024.findings-emnlp.638. Across 51 scenarios, GPT-4o's session success was 35.1% without workflow knowledge and 41.7–43.2% with it as text, code or a flowchart.
- Tsuchida R, Fan Y, Shimari K, Kula RG, Matsumoto K (2026). Do coding agents follow AGENTS.md? The effect of policy wording on instruction compliance. Preprint. arXiv:2611.04367. One policy in four wordings on a set of 86 coding tasks; with the more capable of two Claude models, average compliance was 78.0% for the direct wording and 64.2% for the conditional one.
- Anschütz M, Damaratskaya A, Lee CJ, Schmalz A, Mosca E, Groh G (2025). (Dis)improved?! How simplified language affects large language model performance across languages. Proceedings of the GEM² Workshop, pp. 847–861. https://aclanthology.org/2025.gem-1.70/. Automatically simplified task texts mostly lowered model performance, by up to 20 points in English.
- abryfs (2026). agent-ste. GitHub repository, commit 7a2b81f (11 August 2026). https://github.com/abryfs/agent-ste. Author-reported benchmark: reductions in measured STE violations averaged 95.5% across twelve models on eight writing tasks; downstream task execution is not measured.
- Simpson SG (2010). The Gödel hierarchy and reverse mathematics. In Feferman S, Parsons C, Simpson SG (eds.), Kurt Gödel: Essays for his Centennial. Lecture Notes in Logic 33, Cambridge University Press, pp. 109–127. doi:10.1017/CBO9780511750762.008. Reports that several hundred theorems of core mathematics fall into the five classes, and that artificial theories can be incomparable while natural ones line up.
- Patey L (2016). The reverse mathematics of Ramsey-type theorems. arXiv:1601.04428. Studies theorems outside the Big Five, such as Ramsey's theorem for pairs, and asks (§2.3) whether the Big Five pattern reflects a bias in the statements mathematicians study.
- Cowan N (2001). The magical number 4 in short-term memory: a reconsideration of mental storage capacity. Behavioral and Brain Sciences 24(1): 87–114. doi:10.1017/S0140525X01003922. Estimates the capacity of short-term memory at about four chunks.
- Hájek P, Pudlák P (1993). Metamathematics of First-Order Arithmetic. Perspectives in Mathematical Logic, Springer. Shows that truth for formulas of a given quantifier complexity is definable at that same complexity, while Tarski's theorem rules out a single truth definition for all formulas.
- Nemoto T, Ould MedSalem M, Tanaka K (2007). Infinite games in the Cantor space and subsystems of second order arithmetic. Mathematical Logic Quarterly 53(3): 226–236. doi:10.1002/malq.200610041. Determinacy of open games on the binary tree (Cantor space) is equivalent to WKL₀ over RCA₀.
- Keisler HJ (2006). Nonstandard arithmetic and reverse mathematics. Bulletin of Symbolic Logic 12(1): 100–125. doi:10.2178/bsl/1140640945. Gives a nonstandard counterpart of each Big Five system that is conservative over the corresponding standard system.
- Rathjen M (2017). Proof theory of constructive systems: inductive types and univalence. In Feferman on Foundations, Springer, pp. 385–419. doi:10.1007/978-3-319-63334-3_14; arXiv:1610.02191. Martin-Löf type theory with universes but without W-types, with or without univalence, has the strength of ATR₀ (ordinal Γ₀); with W-types it reaches much stronger set theories.
- Eastaugh B (2024). Reverse mathematics. Stanford Encyclopedia of Philosophy, Metaphysics Research Lab, Stanford University. https://plato.stanford.edu/entries/reverse-mathematics/. Survey of the Big Five, their characteristic theorems and proof-theoretic ordinals (ε₀ for ACA₀, Γ₀ for ATR₀).
- Kumar A, Zhuang V, Agarwal R, Su Y, Co-Reyes JD, Singh A, Baumli K, Iqbal S, Bishop C, Roelofs R, Zhang LM, McKinney K, Shrivastava D, Paduraru C, Tucker G, Precup D, Behbahani F, Faust A (2024). Training language models to self-correct via reinforcement learning. ICLR 2025. arXiv:2409.12917. Trains a model to improve its own answer over two attempts, without external feedback at test time.
- Hong S, Zhuge M, Chen J, Zheng X, Cheng Y, Zhang C, Wang J, Wang Z, Yau SKS, Lin Z, Zhou L, Ran C, Xiao L, Wu C, Schmidhuber J (2023). MetaGPT: meta programming for a multi-agent collaborative framework. ICLR 2024. arXiv:2308.00352. Encodes the standard operating procedures and roles of a human software team into prompt sequences for a team of agents.
- Vijayaraghavan G, Jayachandran P, Murthy A, Govindan S, Subramanian V (2026). If you want coherence, orchestrate a team of rivals: multi-agent models of organizational intelligence. Preprint. arXiv:2601.14351. Describes a production system that reuses a corporate structure of planners, executors and critics with strict role boundaries.
- Haynes AB, Weiser TG, Berry WR, Lipsitz SR, Breizat AH, Dellinger EP, Herbosa T, Joseph S, Kibatala PL, Lapitan MC, Merry AF, Moorthy K, Reznick RK, Taylor B, Gawande AA; Safe Surgery Saves Lives Study Group (2009). A surgical safety checklist to reduce morbidity and mortality in a global population. New England Journal of Medicine 360(5): 491–499. doi:10.1056/NEJMsa0810119. In eight hospitals, 3,733 patients before and 3,955 after a checklist program: deaths fell from 1.5% to 0.8% and complications from 11.0% to 7.0%. The program included training and team briefings; the authors call the mechanism multifactorial and name the before-and-after design as a limit.
- Urbach DR, Govindarajan A, Saskin R, Wilton AS, Baxter NN (2014). Introduction of surgical safety checklists in Ontario, Canada. New England Journal of Medicine 370(11): 1029–1038. doi:10.1056/NEJMsa1308261. In 101 hospitals, adjusted mortality was 0.71% before and 0.65% after adoption (P = 0.13) and complications 3.86% and 3.82% (P = 0.29). The province required public reporting of checklist use; no formal team training was required.
- Southern MG (2026). OpenAI founding member: make LLMs write like an aircraft manual. Search Engine Journal, 2 October 2026. https://www.searchenginejournal.com/karpathy-llm-aircraft-manual-writing/591813/. Reports Andrej Karpathy's post recommending that a model explain things in ASD-STE100, or 80% of the way to it, so that people can read its output more easily.
- Ghinda L (2026). Explain to me in Simple Technical English. All About Coding, 14 August 2026. https://allaboutcoding.ghinda.com/explain-to-me-in-simple-technical-english. A developer's test on four pieces of code, which the author calls not rigorous: against 47 facts kept without a style instruction, Claude lost 4 with a loose instruction and 22 with full ASD-STE100.
- Yu C, Yin L, Fan Q, Yu Y, Zhong RR, Li M (2026). Compile, then page: executable SOP programs and a capability-gated runtime for procedural LLM agents. Preprint. arXiv:2607.11346. On a screened set of 125 SOPBench bank cases with six models, pseudo-code compiled from the procedures' machine-readable constraints, with added verification steps, raised success by 0 to 16.0 points over the official prose; showing only the active step helped the strongest models and harmed the two weakest.
- Xu X, Guo H, Shen G, Cheng S, Zhang X (2026). Natural-language workflows are not software yet: artifact-driven compilation for reliable agent execution. Preprint. arXiv:2608.21341. Compiling workflows so that each step declares the artifacts it reads and writes raised the average task resolve rate from 62%, 46% and 60% to 85%, 82% and 85% on three executor models (488 instances, 11 workflows); a clearer text rewrite reached 66%, 46% and 76%. The compiler also adds checks, explicit control flow, optimization and validation, so the gain belongs to that whole package.
- Yuan S, Song K, Chen J, Tan X, Shen Y, Ren K, Li D, Yang D (2024). EasyTool: enhancing LLM-based agents with concise tool instruction. NAACL 2025. arXiv:2401.06201. Rewriting tool documentation into concise instructions with parameter guidance and examples cut it from 2,530 to 748 tokens on ToolBench and raised ChatGPT's answer-success score, as judged by GPT-4 and averaged over two test sets, from 15.0% to 52.8%.
Our records:
- Biomedical Literature Evidence 1.0.0-public.1 validation record: 19 unittest methods passed; 3 of 3 RIS and CSL-JSON records retained as unverified; synthetic completed-insufficient ledger check (
completed: true, support_ready: false). - Mendelian randomization 1.0.0-public.2 reproduction report: Walker et al. 2019 v2, case 2 (157 of 163 archived rows retained, 19 of 19 headline cells matched); 54 of 54 finite routing, SE, odds-ratio and Benjamini–Hochberg family checks verified in the audited Big Five benchmark (internal benchmark record, no public record yet).
- Meta-analysis 1.0.0 performance report: 28 open-access Cochrane reviews, 1,060 of 1,061 pooled analyses and 10,797 of 11,351 checkable quantitative items reproduced (95%);
ma_validate.pypatched to rejectNaN/±Infand verified on 32 stdlib unit tests plus 35,130 testablesigned_zrows (5,106 excluded for missing values) and stored Holm counts (internal benchmark record, no public record yet). - Scientific Figure Master 1.1.2 skill page: 61 of 61 scored caption fields correct across 20 real captions; general OOXML
word/charts/chart1.xmlandchart2.xmlverifier caught bothFig. S2(OA-10 = 0.9108 < 1.6225) andFig. S3(NON-OA2andOA-1vs.OA-1andOA-8) caption-to-chart outlier contradictions (internal benchmark record, no public record yet). - Grant Writing Master 7.0.0-public.1 validation record and fact-preservation record: null pilot results and non-causal language preserved across revisions and languages; SHA-256 package provenance and character/byte measurements verified (internal benchmark record, no public record yet).
- Scientific Writing 1.1.0 release notes: complete manuscript revision comparison with separate editorial decision record;
audit_numbers.pyupgraded with stdlib.xlsxXML value-versus-formula separation and explicit563 = 209 + 354DEG prose-to-CSV binding (internal benchmark record, no public record yet). - Academic Peer Review 1.0.2 skill page, benchmark and PeerJ 20995 blind review: calibrated against 1,035 comments from 60 open reviews;
screen.pyupgraded with stdlib.docxextraction, sustained monotonic margin-number removal andMethods-stratified animal-sex screening (internal benchmark record, no public record yet). - Big5c SOPBench pilot (internal record; the full study will appear on arXiv): 185 cases across 20 procedures in seven SOPBench domains, on GPT-5.6 Luna and Gemini 3.8 Flash low, comparing generic reflection, flat verification and Big5c on cases handled correctly and tokens per case (184 Gemini cases with complete token records).
- Big5c skill-repair pilot (internal record; the full study will appear on arXiv): six synthetic Python-backed research workflows, each run with a working and a faulty helper script (12 tasks per model), on GPT and Gemini; Big5c and generic reflection each completed 12 of 12 tasks and repaired 6 of 6 faulty helpers, and Big5c used 34.0% fewer tokens on GPT and 22.7% fewer on Gemini across the 12 tasks. The saving came from the six runs with a working helper, where a passing check skips the critique.