Ask a capable AI agent to run a standard analysis and it will usually write the code from scratch: load the data, normalise it, fit the model, and draw the figure. The code often looks right. Sometimes it is. Either way, the work is repeated for the next person who asks the same question, and the one after that, and each rewrite is a fresh chance to get a detail wrong. Even without AI, most researchers have tried and failed to reproduce someone else's result [1]; code rewritten for every run adds a new way to fail each time.
Our tools take a different route. A method is implemented once, checked against a published reference, and kept as a node: a fixed, versioned piece of code with the context an agent needs to use it. In CohortDAG and Tracyte, nodes sit on a canvas; in our skills, such as meta-analysis and Mendelian randomization, they are scripts behind a pipeline. In both, the agent's job is to choose and connect the steps, not to write the analysis. This essay explains why we think that is the better division of labour.
Pay for the method once
Each node starts from a reference we can check against, whether an official tutorial, a package's documentation, or a published paper. We pin the reference version, re-run it, rebuild the procedure as a node, and compare the outputs item by item. Agreement is recorded as exact, equivalent, approximate, or not reproduced, and any gap stays visible as a gap.
In CohortDAG, six study pipelines built from standard nodes reproduce the numbers of their source papers, matching 89 of 91 items to the printed digit and the other 2 within rounding. On a private clinical cohort, the rebuilt pipeline matched the original analysis notebook byte for byte. Executing and auditing every node against that notebook also surfaced 36 defects that the passing test suite had never caught, including a survival-curve median whose confidence bounds were swapped. Each became a required check in the pipeline validator.
The same rule governs our skills. When we checked the Mendelian randomization skill against Walker et al. 2019, it retained 157 instruments from the frozen archive of 163 associations and matched all 19 headline comparison cells. The meta-analysis skill re-ran every analysis of 28 open-access Cochrane reviews from their published data packages and reproduced 10,797 of 11,351 checkable quantitative items (95%), with the 554 misses listed.
This verification is expensive, and we pay for it deliberately, once. From then on, the node carries what an agent needs to use it without re-deriving the method: what it does, its inputs and outputs, its parameters, its scientific context, and its verification record. The code inside stays fixed for that version; any change creates a new version that must be checked again. This applies the old rule of reproducible computation, pinning exact versions and avoiding manual steps [2], to the agent's work as well as the researcher's.
The immediate saving is the tokens, debugging, and setup otherwise spent each time an agent reconstructs an existing method. For standard analyses that saving is large, because the method is most of the work.
Make fewer things the model can get wrong
A language model makes mistakes at a non-zero rate, and in a long block of generated code every line is another draw. When sixteen code-generating models were asked to write code, a measurable share of the packages they imported did not exist at all: at least 5.2% for commercial models and 21.7% for open-source ones, across 576,000 generated samples [3]. The longer a task would take a skilled person, the less often an agent completes it, and the drop is steep [4]. On the 20-disc Towers of Hanoi puzzle, Meyerson and colleagues make the arithmetic explicit [5]: when every step must be right, small per-step error rates compound until a long task is almost certain to fail somewhere. Their answer is to break the task into steps small enough to check and to check each one before moving on; with that, all 1,048,575 moves were played without a single error.
Composing verified nodes applies the same principle to data analysis. Because the procedure lives inside the nodes, the agent writes a short graph specification (which nodes to run, in what order, and with which settings) instead of thousands of lines of code. That narrows the surface where the model can err from every line of implementation to a handful of structural choices. Software checks those choices automatically, and a person reviews them on the canvas in a human-friendly GUI. In CohortDAG, the agent selects from a fixed schema of node types and parameters, and the graph is validated and compiled before anything runs, so an invented node type or a missing parameter is caught immediately rather than surfacing as a quiet error in the results.
The same boundary governs what a model is allowed to write in prose. In the meta-analysis skill, fixed R scripts compute the statistics, a results file supplies every number in the written review, and the writer refuses to emit a sentence whose value is missing. Limits of the evidence (a single study, fewer than ten studies, or one study carrying more than half the weight) are computed into flags before prose is generated; they are never left for the model to judge on the fly.
There is a second benefit, and it may be the larger one. Context the agent does not spend on writing boilerplate code is context it can spend on the science: what the data represents, what the researcher is testing, whether the method's assumptions hold for this dataset, and whether the result makes biological sense. A model that is not busy rewriting a normalisation routine has room to notice when samples are confounded with batch, when a parameter falls outside its validated range, or when an intermediate result looks wrong, and to investigate why. Reproducing a reference proves agreement within its original scope; judging whether the method fits a new question is where we want the model's attention.
Decide where the AI works
Not every step in research is mechanical. Some require judgement about meaning, such as screening abstracts, labelling cell clusters, reading free-text clinical notes, or inferring which samples belong together from file names. Getting that last step wrong can ruin a study: a differential-expression analysis that treats cells rather than samples as the unit of replication can report hundreds of false discoveries [6]. Tasks like these are where a language model is genuinely useful, and where it most needs guardrails.
We therefore treat AI as an explicit step of its own, not as glue spread across the workflow. An AI node is a node like any other, defined by three parts: a customised prompt that states its task, outputs, and limits; the skills and context it loads so that it follows a tested protocol rather than improvising one; and a quality-control gate where a person reviews its output before anything downstream runs. The node's output is recorded with the run, and rejecting it invalidates everything built on top of it.
A custom node holds the researcher's own code for any situation. Once a custom node or an AI node works, the researcher names and documents it, and from then on their agent can discover and reuse it like any built-in node. A lab's specific know-how becomes part of its working vocabulary.
This gives the researcher a dial. A workflow can be entirely deterministic software, or it can include one or several narrowly scoped AI steps. A broad semantic task can be split into smaller stages (classify, check, and summarise) so that each step is narrow enough to audit. The million-step study shows what that control buys [5]. Once the Towers of Hanoi was cut into its smallest steps and each step was checked before the next, a model that still made a mistake on roughly one step in 250 to 450 played all 1,048,575 moves without a single error. The arithmetic behind it says that the number of checks each step needs grows only with the logarithm of the task's length, so the approach keeps working as tasks get longer. The puzzle had a single right move at every step, which made a vote among sampled answers a reliable per-step check; research problems rarely do, which is why the review gate on each AI node is that check, and why the researcher, not the model, sets how small the steps are.
The meta-analysis skill shows the dial in practice. Screening and data extraction run in two independent passes, each in a fresh context window so that neither pass can see the other. Every AI decision is labelled as such in the audit ledger, and an AI screener counts as at most one of the two independent screeners a review requires. Named humans sign the gates between stages, and the software rejects an AI assistant as a signatory. The same rule determines where AI sits when a tool handles sensitive clinical data: the pipeline itself stays deterministic, and the agent operates from outside it through the command line and MCP server, composing tested nodes.
What comes next
The claim above is testable, and we are testing it. Across a benchmark of standard analyses, we are comparing an agent that composes verified nodes against an agent that writes the analysis code from scratch, measuring tokens consumed, wall-clock time, defects caught in review, and agreement with the reference. We will publish the benchmark here in future posts.
References
- Baker M (2016). 1,500 scientists lift the lid on reproducibility. Nature 533: 452–454. doi:10.1038/533452a. A survey of 1,576 researchers: more than 70% had tried and failed to reproduce another scientist's experiments.
- Sandve GK, Nekrutenko A, Taylor J, Hovig E (2013). Ten simple rules for reproducible computational research. PLoS Computational Biology 9(10): e1003285. doi:10.1371/journal.pcbi.1003285. Rules 2 and 3: avoid manual data manipulation; archive the exact versions of all external programs.
- Spracklen J, Wijewickrama R, Sakib AHMN, Maiti A, Viswanath B, Jadliwala M (2024). We have a package for you! A comprehensive analysis of package hallucinations by code generating LLMs. arXiv:2406.10279; USENIX Security 2025. Sixteen models, 576,000 code samples: on average at least 5.2% of the packages generated by commercial models, and 21.7% of those generated by open-source models, did not exist.
- Kwa T, West B, Becker J, et al. (2025). Measuring AI ability to complete long software tasks. arXiv:2503.14499; NeurIPS 2025. Success rate falls with the human time a task takes, well fit by an exponential decline.
- Meyerson E, Paolo G, Dailey R, et al. (2025). Solving a million-step LLM task with zero errors. arXiv:2511.09030. The 20-disc Towers of Hanoi (1,048,575 moves) solved with zero errors by one-step decomposition, a vote of at least three responses per step, and discarding over-long or malformed responses; the model's raw per-step error rate was 0.2–0.4%; the votes needed per step grow only logarithmically with task length. The puzzle has a known correct answer at every step.
- Squair JW, Gautier M, Kathe C, et al. (2021). Confronting false discoveries in single-cell differential expression. Nature Communications 12: 5692. doi:10.1038/s41467-021-25960-2. Methods that ignore variation between biological replicates are biased and prone to false discoveries.
Our records:
- CohortDAG 0.2.0 release notes: six open-data study pipelines scored against their papers' numbers, 89 exact and 2 equivalent of 91 items.
- CohortDAG 0.2.0 tool page, provenance of 17 September 2026: the all-node execute-and-audit, byte-perfect against the notebook on a private clinical cohort, and the 36 defects it surfaced.
- Mendelian randomization 1.0.2 evidence: Walker et al. 2019 v2, case 2; 163 archived rows, 157 retained, 19 of 19 headline cells matched.
- Meta-analysis 1.0.0 performance report: 28 open-access Cochrane reviews, 10,797 of 11,351 checkable quantitative items reproduced (95%).