A mathematics advisory group’s recommendations and a new analysis have raised questions about human review and formal verification of OpenAI’s proofs.
OpenAI released 719 mathematical manuscripts this week, presenting solutions to difficult research problems. The release has drawn scrutiny over how the work was checked and whether mathematicians can independently understand the results. The Advisory Group on Mathematics and Artificial Intelligence (AGMAI), hosted by Princeton University’s Institute for Advanced Study, had published guidance for frontier AI labs in September.
AGMAI’s recommendations call for avoiding tests of advanced open problems on proprietary models and for ensuring that human understanding accompanies AI-generated results. OpenAI’s release says it used proprietary models on open research problems. The group did not provide a full evaluation of the new collection, but the article reports that only 10 manuscripts included the models’ chain of thought. It also reports that 42% of the proofs had not undergone formalization, a process that expresses a proof in a form that can be checked by software.
A separate paper by mathematicians at the University of Cambridge and King’s College London examined an OpenAI solution to a problem derived from the Navier-Stokes equations, which describe fluid motion. The authors identified at least two mismatches between the natural-language explanation and the corresponding Lean code. Lean is used to encode mathematical arguments so their formal structure can be checked. The discrepancies do not, by themselves, establish that either version of the solution is wrong; they do call into question whether a model’s formal translation can be accepted without human scrutiny.
The debate is not only about whether a proof can pass a technical check. Mathematicians typically explain results to peers, answer questions and develop connections to other work. Terence Tao has criticized releases in which people prompting models may not understand the output well enough to engage with the field. Harvard mathematician Melanie Wood told TechCrunch that human understanding may still need to be built after an AI produces a proposed solution. AGMAI has also urged labs to provide machine-readable links between informal explanations and formal proofs, and to support mathematicians doing the work of evaluating and understanding them.