VOILA! Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results by Prof. Elliott Ash

VOILA! AI and Science

Title

VOILA! Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results

Lecturer

Prof. Elliott Ash (ETH Zurich), https://elliottash.com/

Content and organization

In this talk

Recent work has used LLM agents to reproduce empirical social science results with access to both the data and code. We broaden this scope by asking: Can they reproduce results given only a paper’s methods description and original data? We develop an agentic reproduction system that extracts structured methods descriptions from papers, runs reimplementations under strict information isolation — agents never see the original code, results, or paper — and enables deterministic, cell-level comparison of reproduced outputs to the original results. An error attribution step traces discrepancies through the system chain to identify root causes. Evaluating four agent scaffolds and four LLMs on 48 papers with human-verified reproducibility, we find that agents can largely recover published results, but performance varies substantially between models, scaffolds, and papers. Root cause analysis reveals that failures stem both from agent errors and from underspecification in the papers themselves.

Check the article in arXiv

About the speaker

Elliott Ash is Associate Professor of Law, Economics, and Data Science at ETH Zurich’s Center for Law & Economics, Switzerland. Prof. Ash is also a Scientific Lead in the Swiss AI Initiative, CEPR Research Affiliate (Political Economy), Associate Editor at Economic Journal, Co-Editor at Journal of Law and Economics, ETH AI Center Core Faculty, and recipient of a European Research Council Starting Grant. Previously, he held research appointments at New York University (Scholar in Residence), University of Warwick (Assistant Professor) and Princeton University (Postdoc). He received a Ph.D. in Economics and J.D. from Columbia University, a B.A. (Plan II Honors) from University of Texas at Austin, and an LL.M. in international criminal law from University of Amsterdam.

Elliott’s research and teaching focuses on empirical analysis of the law and legal system using techniques from econometrics, natural language processing, and machine learning. His research has been published in American Economic Journal: Applied Economics, American Economic Journal: Economic Policy, Journal of Law and Economics, Annual Review of Economics, Economic Journal, Cornell Law Review, Georgetown Law Journal, Journal of Public Economics, Journal of Politics, and Political Analysis. Elliott’s research has earned grant funding from the European Research Council, Swiss National Science Foundation, Swiss Data Science Center, U.S. National Science Foundation, the Turing Institute, and the Washington Center for Equitable Growth.

Course Duration

1.5

Course Type

Short Course

Participation terms

Attendance is free and open to everyone interested. Please register via the course link, and you will receive the Zoom meeting details one day before the seminar.

Language

English (with subtitles)

Modality (online/in person):

Online

Notes

For AIDA students only : In addition to registering via the course link, please click on the “Enroll in this course” button located at the bottom of the page to ensure that the course appears on your AIDA Certificate of Course Attendance upon successful completion.