Paper reading: AI is better at experiments—but does that mean better research taste?

Mon, 12 Oct 2026 00:00:00 +0000

Paper information

Paper:
Authors: Oliver Jaffe and Dane Sherburn
Source and version: arXiv preprint, 2610.06824v2, revised October 6, 2026
Reading sources: /

Design and findings

These points summarize the paper; the section links locate the supporting passages.

Scope: A fixed problem and shared Coder separate experimental judgment from implementation; problem selection is not measured. ( )
Results: Opus 5.5 has a 2.3× compute multiplier (95% CI 1.15–4.37). Recent efficiency and final-performance trends double roughly every 3 and 14.6 months, respectively. ( )
Novelty: Neither models nor humans produced “Invented” proposals: those containing a structural component absent from the literature. ( )
Verification: Tasks remain private, limiting external scrutiny. ( )

Efficiency, result quality, and novelty are distinct. Not observing a breakthrough does not prove one impossible.

My understanding: doing good research and recognizing it

My first response in discussing this paper was that reaching the same result with fewer experiments can indeed reflect part of research taste. But evaluating taste still requires human judgment; efficiency alone is not the whole concept.

There are two different abilities here: doing research well and recognizing the value of research. Someone may understand a result without yet appreciating the choice of question, the elegance of an experiment, or why a finding matters. Some forms of research beauty become visible only after understanding the problem and method.

Efficient research versus research that clarifies a problem

The following is a hypothetical example, not a TasteVal result.

Two researchers want to improve the same model. Researcher A tunes parameters effectively and raises the score after a few attempts. Researcher B suspects that the model exploits a shortcut in the data and designs a simple control experiment, showing that the high score does not demonstrate the capability they had assumed.

A’s strength is getting the job done efficiently. B’s strength is identifying the crucial issue and making something unclear become clear. B may not raise the score but may change our understanding of the problem.

That is the distinction I want to retain: obtaining a result is not identical to understanding why it is meaningful.

Evaluating taste also takes judgment

An evaluator who looks only at scores might miss B’s contribution. Appreciating the control experiment requires understanding which explanation it rules out and why that explanation could have misled us.

Still, this must not become “anyone who disagrees lacks taste.” Research beauty may resist complete quantification, but we should try to explain its value: What problem does it reveal? What doubt does it resolve? What understanding does it change?

Judgment is not authority exempt from explanation; it is the ability to articulate reasons for value.

The question I retain

I would distinguish at least two layers of research taste: knowing which experiments are worth running, and recognizing which problems, explanations, and findings deserve attention.

This note is not a final verdict on AI’s capabilities. The evaluation principle I want to keep is: acknowledge improvements in efficiency while continuing to ask whether they bring deeper understanding.

Reading and writing note: This article was organized through AI-assisted reading and discussion, with the personal perspective confirmed by the author. Paper findings are separated from reading reflections; the hypothetical example is not research data. This note is neither a reproduction nor an independent validation of the original study.

SocioVerse Challenge 2026

Mon, 12 Oct 2026 00:00:00 +0000

In 2026, I serve as Program Chair of SocioVerse Challenge 2026, an AI4SS challenge on human–AI collaboration and social governance. The challenge brings large language models, agent-based social simulation, and social-science research into a shared research setting. Participants are asked not simply to demonstrate a model, but to formulate a meaningful research question, explain their study design, and deliver results that can be examined and reproduced.

Built around SocioVerse2, the challenge has three tracks covering reconstruction and forecasting, interventions and their effects, and innovations in research methods and systems. This page documents the research agenda, organizing institutions, participation options, and evaluation process.

Official SocioVerse Challenge 2026 poster, retained as originally published. The schedule and award information below reflect the updated challenge website.

Research Agenda

AI4SS stands for AI for Social Science. This challenge asks how agent-based simulation can connect individual decisions, their environments, and interactions to the study of collective behavior, institutions, and social change.

SocioVerse2 organizes study settings around population (P), environment (E), and behavior (B). Participants need to explain who is being modeled, how data and environmental conditions are specified, how actions and interactions arise, and which results can be compared with observations. Platform examples span social media, urban housing, pharmaceutical procurement, and consumer confidence, among other domains. They offer starting points rather than prescribed tasks or conclusions.

The challenge therefore combines substantive research with technical implementation. A working simulation also needs an important question, explicit assumptions, and a clear account of the limits of its conclusions.

Organizers and Joint Exchange

The challenge is jointly organized by:

The Data Intelligence and Social Computing Laboratory (Fudan DISC), Fudan University.
The Natural Language Processing Group, King’s College London.
The Behavioral Science Center, Fudan University.
The Center for Computational and Intelligent Communication, Fudan University.

The Shanghai Innovation Institute provides platform support.

According to the challenge announcement, results will be exchanged jointly with LASS 2026 @ CIKM 2026. The 2nd Workshop on LLM Agents for Social Simulation is scheduled for November 8, 2026, in Rome, Italy, addressing agent modeling, simulation evaluation, safety and ethics, and scalable research infrastructure.

Three Research Tracks

A — Historical Reconstruction and Future Prediction

This track studies how individual decisions and social interactions produce collective distributions and changes over time. Projects may reconstruct an observed social process or examine forecasting performance. The central requirement is not merely a plausible-looking trajectory: researchers must specify the information available at each point in time, validate results against independent observations and suitable baselines, and identify errors across populations, locations, or periods.

B — Counterfactual Intervention and Impact Assessment

This track examines how changes in policies, institutions, information, or other interventions affect social outcomes through individual choices and interactions. Studies need comparable scenarios, an account of the mechanism, and attention to group differences and unintended effects. A simulated difference is not automatically a real-world causal effect; assumptions, robustness, and the scope of inference must remain explicit.

C — Methodological and Systems Innovation

This track focuses on research methods and tools, including modeling approaches, agent mechanisms, computational efficiency, evaluation, platform extensions, and human–system collaboration. A contribution may improve one component or extend the broader platform, but comparisons should establish its effectiveness, resource costs, and appropriate research settings.

Participation and Deliverables

The challenge welcomes university students and faculty, research-institute staff, industry researchers, and independent developers. Teams may have one to five formal members and one or two advisers. Each team chooses one subtask within one main track and submits a complete contribution.

Participants may use the online conversational workbench or develop locally. Both options share the same submission requirements and evaluation criteria. The online option uses platform-provided model credits; local development organizes the study and experiments within the open-source SocioVerse2 research environment.

Final submissions include a research report, a reproducible research package, and demonstration materials. Reports may be written in Chinese or English. The package must enable reviewers to follow and reproduce the experiment, with disclosure of models, random seeds, usage costs, and data conditions. Track C projects that modify the platform core also need the full code and execution instructions. Projects involving real individual-level data must address authorization, de-identification, and leakage risks.

Evaluation Process

The proposal stage determines eligibility for API funding; it does not eliminate teams. Teams without additional funding may continue. Final evaluation starts with reproducibility checks, followed by independent scoring by at least two reviewers, with a third reviewer added for substantial disagreement.

The website specifies five common dimensions, supplemented by track-specific considerations:

Dimension	Weight	Focus
Social-science value and originality	25%	Importance, substantive grounding, and novelty of the question
Methods and experimental design	25%	Study settings, controls, ablations, and sensitivity analysis
Results and theoretical contribution	20%	Evidential support and explanatory value
Reproducibility and engineering quality	20%	Execution instructions, fixed seeds, and cost disclosure; a separate qualifying threshold applies
Presentation and future potential	10%	Reporting, visualization, and opportunities for further work

This combines research judgment with engineering checks. Producing an output does not establish a conclusion, while a worthwhile question still needs a sufficiently clear experiment and evidence to be assessed.

Awards and Research Support

Each of the three tracks awards one first prize, two second prizes, and three third prizes: 18 main awards in total. The current website specifies cash prizes in US dollars:

Award	Teams per track	Prize per team
First prize	1	US$1,500
Second prize	2	US$750
Third prize	3	US$400

The main cash-prize pool is US$12,600**, with a separate **US$5,000 allocation for API and compute support. Compute credits are not cash prizes. The announcement specifies US$30 in initial credits per registered team and US$70 more after a proposal passes the funding review. Teams completing a submission receive written expert feedback, and outstanding work may be included among the platform examples.

The poster retains its original promotional figures. This text uses the current website’s currency and itemized awards, without converting or combining unlike categories.

Schedule

The schedule below was checked against the challenge website on October 13, 2026. Times are UTC+8.

Date	Milestone
September 15, 2026	Launch and registration opening recorded on the website
October 12, 2026, 23:59	First-round proposal deadline
October 15, 2026	Initial funding decisions
October 19 and 26, 2026	Further proposal-funding review batches
October 31, 2026, 23:59	Final submission deadline
November 1–7, 2026	Reproducibility checks and expert review
November 8, 2026	Scheduled awards announcement and ceremony; joint exchange with LASS

The announcement permits registration and submissions after the first-round deadline, with further funding decisions on October 19 and 26. The proposal deadline is not the end of participation.

The original poster lists September 28 as the launch and October 30 as the report deadline. The current website lists September 15 for registration opening and October 31 for final submissions. The poster is preserved without changing its text; updated arrangements follow the official website.

Official Resources

This record draws on the official challenge website and poster. Schedules, awards, and submission rules may change; consult the challenge website for subsequent announcements.

Artificial Intelligence | Dylan Chiang