In this piece
Is a PaperOrchestra manuscript ready to submit?
In one of PaperOrchestra’s public sample papers, every number printed in a results row matches the published original. An entire metric is missing. Compare the column headings in the two papers and the omission is easy to spot.
PaperOrchestra is a Google Cloud AI Research framework that turns an idea brief and completed experimental logs into a LaTeX manuscript. It can find references, draw figures and revise a draft. Before you submit one, check whether the draft includes the measurements a reader needs to evaluate your claim, and whether the venue permits that use of AI.
Reviewed 16 November 2026. We compared a public draft with the original paper, checked a second example against its experimental log, and inspected the released code. Timing and preference scores below come from the authors’ study. We did not run a fresh generation.
What did a PaperOrchestra draft leave out?
The CVPR sample with synthesized figures drops the ordinary collision-rate column from Table 1 of Certified Human Trajectory Prediction. The study compares ways to forecast pedestrian motion. Zoom in on the middle page of the generated sample and find the row for Smoothed D-Pool. Then open Table 1 on page 7 of the original PDF. Generated CVPR manuscript · Bahari et al., Tables 1 and 2
| Metric for Smoothed D-Pool | Published Table 1 | Generated Table 1 |
|---|---|---|
| Final displacement error | 1.23 m | 1.23 m |
| Certified final displacement error | 2.0 m | 2.00 m |
| Collision rate (Col) | 9.0% | Column omitted |
| Certified collision rate (Certified-Col) | 49% | 49% |
The ordinary collision rate, 9.0%, compares a forecast path with the recorded paths of nearby pedestrians. The 49% counts cases in which at least one neighbor falls inside the certified bounds around the forecast. The companion draft with the original figures keeps all four columns. We checked the benchmark input too. Its experimental log records 1.23, 2.0, 9.0 and 49 in that order. To find the row in your copy of the dataset, run the command below. Companion CVPR draft · Versioned benchmark archive
unzip -p datasets.zip \
'*/cvpr2025_5f3288a486/raw_materials/experimental_log.md' | grep 'Smoothed D-Pool'In Table 2 of the synthesized-figures draft, all three remaining-noise values for the diffusion denoiser match the original, at 0.06, 0.14 and 0.24. This spot check tells us exactly what to restore in one table. Two gallery samples cannot establish what caused the omission or how often it occurs.
What does PaperOrchestra need to write a paper?
Start with the question, method and results of an experiment you have already run. The released command-line tool takes a materials directory containing idea_sparse.md and experimental_log.md, plus a directory for the venue’s LaTeX template. Plot generation is optional and defaults to off in the CLI. Released CLI and input filenames
The outline agent plans the argument. The literature agent finds candidate papers and checks their records through Semantic Scholar. With plotting enabled, another agent prepares visuals, including code-generated plots. A section writer assembles the LaTeX, and a refinement agent revises it in response to simulated reviews. If a reviewer asks for another experiment, the researcher still has to run it. Authors’ workflow and sample gallery
A verified bibliographic record helps you find the source. To check support for a sentence, open the paper at its cited result and compare the population, method and scope. For a team deciding how to collect evidence earlier in a project, our Feynman research-agent review covers literature search and replication planning.
How did the drafts compare with human papers?
Eleven AI researchers judged 40 sampled papers in 180 blind paired evaluations. PaperOrchestra scored ahead of both automated writers and behind the published human originals. The authors aggregate votes by paper. The table shows the percentage of paper-level comparisons PaperOrchestra won minus the percentage it lost. Ties count toward neither side of that difference. COLM 2026 study, Figure 7 and Appendix E.6
| Compared with | Literature review | Overall manuscript |
|---|---|---|
| Single-agent writer | +67.6 points | +37.8 points |
| AI Scientist-v2 | +50.0 points | +13.9 points |
| Published human paper | −37.8 points | −75.7 points |
PaperWritingBench starts with 100 accepted CVPR 2025 papers and 100 accepted ICLR 2025 papers. The team worked backwards to reconstruct the ideas and results. Even the short idea briefs averaged roughly 590 words. The standard comparisons supplied the original figures to the systems, while the authors assessed generated visuals separately. This experiment measures how well a system writes up existing evidence. Real conference acceptance and the discovery of new results require different tests. Released PaperWritingBench dataset
How do you check a claim against the experimental log?
Pick the broadest claim in the abstract and look for the row where it might fail. The study’s example log in Appendix D.4 describes TSAM, a system that picks out an object in video from text and sound. The original CVPR paper reports both how well it finds an object and what happens when the object is absent. Radman and Laaksonen, Table 1
On unseen categories, TSAM’s Jaccard score is 54.58% against EEMC’s 49.54%, a gain of 5.04 percentage points. When the requested object is absent, lower is better on the S metric. EEMC scores 0.007 and TSAM scores 0.017. A hypothetical sentence saying TSAM won on every test would fail this check. An author could instead say that TSAM improves segmentation on unseen categories while EEMC does better when there is nothing to segment.
Where is the Streamlit demo, and what does a run cost?
The demo link opens instructions for running the Streamlit app on your own machine. From the repository root, with its Python dependencies installed and credentials configured for your model provider, launch the app with the command below. The frontend invokes the paper-writing CLI. If you only want to inspect a finished result, the project gallery is available in your browser. Official Streamlit launch guide · Google Research repository and setup
streamlit run frontend/app.pyTable 9 of the COLM paper estimates 50 to 70 model calls per manuscript. The authors timed ten single-worker runs, dropped the fastest and slowest, and reported a mean of 39.6 minutes for the remaining eight. They give no dollar figure per paper. To price your own run, record model usage and retries, then add the time an author spends checking results, citations and revisions. The original experiments have their own cost.
What should an author check before submission?
Check the policy for the exact venue, track and year before you generate a full draft. For example, the NeurIPS 2026 position-paper track requires human authors to write the substance of the manuscript and limits AI’s role in the text to copy-editing or similarly minor changes. The main track publishes its own rules on author use of agents. NeurIPS 2026 position-paper rules · NeurIPS 2026 main-track handbook
- Compare the tables and plots with the logs. Check for missing columns and omitted baselines, along with units, split names and which direction is better. Regenerate quantitative plots from the recorded data.
- Read the sources attached to the strongest claims. Record the table, section or passage that supports each one. Narrow the claim if the source only supports it on one dataset.
- Reopen the PDF after the last revision. Inspect captions, cross-references, author information, anonymity and disclosure requirements. Have the person responsible for the experiment approve the wording. Our guide to evaluating agent outputs covers how to retain evidence when a team repeats these checks.
Reviewers already handle substantial volume. ICLR’s chairs counted 19,525 valid submissions and 76,139 reviews for its 2026 cycle. Those figures establish the workload at one conference. The PaperOrchestra study did not measure how a writing tool changes submission volume or reviewer hours. ICLR 2026 review retrospective
