Is a PaperOrchestra manuscript ready to submit?

Back
Kevin Riedl

6 min read · 16 Nov 2026
Last reviewed

Next
Made on your device, with no Instagram connection. We copy the post link for Instagram’s Link sticker.

In one of PaperOrchestra’s public sample papers, every number printed in a results row matches the published original. An entire metric is missing. Compare the column headings in the two papers and the omission is easy to spot.

PaperOrchestra is a Google Cloud AI Research framework that turns an idea brief and completed experimental logs into a LaTeX manuscript. It can find references, draw figures and revise a draft. Before you submit one, check whether the draft includes the measurements a reader needs to evaluate your claim, and whether the venue permits that use of AI.

Reviewed 16 November 2026. We compared a public draft with the original paper, checked a second example against its experimental log, and inspected the released code. Timing and preference scores below come from the authors’ study. We did not run a fresh generation.

What did a PaperOrchestra draft leave out?

The CVPR sample with synthesized figures drops the ordinary collision-rate column from Table 1 of Certified Human Trajectory Prediction. The study compares ways to forecast pedestrian motion. Zoom in on the middle page of the generated sample and find the row for Smoothed D-Pool. Then open Table 1 on page 7 of the original PDF. Generated CVPR manuscript · Bahari et al., Tables 1 and 2

Smoothed D-Pool in the two versions of Table 1
Metric for Smoothed D-PoolPublished Table 1Generated Table 1
Final displacement error1.23 m1.23 m
Certified final displacement error2.0 m2.00 m
Collision rate (Col)9.0%Column omitted
Certified collision rate (Certified-Col)49%49%

The ordinary collision rate, 9.0%, compares a forecast path with the recorded paths of nearby pedestrians. The 49% counts cases in which at least one neighbor falls inside the certified bounds around the forecast. The companion draft with the original figures keeps all four columns. We checked the benchmark input too. Its experimental log records 1.23, 2.0, 9.0 and 49 in that order. To find the row in your copy of the dataset, run the command below. Companion CVPR draft · Versioned benchmark archive

unzip -p datasets.zip \
  '*/cvpr2025_5f3288a486/raw_materials/experimental_log.md' | grep 'Smoothed D-Pool'

In Table 2 of the synthesized-figures draft, all three remaining-noise values for the diffusion denoiser match the original, at 0.06, 0.14 and 0.24. This spot check tells us exactly what to restore in one table. Two gallery samples cannot establish what caused the omission or how often it occurs.

What does PaperOrchestra need to write a paper?

Start with the question, method and results of an experiment you have already run. The released command-line tool takes a materials directory containing idea_sparse.md and experimental_log.md, plus a directory for the venue’s LaTeX template. Plot generation is optional and defaults to off in the CLI. Released CLI and input filenames

The outline agent plans the argument. The literature agent finds candidate papers and checks their records through Semantic Scholar. With plotting enabled, another agent prepares visuals, including code-generated plots. A section writer assembles the LaTeX, and a refinement agent revises it in response to simulated reviews. If a reviewer asks for another experiment, the researcher still has to run it. Authors’ workflow and sample gallery

A verified bibliographic record helps you find the source. To check support for a sentence, open the paper at its cited result and compare the population, method and scope. For a team deciding how to collect evidence earlier in a project, our Feynman research-agent review covers literature search and replication planning.

How did the drafts compare with human papers?

Eleven AI researchers judged 40 sampled papers in 180 blind paired evaluations. PaperOrchestra scored ahead of both automated writers and behind the published human originals. The authors aggregate votes by paper. The table shows the percentage of paper-level comparisons PaperOrchestra won minus the percentage it lost. Ties count toward neither side of that difference. COLM 2026 study, Figure 7 and Appendix E.6

PaperOrchestra’s win–loss margin in blind comparisons (percentage points)
Compared withLiterature reviewOverall manuscript
Single-agent writer+67.6 points+37.8 points
AI Scientist-v2+50.0 points+13.9 points
Published human paper−37.8 points−75.7 points

PaperWritingBench starts with 100 accepted CVPR 2025 papers and 100 accepted ICLR 2025 papers. The team worked backwards to reconstruct the ideas and results. Even the short idea briefs averaged roughly 590 words. The standard comparisons supplied the original figures to the systems, while the authors assessed generated visuals separately. This experiment measures how well a system writes up existing evidence. Real conference acceptance and the discovery of new results require different tests. Released PaperWritingBench dataset

How do you check a claim against the experimental log?

Pick the broadest claim in the abstract and look for the row where it might fail. The study’s example log in Appendix D.4 describes TSAM, a system that picks out an object in video from text and sound. The original CVPR paper reports both how well it finds an object and what happens when the object is absent. Radman and Laaksonen, Table 1

On unseen categories, TSAM’s Jaccard score is 54.58% against EEMC’s 49.54%, a gain of 5.04 percentage points. When the requested object is absent, lower is better on the S metric. EEMC scores 0.007 and TSAM scores 0.017. A hypothetical sentence saying TSAM won on every test would fail this check. An author could instead say that TSAM improves segmentation on unseen categories while EEMC does better when there is nothing to segment.

Where is the Streamlit demo, and what does a run cost?

The demo link opens instructions for running the Streamlit app on your own machine. From the repository root, with its Python dependencies installed and credentials configured for your model provider, launch the app with the command below. The frontend invokes the paper-writing CLI. If you only want to inspect a finished result, the project gallery is available in your browser. Official Streamlit launch guide · Google Research repository and setup

streamlit run frontend/app.py

Table 9 of the COLM paper estimates 50 to 70 model calls per manuscript. The authors timed ten single-worker runs, dropped the fastest and slowest, and reported a mean of 39.6 minutes for the remaining eight. They give no dollar figure per paper. To price your own run, record model usage and retries, then add the time an author spends checking results, citations and revisions. The original experiments have their own cost.

What should an author check before submission?

Check the policy for the exact venue, track and year before you generate a full draft. For example, the NeurIPS 2026 position-paper track requires human authors to write the substance of the manuscript and limits AI’s role in the text to copy-editing or similarly minor changes. The main track publishes its own rules on author use of agents. NeurIPS 2026 position-paper rules · NeurIPS 2026 main-track handbook

  1. Compare the tables and plots with the logs. Check for missing columns and omitted baselines, along with units, split names and which direction is better. Regenerate quantitative plots from the recorded data.
  2. Read the sources attached to the strongest claims. Record the table, section or passage that supports each one. Narrow the claim if the source only supports it on one dataset.
  3. Reopen the PDF after the last revision. Inspect captions, cross-references, author information, anonymity and disclosure requirements. Have the person responsible for the experiment approve the wording. Our guide to evaluating agent outputs covers how to retain evidence when a team repeats these checks.

Reviewers already handle substantial volume. ICLR’s chairs counted 19,525 valid submissions and 76,139 reviews for its 2026 cycle. Those figures establish the workload at one conference. The PaperOrchestra study did not measure how a writing tool changes submission volume or reviewer hours. ICLR 2026 review retrospective

PaperOrchestra FAQ

Can PaperOrchestra start with a research idea alone?
The documented workflow needs experimental results and a venue template as well as an idea brief. A new hypothesis still needs an experiment before the manuscript can report a result.
Where can I see a generated paper without installing anything?
The authors’ project gallery shows four CVPR and ICLR examples. Each venue has a sample with original figures and a sample with synthesized visuals. The Streamlit link is for running the interactive generator locally.
Does PaperOrchestra check whether citations are real?
The literature agent checks bibliographic records through Semantic Scholar. An author still needs to read the cited passage and verify that its findings support the specific claim in the draft.
How long does PaperOrchestra take?
The authors report 39.6 minutes after removing the fastest and slowest of ten sampled runs. That timing covers manuscript generation. Budget separately for the experiments and final author review.
Has a conference accepted the generated manuscripts?
The study reports blind preferences and simulated reviewer decisions. The accepted papers in PaperWritingBench are the human originals used to reconstruct the inputs. The study gives no observed acceptance rate for the generated drafts.

Make the pilot decision with evidence

Bring one workflow, representative examples and the outcome you need to accept. Define an evaluation set, failure limits and the evidence required before scaling an existing AI system.

Relevant service:

Inbox, without the noise

Follow the work that matters to you

Get a short email when we publish something new. Follow the whole blog or only the problems you care about.

What would you like to receive?
Choose your topics

Free, double opt-in, no tracking pixels.

Back
Kevin Riedl

6 min read · 16 Nov 2026
Last reviewed

Next

Get the next Delivery and QA field note

One concise email when we publish. No tracking pixels, and no inbox filler.

Free, double opt-in, no tracking pixels.