---
title: "Is a PaperOrchestra Manuscript Ready to Submit?"
canonical: https://wavect.io/blog/paperorchestra-manuscript-review/
language: en
description: "Review a PaperOrchestra manuscript against its original CVPR results. See the missing metric, human benchmark, cost evidence and checks before submission."
image: "https://wavect.io/img/blog/headers/header_paperorchestra-manuscript-review.png"
author: ["Kevin Riedl"]
published: "2026-11-16"
updated: "2026-11-16"
reviewed: "2026-11-16"
---

# Is a PaperOrchestra manuscript ready to submit?

[**Back**](/blog/overview/)

[![Kevin Riedl](/img/team/kevin.webp)](/team/kevin-riedl/)

[Kevin Riedl](/team/kevin-riedl/) https://linkedin.com/in/wsdt

6 min read · 16 Nov 2026 Last reviewed November 16, 2026

[**Next**](/blog/feynman-open-source-ai-research-agent/)

TL;DR

PaperOrchestra drafts LaTeX from completed experimental work. In a public CVPR example, Table 1 retains a 49% certified collision rate but drops the 9.0% ordinary rate from the original. Human evaluators preferred its writing to two automated alternatives and favored the human originals overall. Compare each important claim with its evidence and check the venue’s author policy before submitting.

In one of PaperOrchestra’s public sample papers, every number printed in a results row matches the published original. An entire metric is missing. Compare the column headings in the two papers and the omission is easy to spot.

PaperOrchestra is a Google Cloud AI Research framework that turns an idea brief and completed experimental logs into a LaTeX manuscript. It can find references, draw figures and revise a draft. Before you submit one, check whether the draft includes the measurements a reader needs to evaluate your claim, and whether the venue permits that use of AI.

*Reviewed 16 November 2026. We compared a public draft with the original paper, checked a second example against its experimental log, and inspected the released code. Timing and preference scores below come from the authors’ study. We did not run a fresh generation.*

## What did a PaperOrchestra draft leave out?

The CVPR sample with synthesized figures drops the ordinary collision-rate column from Table 1 of *Certified Human Trajectory Prediction*. The study compares ways to forecast pedestrian motion. Zoom in on the middle page of the generated sample and find the row for Smoothed D-Pool. Then open Table 1 on page 7 of the original PDF. [Generated CVPR manuscript](https://github.com/yiwen-song/paper_orchestra/blob/342e47ed21b645527bec044e10228f1fc8820cfd/figures/cvpr_ours_ploton.jpg) · [Bahari et al., Tables 1 and 2](https://openaccess.thecvf.com/content/CVPR2025/papers/Bahari_Certified_Human_Trajectory_Prediction_CVPR_2025_paper.pdf#page=7)

| Metric for Smoothed D-Pool | Published Table 1 | Generated Table 1 |
| --- | --- | --- |
| Final displacement error | 1.23 m | 1.23 m |
| Certified final displacement error | 2.0 m | 2.00 m |
| Collision rate (Col) | 9.0% | Column omitted |
| Certified collision rate (Certified-Col) | 49% | 49% |

The ordinary collision rate, 9.0%, compares a forecast path with the recorded paths of nearby pedestrians. The 49% counts cases in which at least one neighbor falls inside the certified bounds around the forecast. The companion draft with the original figures keeps all four columns. We checked the benchmark input too. Its experimental log records 1.23, 2.0, 9.0 and 49 in that order. To find the row in your copy of the dataset, run the command below. [Companion CVPR draft](https://github.com/yiwen-song/paper_orchestra/blob/342e47ed21b645527bec044e10228f1fc8820cfd/figures/cvpr_ours_plotoff.jpg) · [Versioned benchmark archive](https://huggingface.co/datasets/yiwen-song/PaperWritingBench/blob/fbcfcafd042184ca954954b08dfcb92bd7942b51/datasets.zip)

```
unzip -p datasets.zip \
  '*/cvpr2025_5f3288a486/raw_materials/experimental_log.md' | grep 'Smoothed D-Pool'
```

In Table 2 of the synthesized-figures draft, all three remaining-noise values for the diffusion denoiser match the original, at 0.06, 0.14 and 0.24. This spot check tells us exactly what to restore in one table. Two gallery samples cannot establish what caused the omission or how often it occurs.

## What does PaperOrchestra need to write a paper?

Start with the question, method and results of an experiment you have already run. The released command-line tool takes a materials directory containing `idea_sparse.md` and `experimental_log.md`, plus a directory for the venue’s LaTeX template. Plot generation is optional and defaults to off in the CLI. [Released CLI and input filenames](https://github.com/google-research/paper-orchestra/blob/d9bae997e793a015ed38bb1605be9eb58c00284f/paper_writing_cli.py)

The outline agent plans the argument. The literature agent finds candidate papers and checks their records through Semantic Scholar. With plotting enabled, another agent prepares visuals, including code-generated plots. A section writer assembles the LaTeX, and a refinement agent revises it in response to simulated reviews. If a reviewer asks for another experiment, the researcher still has to run it. [Authors’ workflow and sample gallery](https://yiwen-song.github.io/paper_orchestra/)

A verified bibliographic record helps you find the source. To check support for a sentence, open the paper at its cited result and compare the population, method and scope. For a team deciding how to collect evidence earlier in a project, our [Feynman research-agent review](/blog/feynman-open-source-ai-research-agent/) covers literature search and replication planning.

## How did the drafts compare with human papers?

Eleven AI researchers judged 40 sampled papers in 180 blind paired evaluations. PaperOrchestra scored ahead of both automated writers and behind the published human originals. The authors aggregate votes by paper. The table shows the percentage of paper-level comparisons PaperOrchestra won minus the percentage it lost. Ties count toward neither side of that difference. [COLM 2026 study, Figure 7 and Appendix E.6](https://proceedings.colm.cc/paper_files/paper/2026/file/6546d4ab1e224ea9625c7c42ad25d6c8-Paper-Conference.pdf)

| Compared with | Literature review | Overall manuscript |
| --- | --- | --- |
| Single-agent writer | +67.6 points | +37.8 points |
| AI Scientist-v2 | +50.0 points | +13.9 points |
| Published human paper | −37.8 points | −75.7 points |

PaperWritingBench starts with 100 accepted CVPR 2025 papers and 100 accepted ICLR 2025 papers. The team worked backwards to reconstruct the ideas and results. Even the short idea briefs averaged roughly 590 words. The standard comparisons supplied the original figures to the systems, while the authors assessed generated visuals separately. This experiment measures how well a system writes up existing evidence. Real conference acceptance and the discovery of new results require different tests. [Released PaperWritingBench dataset](https://huggingface.co/datasets/yiwen-song/PaperWritingBench)

## How do you check a claim against the experimental log?

Pick the broadest claim in the abstract and look for the row where it might fail. The study’s example log in Appendix D.4 describes TSAM, a system that picks out an object in video from text and sound. The original CVPR paper reports both how well it finds an object and what happens when the object is absent. [Radman and Laaksonen, Table 1](https://openaccess.thecvf.com/content/CVPR2025/papers/Radman_TSAM_Temporal_SAM_Augmented_with_Multimodal_Prompts_for_Referring_Audio-Visual_CVPR_2025_paper.pdf)

On unseen categories, TSAM’s Jaccard score is 54.58% against EEMC’s 49.54%, a gain of 5.04 percentage points. When the requested object is absent, lower is better on the S metric. EEMC scores 0.007 and TSAM scores 0.017. A hypothetical sentence saying TSAM won on every test would fail this check. An author could instead say that TSAM improves segmentation on unseen categories while EEMC does better when there is nothing to segment.

## Where is the Streamlit demo, and what does a run cost?

The demo link opens instructions for running the Streamlit app on your own machine. From the repository root, with its Python dependencies installed and credentials configured for your model provider, launch the app with the command below. The frontend invokes the paper-writing CLI. If you only want to inspect a finished result, the project gallery is available in your browser. [Official Streamlit launch guide](https://github.com/google-research/paper-orchestra/blob/main/frontend/README.md) · [Google Research repository and setup](https://github.com/google-research/paper-orchestra)

```
streamlit run frontend/app.py
```

Table 9 of the COLM paper estimates 50 to 70 model calls per manuscript. The authors timed ten single-worker runs, dropped the fastest and slowest, and reported a mean of 39.6 minutes for the remaining eight. They give no dollar figure per paper. To price your own run, record model usage and retries, then add the time an author spends checking results, citations and revisions. The original experiments have their own cost.

## What should an author check before submission?

Check the policy for the exact venue, track and year before you generate a full draft. For example, the NeurIPS 2026 position-paper track requires human authors to write the substance of the manuscript and limits AI’s role in the text to copy-editing or similarly minor changes. The main track publishes its own rules on author use of agents. [NeurIPS 2026 position-paper rules](https://neurips.cc/Conferences/2026/CallForPositionPapers) · [NeurIPS 2026 main-track handbook](https://neurips.cc/Conferences/2026/MainTrackHandbook)

1. **Compare the tables and plots with the logs.** Check for missing columns and omitted baselines, along with units, split names and which direction is better. Regenerate quantitative plots from the recorded data.
2. **Read the sources attached to the strongest claims.** Record the table, section or passage that supports each one. Narrow the claim if the source only supports it on one dataset.
3. **Reopen the PDF after the last revision.** Inspect captions, cross-references, author information, anonymity and disclosure requirements. Have the person responsible for the experiment approve the wording. Our [guide to evaluating agent outputs](/blog/ai-agent-eval-sandbox-security-checklist/) covers how to retain evidence when a team repeats these checks.

Reviewers already handle substantial volume. ICLR’s chairs counted 19,525 valid submissions and 76,139 reviews for its 2026 cycle. Those figures establish the workload at one conference. The PaperOrchestra study did not measure how a writing tool changes submission volume or reviewer hours. [ICLR 2026 review retrospective](https://blog.iclr.cc/2026/03/31/a-retrospective-on-the-iclr-2026-review-process/)

## PaperOrchestra FAQ

### Can PaperOrchestra start with a research idea alone?

The documented workflow needs experimental results and a venue template as well as an idea brief. A new hypothesis still needs an experiment before the manuscript can report a result.

### Where can I see a generated paper without installing anything?

The authors’ project gallery shows four CVPR and ICLR examples. Each venue has a sample with original figures and a sample with synthesized visuals. The Streamlit link is for running the interactive generator locally.

### Does PaperOrchestra check whether citations are real?

The literature agent checks bibliographic records through Semantic Scholar. An author still needs to read the cited passage and verify that its findings support the specific claim in the draft.

### How long does PaperOrchestra take?

The authors report 39.6 minutes after removing the fastest and slowest of ten sampled runs. That timing covers manuscript generation. Budget separately for the experiments and final author review.

### Has a conference accepted the generated manuscripts?

The study reports blind preferences and simulated reviewer decisions. The accepted papers in PaperWritingBench are the human originals used to reconstruct the inputs. The study gives no observed acceptance rate for the generated drafts.

QA and production readiness

## Read next

Testing, audits, maintenance and hardening practices for reliable production software.

[Start here**QA for AI-Generated Code**](/blog/qa-for-ai-generated-code/)

- [Can Claude Fable 5.2 resume your saved agent sessions?](/blog/claude-fable-5-2-session-replay/)
- [A 2026 AI agent development stack, tested on a frontend regression](/blog/pi-agent-development-stack/)
- [Alibaba Open Code Review returned no comments. Which files did it check?](/blog/alibaba-open-code-review-no-comments/)
- [How can a water utility field-test a LoRaWAN meter reading and leak alarm?](/blog/lorawan-water-meter-field-acceptance-test/)
- [Both software vendors say they’re done. What should you test before signing off?](/blog/multi-vendor-software-acceptance-testing/)

[Next article**Can Claude Fable 5.2 resume your saved agent sessions?**](/blog/claude-fable-5-2-session-replay/)

[**Back**](/blog/overview/)

[![Kevin Riedl](/img/team/kevin.webp)](/team/kevin-riedl/)

[Kevin Riedl](/team/kevin-riedl/) https://linkedin.com/in/wsdt

6 min read · 16 Nov 2026 Last reviewed November 16, 2026

[**Next**](/blog/feynman-open-source-ai-research-agent/)
