Does Scrapling Save Scraped Items in Checkpoints?

8 min read · 3 Nov 2026
Last reviewed November 3, 2026

After our Scrapling crawl paused with record A saved, the resumed run returned B and C. Exporting that result to the same JSON path replaced the file, leaving A absent.

No, Scrapling 0.5 checkpoints do not save scraped items. They save pending requests and seen request fingerprints. Store extracted records separately, give them stable identities, and reconcile the stored output after resume. The restored request queue and the next invocation’s item list serve different purposes, as the checkpoint data structure makes explicit.

Tested on 3 November 2026. We ran Scrapling 0.5, released on 17 October, against three local HTTP pages. Five fresh Python processes covered an uninterrupted run, graceful pause, resume, deliberate checkpoint replay and an injected record-write failure.

What does Scrapling add to an agent’s web workflow?

Scrapling is a Python web-scraping framework that combines HTTP clients, browser fetching, parsing and concurrent spiders. An agent integration can use different fetchers for static and rendered pages while keeping extraction in one library. Our test focuses on what happens to extracted records when a spider resumes.

The release README documents browser impersonation, JavaScript rendering, retries, proxy rotation, CLI extraction and MCP access. The project uses the BSD 3-Clause license.

Adaptive parsing relocates elements after HTML changes. The adaptive parsing guide explains saved properties and similarity matching. Recovering a selector does not persist its extracted record. For the wider retrieval architecture, see our Wigolo local web retrieval review.

Does a Scrapling checkpoint contain scraped items?

A Scrapling 0.5 checkpoint contains pending requests and seen request fingerprints. The collected item list belongs to the current crawl invocation, and your application controls any separate database or export. In the pinned implementation, the crawler engine clears that item list when a crawl starts.

Store	What it holds	What to expect after resume
Scrapling checkpoint	Pending requests and seen fingerprints	Remaining requests can resume from the saved state
`result.items`	Items retained during this invocation	Earlier invocations' collected items are absent
Application record store	Records your code commits to a database or another persistent destination	Retention depends on your storage and write policy

This matters when you export after each invocation. In 0.5, result.items.to_json(path) writes the current list to that path. The export implementation replaces the file; to_jsonl(path) also opens it in write mode. Neither method merges earlier runs for you.

Retain separate per-run exports or commit items to a database. Our fixture demonstrates what happens when you reuse an export path.

What survived our pause, resume and replay test?

Our Scrapling test retained all three expected records in the keyed SQLite store after resume, while the reused JSON export contained only B and C. Deliberately replaying the saved checkpoint emitted B and C again. The append log grew to five writes; the keyed store still contained three records.

The fixture serves immutable records A, B and C over loopback HTTP. It uses one concurrent request, disables periodic checkpoints, and pauses after A’s database commit. Every phase starts a fresh Python worker. The pause-and-resume sequence shares one database and export path; the control and injected-failure runs use separate directories.

Observed phase	Returned items and JSON export	Committed SQLite IDs	Outcome
Uninterrupted control	A, B, C	A, B, C	All expected records present
Graceful pause after A	A	A	Checkpoint retained B and C as pending
Fresh-process resume	B, C	A, B, C	Same-path JSON export was missing A
Deliberate replay of our saved checkpoint	B, C	A, B, C	Append log: A once, B twice, C twice
Separate run with injected B write failure	A, C	A, C	`completed=True`, one recorded callback error, B missing

For replay, we restored our paused run’s checkpoint after the resumed crawl completed. This exercises repeated requests; it does not simulate an unexpected crash or measure what survives a power failure.

In the failure case, our item hook raised before writing B, and on_error logged the exception without retrying. Scrapling finished the remaining crawl. The manifest check exposed the missing record.

Download the Scrapling checkpoint fixture and recorded results. The package includes the local pages, expected-record manifest, worker and runner code, dependency versions, JSON reports, append logs and readable database exports.

How can you reproduce the checkpoint test?

Run the downloaded fixture in a fresh Python environment with Scrapling’s fetchers extra. Its runner starts a local HTTP server, launches the five workers, and checks their exported files and database records independently. Browser binaries, an MCP client and an AI model are unnecessary for this HTTP-only experiment.

python3.12 -m venv .venv
mkdir -p .cache/pip .cache/tmp
PIP_DISABLE_PIP_VERSION_CHECK=1 PIP_CACHE_DIR="$PWD/.cache/pip" TMPDIR="$PWD/.cache/tmp" .venv/bin/python -m pip install -r requirements-tested.txt
.venv/bin/python run_fixture.py --output results-reproduced
.venv/bin/python verify_payloads.py --results results-reproduced

Run those commands from the extracted fixture directory on Linux. The recorded run used Python 3.12.14; requirements-tested.txt pins its package versions. We verified the Scrapling 0.5 wheel against its published SHA-256 before installation:

366f0ec9f1e7e998900e6c5e3a4063ea34062326394ced64e5c389b690572726

Inspect summary.json, payload-validation.json and the phase reports in your output directory. The server stays alive across workers so checkpointed URLs remain valid. Each rerun creates its own checkpoint; process IDs, the local port and checkpoint bytes can differ. The checks compare record identities and payloads with the fixture manifest.

Where should you save extracted records?

Save Scrapling records in an application-owned destination through on_scraped_item, using an identity that matches your collection. Commit successful writes before returning from the hook. Scrapling awaits this hook before retaining the item, but the hook also needs an explicit error policy; a unique database key cannot recover a write that never committed.

The Spider hook API provides the integration point. This is the executed fixture’s database write, with the surrounding instrumentation omitted:

self.database.execute(
    "INSERT INTO items (collection_id, item_id, payload_json) VALUES (?, ?, ?) "
    "ON CONFLICT(collection_id, item_id) DO UPDATE SET payload_json=excluded.payload_json",
    (self.manifest["collection_id"], item["id"], json.dumps(item, sort_keys=True)),
)
self.database.commit()

The primary key is (collection_id, item_id). An upsert inserts a new row or updates the row with the same key, so repeated writes of A, B or C converge within this immutable collection. The download includes the schema and complete hook.

For changing data, choose whether new observations replace or version older ones. Include the observation scope in the identity when history matters. Otherwise, deduplication can overwrite evidence you meant to retain.

A production integration needs a recoverable failed-record ledger or an explicit retry policy, followed by reconciliation. Validate that policy against your sink’s failures; this SQLite fixture does not establish transaction recovery for another database.

How do you know the resumed dataset is complete?

Reconcile Scrapling’s persistent output against the collection you intended to gather, including expected identities and required fields. In our fixture, the independent manifest makes missing A or B unambiguous. On a real website, the expected population needs its own discovery method and scope; a finished request queue cannot supply that definition.

Acceptance question	Evidence to inspect	What the fixture showed
Are expected records missing?	Expected IDs minus stored IDs	A missing from resumed JSON; B missing after the injected write failure
Were records processed again?	Per-ID emission or write counts	B and C each written twice in the replay sequence
Does the final store retain one intended record per key?	Database rows under the chosen collection identity	Three rows after resume and deliberate replay
Do required values match the intended records?	Stored payloads checked against the fixture manifest	IDs, names and source URLs matched after resume and replay
Are any write errors unresolved?	Item-hook errors and recovery records	The injected B failure prevents acceptance despite crawl completion

The advanced crawl guide documents pause/resume, streaming and output hooks. Use those building blocks to preserve data, then decide separately when to release the collection downstream.

If an agent or RAG index consumes only the resumed JSON file in this example, it receives two records. Publish the reconciled collection from your persistent store instead. Process-level recovery is a further concern; our AX runner recovery and self-hosting guide covers that separate boundary.

Does the MCP server change this persistence contract?

Scrapling’s MCP server exposes web and browser operations to compatible agents, but this experiment exercised the Python Spider API. Its checkpoint results therefore do not establish persistence for an MCP workflow. Define where your agent saves returned content, then verify that store across the interruptions your own workflow must tolerate.

The v0.5 MCP guide lists 16 tools, including request sessions, browser actions, targeted extraction, screenshots, network inspection and saved browser state. A saved login and a saved extracted dataset need separate handling.

When would we use Scrapling for agent ingestion?

We would use Scrapling when a Python integration benefits from its fetching, parsing and spider controls, and the application also owns record persistence. Start with a small approved collection, choose item identities, exercise resume and replay, and inspect stored results. Add realistic write failures before relying on the collection downstream.

Use representative pages, the required record schema and restart requirements to define your ingestion acceptance test.

Scrapling checkpoint FAQ

Does to_jsonl append records from earlier runs?

No. Scrapling 0.5 opens the target JSONL file in write mode. Calling to_jsonl with the same path after another invocation replaces that file with the current item list. Preserve separate run files and merge them deliberately, or write records to a persistent sink while crawling. The JSONL extension alone does not imply append behavior.

Does request deduplication prevent duplicate data rows?

Request deduplication and record identity solve different tasks. Different requests can produce the same logical item, and replaying a saved checkpoint can process its pending requests again. In our controlled replay, B and C appeared twice in the successful-write log. A collection-and-item database key kept the final store at three rows; it did not prevent the repeated processing.

Does saved browser login state include a crawl’s extracted data?

No. Scrapling’s documented browser-state export saves cookies, local storage and IndexedDB. It does not contain the Spider’s extracted item collection. The guide also excludes tabs, browser settings and sessionStorage from that browser snapshot. Treat authentication state and the destination for extracted records as separate application resources. Our fixture did not execute a browser-state save or restore.

Does AutoThrottle keep its learned delays after resume?

The official crawl guide says learned AutoThrottle delays are not stored in a checkpoint. A resumed crawl starts from the configured starting delay. Preserve the intended throttle configuration in your spider and observe its behavior after restart. This is a documented implementation boundary; our single-request-concurrency fixture did not benchmark or exercise AutoThrottle adaptation.

Does this test prove recovery after a power failure?

No. We used an explicit graceful pause, new Python worker processes and deliberate replay of our own saved checkpoint. We did not kill a process during a database commit, interrupt storage writes or cut power. The result establishes the observed checkpoint, export and keyed-store behavior for these controlled cases. Hard-crash durability needs a separate test against the actual storage and runtime.

Can a manifest prove that you found every page on a website?

A manifest checks completeness relative to the population it lists. Our fixture defines three immutable records, so missing identities are directly measurable. An unknown website may have undiscovered pagination, changed records or inaccessible sections. Establish the crawl scope and discovery method independently, then reconcile the records that scope requires. A passing three-record test cannot establish complete coverage of another site.