---
title: "Does Scrapling Save Scraped Items in Checkpoints?"
canonical: https://wavect.io/blog/scrapling-checkpoints-scraped-items/
language: en
description: "Does Scrapling save scraped items in checkpoints? See a reproducible pause, resume and replay test, JSON export behavior, and checks for missing records."
image: "https://wavect.io/img/blog/headers/header_scrapling-checkpoints-scraped-items.png"
author: ["Kevin Riedl"]
published: "2026-11-03"
updated: "2026-11-03"
reviewed: "2026-11-03"
---

# Does Scrapling Save Scraped Items in Checkpoints?

[**Back**](/blog/overview/)

[![Kevin Riedl](/img/team/kevin.webp)](/team/kevin-riedl/)

[Kevin Riedl](/team/kevin-riedl/) https://linkedin.com/in/wsdt

8 min read · 3 Nov 2026 Last reviewed November 3, 2026

[**Next**](/blog/wigolo-local-web-intelligence-mcp-review/)

TL;DR

Scrapling 0.5 checkpoints save pending requests and seen fingerprints, not collected items. In our local test, a fresh-process resume returned B and C; exporting to the same JSON path removed the earlier A record. A separately committed SQLite store retained all three. Deliberate checkpoint replay duplicated writes, while stable keys preserved three final rows. With a logging-only error handler, an injected write failure left a missing record despite crawl completion. Persist items explicitly and reconcile the intended collection before an agent consumes it.

After our Scrapling crawl paused with record A saved, the resumed run returned B and C. Exporting that result to the same JSON path replaced the file, leaving A absent.

**No, Scrapling 0.5 checkpoints do not save scraped items.** They save pending requests and seen request fingerprints. Store extracted records separately, give them stable identities, and reconcile the stored output after resume. The restored request queue and the next invocation's item list serve different purposes, as the [checkpoint data structure](https://github.com/D4Vinci/Scrapling/blob/3a8bf4e4d3ed91257ba0aa5b7dc66241d1e23840/scrapling/spiders/checkpoint.py) makes explicit.

**Tested on 3 November 2026.** We ran [Scrapling 0.5, released on 17 October](https://github.com/D4Vinci/Scrapling/releases/tag/v0.5), against three local HTTP pages. Five fresh Python processes covered an uninterrupted run, graceful pause, resume, deliberate checkpoint replay and an injected record-write failure.

## What does Scrapling add to an agent's web workflow?

Scrapling is a Python web-scraping framework that combines HTTP clients, browser fetching, parsing and concurrent spiders. An agent integration can use different fetchers for static and rendered pages while keeping extraction in one library. Our test focuses on what happens to extracted records when a spider resumes.

The [release README](https://github.com/D4Vinci/Scrapling/blob/3a8bf4e4d3ed91257ba0aa5b7dc66241d1e23840/README.md) documents browser impersonation, JavaScript rendering, retries, proxy rotation, CLI extraction and MCP access. The project uses the [BSD 3-Clause license](https://github.com/D4Vinci/Scrapling/blob/3a8bf4e4d3ed91257ba0aa5b7dc66241d1e23840/LICENSE).

Adaptive parsing relocates elements after HTML changes. The [adaptive parsing guide](https://github.com/D4Vinci/Scrapling/blob/3a8bf4e4d3ed91257ba0aa5b7dc66241d1e23840/docs/parsing/adaptive.md) explains saved properties and similarity matching. Recovering a selector does not persist its extracted record. For the wider retrieval architecture, see our [Wigolo local web retrieval review](/blog/wigolo-local-web-intelligence-mcp-review/).

## Does a Scrapling checkpoint contain scraped items?

A Scrapling 0.5 checkpoint contains pending requests and seen request fingerprints. The collected item list belongs to the current crawl invocation, and your application controls any separate database or export. In the pinned implementation, the [crawler engine](https://github.com/D4Vinci/Scrapling/blob/3a8bf4e4d3ed91257ba0aa5b7dc66241d1e23840/scrapling/spiders/engine.py) clears that item list when a crawl starts.

| Store | What it holds | What to expect after resume |
| --- | --- | --- |
| Scrapling checkpoint | Pending requests and seen fingerprints | Remaining requests can resume from the saved state |
| `result.items` | Items retained during this invocation | Earlier invocations' collected items are absent |
| Application record store | Records your code commits to a database or another persistent destination | Retention depends on your storage and write policy |

This matters when you export after each invocation. In 0.5, `result.items.to_json(path)` writes the current list to that path. The [export implementation](https://github.com/D4Vinci/Scrapling/blob/3a8bf4e4d3ed91257ba0aa5b7dc66241d1e23840/scrapling/spiders/result.py) replaces the file; `to_jsonl(path)` also opens it in write mode. Neither method merges earlier runs for you.

Retain separate per-run exports or commit items to a database. Our fixture demonstrates what happens when you reuse an export path.

## What survived our pause, resume and replay test?

Our Scrapling test retained all three expected records in the keyed SQLite store after resume, while the reused JSON export contained only B and C. Deliberately replaying the saved checkpoint emitted B and C again. The append log grew to five writes; the keyed store still contained three records.

The fixture serves immutable records A, B and C over loopback HTTP. It uses one concurrent request, disables periodic checkpoints, and pauses after A's database commit. Every phase starts a fresh Python worker. The pause-and-resume sequence shares one database and export path; the control and injected-failure runs use separate directories.

| Observed phase | Returned items and JSON export | Committed SQLite IDs | Outcome |
| --- | --- | --- | --- |
| Uninterrupted control | A, B, C | A, B, C | All expected records present |
| Graceful pause after A | A | A | Checkpoint retained B and C as pending |
| Fresh-process resume | B, C | A, B, C | Same-path JSON export was missing A |
| Deliberate replay of our saved checkpoint | B, C | A, B, C | Append log: A once, B twice, C twice |
| Separate run with injected B write failure | A, C | A, C | `completed=True`, one recorded callback error, B missing |

For replay, we restored our paused run's checkpoint after the resumed crawl completed. This exercises repeated requests; it does not simulate an unexpected crash or measure what survives a power failure.

In the failure case, our item hook raised before writing B, and `on_error` logged the exception without retrying. Scrapling finished the remaining crawl. The manifest check exposed the missing record.

[Download the Scrapling checkpoint fixture and recorded results](/downloads/scrapling-checkpoints-scraped-items-fixture.zip). The package includes the local pages, expected-record manifest, worker and runner code, dependency versions, JSON reports, append logs and readable database exports.

## How can you reproduce the checkpoint test?

Run the downloaded fixture in a fresh Python environment with Scrapling's fetchers extra. Its runner starts a local HTTP server, launches the five workers, and checks their exported files and database records independently. Browser binaries, an MCP client and an AI model are unnecessary for this HTTP-only experiment.

```
python3.12 -m venv .venv
mkdir -p .cache/pip .cache/tmp
PIP_DISABLE_PIP_VERSION_CHECK=1 PIP_CACHE_DIR="$PWD/.cache/pip" TMPDIR="$PWD/.cache/tmp" .venv/bin/python -m pip install -r requirements-tested.txt
.venv/bin/python run_fixture.py --output results-reproduced
.venv/bin/python verify_payloads.py --results results-reproduced
```

Run those commands from the extracted fixture directory on Linux. The recorded run used Python 3.12.14; `requirements-tested.txt` pins its package versions. We verified the [Scrapling 0.5 wheel](https://pypi.org/project/scrapling/0.5/#files) against its published SHA-256 before installation:

```
366f0ec9f1e7e998900e6c5e3a4063ea34062326394ced64e5c389b690572726
```

Inspect `summary.json`, `payload-validation.json` and the phase reports in your output directory. The server stays alive across workers so checkpointed URLs remain valid. Each rerun creates its own checkpoint; process IDs, the local port and checkpoint bytes can differ. The checks compare record identities and payloads with the fixture manifest.

## Where should you save extracted records?

Save Scrapling records in an application-owned destination through `on_scraped_item`, using an identity that matches your collection. Commit successful writes before returning from the hook. Scrapling awaits this hook before retaining the item, but the hook also needs an explicit error policy; a unique database key cannot recover a write that never committed.

The [Spider hook API](https://github.com/D4Vinci/Scrapling/blob/3a8bf4e4d3ed91257ba0aa5b7dc66241d1e23840/scrapling/spiders/spider.py) provides the integration point. This is the executed fixture's database write, with the surrounding instrumentation omitted:

```
self.database.execute(
    "INSERT INTO items (collection_id, item_id, payload_json) VALUES (?, ?, ?) "
    "ON CONFLICT(collection_id, item_id) DO UPDATE SET payload_json=excluded.payload_json",
    (self.manifest["collection_id"], item["id"], json.dumps(item, sort_keys=True)),
)
self.database.commit()
```

The primary key is `(collection_id, item_id)`. An upsert inserts a new row or updates the row with the same key, so repeated writes of A, B or C converge within this immutable collection. The download includes the schema and complete hook.

For changing data, choose whether new observations replace or version older ones. Include the observation scope in the identity when history matters. Otherwise, deduplication can overwrite evidence you meant to retain.

A production integration needs a recoverable failed-record ledger or an explicit retry policy, followed by reconciliation. Validate that policy against your sink's failures; this SQLite fixture does not establish transaction recovery for another database.

## How do you know the resumed dataset is complete?

Reconcile Scrapling's persistent output against the collection you intended to gather, including expected identities and required fields. In our fixture, the independent manifest makes missing A or B unambiguous. On a real website, the expected population needs its own discovery method and scope; a finished request queue cannot supply that definition.

| Acceptance question | Evidence to inspect | What the fixture showed |
| --- | --- | --- |
| Are expected records missing? | Expected IDs minus stored IDs | A missing from resumed JSON; B missing after the injected write failure |
| Were records processed again? | Per-ID emission or write counts | B and C each written twice in the replay sequence |
| Does the final store retain one intended record per key? | Database rows under the chosen collection identity | Three rows after resume and deliberate replay |
| Do required values match the intended records? | Stored payloads checked against the fixture manifest | IDs, names and source URLs matched after resume and replay |
| Are any write errors unresolved? | Item-hook errors and recovery records | The injected B failure prevents acceptance despite crawl completion |

The [advanced crawl guide](https://github.com/D4Vinci/Scrapling/blob/3a8bf4e4d3ed91257ba0aa5b7dc66241d1e23840/docs/spiders/advanced.md) documents pause/resume, streaming and output hooks. Use those building blocks to preserve data, then decide separately when to release the collection downstream.

If an agent or RAG index consumes only the resumed JSON file in this example, it receives two records. Publish the reconciled collection from your persistent store instead. Process-level recovery is a further concern; our [AX runner recovery and self-hosting guide](/blog/google-ax-agent-executor-security-self-hosting/) covers that separate boundary.

## Does the MCP server change this persistence contract?

Scrapling's MCP server exposes web and browser operations to compatible agents, but this experiment exercised the Python Spider API. Its checkpoint results therefore do not establish persistence for an MCP workflow. Define where your agent saves returned content, then verify that store across the interruptions your own workflow must tolerate.

The [v0.5 MCP guide](https://github.com/D4Vinci/Scrapling/blob/3a8bf4e4d3ed91257ba0aa5b7dc66241d1e23840/docs/ai/mcp-server.md) lists 16 tools, including request sessions, browser actions, targeted extraction, screenshots, network inspection and saved browser state. A saved login and a saved extracted dataset need separate handling.

## When would we use Scrapling for agent ingestion?

We would use Scrapling when a Python integration benefits from its fetching, parsing and spider controls, and the application also owns record persistence. Start with a small approved collection, choose item identities, exercise resume and replay, and inspect stored results. Add realistic write failures before relying on the collection downstream.

Use representative pages, the required record schema and restart requirements to define your ingestion acceptance test.

## Scrapling checkpoint FAQ

### Does to_jsonl append records from earlier runs?

No. Scrapling 0.5 opens the target JSONL file in write mode. Calling to_jsonl with the same path after another invocation replaces that file with the current item list. Preserve separate run files and merge them deliberately, or write records to a persistent sink while crawling. The JSONL extension alone does not imply append behavior.

### Does request deduplication prevent duplicate data rows?

Request deduplication and record identity solve different tasks. Different requests can produce the same logical item, and replaying a saved checkpoint can process its pending requests again. In our controlled replay, B and C appeared twice in the successful-write log. A collection-and-item database key kept the final store at three rows; it did not prevent the repeated processing.

### Does saved browser login state include a crawl's extracted data?

No. Scrapling's documented browser-state export saves cookies, local storage and IndexedDB. It does not contain the Spider's extracted item collection. The guide also excludes tabs, browser settings and sessionStorage from that browser snapshot. Treat authentication state and the destination for extracted records as separate application resources. Our fixture did not execute a browser-state save or restore.

### Does AutoThrottle keep its learned delays after resume?

The official crawl guide says learned AutoThrottle delays are not stored in a checkpoint. A resumed crawl starts from the configured starting delay. Preserve the intended throttle configuration in your spider and observe its behavior after restart. This is a documented implementation boundary; our single-request-concurrency fixture did not benchmark or exercise AutoThrottle adaptation.

### Does this test prove recovery after a power failure?

No. We used an explicit graceful pause, new Python worker processes and deliberate replay of our own saved checkpoint. We did not kill a process during a database commit, interrupt storage writes or cut power. The result establishes the observed checkpoint, export and keyed-store behavior for these controlled cases. Hard-crash durability needs a separate test against the actual storage and runtime.

### Can a manifest prove that you found every page on a website?

A manifest checks completeness relative to the population it lists. Our fixture defines three immutable records, so missing identities are directly measurable. An unknown website may have undiscovered pagination, changed records or inaccessible sections. Establish the crawl scope and discovery method independently, then reconcile the records that scope requires. A passing three-record test cannot establish complete coverage of another site.

Models and infrastructure

## Read next

Model selection, inference economics, local deployment, compression and serving architecture.

[Start here**Self-Hosting LLMs in the EU: When Open Weights Actually Pay Off**](/blog/self-hosting-llms-eu-cost/)

- [How to Validate Open Ontologies Derivation Certificates](/blog/open-ontologies-proof-verification/)
- [Cue Turn Detection: Phone Numbers, Benchmarks and Limits](/blog/cue-turn-detection-phone-numbers/)
- [Google SAM is now Agent Mesh: MCP tools across machines](/blog/google-sam-agent-mesh-mcp-tools/)
- [EdgeQuake: PostgreSQL GraphRAG for MCP agent memory](/blog/edgequake-postgresql-graphrag-mcp-agent-memory/)
- [BitNet on CPU: 2B4T, real RAM needs and the 100B claim](/blog/bitnet-cpu-llm-ram-requirements/)

[Previous article**How to Validate Open Ontologies Derivation Certificates**](/blog/open-ontologies-proof-verification/) [Next article**Cue Turn Detection: Phone Numbers, Benchmarks and Limits**](/blog/cue-turn-detection-phone-numbers/)

[**Back**](/blog/overview/)

[![Kevin Riedl](/img/team/kevin.webp)](/team/kevin-riedl/)

[Kevin Riedl](/team/kevin-riedl/) https://linkedin.com/in/wsdt

8 min read · 3 Nov 2026 Last reviewed November 3, 2026

[**Next**](/blog/wigolo-local-web-intelligence-mcp-review/)