Fix n8n Pagination Missing Records: Retrieve Every API Item with Offset/Cursor/Next-URL (HTTP Request) — for Automation Builders

If your n8n pagination is missing records, you can fix it by choosing the right pagination model (offset, cursor, or next-URL), enforcing stable ordering, and configuring stop conditions so n8n retrieves every API item instead of silently stopping early or looping on the same page.

Many “missing records” problems are not random—they come from a small set of repeatable causes: a next pointer that doesn’t change, an offset that doesn’t increment, unstable sorting that reshuffles results between pages, or a workflow that accidentally drops items during mapping and merging.

You’ll also want a verification method that proves completeness, because “the workflow ran without errors” is not the same as “we captured all data,” especially when APIs are real-time and the dataset can change mid-run.

Introduce a new idea: once you’ve locked down a reliable pagination setup, you can harden it further with edge-case defenses (eventual consistency, concurrency, and checkpointing) so your workflow stays accurate even under n8n timeouts and slow runs.

What does “n8n pagination missing records” mean in the HTTP Request node?

“n8n pagination missing records” means your workflow is paginating an API endpoint but still returns fewer items than the source actually has, usually because the next-page logic, ordering, or stop condition causes n8n to skip pages, stop early, or repeatedly fetch the same page.

Next, it helps to recognize the symptom patterns—because each pattern points to a different root cause.

In practice, “missing records” shows up in a few classic ways:

• The count is lower than expected (you expected 25,000 items, you got 24,300).
• There are gaps (IDs jump, timestamps skip, or a known customer/order never appears).
• You see duplicates—and duplicates often hide missing items because the total item count might look “close enough.”
• Only the first page returns (page 1 works, page 2 never loads).
• The pagination stops with a safety message when the response repeats (common in n8n community threads about repeated responses and pagination stopping).

Most importantly: missing records can be caused by pagination configuration or by workflow handling after pagination. A perfect pagination loop can still “lose” records if your field mapping collapses arrays, merges incorrectly, or hits an n8n field mapping failed scenario in downstream nodes.

Why do paginated API pulls miss items even when pagination is enabled?

Paginated API pulls miss items because pagination is not only about “looping”—it’s about looping correctly under change, and that requires four things to align: a stable sort, a correct next pointer, a correct stop condition, and safe downstream handling.

Next, it helps to separate causes by where the failure happens.

A) Next pointer / page progression fails

• The expression references the wrong response field (e.g., next.after is nested but you read a top-level after).
• The next pointer exists, but it’s not carried forward (cursor resets to the initial value).
• Offset does not increment (page 1 repeats, so page 2 is never reached).

B) Stop condition ends too early

• “Stop when response is empty” triggers because the response path is wrong (you’re checking a field that is sometimes empty).
• “Stop when next is missing” triggers because you’re checking the wrong property.
• A safety guard stops repeated identical responses (you keep fetching the same page, so n8n halts).

C) Ordering changes mid-run

• Offset pagination is vulnerable when new items insert or old items delete between requests, causing duplicates and missing entries across page boundaries.
• Your sort key is not deterministic (sorting by a timestamp alone when multiple records share the same timestamp).

D) Downstream workflow drops items

• A Merge node keeps only one branch.
• A mapping step flattens an array incorrectly.
• An Item Lists / Code step filters out nulls that are actually valid rows.
• A partial failure leads to retries without deduplication, creating duplicates that distract from missing records.

This is why good n8n troubleshooting treats pagination as a system: request → response parsing → next request → termination → validation.

Is missing records usually caused by n8n or by the API you’re calling?

The API is often the deeper cause, while n8n is often the place where the weakness becomes visible—so the right answer depends on what you observe first.

Next, you can separate “n8n configuration” from “API behavior” using these indicators.

More likely an n8n configuration issue

• Page 1 repeats over and over (same payload size, same first/last IDs).
• The cursor/offset value in your requests never changes.
• Pagination stops with “identical response” after several attempts, which points to page progression not changing.

More likely an API behavior issue

• The API does not guarantee stable ordering unless you specify a sort parameter.
• The dataset is real-time (items inserted/deleted) while you paginate with offset.
• The API has eventual consistency or replication lag, so successive reads can disagree temporarily.
• The API returns a next link that expires or is rate-limited, causing mid-run partial results.

A practical stance is: treat n8n as a deterministic executor. If your request parameters progress correctly and your stop condition is correct, but you still see missing records, you’re likely dealing with API ordering, API consistency, or API limits.

How do you set up n8n pagination to retrieve every API item reliably?

You set up reliable n8n pagination by applying one method in 6 steps—choose the pagination model, enforce stable ordering, extract the next pointer correctly, set a safe stop rule, add dedupe/guardrails, and validate completeness—so every page is fetched exactly once and every item survives downstream handling.

Then, you can translate the method into the HTTP Request node’s pagination UI and expressions.

A reliability-first checklist (use this before you tweak anything else):

1. Decide pagination type: offset, cursor/token, or next-URL.
2. Set a stable sort (when the API supports it): order by a monotonic field + tie-breaker.
3. Map the next pointer from the response into the next request (correct path, correct scope).
4. Pick the stop condition that matches the API contract.
5. Add guardrails: max pages, dedupe by ID, retry strategy.
6. Validate completeness: log page metadata and compare counts/IDs/timestamps.

If you’re seeing missing records, don’t start by “increasing page size.” Start by proving your workflow can reliably walk page boundaries.

How do you configure offset-based pagination (offset + limit) without skipping or repeating items?

Offset pagination works best when the dataset is stable; when it isn’t, you must add stability measures to prevent duplicates and missing entries.

A safe offset configuration looks like this:

• Inputs: limit (page size), offset (records to skip)
• Progression rule: offset = offset + limit
• Stop rule: stop when returned items < limit (or when offset >= total, if total exists)

In n8n’s HTTP Request pagination, that usually means:

• Pagination Mode: Update a Parameter in Each Request
• Parameter name: offset (or page, depending on API)
• Parameter value expression: increment based on the previous request state (e.g., page count × limit)
• Complete when: response is empty or returned count < limit

But offset has a known correctness weakness when data changes between page calls: inserts and deletes can shift the “window,” causing duplicates and missing rows.

Next, reduce skipping/repeating with offset by applying three defenses:

1. Force deterministic sorting (if supported): sort=created_at,id or sort=updated_at,id.
2. Use a tie-breaker: if multiple records share the same timestamp, add id as the secondary sort.
3. Prefer “since checkpoint” filters: paginate within a time window (e.g., created_after=checkpoint) rather than across the whole historical dataset every run.

If the API offers cursor-based pagination, treat it as the default choice for dynamic datasets.

How do you configure cursor-based pagination (page token) so the cursor never resets?

Cursor pagination retrieves “the next set” based on a token (cursor) returned by the API, which makes it naturally resistant to inserts/deletes that break offset pagination.

Next, a correct cursor setup requires two truths:

• The cursor must come from the latest response.
• The next request must use exactly that cursor, not a recomputed guess.

In n8n terms, cursor pagination is usually:

• Parameter name: after, cursor, page_token, or similar
• Parameter value: an expression referencing the previous response, like $response.body.paging.next.after
• Stop condition: cursor missing / null / empty

Here’s the mental model that prevents “cursor resets”:

• First request: omit cursor (or set to null)
• Response: extract next_cursor
• Second request: set cursor = next_cursor
• Repeat until the API stops returning next_cursor

If you ever see page 1 repeating, the cursor is either not extracted correctly, not injected into the next request, or overwritten by a default value.

In practical debugging, log the cursor each iteration (first 10 chars is enough). If the cursor value doesn’t change, your workflow is not progressing—don’t “wait longer,” fix the pointer.

How do you configure next-URL pagination (link/next field) so n8n always follows the right link?

Next-URL pagination is often the most reliable because the API hands you the exact URL for the next page—including any filters, sorts, and opaque tokens.

To implement it reliably:

1. Read next from the response (or from response headers, depending on API).
2. Use that URL verbatim (avoid rebuilding query parameters manually).
3. Preserve headers/auth when making the next request.
4. Stop when next is null/empty.

The most common mistake is partial reconstruction: you extract page=2 but forget a filter or cursor parameter, so you fetch the wrong slice.

In n8n, next-URL pagination is also where people trip on response parsing scope: if you reference $response before it exists on the first run, expressions can fail. When you see “undefined on first run,” use a conditional expression that falls back to the initial URL, then switches to the next link once available.

If you can follow the API’s next link directly, do it—especially for complex endpoints.

Which stop conditions prevent early termination and infinite loops in n8n pagination?

There are 5 main “stop when…” strategies that prevent early termination and infinite loops in n8n pagination: (1) next pointer missing, (2) response empty, (3) returned count < limit, (4) max pages cap, and (5) repeated-response detection—choose based on what the API guarantees.

Next, connect stop conditions to the “missing records” symptom, because the wrong stop rule is the easiest way to silently lose data.

What are the safest “stop when…” rules for APIs that don’t return totals?

When the API does not return a total count, you must stop based on signals that are always meaningful.

Next, use this priority order:

1. Stop when next pointer is missing
   Best for cursor/next-URL APIs: if next or next_cursor is absent, you’re done.
2. Stop when response array is empty
   Works for many offset APIs: when the endpoint returns [], no more pages remain.
3. Stop when returned count < limit
   Very common: if you asked for 100 and you got 17, that’s the last page.
4. Add a max-pages cap as a safety fuse
   This is not your “primary stop”; it’s your “prevent runaway” stop.

A solid implementation uses the pointer or empty-array signal as the main stop, plus a max-pages fuse as a protection. That way, if the API misbehaves, you fail safe instead of looping forever—or worse, stopping early without noticing.

How do you fix the “identical response returned 5x” stop so it doesn’t cut off valid data?

You fix the “identical response returned 5x” stop by ensuring the next request actually changes—because that message is almost always telling you the pagination pointer (page/offset/cursor/next URL) is not progressing.

Next, treat it like a debugging checklist, not a mystery:

1) Confirm the pointer changes

• If offset pagination: does offset increase every iteration?
• If cursor pagination: does the cursor token change?
• If next-URL: does the requested URL change?

2) Confirm you reference the right response

• If you read a pointer from the wrong node or wrong scope, you may reuse an old cursor.
• If the response field path is wrong, you may fall back to a default value each time.

3) Watch for caching

Some APIs cache responses unless you include cache-control headers. If you truly request different pages but receive identical payloads, add a header like Cache-Control: no-cache (only if the API honors it).

4) Use a deterministic stop rule

If your response is identical because the API returns a “last page” repeatedly, then a stop rule based on pointer presence is safer than “response empty.”

5) Build in a progress invariant

A progress invariant is a tiny assertion: “the first ID of page N must differ from page N−1” or “cursor must change.” If it fails, stop and alert.

If you also experience n8n timeouts and slow runs, repeated-response loops will make it worse because you burn runtime without gaining new items—fixing pointer progression is the best performance optimization you can make.

How can you validate you truly retrieved every item in n8n?

There are 4 practical ways to validate you retrieved every item in n8n: compare counts against a known total (if available), log page-by-page boundaries, deduplicate and reconcile IDs, and run a controlled backfill test—together they prove completeness instead of assuming it.

Next, validation must cover both halves of the system: pagination correctness and downstream item preservation.

A simple but strong validation workflow looks like this:

• During pagination: capture metadata per page (cursor/offset, returned count, first ID, last ID).
• After pagination: compute totals, unique IDs, duplicates, and missing intervals.
• Downstream: verify the count entering your sink node matches the count leaving the HTTP Request pagination stage.

This is where many teams discover the real issue wasn’t pagination—it was a mapping/merge step that dropped items.

Can you prove completeness (Yes/No) without an API total count?

Yes, you can prove completeness without a total count because (1) stable ordering confirms you walked the entire sequence, (2) a monotonic cursor/offset confirms you progressed without gaps, and (3) ID-based dedupe confirms you captured unique items—although you still need checkpointing if the dataset changes during the run.

Next, define what “complete” means for your use case:

• Complete for a historical backfill: you retrieved all items that existed at the start of the backfill window.
• Complete for an ongoing sync: you retrieved all changes since the last checkpoint.

Without totals, you rely on invariants:

• Cursor must progress until the API stops returning next.
• Each page’s first/last ID must move consistently.
• Deduped unique ID count should equal raw count unless duplicates are expected.

If you paginate by timestamp, beware of n8n timezone mismatch pitfalls: you might think you’re pulling “since midnight UTC,” but you’re actually pulling “since midnight local time,” which can create apparent “missing records” that are really filter boundary errors. Make timestamps explicit (ISO 8601 with timezone), and treat time boundaries as part of your completeness proof.

What data should you log on every page request to catch missing records fast?

You should log 8 fields on every page request: request URL (or params), page number/offset, cursor token (shortened), response item count, first item ID, last item ID, first timestamp, and last timestamp—because those fields reveal pointer progress, ordering stability, and where gaps start.

Next, the table below shows what to log and why it matters.

Log field	Example	Why it matters
Request pointer	offset=2000 / after=abc…	Proves progression
Page index	page=21	Helps debugging + replay
Returned count	100	Validates stop rule
First ID / Last ID	id=901… / id=1000…	Detects gaps + repeats
First/Last timestamp	2026-01-01T…Z	Detects ordering drift
Next pointer	next_after=def…	Confirms continuation
Retry count	2	Reveals instability
Run ID	execution ID	Enables reconciliation

If you keep these logs, “missing records” stops being a feeling—you can point to the exact page boundary where the loss happened.

Also watch for downstream warnings that are not pagination errors but still cause loss, such as an n8n field mapping failed incident that prevents some items from being written even though they were fetched.

What is the best pagination approach in n8n for your API: offset vs cursor vs next-url?

Offset wins for simplicity, cursor is best for correctness on changing datasets, and next-URL is optimal when the API provides canonical navigation links—choose based on ordering stability, dataset churn, and how much you trust the API’s pagination contract.

Next, use a decision framework so you stop “trying random settings” and instead select the strategy that matches your API.

The table below compares the approaches across common criteria.

Criterion	Offset	Cursor/Token	Next-URL
Works on changing datasets	Weak	Strong	Strong (if API is good)
Easy to implement	Strong	Medium	Medium
Jump to page N	Strong	Weak	Weak
Risk of missing/duplicate entries	Higher when data changes	Lower	Lower
Best for backfills	Sometimes	Yes	Yes
Best for ongoing sync	Only with filters/checkpoints	Yes	Yes

Offset’s “missing/duplicate” risk under change is well-known; inserts or deletes between page loads can create duplicates or missing entries.

When does offset pagination break down compared to cursor pagination?

Offset pagination breaks down when the underlying dataset changes while you paginate—because the offset is a position, not an identity—while cursor pagination stays stable because it uses a continuation token tied to a specific ordering boundary.

Next, here are the break-down triggers you can recognize immediately:

• High write volume: new records inserted frequently.
• Deletes/updates: records removed or re-ordered.
• Sorting by a field with ties: many records share the same timestamp.
• Long-running backfills: the longer the run, the more the dataset changes during the run.

In these cases, cursor pagination is safer because it advances from “last seen boundary” instead of “skip N rows.” If you must use offset (because the API only supports offset), then pair it with a stable sort, a time-window filter, and dedupe by ID.

This combination is often enough to eliminate practical missing-record issues.

When is next-url pagination more reliable than building params yourself?

Next-URL pagination is more reliable when the API provides the next link because the API is the source of truth for token encoding, filtering, and continuation rules—so you avoid mistakes like missing a filter parameter, mis-encoding a cursor, or incrementing the wrong value.

Next, next-URL shines when:

• the API uses opaque cursors (you cannot safely compute them yourself),
• the endpoint uses complex filters,
• the API returns different pagination rules per query,
• you need to preserve server-side ordering guarantees exactly.

If the API offers next URLs, follow them. If it doesn’t, prefer cursor tokens. If you only have offset, add guardrails and verification.

What advanced edge cases cause missing records in n8n pagination, and what are the best workarounds?

There are 4 advanced edge cases that cause missing records in n8n pagination—non-deterministic ordering, eventual consistency during reads, pagination vs incremental sync mismatch, and retry/duplicate handling—and each has a specific workaround that improves correctness without overcomplicating your workflow.

Next, these are “micro semantics” problems: they don’t show up in every workflow, but when they do, they can silently damage data quality.

How do you prevent missing records caused by non-deterministic ordering (unstable sort keys)?

You prevent missing records from non-deterministic ordering by enforcing a deterministic sort with a tie-breaker—because if the same record can appear in different positions between requests, pagination can never be reliably complete.

Next, apply these rules:

1. Always specify ordering parameters if the API supports them. Example concept: sort=created_at is not enough if many rows share the same second; use sort=created_at,id.
2. Use a monotonic field (increasing ID, created timestamp, updated timestamp).
3. Treat equal timestamps as a first-class case by using a tie-breaker key such as id.
4. Don’t paginate an unordered endpoint; switch to filters, incremental windows, or an export endpoint instead.

If you’re seeing missing records only sometimes, unstable ordering is one of the top suspects—especially when your dataset is busy.

How do you handle eventual consistency and “data changing while you paginate”?

You handle eventual consistency and data-changing mid-pagination by using checkpointed windows and overlap backfills—because the goal shifts from “one run sees everything perfectly” to “the system converges to completeness over repeated runs.”

Next, use a resilient pattern.

Pattern: Sliding window + checkpoint

• Keep a checkpoint (last seen timestamp/ID).
• Each run pulls from checkpoint - overlap to now.
• Deduplicate by unique ID on write.
• Update checkpoint after successful write.

This pattern tolerates replication lag, late-arriving data, and out-of-order updates.

According to a study by Princeton University from the Department of Computer Science, in 2015, researchers measuring Facebook’s replicated storage reported measurable anomaly rates under strong consistency models, showing anomalies are rare but quantifiable in large-scale systems.

Also note how easy it is to confuse this with a filtering bug: an n8n timezone mismatch can shift your “since” window and look exactly like eventual consistency. When debugging, isolate the cause by testing a fixed historical window where data is not changing to confirm your pagination logic is sound.

What is the difference between pagination and incremental sync (and when should you switch)?

Pagination wins for one-time historical backfills, incremental sync wins for ongoing reliability, and the hybrid (backfill + incremental) is optimal for most production automations because it reduces missing records under change and makes verification easier.

Next, here’s the practical difference:

• Pagination answers: “Give me everything in this dataset, in chunks.”
• Incremental sync answers: “Give me everything that changed since my last checkpoint.”

When to switch to incremental sync:

• The dataset is large and constantly changing.
• Offset pagination causes duplicates/missing entries.
• Runs are long enough that the data changes mid-run.
• You care about ongoing correctness more than “jump to page 10,000.”

A hybrid plan that works well in n8n:

1. Run a historical backfill with cursor/next-URL pagination (or offset + stability).
2. Store the last-seen boundary (timestamp/ID).
3. Schedule an incremental workflow that pulls only changes since that boundary.
4. Reconcile periodically with a small overlap window.

This also reduces operational pain from n8n timeouts and slow runs because incremental runs are smaller and faster.

Which reliability patterns guarantee no missing records even under retries and duplicates?

You guarantee no missing records under retries and duplicates by designing for idempotency and checkpointing—because retries are inevitable, and “exactly once” is achieved at the storage layer, not the network layer.

Next, use these reliability patterns:

1. Idempotent upsert (recommended)
   • Write by unique ID.
   • If the ID exists, update; else insert.
   • This makes retries safe.
2. Deduplicate by stable key before writing
   • Use a Code node to keep a set of IDs (or use database constraints).
   • This is essential when the API can return duplicates across page boundaries.
3. External checkpoint store
   • Store cursor/timestamp in a DB, KV store, or a simple table.
   • Resume from checkpoint after failures.
4. Reconciliation runs
   • Periodically re-fetch a known window and compare counts/IDs.
   • This catches silent issues like mapping drops or partial writes.

If you see missing items despite correct pagination, inspect the last mile: writes and mappings. A single n8n field mapping failed event can drop items without breaking the whole run, so treat write success rate as part of your completeness definition.