Scraping Pagination is the process of collecting data that a website distributes across multiple pages or batches. It is a fundamental part of web scraping because product catalogs, search results, news archives, directories, job listings, and marketplaces rarely place every available record on a single page.

Scraping the first page is usually straightforward. The real challenge begins when a scraper needs to determine where the next page is, recognize when the dataset ends, avoid collecting the same record twice, and continue reliably when a crawl contains hundreds or thousands of pages.

There is no universal pagination system. Traditional websites may use numbered URLs, while APIs often rely on offsets or cursors. Modern JavaScript applications may hide pagination behind infinite scroll or a Load More button.

A reliable scraper should therefore identify the website’s pagination model before deciding how to collect the data.

This guide covers seven common pagination methods and explains how to turn them into a reliable multi-page data collection workflow.

How Pagination Works in Web Scraping

Pagination allows a website to divide a large dataset into smaller responses.

Suppose an ecommerce category contains 10,000 products. Returning every product in a single response would be inefficient, so the website might display 50 products at a time.

The logical structure becomes:

Products 1–50 → Products 51–100 → Products 101–150 → ...

The visible interface may show page numbers, a Next button, infinite scrolling, or no obvious pagination control at all.

For a scraper, the important question is not what the pagination looks like to a user. It is how the application requests the next batch of data.

The seven most common patterns are:

Pagination Method	Typical Pattern	Common Scenario
Numbered pages	?page=2	Catalogs and directories
Next-page links	Next → URL	Blogs and archives
Offset pagination	?offset=50&limit=50	APIs and catalogs
Cursor pagination	?cursor=abc123	Dynamic APIs
Infinite scroll	Scroll triggers another request	Feeds and modern websites
Load More	Button requests another batch	Product/search pages
API pagination	JSON response returns next page/token	JavaScript applications

Before building a pagination loop, inspect how the website actually loads additional records. Assuming that every site supports ?page=2 is one of the easiest ways to build a scraper that works on the first page and fails everywhere else.

7 Ways to Scrape Paginated Data

1. Scrape Numbered Pages

Numbered pagination is the most straightforward model.

A website might expose URLs such as:

https://example.com/products?page=1
https://example.com/products?page=2
https://example.com/products?page=3

Once this pattern is identified, the scraper can increment the page number after each successful request.

A basic Python implementation might look like this:

import requests
from bs4 import BeautifulSoup

page = 1

while True:
    url = f"https://example.com/products?page={page}"
    response = requests.get(url, timeout=20)
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")
    products = soup.select(".product")

    if not products:
        break

    for product in products:
        print(product.get_text(strip=True))

    page += 1

The loop itself is simple. The more important part is deciding when it should stop.

Some websites return an empty page after the final result. Others redirect an invalid page number back to the last valid page. Some may continue returning a successful HTTP status even when no useful records remain.

For that reason, the scraper should validate the returned content rather than assuming that an HTTP error will indicate the end of pagination.

Numbered pagination works particularly well when the URL structure is stable and the total number of pages changes infrequently.

2. Follow Next-Page Links

Not every website exposes a predictable numerical pattern.

Instead, the current HTML document may contain a link pointing to the next page:

<a class="next" href="/products?page=2">Next</a>

In this case, the scraper does not need to generate the next URL. It can extract it directly from the current page.

The process becomes:

Fetch Current Page → Extract Records → Find Next Link → Follow Next URL

This can be more robust than manually constructing URLs because the website itself provides the navigation path.

It is particularly useful when pagination URLs contain category identifiers, sorting parameters, filters, or other values that would otherwise need to be reconstructed manually.

The crawler should still keep track of URLs it has already visited. If a website contains a broken pagination link that points back to an earlier page, visited-URL tracking can prevent an endless loop.

The crawl ends naturally when no valid next-page link remains.

3. Use Offset and Limit Pagination

Offset pagination is common in APIs and data-heavy websites.

Instead of requesting a specific page, the client tells the server where the next batch should begin and how many records it wants.

For example:

?offset=0&limit=50
?offset=50&limit=50
?offset=100&limit=50

If the limit is 50, the crawler can increase the offset by 50 after each successful batch.

Conceptually:

Request 1 → Records 1–50
Request 2 → Records 51–100
Request 3 → Records 101–150

Offset pagination is easy to implement, but it introduces an important data-quality issue when the underlying dataset changes during the crawl.

Suppose a marketplace adds several new listings while a long scraping job is running. Existing records may shift between offsets. A record previously located in one batch may move into another.

This can result in duplicate or missing records.

The solution is not simply to change the pagination logic. The scraper should also identify each collected record using a stable key such as a product ID, listing ID, article ID, or canonical URL.

Pagination determines which batch to request.

A stable identifier determines which record has actually been collected.

Keeping those responsibilities separate makes the final dataset much more reliable.

4. Handle Cursor-Based Pagination

Cursor pagination works differently from numbered pages and offsets.

Instead of calculating where the next batch begins, the client receives a continuation token from the server.

A simplified response might look like:

{
  "items": [
    {"id": 101},
    {"id": 102}
  ],
  "next_cursor": "eyJpZCI6MTAyfQ=="
}

The scraper sends next_cursor with the following request and receives another batch plus another cursor.

The process becomes:

Initial Request → Records + Cursor → Next Request → Records + New Cursor

The important rule is that the scraper should normally treat the cursor as opaque data.

It does not need to understand or calculate what the token means. It only needs to preserve the value and send it back according to the endpoint’s expected pagination flow.

Cursor-based pagination is often useful for dynamic datasets because the continuation token can represent a more specific position than a simple page number, depending on the implementation.

It also has implications for crawl recovery.

If the collection process stops after thousands of records, preserving the last valid cursor may help a crawler resume from a known continuation point when the endpoint supports reusable cursors.

However, developers should not assume that every cursor remains valid indefinitely. Some services expire continuation tokens or tie them to session state.

5. Scrape Infinite Scroll

Infinite scroll removes visible page numbers entirely.

As a visitor moves toward the bottom of the page, JavaScript requests another batch of content and inserts the new records into the document.

From the user’s perspective:

Scroll → More Items Appear → Scroll → More Items Appear

From the application’s perspective, the process is often closer to:

Scroll Event → JavaScript → XHR/Fetch Request → Data Response → Render New Items

This distinction matters for scraping.

One approach is to use a browser automation tool such as Playwright, Selenium, or Puppeteer, repeatedly scroll the page, and extract newly rendered elements.

That is sometimes necessary, especially when JavaScript execution and browser state are essential to the application.

But it should not automatically be the first solution.

Inspecting browser network activity may reveal that infinite scrolling is simply calling a paginated data endpoint in the background.

If the underlying request is appropriate to use and can be reproduced reliably, collecting the structured response can be considerably more efficient than repeatedly rendering the entire page in a browser.

The choice therefore depends on the application:

Browser-based approach: useful when rendering and interaction are required.

Request-based approach: useful when pagination is exposed through a suitable data request.

The key is to understand what the infinite-scroll interface is doing underneath.

6. Handle Load More Buttons

A Load More button is closely related to infinite scrolling, except the visitor explicitly triggers the next batch.

A page might initially show 20 products:

20 Products → Load More → 40 Products → Load More → 60 Products

The button itself is only the visible interface.

Clicking it may send a page number, an offset, a cursor, or another continuation parameter to the server.

This means a scraper has two options.

If browser interaction is genuinely required, automation can click the button and wait for new records to appear.

Alternatively, if the button triggers an accessible background request suitable for the collection workflow, the scraper may be able to work with the underlying pagination mechanism directly.

In either case, the crawler needs a stopping condition.

The Load More button disappearing is one obvious signal. Another is an API response containing no continuation token.

A particularly useful safeguard is checking whether the latest request produced any new unique records.

If repeated interactions return the same record IDs, continuing indefinitely only wastes requests.

7. Use API Pagination

Many modern websites retrieve visible data through APIs behind the frontend.

The initial HTML may contain relatively little useful information. JavaScript then requests product, article, search, or listing data and renders it into the page.

If that data source is publicly accessible as part of the normal web application and its use is compatible with the site’s rules, understanding its pagination can make collection more efficient.

Instead of:

Download HTML → Execute JavaScript → Render Page → Parse DOM

the workflow may become:

Request Structured Data → Parse Response → Follow Pagination → Store Records

An API response may already provide fields such as:

product_id
title
price
currency
availability
next_cursor

This can reduce the amount of HTML parsing required.

API pagination can still use any of the mechanisms discussed earlier. Some endpoints use page numbers, others use offsets, and many modern APIs use cursors.

The important point is that API pagination is a delivery method, not necessarily a unique pagination algorithm.

Developers should also remember that internal application endpoints may change without notice and should not be treated as guaranteed public APIs. Authentication and access controls must be respected.

How to Make Pagination Scraping Reliable

Once the scraper can reach the next page, the next challenge is making sure it collects the entire dataset correctly.

This becomes increasingly important as a crawl grows.

Fetching five pages may require little more than a loop. Fetching 50,000 pages means requests can time out, workers can restart, records can move between pages, and temporary failures can occur before the job finishes.

A more reliable workflow looks like this:

Fetch → Validate → Parse → Deduplicate → Save → Checkpoint → Continue

The first important principle is deduplication.

Page position should never be treated as the identity of a record. A product appearing as item eight on page five today may appear on page four tomorrow.

Whenever possible, use a stable identifier such as a product ID, listing ID, article ID, or canonical URL.

The second principle is explicit stopping logic.

Depending on the pagination model, the end may be indicated by the disappearance of a Next link, an empty response, the absence of a cursor, fewer records than the requested limit, or no new unique records.

A crawler should recognize the actual end condition instead of continuing until an arbitrary page number.

The third principle is checkpointing.

For long crawls, progress should be saved periodically. A worker that fails after processing 30,000 pages should not require the entire job to restart from page one.

Useful state can include the last completed page, current offset, continuation cursor, completed URLs, or queued jobs.

Finally, temporary failures must not be confused with the end of pagination.

If page 138 times out, that does not mean page 138 contains no records.

A production crawler should distinguish outcomes such as:

PAGINATION_END

NETWORK_TIMEOUT

HTTP_ERROR

PARSE_ERROR

PROXY_CONNECTION_ERROR

Only a confirmed pagination-ending condition should terminate the crawl. Temporary network or application failures can follow an appropriate retry policy.

These safeguards turn pagination from a simple loop into a reliable data collection process without making the architecture unnecessarily complicated.

Using Residential Proxies for Large Pagination Jobs

Pagination and proxies solve different problems.

Pagination answers:

Where is the next batch of data?

A proxy answers:

Through which network route is the request being sent?

For a small public dataset with a few pages, proxy infrastructure may not be necessary.

For larger authorized collection projects involving many pages, multiple markets, or geographically distributed observations, network routing can become part of the overall scraping architecture.

A typical workflow might look like:

Pagination Queue → Scraping Worker → Residential Proxy → Target Page → Parser → Dataset

The crawler remains responsible for discovering pages, managing cursors, controlling concurrency, parsing responses, deduplicating records, saving progress, and retrying temporary failures.

The residential proxy provides the network route.

This distinction also helps when choosing between rotating and sticky proxy sessions.

Changing IP addresses after every paginated request is not automatically better.

Suppose pages 1 through 10 belong to one logical session and the website maintains relevant session state. Keeping a consistent network route during that sequence may provide more stable observations:

Page 1 → IP A
Page 2 → IP A
Page 3 → IP A
...

For independent and stateless jobs, different sessions may be appropriate.

The correct strategy depends on the collection workflow rather than on a rule that IP addresses should always rotate as frequently as possible.

ColaProxy can provide the residential proxy network layer for compatible and authorized multi-page data collection. For regional projects, separate crawl jobs can use residential routes associated with their intended target markets, while sticky sessions can maintain network consistency where a logical multi-page workflow requires it.

A clean division of responsibilities looks like this:

Crawler: pagination, scheduling, retries, parsing and storage
Proxy layer: network routing and session routing
Dataset layer: validation, deduplication and persistence

Keeping these layers separate makes a large scraping system easier to troubleshoot and scale.

A Practical Multi-Page Scraping Workflow

A good pagination scraper starts by understanding the target website rather than immediately writing a for loop.

First determine how additional data is requested.

Does the URL contain a page number? Does the HTML provide a Next link? Does the application use an offset? Does the response contain a cursor? Does scrolling trigger an XHR request?

Once the mechanism is identified, define the stopping condition before scaling the crawl.

Then decide how records will be uniquely identified. This is particularly important for datasets that can change while pagination is in progress.

For a larger job, store crawl progress and separate temporary failures from successful completion.

The complete process can be summarized as:

Discover Pagination → Fetch → Validate → Parse → Deduplicate → Save → Checkpoint → Continue

If geographic network context matters, the proxy layer can be inserted into the fetch stage:

Discover Pagination → Worker → Residential Proxy → Fetch → Validate → Parse → Deduplicate → Save

This architecture works whether the website uses numbered pages, offsets, cursors, or JavaScript-driven pagination.

Only the mechanism for discovering the next batch changes.

That is the central idea behind scalable pagination scraping: the pagination model may change, but the surrounding data pipeline can remain consistent.

Conclusion

Scraping Pagination becomes much easier once the scraper stops treating every website as a sequence of numbered pages.

Some websites expose predictable ?page= URLs. Others provide Next links or offset parameters. Modern applications increasingly use cursors, infinite scrolling, Load More interactions, and structured API responses.

The seven approaches covered here address the most common models:

Numbered Pages → Next Links → Offset → Cursor → Infinite Scroll → Load More → API Pagination

For small projects, identifying the correct pagination method may be enough.

For larger projects, reliability becomes equally important. Stable record identifiers, clear stopping conditions, checkpoints, error classification, and controlled retries help prevent incomplete or duplicated datasets.

Residential proxies can then provide a network routing layer when authorized multi-page collection requires distributed or regional network observations, while the crawler remains responsible for pagination itself.

The goal is not to make pagination complicated.

It is to build a workflow where the scraper always knows three things:

where to request next, whether the returned data is valid, and when the collection is actually complete.

FAQ

What is scraping pagination?

Scraping pagination is the process of collecting data distributed across multiple pages or batches instead of extracting only the first page of a website.

How do I scrape multiple pages?

First identify how the website loads the next batch. Common methods include numbered URLs, Next links, offsets, cursors, infinite scrolling, Load More buttons, and paginated API responses.

How do I know when pagination ends?

The stopping condition depends on the website. Common signals include no Next link, no continuation cursor, an empty result set, fewer records than the requested limit, or no new unique records.

What is cursor pagination?

Cursor pagination uses a server-provided continuation token to identify where the next request should continue. The scraper normally preserves and returns that token rather than calculating the next page itself.

How do you scrape infinite scroll?

Browser automation can scroll the page and collect rendered elements. Alternatively, developers can inspect whether scrolling triggers a suitable XHR or fetch request that exposes the underlying pagination more directly.

How do I prevent duplicate data when scraping multiple pages?

Use a stable identifier such as a product ID, listing ID, article ID, or canonical URL instead of treating a record’s page position as its identity.

Do I need residential proxies for pagination scraping?

Not necessarily. Pagination and proxy routing solve different problems. Residential proxies can be useful for compatible large-scale or geographically distributed collection workflows, but a proxy is not required simply because a website contains multiple pages.

Can ColaProxy support multi-page scraping?

ColaProxy can provide residential proxy routing for compatible and authorized web data collection. The crawler should separately manage pagination, request scheduling, retries, parsing, deduplication, and storage.