Pagination and Rate Limiting

Calling REST APIs with Python covers one request. Real ingestion rarely stops at one — a source with more than a page of results, and a source that will refuse to answer if you ask too fast, are both the normal case rather than the exception.

Offset pagination breaks under writes#

The simplest pagination scheme uses a page number or a row offset:

GET /orders?offset=0&limit=100GET /orders?offset=100&limit=100GET /orders?offset=200&limit=100

This is easy to reason about and fails silently the moment rows are inserted or deleted between requests. If a row is deleted from page 1 while you are still reading it, every later page shifts left by one — the row you would have read at offset 250 is now at offset 249, and one row gets skipped entirely. Offset pagination over a table that is actively being written to will eventually skip or duplicate rows; there is no client-side fix, because the server does not remember where "page 3" was relative to the data as it existed when page 1 was fetched.

Cursor pagination survives writes#

Cursor-based pagination returns an opaque token alongside each page that means "everything after this specific record," rather than "the n-th through (n+100)-th row right now":

def fetch_all_orders(client: "OrdersAPIClient") -> list[dict]:    orders = []    cursor = None
    while True:        page = client.get_orders(cursor=cursor, limit=100)        orders.extend(page["results"])        cursor = page.get("next_cursor")        if cursor is None:            break
    return orders

Because the cursor is anchored to a record — often that record's primary key or creation timestamp — rows inserted after you started reading do not shift anything you have already fetched, and rows deleted after you started do not cause a skip. GitHub's REST API and Stripe's API both default to this style; when a source offers both, prefer cursor pagination for anything that needs to run unattended.

Rate limiting: 429 is not a failure to retry blindly#

A 429 Too Many Requests response means the source is asking you to slow down, not that anything is broken. Most APIs that rate-limit include a Retry-After header — defined in RFC 9110, the current HTTP semantics specification — telling you exactly how many seconds to wait:

import time
def request_with_rate_limit_handling(session, url, **kwargs):    response = session.get(url, **kwargs)    if response.status_code == 429:        wait_seconds = int(response.headers.get("Retry-After", 5))        time.sleep(wait_seconds)        response = session.get(url, **kwargs)    response.raise_for_status()    return response

This handles one retry explicitly. A source under sustained load can return 429 more than once in a row, which is where a general backoff strategy becomes necessary rather than optional.

Exponential backoff with jitter#

When a source does not provide Retry-After — or for 5xx errors, where there is no header telling you how long the problem will last — exponential backoff increases the wait time after each consecutive failure, so a struggling server gets progressively more breathing room instead of a constant hammer of retries:

import randomimport time
def call_with_backoff(fn, max_attempts=5, base_delay=1.0):    for attempt in range(max_attempts):        try:            return fn()        except requests.HTTPError as exc:            status = exc.response.status_code            if status < 500 and status != 429:                raise            if attempt == max_attempts - 1:                raise            delay = base_delay * (2 ** attempt) + random.uniform(0, 1)            time.sleep(delay)

Two details matter more than the formula itself. First, the status < 500 and status != 429 check: a 400 Bad Request will not succeed by waiting and retrying, so it is re-raised immediately instead of burning through the attempt budget — the same client-error-vs-server-error distinction introduced in Calling REST APIs with Python. Second, random.uniform(0, 1) — the jitter. Without it, many clients that all failed at the same moment (a brief server outage, for instance) all retry at exactly the same moment too, recreating the load spike that caused the failure. Adding a small random delay spreads retries out over time instead.

Combining pagination and retries#

The two concerns compose: pagination decides what to fetch next, retry logic decides what to do when a single fetch fails. Wrapping the page fetch in call_with_backoff keeps them separate instead of tangled into one function:

def fetch_all_orders(client: "OrdersAPIClient") -> list[dict]:    orders = []    cursor = None
    while True:        page = call_with_backoff(            lambda: client.get_orders(cursor=cursor, limit=100)        )        orders.extend(page["results"])        cursor = page.get("next_cursor")        if cursor is None:            break
    return orders

Common mistakes#

Paging by offset against a live table#

Works fine in testing against a static fixture, then silently skips or duplicates rows in production once real writes are happening concurrently. Use a cursor when the API offers one.

Retrying 4xx errors the same way as 5xx#

A malformed request or an expired token will fail identically on every retry. Burning through a retry budget on an error that cannot self-resolve just delays the point at which you notice and fix the real problem.

No jitter on backoff#

Synchronized retries from many clients recreate the load spike that caused the original failures. A few lines of randomness fix this for the cost of slightly less predictable wait times.

Ignoring Retry-After when the server provides it#

Computing your own backoff delay when the server already told you exactly how long to wait wastes the more accurate information — and can retry too early, extending the throttling period further.

Further reading#

• RFC 9110: HTTP Semantics — the current specification defining Retry-After and the full status code taxonomy.
• Exponential backoff — the general algorithm behind the retry loop above.
• Rate limiting — why sources impose limits and the common strategies for enforcing them.
• Using pagination in the REST API — GitHub's cursor-based implementation, as a concrete worked example.
• How pagination works — Stripe's version of the same pattern.

See also#