Calling REST APIs with Python and Pagination and Rate Limiting both assume a request gets an answer in one round trip, maybe repeated across pages. Large exports — a year of ad spend, a CRM's full contact list, an analytics platform's raw event log — are usually too big to compute synchronously. These sources hand back a job, not an answer, and expect you to check back later. Google's API design guide calls this shape a long-running operation; the concrete mechanics are the same regardless of what a specific vendor calls it.
Three steps: create, poll, download#
def start_export(client, report_type: str, date_range: dict) -> str: response = client.post("/exports", json={"type": report_type, "range": date_range}) return response.json()["job_id"]The job ID is the only thing this call returns — not the data. Everything else happens in two more round trips, spread out over however long the export actually takes to compute on the source's side.
Polling is not the retry loop from before#
It is tempting to reach for the exact backoff loop from Pagination and Rate Limiting, but polling for job status solves a different problem than retrying a failed request. A retry backs off after a failure that may or may not happen again. Polling checks in on a job that has not failed — it simply is not done yet — so the wait is not a response to an error at all:
import time
def wait_for_export(client, job_id: str, poll_interval=10, timeout=1800) -> dict: deadline = time.time() + timeout while time.time() < deadline: status = client.get(f"/exports/{job_id}").json() if status["state"] in ("completed", "failed"): return status time.sleep(poll_interval) raise TimeoutError(f"Export {job_id} did not finish within {timeout}s")Two details separate this from a retry loop. First, it checks a state
field in a successful response, not an exception — a 200 response saying
"state": "running" is not a failure, so there is nothing to catch. Second,
it tracks an overall deadline (timeout) rather than a number of
attempts, because how long an export takes is a property of the job, not of
the network — a fixed attempt count would either give up too early on a
large export or, at a short poll interval, hammer the status endpoint with
hundreds of checks on a small one.
A fixed interval is usually fine here
Unlike a failed request — where exponential backoff avoids piling retries onto a struggling server — a job's remaining time does not get shorter because you asked less often. A fixed poll interval, long enough that the status endpoint is not under meaningful load, is simpler and just as effective. Growing the interval only helps if the source's own docs specifically ask for backoff on the status check itself.
Distinguishing "still running" from "failed"#
The loop above already treats "failed" as a terminal state, same as
"completed" — both stop the poll. What happens next has to differ:
def download_export(client, job_id: str): status = wait_for_export(client, job_id) if status["state"] == "failed": raise RuntimeError(f"Export {job_id} failed: {status.get('error')}") return client.get(status["result_url"])A job that failed is not "still working, check again later" — treating
every non-"completed" state as pending is the single most common bug in
poll-until-ready code, because it turns a job that failed once, cleanly,
into one that appears to hang until the overall timeout finally fires
with a far less useful error message than the one the source already
handed you in status["error"].
The downloaded result is a file, not a response body#
result_url is usually a signed link to a file — CSV, JSON lines, or a
compressed archive — served separately from the API itself, sometimes with
its own short expiry. Once downloaded, that file goes through the same
checks as any other file-based source:
Ingesting Messy Files covers
encoding, malformed rows, and schema drift for exactly this kind of export.
Common mistakes#
Polling faster than the source's rate limit allows#
A tight poll loop against a job that takes twenty minutes can burn through rate limits meant for actual API usage. A poll interval of ten to thirty seconds is reasonable for most exports; check the source's documentation for a recommended minimum.
No overall deadline#
Without timeout, a job that silently stalls on the source's side — not
uncommon under their own load — polls forever. An overall deadline turns an
invisible hang into a clear, logged failure.
Re-creating the job on every retry of the calling code#
If the code that calls start_export retries on a transient failure
without checking whether a job was already started, a source without its
own idempotency key on job
creation ends up computing the same expensive export twice. Persist the
job_id as soon as it is returned, before the poll loop even begins, so a
retry of the calling code can resume watching the existing job instead of
starting a new one.
Further reading#
- AIP-151: Long-running operations — Google's API design guide entry for this exact pattern, vendor-neutral in its reasoning even where a specific API differs in its field names.
- Polling (computer science) — the general technique behind the status-check loop.
- Rate limiting — why the poll interval can't be arbitrarily short.
- Exponential backoff — the retry-specific technique this pattern deliberately does not reuse.
- Idempotence — why the job ID needs to be persisted before polling begins.