Testing Ingestion Code Without Hitting the Network

Every client built across this section eventually needs tests, and the obvious approach — call the real API from the test suite — is usually the wrong one. A real call is slow, requires live credentials in CI, is subject to the source's own rate limits, and cannot reliably produce a 500 or a timeout on demand to prove the retry logic in Pagination and Rate Limiting actually works. The fix is not to skip testing the client — it is to test it against a stand-in for the network instead of the network itself.

Mocking at the right layer#

A test double can sit at several different depths. Patching requests.Session.get directly with unittest.mock works, but couples the test to how the client happens to call requests internally — refactor session.get(...) into session.request("GET", ...) and every test breaks, even though the client's actual behavior did not change. Mocking one layer lower, at the HTTP boundary itself, avoids that:

import responses
@responses.activatedef test_get_orders_returns_parsed_results():    responses.add(        responses.GET, "https://api.example.com/orders",        json={"results": [{"order_id": 1}], "next_cursor": None},        status=200,    )    client = OrdersAPIClient("https://api.example.com", api_key="test")    assert client.get_orders() == [{"order_id": 1}]

responses intercepts at the socket level requests itself uses, so the real Session, raise_for_status, and JSON parsing all actually run — the test exercises the client's real code path, just against a fake server response instead of a real one.

Testing the failure paths on purpose#

The whole reason Pagination and Rate Limiting built a retry loop was to survive a 500 that resolves on the next attempt. A mocked response can simulate exactly that sequence, deterministically, which a real API generally will not do on request:

@responses.activatedef test_retries_after_server_error():    responses.add(responses.GET, "https://api.example.com/orders", status=500)    responses.add(        responses.GET, "https://api.example.com/orders",        json={"results": [], "next_cursor": None}, status=200,    )    client = OrdersAPIClient("https://api.example.com", api_key="test")    assert client.get_orders() == []

responses.add called twice registers two responses in order — the first call to the URL gets the 500, the second gets the 200. If _request_with_backoff from Building a Reusable Ingestion Client did not actually retry, this test would raise requests.HTTPError instead of returning an empty list, failing loudly instead of passing by accident.

What mocked tests cannot catch#

A mocked response encodes an assumption about the API's shape as it was understood when the test was written. If the real API renames a field or changes what a status code means, every mocked test keeps passing — they were never actually checking against the live contract, only against a frozen copy of it. This gap is real and worth naming rather than pretending mocks make integration tests unnecessary.

Two ways teams close it, in increasing order of investment:

• A small, separate smoke-test tier. One or two tests that hit a real sandbox or staging endpoint, run manually or on a slow schedule rather than on every commit, specifically to catch drift the mocks cannot see — deliberately small because these tests are slower and flakier by nature, not because they matter less.
• Consumer-driven contract testing. Tools like Pact formalize the same idea: the consumer's expectations become a shareable contract the provider's own test suite can verify against, closing the gap without either side manually re-checking the other by hand. This is more infrastructure than most single-team ingestion code needs, and earns its place once several teams depend on the same internal API.

Common mistakes#

Mocking inside the client instead of at the HTTP boundary#

Patching the client's own methods (OrdersAPIClient.get_orders, mocked directly) tests nothing but the mock itself — the client's actual retry logic, error handling, and parsing never run. Mock at the network boundary, not above it.

Only testing the happy path#

A suite that only registers 200 responses proves the client can parse a well-formed reply. It says nothing about whether the retry loop, the signature verification, or the error-wrapping logic actually behave the way Common mistakes in earlier articles warned they could fail to.

Treating a green mocked suite as proof the integration still works#

Mocked tests catch regressions in this codebase's handling of a known response shape. They cannot catch the source changing that shape underneath you. A passing suite and a working integration are related, not identical, claims.

Further reading#

• Test double and Mock object — the general vocabulary for stand-ins used in the examples above.
• responses — the library used throughout this article to mock at the HTTP boundary.
• unittest.mock — the standard-library alternative, better suited to mocking non-HTTP dependencies.
• Pact — consumer-driven contract testing, for closing the gap mocked tests structurally cannot.

See also#