Webhooks and Event-Driven Ingestion

Everything so far in this section has been pull: your code decides when to ask. A webhook flips that — the source decides when to send, and your system has to be listening, correctly, at all times. Data Ingestion Fundamentals names this the push side of the first axis; this article covers what receiving push traffic actually requires that pulling never had to.

An endpoint is not enough by itself#

A webhook receiver is, at minimum, an HTTP endpoint that accepts a POST request:

from flask import Flask, request
app = Flask(__name__)
@app.post("/webhooks/orders")def receive_order_event():    payload = request.get_json()    process_order_event(payload)    return "", 200

This works in a demo and is unsafe in production for a specific reason: it processes and trusts anything posted to that URL. Anyone who finds the endpoint — and public webhook URLs are not secret — can send a fabricated payload and have it processed exactly like a real event.

Verifying the signature#

Almost every serious webhook provider signs each payload with a shared secret using HMAC, and sends the signature in a header. The receiver recomputes the same signature from the raw body and compares:

import hashlibimport hmac
def verify_signature(raw_body: bytes, signature_header: str, secret: str) -> bool:    expected = hmac.new(        secret.encode("utf-8"),        raw_body,        hashlib.sha256,    ).hexdigest()    return hmac.compare_digest(expected, signature_header)

Two details make this function correct rather than just plausible. First, it signs raw_body — the exact bytes as received — not a re-serialized version of the parsed JSON. Re-serializing can reorder keys or change whitespace, producing a signature that never matches even for a genuine payload. Second, hmac.compare_digest instead of == — a plain string comparison exits as soon as it finds a mismatched character, which leaks, through response timing, how many leading characters were correct. compare_digest always takes the same time regardless of where the strings first differ, closing that side channel.

@app.post("/webhooks/orders")def receive_order_event():    raw_body = request.get_data()    signature = request.headers.get("X-Signature", "")
    if not verify_signature(raw_body, signature, WEBHOOK_SECRET):        return "invalid signature", 401
    payload = request.get_json()    process_order_event(payload)    return "", 200

Idempotency: the same event, delivered twice#

Webhook senders generally guarantee at-least-once delivery — a network timeout on their end, even after your endpoint successfully processed the event, causes a resend. Data Ingestion Fundamentals covers why this makes idempotence a requirement, not an optimization, for anything receiving push traffic. Most providers include a unique event ID in the payload for exactly this purpose:

def process_order_event(payload: dict) -> None:    event_id = payload["event_id"]
    if already_processed(event_id):        return
    apply_order_event(payload)    mark_processed(event_id)

already_processed and mark_processed need to share storage that survives a process restart — a database table with event_id as a unique key is the usual choice, since the uniqueness constraint doubles as the deduplication check under concurrent delivery.

Respond fast, process later#

A webhook sender typically enforces its own timeout — often a few seconds — and treats a slow response the same as a failed one, triggering a retry of an event your system actually already received and is still working on. The fix is to acknowledge receipt immediately and do the real work somewhere else:

@app.post("/webhooks/orders")def receive_order_event():    raw_body = request.get_data()    signature = request.headers.get("X-Signature", "")
    if not verify_signature(raw_body, signature, WEBHOOK_SECRET):        return "invalid signature", 401
    enqueue_for_processing(request.get_json())    return "", 200

enqueue_for_processing writes the payload to a queue or a staging table and returns immediately; a separate worker processes the queue at its own pace, decoupled from the sender's timeout entirely. This is the same push-then-batch composition Data Ingestion Fundamentals mentions — push arrives continuously, but the actual processing runs as its own batch job.

When processing keeps failing: dead letters#

A payload that fails processing every time — a malformed event, a bug triggered by one specific shape of data — should not retry forever and should not silently vanish either. After a bounded number of attempts, move it to a dead letter queue: a separate place to hold events that could not be processed, for a human to inspect:

def process_queued_event(payload: dict, attempt: int) -> None:    try:        apply_order_event(payload)    except Exception:        if attempt >= MAX_ATTEMPTS:            move_to_dead_letter(payload)        else:            requeue(payload, attempt + 1)        raise

This keeps one bad event from blocking every event behind it in the queue, while still preserving the payload — and the fact that it failed — instead of losing it.

Common mistakes#

Trusting the payload without verifying the signature#

The single most common webhook vulnerability: an endpoint that processes whatever is posted to it, from anyone, as if it were genuine. Verify the signature before touching the payload, every time.

Comparing signatures with == instead of compare_digest#

A theoretical timing side-channel, but a free fix — there is no reason to use the vulnerable comparison once the safe one is one import away.

Doing real work inside the request handler#

Slow processing inside the handler itself risks the sender's timeout, which triggers a retry of an event that is, from the sender's point of view, still "in flight" — even though your system already has it.

No idempotency key check#

Without one, a retried delivery — expected, not exceptional, under at-least-once guarantees — applies the same event twice: a double charge, a duplicated row, a counter incremented one too many times.

Further reading#

• Webhook — the general pattern, including the security considerations summarized above.
• HMAC — the signing scheme almost every webhook provider uses.
• hmac — Python documentation — compare_digest and the rest of the standard library module used here.
• Dead letter queue — the general pattern for handling permanently failing messages.
• Flask — the minimal framework used for the endpoint examples above; any HTTP framework works the same way.

See also#