Data Ingestion Fundamentals

Data Engineering Overview draws the data path as Sources → Ingestion → Storage → Transformation → Serving. This section is about the first arrow: how data actually gets from a source you do not control into a system you do. Every ingestion problem, regardless of the source, is a combination of two choices.

Two axes: who initiates, and how often#

Pull means your system reaches out and asks for data: polling a REST API on a schedule, querying a database, downloading a file. You control the timing; the source only has to sit there and answer. Push means the source initiates: a webhook firing on an event, a message landing on a queue. The source controls the timing; your system has to be listening.

Batch means data moves in discrete chunks — once an hour, once a day, on demand. Streaming means data moves continuously, record by record or in small micro-batches, as it becomes available.

Most real systems are not purely one cell. A webhook (push, streaming) often writes into a queue that a batch job drains every few minutes — push arrives continuously, but processing happens in batches. The two axes describe the transfer, not necessarily the processing that follows it.

The ingestion contract#

Before either side writes a line of code, a source and its consumer need to agree — explicitly or by convention — on four things:

1. Schema
   What fields exist, what type each one is, and which are optional. An API that adds a field silently is usually fine; one that removes or retypes a field silently breaks every consumer downstream.
2. Freshness
   How stale is acceptable. A dashboard refreshed nightly and a fraud check that needs the last 30 seconds of events have completely different ingestion designs, even if the source is identical.
3. Ordering
   Whether records can arrive out of order, and whether that matters. A stream of independent user signups usually does not care; a stream of account balance updates usually does.
4. Delivery guarantee
   Whether a record can be delivered more than once, less than once, or exactly once. Most real systems give you at-least-once and expect you to handle duplicates — see Idempotence below.

None of these need a formal document for an internal source. But they need an answer, even an implicit one, because the ingestion code you write encodes assumptions about all four whether you state them or not.

Idempotency: the answer to "what if this runs twice"#

An operation is idempotent if running it twice produces the same result as running it once. UPDATE orders SET status = 'shipped' WHERE id = 42 is idempotent — running it five times leaves the row in the same state as running it once. INSERT INTO orders VALUES (...) without a uniqueness constraint is not — five runs create five rows.

The practical consequence: ingestion code should either write in an idempotent way (upsert on a natural or explicit key, rather than blind insert) or track what has already been processed (a watermark, a set of seen IDs) and skip duplicates before they reach storage. Batch and Incremental Processing covers the watermark pattern in more depth on the processing side; this article is about recognizing, at the ingestion boundary, that the question will come up at all.

Reading a source before integrating it#

A short checklist, applied once per new source, catches most surprises before they reach production:

• What guarantees does the source actually document? "Real-time" and "near real-time" are marketing terms; look for an actual freshness SLA — see Service-level agreement.
• What happens on the source's bad day? Does a failed webhook retry itself, or does the event simply vanish? Does a paginated API expose a cursor you can resume from, or only page numbers that shift as new rows are inserted underneath you (see Pagination and Rate Limiting)?
• Who is responsible for replay? If your system is down for an hour, does the source resend what you missed, or is that window gone forever unless you built your own buffer?

Common mistakes#

Treating "push" as "reliable" and "pull" as "safe"#

Push feels more real-time and pull feels more controllable, but neither property follows automatically. A poorly implemented webhook sender that does not retry on failure is less reliable than a five-minute polling job that always runs. Evaluate the actual guarantees, not the shape of the integration.

Skipping the contract because the source is "internal"#

An internal API maintained by another team in the same company still changes without warning to you specifically. The four-part contract above is cheaper to write down once than to reverse-engineer after a field silently changes type in production.

Assuming exactly-once without building for at-least-once#

Most transports — HTTP retries, webhook redelivery, Kafka's default guarantees — hand you at-least-once. Code that assumes each record arrives exactly once will eventually double-count something. Build the idempotency in from the start; it is much harder to retrofit after storage already holds duplicates.

Further reading#

• Webhook — the push side of the first axis, covered in depth in Webhooks and Event-Driven Ingestion.
• Idempotence — the property that makes at-least-once delivery safe to build on.
• Service-level agreement — the formal shape of a freshness guarantee.
• Extract, transform, load — where ingestion sits in the broader pipeline vocabulary.
• Topics, Partitions, and Offsets — a concrete, production system built entirely around the push/streaming cell of the grid above.

See also#