All topics

What DataCraft is, how it is built, and where to start.

How the sections, articles, navigation, and search fit together.

A recommended path through DataCraft and what each section gives you.

A high-level mental model for how data moves from a source to a reliable consumer.

What happens to data from creation and collection through processing, consumption, retention, archival, and deletion.

How data moves from sources through layered storage and transformation stages to reliable consumers.

Why coordinating several machines is a fundamentally different problem than running logic on one, and the assumptions a network call quietly breaks that a function call never did.

What Consistency, Availability, and Partition tolerance actually mean in Brewer's theorem, why "pick two" is a misleading way to remember it, and the PACELC extension that covers what CAP leaves out.

Splitting data across many nodes so no single one has to hold or serve all of it, why the partition key decides whether that split is even, and how consistent hashing keeps adding a node from reshuffling everything else.

Keeping copies of the same data on multiple nodes for durability and availability, leader-based versus quorum-based replication, and the spectrum from strong to eventual consistency each one lands on.

Why getting several nodes to agree on one thing — who the leader is, what the next value is — needs a proven algorithm instead of a heuristic, and how Raft solves it in a way that stays understandable.

Why a distributed system can never directly observe that a node is down — only the absence of a response, which a dead node and a slow one produce identically — and the timeout trade-off that follows from that.

Why comparing timestamps from two different machines cannot reliably tell you which event happened first, and the logical clocks distributed systems use instead of wall-clock time to establish order.

A compact lookup for CAP/PACELC, partitioning strategies, replication modes, consensus, failure detection, and logical clocks.

Practical patterns for reading ClickHouse data into Pandas, transforming it, validating results, and writing it back.

Detect abnormal DataFrame records, separate them from valid data, and log them for investigation.

How Python and Pandas process data in repeatable batches and continue from the last successfully processed point.

Deriving calendar features from a timestamp, and building the same rollup at several time granularities from one reusable function instead of duplicated code.

Turning a wide export — one column per category — into tidy long-format rows with melt, extracting clean ids from composite labels, and combining two same-shaped tables.

Joining two parallel measurements of the same thing and quantifying the gap between them with MAE, RMSE, MAPE, and correlation.

Why a Python for-loop over a large array is slow, how a vectorized NumPy operation avoids it, and the broadcasting rules that let arrays of different shapes combine without one.

What a dtype actually costs per value, downcasting numeric columns safely, dictionary-encoding repeated strings with the category dtype, and processing a file too large to hold in memory at all.

A trend that reverses when data is split by segment, a filter that quietly biases a sample, a correlation mistaken for a cause, and a percentile whose value depends on which of several equally standard methods computed it.

Why the same script can produce different results on two runs, seeding every random source a pipeline actually uses instead of just one of them, and what a fixed seed does not guarantee on its own.

Why print() stops working once a pipeline has more than one step, what a structured log line makes possible that a plain sentence doesn't, and threading one run ID through every stage so its log lines can be found together later.

Keeping what changes per environment out of the code that doesn't, validating configuration once at startup with a typed schema instead of scattered os.environ.get calls, and where the credentials advice repeated across this site actually comes from.

Turning a script with values hardcoded at the top into a command a teammate can run with --help, argparse compared to Typer, and where argument parsing ends and configuration loading begins.

What a GeoDataFrame adds on top of Pandas, what a coordinate reference system actually is, and how a spatial join assigns points to the polygon they fall inside.

Building geometry from plain coordinate columns and matching each point to the closest feature in another layer with sjoin_nearest, instead of a containment predicate.

The difference between reading spreadsheet data with Pandas and editing an existing workbook in place with openpyxl, and how to write computed values into a fixed template safely.

A compact reference for common Pandas DataFrame operations used in data engineering.

The two axes every ingestion problem falls on — push vs. pull, batch vs. streaming — and the contract a source and its consumer need to agree on before any code gets written.

Building a small, reusable HTTP client around requests — a shared Session, timeouts that are never left to their default, and turning a failed response into an exception before it reaches your transformation code.

Walking a paginated API to completion without missing or duplicating rows, and backing off correctly when the source asks you to slow down.

API keys, Basic Auth, and the OAuth 2.0 client credentials flow compared — what each one actually proves, and how to cache a token instead of fetching one on every call.

The checks that belong at the file-reading boundary, before a DataFrame reaches any business-rule validation — encoding, unparseable rows, and columns that quietly changed shape.

Receiving pushed data safely — verifying a webhook actually came from who it claims, deduplicating retried deliveries, and responding fast enough that the sender does not retry anyway.

A third ingestion shape, beyond a single call and a paginated one — start an asynchronous export job, poll until it finishes, then download the result, without hammering the status endpoint while you wait.

Folding session reuse, timeouts, retries, pagination, and token caching into one client class, instead of re-deriving all five for every new API a pipeline needs to talk to.

Why a test suite that calls the real API is the wrong default, mocking at the HTTP boundary instead of deeper, deliberately testing the failure paths, and the narrow gap even good mocks leave open.

A compact lookup for requests sessions, retry and backoff, cursor pagination, auth headers, webhook signature verification, poll-until-ready jobs, and mocked HTTP tests.

What a database, table, and schema are, why analytical systems use a column store, and how OLAP differs from OLTP.

The difference between a raw psycopg connection and a SQLAlchemy engine, why a project usually needs both, and where credentials belong.

Defining tables as small, repeatable Python functions instead of a hand-run SQL script, and rebuilding a database from scratch safely in a development environment.

Column-oriented storage, creating databases and tables, core data types, and reading and writing data in ClickHouse.

Using clickhouse-connect to run statements, read query results, pull single values, pass query parameters, and load data.

SELECT, WHERE, GROUP BY, HAVING, ORDER BY, LIMIT, and the aggregate functions used to summarize data.

How the MergeTree engine stores data, what ORDER BY and PARTITION BY actually do, and how to choose a sort key that matches your queries.

Turning raw rows into query-ready tables with INSERT ... SELECT, JOIN, CTEs, and an idempotent incremental load.

A compact lookup for SELECT structure, joins, aggregates, CTEs, ClickHouse table syntax, time functions, and the clickhouse-connect client.

Why the format a file is saved in matters as much as the data inside it, the difference between a text and a binary format, and the two questions that decide which one a pipeline should reach for.

The physical layout decision that explains why an analytical query only pays for the columns it touches, and why a system built for single-row updates almost always stores data by row instead.

The row group and column chunk structure behind every Parquet file, why the footer is read before any actual data, and the small-files problem that catches most teams new to writing it.

Encoding and compression are two separate layers, not one feature — dictionary and run-length encoding exploit what a column's values actually look like, while a compression codec squeezes the result further, at a real CPU cost on read.

How a query engine skips whole row groups and unreferenced columns without reading a single byte of their data — and the common way a query accidentally disables this and nobody notices.

Encoding a column's value into a directory path so a query engine can skip entire files without opening any of them, the Hive-style layout most engines expect, and why partitioning by the wrong column recreates the small-files problem.

Why a row-oriented, self-describing format fits a stream of individual messages better than Parquet does, and the precise backward/forward rules that let producers and consumers upgrade a schema independently without breaking each other.

A compact lookup for row vs. column layout, Parquet structure, encoding and compression, pushdown, disk partitioning, and Avro compatibility rules.

What actually happens between typing SQL and getting rows back — parsing, planning, optimizing, and executing — and why the optimizer's plan, not the query text, is what determines how fast a query runs.

A plan is a tree that executes bottom-up, not a script that runs top-down, and the single most useful diagnostic in it is the gap between how many rows a node expected and how many it actually saw.

What a B-tree index actually trades away to make lookups fast, why a function wrapped around the indexed column silently disables it, and why having an index does not guarantee the optimizer will use it.

Nested loop, hash, and sort-merge joins all produce the identical result set from different amounts of work — which one the optimizer picks, and why, is often the single biggest lever on a query's actual cost.

How the optimizer estimates a plan's cost from statistics collected ahead of time, why an estimation error at one step compounds through every step built on top of it, and what stale statistics actually do to a plan.

A repeatable method for finding what actually makes a query slow — one hypothesis, one change, one controlled re-measurement at a time — instead of changing several things at once and hoping.

N+1 queries, an implicit type conversion that silently disables an index, and pulling an unbounded result set into application memory — three patterns worth checking early, before a longer profiling session.

A compact lookup for plan nodes, index selectivity, join algorithms, cost estimation, the profiling method, and the three most common anti-patterns.

The problem Kafka solves, the append-only log at its core, and how brokers, topics, partitions, producers, and consumers fit together.

How a topic is split into partitions, how keys decide placement, what offsets mean, and how retention, compaction, and replication behave.

A single-broker Kafka and Kafka UI, the listener model that makes clients connect, and the CLI and UI tools for operating a broker.

Sending events to a Kafka topic with confluent-kafka — the async buffering model, keys and partitioning, delivery reports, acks, and batching.

Reading a Kafka topic with confluent-kafka — the poll loop, consumer groups and rebalancing, offset management, and batching before a downstream write.

Where Kafka sits in a data platform — as an ingestion buffer between producers and the warehouse — and how to make the end-to-end flow reliable.

A compact lookup for the confluent-kafka producer and consumer, config keys, topic settings, the CLI, and common errors.

What actually runs out first when a Pandas job stops working — memory or a single CPU core — and why the fix is the same partitioning idea covered elsewhere on this site, just applied to computation instead of storage.

Why a chain of Spark transformations returns instantly with no output, what actually triggers computation, and how deferring execution lets Spark optimize the whole plan before running any of it.

Why Spark's DataFrame API can be optimized automatically and its older RDD API mostly can't — the difference comes down to whether Spark can see inside what you asked it to do.

Why the number of partitions a job runs with matters as much as the cluster size behind it, the default that surprises almost everyone new to Spark, and when repartition and coalesce are not interchangeable.

Why a filter costs almost nothing and a groupBy can dominate a job's entire runtime — the narrow/wide distinction between transformations, and what a shuffle actually moves across the network.

Why schema inference costs an extra pass over the data, how Spark's Parquet reader gets pushdown and partition pruning for free, and the retry that silently duplicates a table because a write used the wrong mode.

Pulling a distributed dataset into the driver's single process, a Python UDF that pays a cross-process cost on every row, and the skewed partition that leaves one task running long after everything else finished.

A compact lookup for lazy evaluation, the DataFrame vs RDD choice, partition sizing, narrow vs wide transformations, broadcast joins, and the three most common PySpark performance pitfalls.

What a container is, how it differs from an image and from a virtual machine, and why data infrastructure is almost always run in containers.

When a stock image is not enough, and how to build a custom one with FROM, RUN, USER, and layer caching.

The Compose file format — services, images, ports, environment, volumes, depends_on, healthcheck — and the network that connects them.

Assembling a full local data platform in one Compose file, one service at a time, with the reasoning for every setting.

A compact lookup for Dockerfile instructions, Compose keys, and the docker and docker compose commands.

What workflow orchestration is, the problem it solves, and how Airflow's DAGs, tasks, scheduler, and executor fit together.

The with DAG(...) block, default_args, the schedule, catchup, and a first PythonOperator that calls a plain function.

What an operator is, PythonOperator in depth, the task context, other common operators, the @task decorator, and writing retry-safe tasks.

How schedule and start_date define data intervals, what catchup does, how to backfill safely, and why tasks must use the interval instead of now().

Setting task order with >> and lists, fan-out and fan-in patterns, trigger rules, branching, and keeping a DAG readable.

Passing small values between tasks with XCom, storing credentials as Connections, and using Hooks instead of hardcoding clients.

The web UI, reading task logs, pausing and re-running, how DAG parsing works, retries and alerting, and the operational failures you will actually hit.

A full worked DAG for an orders pipeline — watermark, interval-aware load, parallel marts, a rollup, a quality check, and a failure alert — with the complete file at the end.

A compact lookup for the DAG skeleton, schedule options, operators, dependencies, trigger rules, XCom, connections, the task context, and the CLI.

Why "is this data correct" is not one question but six, and why quality is something measured against an explicit expectation rather than a property data either has or doesn't.

Formalizing what a producer promises and a consumer can rely on — schema, semantics, and guarantees — as an explicit, checked agreement instead of something every consumer has to reverse-engineer from whatever the table currently looks like.

Stating what should be true about a dataset declaratively, separate from the code that produces it, instead of scattering ad hoc assertions through transformation logic where only a code-reader will ever find them.

The classic unit/integration/end-to-end pyramid only covers code tested before it ships — a data pipeline needs a fourth layer on top, checking the real data continuously, because no pre-deployment test can see what a live upstream source does tomorrow.

Catching the problem no fixed rule was written to anticipate — a row count that quietly dropped, an average that drifted outside its normal range — by comparing today's data against its own recent history instead of a hardcoded threshold.

Tracking quality as a trend over time instead of a single pass/fail check, the difference between a violation that should halt a pipeline and one that should just get logged, and who actually decides where that line sits.

The over-sensitive alert nobody trusts anymore, a check placed only at the end of a pipeline that can't say where a bad value actually entered, and an expectation nobody has ever confirmed actually catches what it claims to.

A compact lookup for the six quality dimensions, data contracts, expectation-based checks, the four-layer testing pyramid, statistical anomaly detection, and hard vs. soft failure severity.

The problem governance actually solves — not "how do we protect data" in the abstract, but "who is allowed to do what with which data, and who is accountable when that goes wrong" — once an organization outgrows everyone simply trusting everyone.

Attaching permissions to roles versus computing them from attributes at request time — two different answers to "who can access what," and the principle that should bound either one regardless of which is chosen.

Why access control needs a map of what's actually sensitive before it can protect anything, the quasi-identifiers that re-identify someone without a single obviously-sensitive field, and why classification is ongoing work, not a one-time pass.

Two different threats with two different protections — a stolen disk versus an intercepted network connection — and why encrypting data solves neither problem an authorized process creates once it legitimately decrypts what it was trusted to read.

You cannot protect what you cannot find — tracking where sensitive data actually flows once it leaves its source table, and why a searchable catalog beats tribal knowledge once an organization has more than a handful of pipelines.

Two techniques that look similar and mean very different things legally and practically — one is reversible with a separate key, the other is meant to never be reversible at all — plus the classic mistake of treating a hash of a guessable value as if it were either.

The engineering principles behind the regulation, not the legal text — why collecting less data is often better than protecting more of it, and why fulfilling a deletion request is only possible if the previous article's lineage tracking was already in place.

A compact lookup for access control models, PII classification tiers, encryption threat models, lineage and cataloging, anonymization vs. pseudonymization, and the core GDPR principles.