Index
Everything published on DataCraft so far. New sections are added as they are written.
A high-level mental model for how data moves from a source to a reliable consumer.
What happens to data from creation and collection through processing, consumption, retention, archival, and deletion.
How data moves from sources through layered storage and transformation stages to reliable consumers.
Practical patterns for reading ClickHouse data into Pandas, transforming it, validating results, and writing it back.
Detect abnormal DataFrame records, separate them from valid data, and log them for investigation.
How Python and Pandas process data in repeatable batches and continue from the last successfully processed point.
A compact reference for common Pandas DataFrame operations used in data engineering.
What a database, table, and schema are, why analytical systems use a column store, and how OLAP differs from OLTP.
Column-oriented storage, creating databases and tables, core data types, and reading and writing data in ClickHouse.
Using clickhouse-connect to run statements, read query results, pull single values, pass query parameters, and load data.
SELECT, WHERE, GROUP BY, HAVING, ORDER BY, LIMIT, and the aggregate functions used to summarize data.
How the MergeTree engine stores data, what ORDER BY and PARTITION BY actually do, and how to choose a sort key that matches your queries.
Turning raw rows into query-ready tables with INSERT ... SELECT, JOIN, CTEs, and an idempotent incremental load.
A compact lookup for SELECT structure, joins, aggregates, CTEs, ClickHouse table syntax, time functions, and the clickhouse-connect client.
The problem Kafka solves, the append-only log at its core, and how brokers, topics, partitions, producers, and consumers fit together.
How a topic is split into partitions, how keys decide placement, what offsets mean, and how retention, compaction, and replication behave.
A single-broker Kafka and Kafka UI, the listener model that makes clients connect, and the CLI and UI tools for operating a broker.
Sending events to a Kafka topic with confluent-kafka — the async buffering model, keys and partitioning, delivery reports, acks, and batching.
Reading a Kafka topic with confluent-kafka — the poll loop, consumer groups and rebalancing, offset management, and batching before a downstream write.
Where Kafka sits in a data platform — as an ingestion buffer between producers and the warehouse — and how to make the end-to-end flow reliable.
A compact lookup for the confluent-kafka producer and consumer, config keys, topic settings, the CLI, and common errors.
What a container is, how it differs from an image and from a virtual machine, and why data infrastructure is almost always run in containers.
When a stock image is not enough, and how to build a custom one with FROM, RUN, USER, and layer caching.
The Compose file format — services, images, ports, environment, volumes, depends_on, healthcheck — and the network that connects them.
Assembling a full local data platform in one Compose file, one service at a time, with the reasoning for every setting.
A compact lookup for Dockerfile instructions, Compose keys, and the docker and docker compose commands.
What workflow orchestration is, the problem it solves, and how Airflow's DAGs, tasks, scheduler, and executor fit together.
The with DAG(...) block, default_args, the schedule, catchup, and a first PythonOperator that calls a plain function.
What an operator is, PythonOperator in depth, the task context, other common operators, the @task decorator, and writing retry-safe tasks.
How schedule and start_date define data intervals, what catchup does, how to backfill safely, and why tasks must use the interval instead of now().
Setting task order with >> and lists, fan-out and fan-in patterns, trigger rules, branching, and keeping a DAG readable.
Passing small values between tasks with XCom, storing credentials as Connections, and using Hooks instead of hardcoding clients.
The web UI, reading task logs, pausing and re-running, how DAG parsing works, retries and alerting, and the operational failures you will actually hit.
A full worked DAG for an orders pipeline — watermark, interval-aware load, parallel marts, a rollup, a quality check, and a failure alert — with the complete file at the end.
A compact lookup for the DAG skeleton, schedule options, operators, dependencies, trigger rules, XCom, connections, the task context, and the CLI.
Not written yet.
Not written yet.