Practical data engineering
Build systems.
Work with data.
Create value.
A clear path from raw data to reliable systems. Learn the principles, tools, and decisions that make data work in the real world.
- Practical
- Built around real systems
- Structured
- Concepts that connect
- Independent
- Tools serve the thinking
Start here
Core guides across the curriculum
- Get StartedWelcomeWhat DataCraft is, how it is built, and where to start.
- FundamentalsData Engineering OverviewA high-level mental model for how data moves from a source to a reliable consumer.
- Distributed SystemsDistributed Systems FundamentalsWhy coordinating several machines is a fundamentally different problem than running logic on one, and the assumptions a network call quietly breaks that a function call never did.
- PythonData Transformations with PandasPractical patterns for reading ClickHouse data into Pandas, transforming it, validating results, and writing it back.
- Data Ingestion & APIsData Ingestion FundamentalsThe two axes every ingestion problem falls on — push vs. pull, batch vs. streaming — and the contract a source and its consumer need to agree on before any code gets written.
- Databases & SQLDatabases and Analytical StorageWhat a database, table, and schema are, why analytical systems use a column store, and how OLAP differs from OLTP.
- File Formats & StorageFile Formats FundamentalsWhy the format a file is saved in matters as much as the data inside it, the difference between a text and a binary format, and the two questions that decide which one a pipeline should reach for.
- Query PerformanceQuery Performance FundamentalsWhat actually happens between typing SQL and getting rows back — parsing, planning, optimizing, and executing — and why the optimizer's plan, not the query text, is what determines how fast a query runs.
- BrokersKafka FundamentalsThe problem Kafka solves, the append-only log at its core, and how brokers, topics, partitions, producers, and consumers fit together.
- Distributed ProcessingDistributed Processing FundamentalsWhat actually runs out first when a Pandas job stops working — memory or a single CPU core — and why the fix is the same partitioning idea covered elsewhere on this site, just applied to computation instead of storage.
- ContainersWhy ContainersWhat a container is, how it differs from an image and from a virtual machine, and why data infrastructure is almost always run in containers.
- OrchestrationOrchestration and Airflow FundamentalsWhat workflow orchestration is, the problem it solves, and how Airflow's DAGs, tasks, scheduler, and executor fit together.
- Data Quality & TestingData Quality FundamentalsWhy "is this data correct" is not one question but six, and why quality is something measured against an explicit expectation rather than a property data either has or doesn't.
- Data Governance & SecurityData Governance FundamentalsThe problem governance actually solves — not "how do we protect data" in the abstract, but "who is allowed to do what with which data, and who is accountable when that goes wrong" — once an organization outgrows everyone simply trusting everyone.