Roadmap

DataCraft is built so you can drop in anywhere, but if you want a route, this is it. The order below builds from the ideas that everything else depends on toward the systems that combine them.

The core learning path#

1. Fundamentals
   Learn the shared vocabulary: sources and sinks, ETL vs. ELT, batch vs. streaming, and what "data quality" is measured against. Everything later assumes these terms.
2. Python
   Get comfortable moving data in code — reading and writing common file formats, reshaping tables with pandas, and pulling from APIs. This is the glue language for the rest of the platform.
3. Databases & SQL
   Go from SELECT and WHERE to GROUP BY, HAVING, JOIN, and CTEs, then design ClickHouse tables and build data marts with INSERT ... SELECT. SQL is where most transformation actually happens.
4. Data Pipelines
   Turn one-off scripts into repeatable pipelines: ETL and ELT patterns, incremental loading, idempotency, and safe backfills. The focus is on runs you can trust to re-run.
5. Orchestration
   Schedule pipelines and express dependencies between them with Airflow — DAGs, operators, scheduling semantics, XCom, and connections. This is how the pieces run in the right order, on time.
6. Infrastructure
   Understand where data systems live: object storage, compute, containers, and the cost model behind each choice. Enough to make sensible decisions, not to become a platform team.
7. Analytics
   Model warehouse data for analysis, define metrics that stay consistent across reports, and see where the analytics engineering layer fits between raw tables and dashboards.
8. Machine Learning
   Prepare data for models: feature pipelines, the train/serve split, and the MLOps basics that keep a model reproducible after it ships.

How to use this path#

The path is a suggestion, not a gate. A few ways to work with it:

• Skip what you know. Each article is self-contained and links to any concept it depends on, so you can start mid-path and follow references backward when needed.
• Read by difficulty. Within a section, Beginner articles come first. It is fine to read every Beginner article across sections before going deeper anywhere.

If you're short on time#

For a working mental model of an end-to-end data system without reading everything, follow just these four stops:

1. Fundamentals — Data Engineering Overview
   The whole picture in one article: what the pieces are and how data flows between them.
2. Databases & SQL — SQL Querying and Aggregations
   The clauses behind nearly every transformation you will write. See SQL Querying and Aggregations.
3. Data Pipelines — Idempotency
   Why re-running a job should be safe, and how to make it so.
4. Airflow — Orchestration and Fundamentals
   How scheduled work and dependencies are managed in practice. See Orchestration and Airflow Fundamentals.

When you are ready for the full route, go back to the top of this page — or start from Welcome.