DataCraft is a practical, open resource for learning data engineering — the work of moving and shaping data so it is reliable, timely, and usable. It covers the path from working with data and SQL to building pipelines, orchestration, infrastructure, analytics, and machine learning, with short explanations, working examples, recipes, and troubleshooting instead of long theory.
The idea in one line
Build data systems. Understand how they work.
What is DataCraft#
Most data problems are not really about a single tool. They are about how the pieces fit together: where data comes from, how it is stored, how it is transformed, how work is scheduled, and how you know the result is correct. DataCraft is organized around that thinking.
Three habits shape every article:
The material is documentation-first and tool-aware but not tool-worship: a tool is introduced to solve a problem, and the underlying idea is always kept separate from the specific product.
A short intro to Data Engineering#
Data engineering exists because raw data is rarely in the shape an analyst, a dashboard, or a model needs. Someone has to collect it, clean it, combine it, and deliver it on a schedule people can trust.
It helps to picture data moving along a path:
- SourcesApplication databases, event streams, third-party APIs, files, and logs.
- IngestionData is copied out of the sources — in batches or continuously — into a place you control.
- StorageA data warehouse or lake holds the raw and processed data cheaply and durably.
- TransformationRaw data is cleaned, joined, aggregated, and modeled into tables that answer real questions.
- ServingThe modeled data is exposed to BI tools, APIs, notebooks, and ML pipelines.
- ConsumptionDashboards, reports, experiments, and models — the reason the pipeline exists.
A single transformation step is often just a query — take raw rows, filter, join, aggregate, and land a table someone can actually use:
-- Daily paid revenue per countrySELECT o.country, date_trunc('day', o.created_at) AS day, sum(o.amount) AS revenueFROM orders AS oWHERE o.status = 'paid'GROUP BY o.country, dayORDER BY day;Work happens either in batch (run every hour or every night over a chunk of data) or as a stream (process each event within seconds). Most teams use both.
Data engineering vs. adjacent roles
Data engineering builds and operates the pipelines and storage. Analytics engineering models warehouse data into clean, documented tables. Data analysis and data science use those tables to answer questions and train models. The lines blur, and one person often wears several hats.
What the platform covers#
Navigation is organized by subject area. Get Started, Fundamentals, Python, Databases & SQL, Kafka, Docker, and Airflow are live; the rest are being written.
Where to start#
Pick the entry point that matches where you are now:
- New to data, comfortable with code — start with Fundamentals, then Databases and SQL.
- Analyst moving toward engineering — skim Fundamentals, then go deep on Data Pipelines and Orchestration.
- Know the tools, want the system view — read the Roadmap and jump between sections as needed.
Whatever your background, these two pages orient you first:
- Read How to use DataCraftIt explains the article template, difficulty levels, and how search and navigation work — see How to use DataCraft.
- Follow the core path in the RoadmapThe Roadmap lays out the sections in a recommended order and says what each one gives you.