A single throwaway script can get away with print("done"). A pipeline
with several stages, running on a schedule, unattended, cannot — by the
time something goes wrong, the person investigating was not watching the
terminal, and "done" printed to a scrollback buffer that no longer exists
is not evidence of anything.
What print cannot do#
Logging (computing)
exists as its own discipline, separate from a program's normal output,
specifically because a print statement has no severity, no timestamp, and
no destination other than wherever standard output happens to be routed
that day. Python's logging
module adds all three:
import logging
logger = logging.getLogger(__name__)logging.basicConfig(level=logging.INFO)
logger.info("Starting export for %s", report_date)logger.warning("Retrying after transient failure, attempt %d", attempt)getLogger(__name__) scopes the logger to the current module, so log lines
can be filtered or silenced per module later without touching the code that
emits them. The severity — info, warning, error — lets a single
level setting control how much detail comes out, from full debug traces
in development to only warnings and above in production, without editing
the calls themselves.
Structured logs: fields instead of sentences#
A log line written as a sentence — "Processed 1204 orders for customer 88 in 3.2s" — is easy for a person to read and hard for a machine to query. A
structured log line
keeps the same information as named fields instead:
import json
def log_json(logger, level, message, **fields): logger.log(level, json.dumps({"message": message, **fields}))
log_json(logger, logging.INFO, "processed_orders", customer_id=88, order_count=1204, duration_s=3.2)Once logs are lines of JSON rather than sentences, a log aggregation system
can filter on customer_id=88 or compute the average duration_s across
every run — questions that would otherwise require parsing free text with
regular expressions, chasing whatever slightly different wording each log
call happened to use.
One run ID, threaded through every stage#
A pipeline with an ingestion stage, a transformation stage, and a load stage produces log lines from three different places for what is conceptually one execution. Without something tying them together, reconstructing "what happened during last night's run" means correlating timestamps by hand:
import uuid
run_id = str(uuid.uuid4())
log_json(logger, logging.INFO, "ingestion_complete", run_id=run_id, rows=50_000)log_json(logger, logging.INFO, "transform_complete", run_id=run_id, rows=49_812)log_json(logger, logging.INFO, "load_complete", run_id=run_id, rows=49_812)Generating one run_id at the start and passing it into every stage —
function parameter, environment variable, or thread-local, depending on how
the pipeline is structured — means every log line from one execution can be
found with a single filter, regardless of which stage or which machine
produced it. Data Pipeline covers the
multi-stage architecture this pattern is meant to make observable.
Common mistakes#
Logging secrets or personal data because they were in scope#
A log line built from **fields or a raw payload dict can capture an API
key or a customer's email address without anyone deciding to log it on
purpose. Configuration and Secrets Management
covers keeping credentials out of code in the first place; the same
discipline extends to keeping them out of logs, which are usually retained
longer and read by more people than the code itself.
A bare "Error occurred" with no identifying context#
A log line without a run_id, a relevant entity ID, or the actual
exception message is close to useless once more than one thing could have
gone wrong. Every error-level log line should carry enough fields to locate
the specific record or run it refers to.
Using print for anything meant to be read later#
print output is easy to lose — redirected away, not captured by a process
manager, rotated out with no severity to filter by. Reach for logging
as soon as output needs to survive past the current terminal session.
Further reading#
logging— Python documentation — the standard library module used throughout this article.- Logging Cookbook — worked examples, including JSON formatters and multi-module setups.
- Logging (computing) — the general discipline this article's structured-log section builds on.
- uuid — Python documentation
—
uuid4(), used above to generate each run's correlation ID. - json — Python documentation
—
dumps, used above to serialize each structured log line.