Python Logging Best Practices - Obvious and Not-So-Obvious

The logging module is a core part of Python's standard library, but its default configuration is insufficient for serious application development.

This guide provides a comprehensive overview of best practices for instrumenting your code effectively. It begins with the "obvious" configuration and usage patterns for basic scripts and services, then progresses to the "not-so-obvious" techniques: including structured formatting, exception handling, and telemetry correlation, that are essential for building robust, observable, and easily debuggable systems.

The Obvious: Foundational Python Logging Best Practices

A reliable logging system is built upon a few non-negotiable practices. The following section discusses those foundations.

1. Use the logging Module, Not print()

The most basic step is to move away from using print() for debugging or tracking events. While print() is great for simple scripts or immediate feedback in a terminal, it's inflexible for real applications.

The built-in logging module offers several advantages:

• Log Levels: You can categorize messages by severity (e.g., DEBUG, INFO, WARNING, ERROR), allowing you to filter out noise.
• Configurability: You can easily change the logging output to the console, a file, or a network service, without changing your application code.
• Contextual Information: Automatically include rich information like timestamps, module names, and line numbers.

2. Avoid the Root Logger (and Use __name__)

When you call logging.warning("..."), you're using the root logger. This is a bad habit because the root logger is a global, shared resource. Any library your project uses might also use the root logger, and configuration changes can have unintended side effects.

The best practice is to create a logger for each module. Use the __name__ special variable, which automatically resolves to the module's path in dot notation (e.g., my_app.utils.helpers).

Here is an example of how to get a logger for a module:

import logging

# Correct way to get a logger
logger = logging.getLogger(__name__)

def my_function():
    logger.info("Starting my_function...")

Note: Avoid calling the root logger in application code, but do configure the root logger once at startup so third-party libraries inherit sane defaults.

3. Use the Correct Log Levels

Log levels give your log messages meaning and allow you to filter them based on importance. Using them correctly is key to avoiding a flood of useless information.

Here’s a simple guide on when to use each level:

• DEBUG: Detailed, diagnostic information. Useful only for developers when debugging a specific problem. Example: Sending request to API with body: {'key': 'value'}
• INFO: Confirmation that things are working as expected. Used for tracking the normal flow of the application. Example: User 'alice' logged in successfully.
• WARNING: An indication that something unexpected happened, but the application is still working. It might be a potential problem in the future. Example: API response time is slow: 2.5s.
• ERROR: A serious problem occurred, and the software was unable to perform a specific function. Example: Failed to connect to the database.
• CRITICAL: A very severe error that might cause the application to terminate. Example: Application is out of memory.

In development, you might set the level to DEBUG, but in production, you'd likely set it to INFO or WARNING to reduce noise.

Note: Python log levels also have numeric values used for filtering and formatting — NOTSET=0, DEBUG=10, INFO=20, WARNING=30, ERROR=40, CRITICAL=50. logger.setLevel() and handler levels accept either names or numbers, and you can include the numeric level in output with the %(levelno)s formatter. For custom levels, pick a number between existing ones (e.g., TRACE=5 or TRACE=15). If you forward logs to systems like syslog, be aware Python’s numeric levels don’t map 1:1 to syslog severities.

4. Centralize Your Logging Configuration

Don't sprinkle logging configuration code throughout your application. Configure it once at the application's entry point. Using logging.config.dictConfig is the most flexible and recommended way to do this.

Here’s a simple centralized configuration that logs INFO-level messages to the console:

# In your application's main entry point (e.g., main.py or __init__.py)
import logging.config

LOGGING_CONFIG = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "standard": {
            "format": "%(asctime)s [%(levelname)s] %(name)s: %(message)s",
            "datefmt": "%Y-%m-%dT%H:%M:%S%z"
        },
    },
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
            "formatter": "standard",
            "level": "INFO",
            "stream": "ext://sys.stdout",
        },
    },
    "loggers": {
        "": {  # Root logger
            "handlers": ["console"],
            "level": "INFO",
            "propagate": True,
        },
    }
}

logging.config.dictConfig(LOGGING_CONFIG)

# Now any other module can just get its logger and use it
logger = logging.getLogger(__name__)
logger.info("Logging is configured.")

This code uses a dictionary to centrally configure your application's logging.

It sets up a single rule: send all logs of INFO level and higher to the console, formatted with a timestamp, level, and module name. This ensures every part of your app logs messages in a consistent and predictable way.

5. Write Meaningful Log Messages

A log message that says "Error occurred" is useless. A good log message provides context and is actionable.

• Bad: logger.error("Couldn't save.")
• Good: logger.error("Failed to save order to database for user_id=%s, order_id=%s", user_id, order_id)

Key tips for great log messages:

• Be clear and concise.
• Provide context. Include relevant IDs (user ID, request ID, transaction ID) and parameters.
• Use placeholders. Use argument-based formatting (e.g., logger.info("User %s", username)) so formatting is deferred unless the message will be emitted. If you prefer f-strings, ensure the logger level permits emission to avoid needless work.

The Not-So-Obvious: Advanced Python Logging Best Practices

With the fundamentals covered, the next step is to structure your logs for complex systems, making them queryable, context-rich, and secure.

These techniques will elevate your logging from a simple diagnostic tool to a powerful observability platform.

6. Embrace Structured Logging with JSON

As applications scale, logs become a source of data to be queried and analyzed. Plain text logs are difficult for machines to parse reliably. Structured logging is the solution, transforming log entries into machine-readable formats like JSON.

By logging in JSON, you can easily filter, search, and visualize your logs in tools like SigNoz, Elasticsearch (ELK), or Datadog. While Python's standard logging module can achieve structured output with extra libraries like python-json-logger, structlog is a powerful, highly performant, and Pythonic library designed from the ground up for structured logging. It simplifies the process significantly by treating log entries as dictionaries that are processed in a chain before final rendering.

Why structlog?

structlog's core concept revolves around event dictionaries and processor chains:

• Context Building: You can incrementally bind key-value pairs to your logger's context using log.bind(). This context is then automatically included in every subsequent log message from that bound logger, preventing repetitive code and missed data.
• Processors: structlog uses a chain of small, single-purpose functions (processors) to manipulate the event dictionary before it's rendered. This allows for flexible modifications like adding timestamps, converting exception info, or merging context variables.
• Flexible Rendering: The final processor in the chain is usually a renderer, which converts the event dictionary into the desired output format, such as JSON, plain text, or a custom format.

First, ensure structlog is installed:

python -m pip install structlog

Here's how to configure structlog to produce JSON output. This configuration is typically done once at your application's startup.

# In your application's main entry point (e.g., main.py or __init__.py)
import logging
import structlog
import sys

# Configure structlog to use a processor chain that ends with JSONRenderer
structlog.configure(
    processors=[
        structlog.stdlib.add_logger_name,
        structlog.stdlib.add_log_level,
        structlog.processors.TimeStamper(fmt="iso", utc=True),
        structlog.processors.StackInfoRenderer(),
        structlog.dev.set_exc_info, # Captures exc_info for exceptions
        structlog.processors.format_exc_info, # Formats exception info
        structlog.processors.JSONRenderer(), # Renders the event dict as JSON
    ],
    logger_factory=structlog.stdlib.LoggerFactory(), # Uses standard library loggers
    wrapper_class=structlog.stdlib.BoundLogger,
    cache_logger_on_first_use=True,
)

# You'll still configure a standard library handler, but it will be simpler
# as structlog's processors handle most of the formatting.
logging.basicConfig(
    format="%(message)s", # structlog handles the full message format
    stream=sys.stdout,
    level=logging.INFO,
)

# Now, get a structlog logger instance
logger = structlog.get_logger(__name__)
logger.info("Structlog is configured for structured logging.")

Now, a log call with structlog becomes even cleaner and allows for dynamic context.

# Bind context once for the current scope (e.g., per request)
request_logger = logger.bind(user_id=123, ip_address='192.168.1.100')

# Log the event. The bound context is automatically included.
request_logger.info("User logged in")

This will produce a clean JSON output (the exact timestamp will vary):

{"event": "User logged in", "logger": "my_app.auth", "level": "info", "timestamp": "2025-09-29T10:00:00.123456Z", "user_id": 123, "ip_address": "192.168.1.100"}

With structlog, you get powerful structured logging that's easy to read during development and perfectly machine-readable for production.

7. Add Context Dynamically with structlog and contextvars

In web applications or microservices, it's crucial to trace a single request as it moves through the system. The best way to do this is with a Correlation ID (or Request ID) that is automatically injected into every log message for that request.

While the standard logging library requires a custom Filter object, structlog makes this pattern much simpler using its built-in support for Python's contextvars. A ContextVar object (from the contextvars module) holds data that is local to a specific task or thread, making it perfect for storing request-scoped information.

First, ensure the merge_contextvars processor is included in your structlog configuration. This processor automatically adds any data from structlog's context into your log's event dictionary.

# In your structlog configuration
import structlog

structlog.configure(
    processors=[
        # ... other processors
        structlog.contextvars.merge_contextvars, # <-- Add this processor
        structlog.processors.JSONRenderer(),
    ],
    # ... other config
)

Next, in your web framework's middleware (or at the entry point of a request), you can generate a unique ID and bind it to the context.

import structlog
import uuid

# In a web framework middleware
def request_middleware(request):
    # This function is called at the start of each request
    structlog.contextvars.bind_contextvars(
        correlation_id=str(uuid.uuid4())
    )
    # ... handle the request ...

Now, any log call made during that request's lifecycle will automatically include the correlation_id without any extra effort.

# In some other part of your application
logger = structlog.get_logger(__name__)

def process_order(order_id):
    # This log call automatically gets the correlation_id from the context
    logger.info("Processing order", order_id=order_id)

The resulting JSON output will be cleanly enriched:

{"event": "Processing order", "order_id": 451, "correlation_id": "a1b2c3d4-e5f6-7890-1234-567890abcdef"}

This pattern is a powerful step up for diagnostics.

For fully distributed systems, this concept is formalized and automated by OpenTelemetry, which uses standardized trace_id and span_id fields to provide even richer, cross-service tracing, as discussed in section 11.

8. Log Exceptions Correctly with Stack Traces

When an exception occurs, logging just the message is not enough. You need the full stack trace to understand where the error happened.

Never do this:

try:
    # ... some failing code
except Exception as e:
    logger.error(f"An error occurred: {e}") # Bad! No stack trace.

Instead, use logger.exception() inside an except block. It automatically logs the message at the ERROR level and includes the full exception info and stack trace.

try:
    result = 1 / 0
except Exception:
    logger.exception("An unhandled exception occurred during division.") # Good!

This is equivalent to logger.error("...", exc_info=True), but is more concise and clear in its intent.

9. Keep Sensitive Information Out of Logs

Logs can be a security liability if they contain sensitive data like passwords, API keys, or personal information. Never log sensitive data in plain text.

You can use a custom logging.Filter to automatically redact sensitive information from your log messages.

Here is a simple filter that masks a field named credit_card:

import logging
import re

class SensitiveDataFilter(logging.Filter):
    CC_RE = re.compile(r"\\b(?:\\d[ -]*?){13,19}\\b")
    def filter(self, record: logging.LogRecord) -> bool:
        # Safely compute the message with args
        msg = record.getMessage()
        msg = self.CC_RE.sub("[REDACTED-CC]", msg)
        record.msg, record.args = msg, ()
        return True

# Remember to add this filter to your handler configuration!

Extend the regex set to cover common secrets (API tokens, emails, SSNs, JWTs), or prefer a structured logger/processor that redacts specific keys in the event dict to avoid brittle string matching.

In OpenTelemetry pipelines, you can also add a log processor to scrub or transform sensitive attributes before export (see Section 11), which helps ensure PII never leaves the application while still benefiting from log/trace correlation.

10. Log to stdout and Let the Environment Handle It

While Python offers handlers for rotating files (RotatingFileHandler), in modern cloud and container-based environments (like Docker and Kubernetes), it's a best practice to log directly to standard output (stdout).

Why?

• Simplicity: Your application doesn't need to worry about file paths, permissions, or log rotation.
• Decoupling: The execution environment is responsible for collecting, routing, and storing logs. Docker's logging drivers or a Kubernetes sidecar (like Fluentd) can capture the stdout stream and forward it to a centralized logging service.
• Immutability: In containerized environments, the filesystem is often ephemeral. Writing to stdout ensures logs persist beyond the life of the container.

Let your application do what it does best, and delegate the complexities of log management to the platform.

11. Beyond Logging: Unifying Logs, Metrics, and Traces with OpenTelemetry

While powerful, logging is only one piece of the observability puzzle. To get a complete picture of your application's behavior, you need to correlate logs with metrics (e.g., CPU usage, request rates) and traces (the end-to-end journey of a request through your system).

This is where OpenTelemetry (OTel) comes in. OTel is a vendor-neutral, open-source standard for generating and collecting all three types of telemetry data: logs, metrics, and traces.

By instrumenting your Python application with OpenTelemetry, you can:

• Automatically Correlate Data: OTel automatically injects trace_id and span_id into your logs. This means you can instantly jump from a specific error log to the exact distributed trace that caused it, seeing the full sequence of operations across all your microservices.
• Avoid Vendor Lock-in: OTel is an open standard. You can instrument your code once and send your data to any OTel-compatible backend.
• Leverage Auto-Instrumentation: OTel provides libraries that can automatically instrument popular frameworks (like Flask, Django, FastAPI) and libraries (like requests, SQLAlchemy). This captures a wealth of telemetry data with minimal code changes.

Integrating with SigNoz for Full-Stack Observability

Once you're generating telemetry data with OpenTelemetry, you need a backend to collect, store, and visualize it. SigNoz is an OTel-native observability platform that provides a unified experience for logs, metrics, and traces in a single application.

With SigNoz, you can use the rich, structured logs you've configured to:

• Build dashboards to monitor log patterns and error rates.
• Set up alerts based on log content.
• Seamlessly correlate logs with traces to find the root cause of issues in seconds, not hours.

SigNoz offers auto-instrumentation for Python logs, making it straightforward to get started. For detailed instructions, you can follow their guide on Python Logs Auto-Instrumentation.

Get Started with SigNoz

You can choose between various deployment options in SigNoz. The easiest way to get started with SigNoz is SigNoz cloud. We offer a 30-day free trial account with access to all features.

Those who have data privacy concerns and can't send their data outside their infrastructure can sign up for either enterprise self-hosted or BYOC offering.

Those who have the expertise to manage SigNoz themselves or just want to start with a free self-hosted option can use our community edition.

Hope we answered all your questions regarding Python logging best practices. If you have more questions, feel free to use the SigNoz AI chatbot, or join our slack community.

You can also subscribe to our newsletter for insights from observability nerds at SigNoz, get open source, OpenTelemetry, and devtool building stories straight to your inbox.