Python Logging—From Setup to Monitoring with Best Practices

Python logging is the foundation of observability for any Python application. While scattering print() statements might work for quick scripts, production-grade applications require a robust mechanism to understand what the application is doing at runtime. The logging module, built directly into the Python standard library, provides a consistent and thread-safe way to record diagnostics, warnings, errors, and operational events.

As your Python application scales—whether it's a Django monolith or a fleet of FastAPI microservices—logging becomes absolutely essential for debugging, incident response, performance analysis, and security auditing. A well-architected logging setup makes it significantly easier to route logs to files, terminals, or centralized platforms like SigNoz for monitoring and alerting.

In this comprehensive tutorial, we will walk you through the process of implementing Python logging from scratch, moving on to Python log management best practices and advanced techniques, and how to gain deep insights into your application's behavior.

What is Python logging and why is it critical?

Python’s logging module provides a highly flexible event-logging framework for applications and libraries. Compared to simple print statements, it provides many benefits:

1. Severity levels: Separate verbose debug noise from critical production issues.
2. Consistent, rich formatting: Automatically inject timestamps, logger names, thread IDs, and contextual fields.
3. Multiple destinations: Seamlessly route logs to the console, rotating files, syslog, email, or asynchronous queues.
4. Centralized configuration: Change logging behavior based on the environment (e.g., development vs. production) without rewriting a single line of application code.

Logging is especially useful for:

• Debugging: Understand control flow and trace failures to their root cause.
• Operations: Surface warnings, errors, and overall service health signals to your SRE team.
• Performance analysis: Record timings and identify bottlenecks in your code.
• Security and audit trails: Capture important user actions or system events securely.
• Centralized observability: Ship logs to OpenTelemetry backends to search, correlate with traces, and trigger alerts.

Getting started with Python logging

As part of a standard pattern, let's set up a module-level logger and configure logging once at your application's entry point.

import logging

# Configure the root logger
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S"
)

# Create a logger for this module
logger = logging.getLogger(__name__)

logger.info("Application started successfully")
logger.debug("This won't print because the level is set to INFO")

This configuration is perfectly fine for small scripts or single-file applications. However, for larger codebases, you should move this configuration into a dedicated setup function or use a dictConfig() block to easily swap settings based on the environment.

Logging levels

Python defines six pre-defined severity levels. In day-to-day application code, the five levels below NOTSET are the ones you will use most often to control verbosity.

Core building blocks of the logging module

To master Python logging, you need to understand its four foundational components: loggers, handlers, formatters, and filters. They work together to process and route your log messages.

Loggers

A Python logger is the interface your code interacts with to emit messages. It acts as the gatekeeper, deciding whether a message meets the required severity level before passing it along.

import logging

# The __name__ variable evaluates to the module's name (e.g., 'app.database')
logger = logging.getLogger(__name__)
logger.info("Fetching data from the database")

Logger names are hierarchical and use dot notation, usually mirroring your Python module structure. For example, my_app.api is considered a child of my_app. This hierarchy is vital because, by default, log records propagate upward to their ancestor's handlers.

A solid best practice is to:

• Create one logger per module using logging.getLogger(__name__).
• Configure handlers only once, centrally, at the application root.
• Rely on propagation to bubble logs up to the root handlers, avoiding duplicate output.

Handlers

While loggers capture the events, Handlers decide where those log records actually go. You can attach multiple handlers to a single logger, allowing you to route the same message to multiple destinations.

Common built-in handlers include:

• StreamHandler: For console output (stdout or stderr).
• FileHandler: For writing to a local file.
• RotatingFileHandler: For size-based log rotation (e.g., keep 5 files of 10MB each).
• TimedRotatingFileHandler: For time-based rotation (e.g., create a new log file every midnight).
• SysLogHandler: For sending logs to the local or remote syslog daemon, enabling Python syslog integration.
• QueueHandler and QueueListener: For non-blocking, high-throughput asynchronous logging.

Here is an example of configuring a logger to write to both the console and a file simultaneously:

import logging

logger = logging.getLogger("my_app")
logger.setLevel(logging.DEBUG)

# Create handlers
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.INFO) # Console gets INFO and above

file_handler = logging.FileHandler("app.log", encoding="utf-8")
file_handler.setLevel(logging.DEBUG) # File gets everything (DEBUG and above)

# Attach handlers to the logger
logger.addHandler(console_handler)
logger.addHandler(file_handler)

logger.info("this will log for both handlers")

Formatters

Formatters control the textual layout of each log entry. They dictate how the internal LogRecord object is transformed into a readable string.

import logging

# Define a formatter that includes the timestamp, logger name, severity, and the actual message
formatter = logging.Formatter(
    fmt="%(asctime)s | %(name)s | %(levelname)s | %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S"
)

You can attach the same formatter to multiple handlers, or create distinct formatters. For instance, you might want detailed formatters with file names (%(filename)s) and line numbers (%(lineno)d) for file logs, but a simpler, cleaner format for console output.

console_handler.setFormatter(formatter)
file_handler.setFormatter(formatter)

Filters

Filters give you surgical control over which records are emitted. While log levels filter purely based on severity, Filters can inspect the actual content of the log record or any custom attributes.

import logging

class SensitiveDataFilter(logging.Filter):
    def filter(self, record):
        # Drop any log record that contains the word "password"
        if "password" in record.getMessage().lower():
            return False # Reject the record
        return True # Allow the record

logger = logging.getLogger("my_app")
logger.setLevel(logging.DEBUG)

handler = logging.StreamHandler()
handler.addFilter(SensitiveDataFilter())
logger.addHandler(handler)

logger.info("User logged in successfully") # This will be logged
logger.debug("Checking password hash for user") # This will be DROPPED

Filters are incredibly powerful. You can use them to drop noisy events from third-party libraries, route specific business events to dedicated auditing handlers, or proactively redact Personally Identifiable Information (PII) before logs leave your application.

Recommended Python logging configuration patterns

While basicConfig() is handy for getting started, production applications thrive on centralized configuration. It is easy to use multiple handlers for multiple files when working on a project initially, or when multiple developers are contributing at the same time.

However, this approach typically leads to conflicting configuration and unwanted behaviour. A well-organized setup reduces boilerplate and ensures consistency across your entire codebase, making debugging and configuration management easier.

Use dictConfig() for centralizing configuration logic

The logging.config.dictConfig() function is the recommended way to configure logging for any non-trivial application. It allows you to declare loggers, handlers, formatters, and policies in a single dictionary, which acts as the source of truth, often loaded from a YAML or JSON file.

import logging.config

LOGGING_CONFIG = {
    "version": 1,
    "disable_existing_loggers": False, # Crucial: Keeps third-party loggers active
    "formatters": {
        "standard": {
            "format": "%(asctime)s [%(levelname)s] %(name)s: %(message)s"
        },
        "detailed": {
            "format": "%(asctime)s [%(levelname)s] %(name)s:%(lineno)d: %(message)s"
        }
    },
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
            "formatter": "standard",
            "level": "INFO",
        },
        "file": {
            "class": "logging.handlers.RotatingFileHandler",
            "formatter": "detailed",
            "level": "DEBUG",
            "filename": "app.log",
            "maxBytes": 10_485_760, # 10MB
            "backupCount": 5,
            "encoding": "utf-8",
        },
    },
    "loggers": {
        # Specific configuration for a noisy third-party library
        "urllib3": {
            "level": "WARNING",
            "handlers": ["console"],
            "propagate": False,
        }
    },
    "root": {
        "level": "INFO",
        "handlers": ["console", "file"],
    },
}

logging.config.dictConfig(LOGGING_CONFIG)
logger = logging.getLogger(__name__)
logger.info("Logging configured successfully via dictConfig")

Configure by environment

Development, staging, and production environments have vastly different logging needs. You can easily manage this by reading an environment variable.

import logging
import os

logger = logging.getLogger(__name__)

env = os.getenv("APP_ENV", "development")

if env == "production":
    # Less verbose, perhaps outputting JSON to stdout
    logging.basicConfig(level=logging.WARNING)
else:
    # Verbose, plain-text output for local debugging
    logging.basicConfig(level=logging.DEBUG)

logger.debug("enabled debug mode")

Avoid duplicate log lines

A very common pitfall in Python logging is seeing the same log line printed multiple times. This usually occurs when handlers are attached to both a child logger and one of its ancestors (like the root logger) while propagation is enabled.

As a rule of thumb:

• Attach handlers high in the logger hierarchy, ideally only on the root logger.
• Let child loggers propagate upward naturally.
• Only set propagate = False when you intentionally want a sub-system to have completely independent logging output.

Structured logging and contextual data

Plain-text logs are easy for humans to read in a terminal, but you must go through many hoops to query them at scale. Structured logging—typically emitting logs as JSON objects—makes your telemetry highly queryable and analytical in observability platforms like SigNoz, simplifying your Python log analysis efforts.

Emitting JSON logs with the standard library

You don't necessarily need third-party packages to start emitting structured JSON logs. You can implement a custom Formatter that converts LogRecord attributes into a JSON payload.

import json
import logging

class JsonFormatter(logging.Formatter):
    def format(self, record):
        # Base structured payload
        payload = {
            "timestamp": self.formatTime(record, self.datefmt),
            "level": record.levelname,
            "logger": record.name,
            "message": record.getMessage(),
        }

        # Inject extra context fields dynamically if they exist
        for field in ("service", "request_id", "user_id"):
            if hasattr(record, field):
                payload[field] = getattr(record, field)

        # Include stack traces if an exception occurred
        if record.exc_info:
            payload["exception"] = self.formatException(record.exc_info)

        return json.dumps(payload)

handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())

logger = logging.getLogger("api")
logger.addHandler(handler)
logger.setLevel(logging.INFO)

# Using 'extra' to append structured context
logger.info("User checked out cart", extra={"request_id": "req-987", "user_id": 42})

The output looks like this:

{"timestamp": "2024-08-29 14:02:11", "level": "INFO", "logger": "api", "message": "User checked out cart", "request_id": "req-987", "user_id": 42}

Using LoggerAdapter for repeated context

If you are logging within a specific context (e.g., handling a web request or a background task) and want to attach the same contextual fields to every log call, LoggerAdapter is much cleaner than passing the extra dict repeatedly.

import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] [Service: %(service)s] %(message)s",
)
base_logger = logging.getLogger("payments")

# Wrap the logger, injecting a 'service' field on all calls
logger = logging.LoggerAdapter(base_logger, {"service": "checkout-service"})

logger.info("Charge created successfully")
logger.error("Payment gateway timeout")

Both of these log entries will now automatically include the service: checkout-service metadata, like so:

2026-05-18 12:22:25,008 [INFO] [Service: checkout-service] Charge created successfully
2026-05-18 12:22:25,008 [ERROR] [Service: checkout-service] Payment gateway timeout

Avoid basicConfig in Libraries

If you are writing a Python library rather than an application, do not call logging.basicConfig(). Calling it will globally configure logging for any application that imports your library, which can cause unexpected behavior and frustrate your users. Instead, add a NullHandler to your library's root logger.

The official Python documentation has a dedicated section on configuring logging for a library, that has vital information on utilizing the standard logging module for Python library.

Python logging performance, exceptions, and concurrency

When building applications for serving live traffic in proudction, it becomes to have robust error handling and reliable performance. Optimizing Python logging performance in this regard will help you get the most out of your applications.

Log exceptions with full stack traces

When handling an exception, using logger.exception() is significantly better than logger.error(). The exception() method automatically sets the log level to ERROR and seamlessly appends the complete stack trace to the log message.

The following code snippet to trigger a ZeroDivisionError.

import logging

logger = logging.getLogger(__name__)

def process_data(key, divisor):
    try:
        return key / divisor
    except ZeroDivisionError:
        logger.exception("Failed to process data: Division by zero")
        raise

process_data(10, 0)

Running it produces the following output in our terminal:

❯ python3 data/blog/main.py
Failed to process data: Division by zero
Traceback (most recent call last):
  File "/Users/dhruv/code/projects/signoz.io/data/blog/main.py", line 7, in process_data
    return key / divisor
           ~~~~^~~~~~~~~
ZeroDivisionError: division by zero
Traceback (most recent call last):
  File "/Users/dhruv/code/projects/signoz.io/data/blog/main.py", line 12, in <module>
    process_data(10, 0)
    ~~~~~~~~~~~~^^^^^^^
  File "/Users/dhruv/code/projects/signoz.io/data/blog/main.py", line 7, in process_data
    return key / divisor
           ~~~~^~~~~~~~~
ZeroDivisionError: division by zero

Optimize performance in hot paths

Logging operations can introduce latency. When logging inside performance-sensitive code paths or high-frequency loops:

• Use parameterized logging: Always use logger.info("User %s logged in", user_id) instead of f-strings logger.info(f"User {user_id} logged in"). Parameterized logging defers string interpolation until after verifying the log level, saving CPU cycles if the record is discarded.
• Guard expensive formatting: If generating the log context is computationally heavy, guard it with isEnabledFor().

import logging
import time

logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.DEBUG)

def serialize_complex_object(ds):
    time.sleep(2)
    return ds

if logger.isEnabledFor(logging.DEBUG):
    # This expensive serialization only happens if DEBUG is active
    expensive_payload = serialize_complex_object({"a": "b", "b": "a"})
    logger.debug("Payload details: %s", expensive_payload)

High-throughput applications: use queue-based logging

While Python’s logging module is thread-safe, I/O bound handlers (like writing to a slow disk or a network socket) can block your main application threads, becoming a serious bottleneck. For high-throughput services, use queue-based logging to move handler work entirely off the request path.

import logging
import logging.handlers
import queue
import time

# Create an unbounded or bounded queue
log_queue = queue.Queue(maxsize=2)

# The handler that the application will write to (very fast)
queue_handler = logging.handlers.QueueHandler(log_queue)

# The handler that actually does the slow I/O
stream_handler = logging.StreamHandler()
file_handler = logging.FileHandler("heavy_app.log")

# The listener runs on a separate background thread
listener = logging.handlers.QueueListener(
    log_queue, stream_handler, file_handler, respect_handler_level=True
)

logger = logging.getLogger("worker")
logger.setLevel(logging.INFO)
logger.addHandler(queue_handler)

listener.start()
try:
    logger.info("High-throughput processing started")
    # Simulate heavy workload
    for i in range(5):
        logger.info(f"Processing item {i}")
        # Since we sleep before processing, we can use a smaller queue size (2).
        # For higher-throughput workloads, consider increasing the queue size
        # to prevent data loss as the queue hits full capacity.
        time.sleep(0.1)
finally:
    # Ensure the queue flushes before exiting
    listener.stop()

Centralized logging with an observability platform

Writing logs locally to a file or stdout is highly effective during local development. However, relying on greping raw files across dozens of servers in production is an anti-pattern. Monitoring logs centrally is what truly reduces Mean Time To Resolution (MTTR) during real incidents.

Why implement Python centralized logging?

Effective Python log management via a centralized platform helps with:

1. Distributed Tracing: Automatically link your Python logs to distributed traces, allowing you to trace a single request's lifecycle across multiple microservices.
2. Cross-service debugging: Aggregate logs from your frontend, backend, databases, and message queues in one platform, reducing contextual fatigue.
3. Alerting and automation: Proactively trigger alerts via Slack or PagerDuty if the error rate spikes.

Monitoring Python logs with SigNoz

SigNoz is a full-stack, OpenTelemetry-native observability platform that provides logs, metrics, and traces within a unified interface. By adopting an OpenTelemetry backend like SigNoz, you eliminate vendor lock-in and utilize an industry-standard telemetry format.

SigNoz seamlessly integrates with Python applications: you can ingest logs via OpenTelemetry SDKs, or use an OTel Collector to scrape your JSON-formatted stdout logs.

Setup SigNoz

SigNoz Cloud is the easiest way to run SigNoz. Sign up for a free account and get 30 days of unlimited access to all features.

You can also install and self-host SigNoz yourself since it is open-source. With 24,000+ GitHub stars, open-source SigNoz is loved by developers. Find the instructions to self-host SigNoz.

For detailed steps on instrumenting your application, refer to the Python OpenTelemetry instrumentation guide.

Investigate Python Telemetry from Your IDE

With your Python app emitting logs and telemetry into SigNoz, you can also investigate that data directly from your IDE workflow. By connecting the SigNoz MCP Server, you can prompt your AI assistant to fetch the latest error logs, summarize exceptions, or query endpoint latency in natural language—all without context-switching to a dashboard.

Troubleshooting common logging issues

No logs are appearing

Check these first:

• The logger's configured level (defaults to WARNING if not set).
• The handler's configured level.
• Whether the handler is correctly attached.
• Ensure that propagate = False isn't accidentally suppressing messages.

You are seeing duplicate log lines

This almost always means the exact same LogRecord is being processed multiple times. Common causes include:

• A handler attached to both a child logger and the root logger, combined with propagate=True.
• Accidentally calling logging.basicConfig() or your setup function more than once.

Log files grow without bound

Never use a plain FileHandler in production. Always utilize RotatingFileHandler (size-based) or TimedRotatingFileHandler (time-based) to ensure your application doesn't exhaust disk space, or ship logs to an external platform.

Key takeaways

• Python’s standard logging module is powerful and entirely sufficient for most production applications.
• Always use module-level loggers created with logging.getLogger(__name__).
• Leverage dictConfig() to maintain a clean, centralized configuration file.
• Emit structured JSON logs in production to make querying effortless.
• Utilize logger.exception() to capture full stack traces.
• Avoid performance bottlenecks by using QueueHandler for high-throughput services.
• Ship your logs to a centralized OpenTelemetry-native platform like SigNoz to correlate them with traces and metrics.

Conclusion

By mastering Python logging, you equip your applications with the self-reporting capabilities required to survive in production. Good logging practices transform debugging from a frustrating hunt into a methodical, data-driven process.

Start with a simple configuration, incrementally add structured JSON formatting, and ultimately integrate with a platform like SigNoz to achieve true observability.

Resources

• Python Logging Documentation
• Python Logging HOWTO
• SigNoz Documentation
• SigNoz Logs Management Overview

FAQs

What is logging in Python?

Logging in Python is the process of recording runtime events using the standard-library logging module, allowing you to debug, monitor, and audit application behavior effectively.

Can I log to multiple destinations at the same time?

Yes. You can attach multiple handlers (e.g., a StreamHandler and a FileHandler) to the same logger or configure them centrally using dictConfig().

How do I add custom fields to my log messages?

Use the extra dictionary for one-off fields, or wrap your logger in a LoggerAdapter when you want to consistently reuse the same contextual data (like a request_id) across many log calls.

What is a good production logging baseline?

A practical, production-ready baseline includes:

• Module-level loggers using getLogger(__name__).
• Centralized configuration via dictConfig().
• Structured JSON logs.
• Appropriate file rotation or centralized log shipping.
• Aggressive redaction of sensitive PII data using Filters.