Python Logging Best Practices - Expert Tips with Practical Examples

Struggling with Python logging? You're not alone. Common headaches include:

• Logs that are too verbose or too sparse
• Missing critical info when debugging
• Performance hits from excessive logging
• Security risks from logging sensitive data

Let's fix that. Here are the best practices we'll cover:

• Use appropriate log levels: Debug, Info, Warning, Error, Critical
• Create custom loggers: Tailor logging to your app's needs
• Format logs effectively: Make them readable and useful
• Centralize logging config: Maintain consistency across your app
• Rotate logs: Prevent massive log files
• Handle exceptions gracefully: Log errors without crashing
• Implement async logging: Boost performance
• Structure your logs: Make them easily searchable

Let’s dive deep into these best practices of logging in Python! (With Practical Examples)

Loggers are Singletons

In Python logging practices, loggers operate as singletons, ensuring that logging.getLogger(name) consistently returns the same logger instance throughout the application. This design promotes uniformity in logging configurations, preventing conflicts and optimizing resource usage. By maintaining a single logger instance per name, developers benefit from streamlined logging operations and enhanced code readability, making it easier to manage and troubleshoot logging behaviours across complex applications.

Let's explore examples to see how multiple instances of non-singleton loggers with the same name can lead to conflicting configurations and inconsistent logging behavior. We'll then demonstrate how singleton loggers ensure uniformity in logging settings and prevent such conflicts, thereby maintaining clarity and reliability in log output across the application.

Example: Non-Singleton Loggers (Hypothetical)

If loggers were not singletons, you might end up with different logger instances with the same name, each with its own configuration. This could lead to conflicts and unexpected logging behavior.

Hypothetical Non-Singleton Logger Example:

import logging

# Setup first logger instance
logger1 = logging.getLogger('exampleLogger')
logger1.setLevel(logging.DEBUG)
console_handler1 = logging.StreamHandler()
formatter1 = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
console_handler1.setFormatter(formatter1)
logger1.addHandler(console_handler1)

# Setup second logger instance with the same name but different configuration
logger2 = logging.getLogger('exampleLogger')
logger2.setLevel(logging.ERROR)
console_handler2 = logging.StreamHandler()
formatter2 = logging.Formatter('%(name)s - %(levelname)s - %(message)s')
console_handler2.setFormatter(formatter2)
logger2.addHandler(console_handler2)

# Logging messages
logger1.debug("Debug message from logger1")
logger2.error("Error message from logger2")

In this hypothetical scenario, if logger1 and logger2 were not singletons, you would get inconsistent output depending on which logger instance handled the log message. The two instances might have different log levels and formats, leading to confusion and difficulty in managing log output.

Example: Singleton Loggers (Real Scenario)

In reality, Python loggers are singletons. Let's demonstrate how this design helps maintain consistency and avoid conflicts.

Singleton Logger Example:

import logging

# Setup logger instance
logger = logging.getLogger('exampleLogger')
logger.setLevel(logging.DEBUG)
console_handler = logging.StreamHandler()
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
console_handler.setFormatter(formatter)
logger.addHandler(console_handler)

# Retrieve the same logger instance elsewhere in the application
same_logger = logging.getLogger('exampleLogger')

# Confirm they are the same instance
print(logger is same_logger)  # Output: True

# Setup another handler to demonstrate consistency
file_handler = logging.FileHandler('example.log')
file_handler.setFormatter(formatter)
same_logger.addHandler(file_handler)

# Logging messages
logger.debug("Debug message from logger")
same_logger.error("Error message from same_logger")

Explanation:

1. Singleton Logger Setup:
   • logging.getLogger('exampleLogger') is called twice, but both logger and same_logger refer to the same logger instance due to the singleton nature.
2. Configuration Consistency:
   • Both logger and same_logger share the same configuration. Adding handlers or changing log levels on one affects the other.
3. Log Messages:
   • The log messages will be handled by the same set of handlers, ensuring consistent log output. Both messages will appear in the console and the example.log file with the same format.

Output:

2024-07-12 12:00:00,000 - exampleLogger - DEBUG - Debug message from logger
2024-07-12 12:00:00,000 - exampleLogger - ERROR - Error message from same_logger
True

And in example.log:

2024-07-12 12:00:00,000 - exampleLogger - DEBUG - Debug message from logger
2024-07-12 12:00:00,000 - exampleLogger - ERROR - Error message from same_logger

Use Appropriate Log Levels

The use of appropriate log levels is the key best practice in python logging. It helps in categorizing log messages based on their importance and severity. The various log levels are as follows:

1. DEBUG: Used for granular details regarding program execution giving detailed information typically of interest when diagnosing problems.
2. INFO: Confirms the proper working of the program. Provides general operational messages that highlight the application's progress.
3. WARNING: Indicates any unexpected happenings or problems that one may encounter in the future
4. ERROR: Reports a serious issue, e.g. the software is unable to perform some function.
5. CRITICAL: Indicates a serious error reporting that the program can't work any longer.

Example:

import logging
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
logger.debug("This is a debug message.")
logger.info("This is an info message.")
logger.warning("This is a warning message.")
logger.error("This is an error message.")
logger.critical("This is a critical message.")

In this example:

• logging.basicConfig is used for configuring the logging system. The level parameter sets the threshold for the logger to DEBUG, which means that all messages with the set level or higher will be logged.
• logging.getLogger(name) is used to create a logger object with the same name as that of the current module.
• Various log messages are logged using different log levels(debug, info, warning, error, and critical).

On running the above program the output will be:

2024-07-08 00:16:01,199 - DEBUG - This is a debug message.
2024-07-08 00:16:01,199 - INFO - This is an info message.
2024-07-08 00:16:01,199 - WARNING - This is a warning message.
2024-07-08 00:16:01,199 - ERROR - This is an error message.
2024-07-08 00:16:01,199 - CRITICAL - This is a critical message.

Signoz is an open-source observability platform that enhances logging by effectively, enabling developers to quickly identify and address issues. By integrating with Signoz, teams can monitor logs in real-time, correlate them with system performance, and gain insights into application behavior, which streamlines troubleshooting and maintenance.

An elaborate step by step guide to send your python logs to Signoz by using OpenTelemetry can be found here.

Create a Custom Logger

Custom loggers let you tailor the logging behaviour to fit specific needs. It lets you define custom formats, handlers, and even log-level thresholds for various application parts.

Let us now see an example of combining the configuration and usage of the custom logger:

import logging
import logging.config

# Custom logger configuration
custom_logger_config = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "customFormatter": {
            "format": "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
        }
    },
    "handlers": {
        "consoleHandler": {
            "class": "logging.StreamHandler",
            "formatter": "customFormatter",
            "level": "DEBUG"
        },
        "fileHandler": {
            "class": "logging.FileHandler",
            "formatter": "customFormatter",
            "level": "INFO",
            "filename": "app.log",
            "mode": "w"
        }
    },
    "loggers": {
        "customLogger": {
            "handlers": ["consoleHandler", "fileHandler"],
            "level": "DEBUG",
            "propagate": False
        }
    }
}

# Apply the custom logger configuration
logging.config.dictConfig(custom_logger_config)

# Create the custom logger
logger = logging.getLogger("customLogger")

# Example log messages
logger.debug("This is a debug message.")
logger.info("This is an info message.")
logger.warning("This is a warning message.")
logger.error("This is an error message.")
logger.critical("This is a critical message.")

Explanation:

• Configuration Dictionary: Used for defining the formatters, handlers, and loggers. Each handler uses a formatter, and each logger uses one or more handlers.
• Custom Formatter: Defines the format of log messages, like the timestamp, logger name, log level, and message.
• Handlers: The consoleHandler give log output to the console, while the fileHandler writes logs to a file (app.log).
• Custom Logger: The customLogger uses the defined handlers and has a log level of DEBUG, meaning it will log messages at this level and higher.

Format Log messages

You can format log messages in Python using formatters. Formatters define the structure and content of the log output. They include details like timestamps, log levels, logger names, and the log message. It provides consistent and informative log entries which is important for debugging and monitoring.

In Python's logging module, you can create a formatter by defining a format string and passing it to the logging.Formatter class. The format string has various placeholders which can be replaced with corresponding log record attributes. Common placeholders are as follows:

• %(asctime)s: Provides the time of the creation of the log message.
• %(name)s: Provides logger name
• %(levelname)s: Gives log level
• %(message)s: Provides the actual log message
• %(filename)s: Filename where the log message originated
• %(lineno)d: The line number in the source file where the log message was created.

Let us now see the creation and use of formatters in Python logging:

import logging

# Define the formatter
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# Create handlers
console_handler = logging.StreamHandler()
file_handler = logging.FileHandler('formatted_app.log')

# Set the formatter for the handlers
console_handler.setFormatter(formatter)
file_handler.setFormatter(formatter)

# Create a logger
logger = logging.getLogger('formattedLogger')

# Set the log level for the logger
logger.setLevel(logging.DEBUG)

# Add handlers to the logger
logger.addHandler(console_handler)
logger.addHandler(file_handler)

# Example log messages
logger.debug("This is a debug message.")
logger.info("This is an info message.")
logger.warning("This is a warning message.")
logger.error("This is an error message.")
logger.critical("This is a critical message.")

The output for the above script will be:

2024-07-08 00:17:35,879 - formattedLogger - DEBUG - This is a debug message.
2024-07-08 00:17:35,879 - formattedLogger - INFO - This is an info message.
2024-07-08 00:17:35,879 - formattedLogger - WARNING - This is a warning message.
2024-07-08 00:17:35,879 - formattedLogger - ERROR - This is an error message.
2024-07-08 00:17:35,879 - formattedLogger - CRITICAL - This is a critical message.

Centralize your logging configuration

In order to simplify, manage and maintain logging settings across your application, you should centralize your logging information. It helps to easily adjust the logging configuration, like log levels, formats, and handlers in place. This provides consistency and reduces errors.

Centralization of logging information can be done via configuring the file with JSON/YAML. Let us understand both these approaches separately.

JSON Configuration

In this example, you shall learn how to use a JSON configuration for logging:

{
  'version': 1,
  'disable_existing_loggers': false,
  'formatters':
    { 'standard': { 'format': '%(asctime)s - %(levelname)s - %(name)s - %(message)s' } },
  'handlers':
    {
      'console':
        {
          'class': 'logging.StreamHandler',
          'formatter': 'standard',
          'level': 'DEBUG',
          'stream': 'ext://sys.stdout',
        },
    },
  'loggers': { '': { 'handlers': ['console'], 'level': 'DEBUG', 'propagate': true } },
}

Further, use this configuration in Python script:

import logging
import logging.config
import json

# Load logging configuration from JSON file
with open('logging_config.json', 'r') as f:
    config = json.load(f)

# Configure logging
logging.config.dictConfig(config)

# Create a logger
logger = logging.getLogger(__name__)

# Example log messages
logger.debug("This is a debug message.")
logger.info("This is an info message.")
logger.warning("This is a warning message.")
logger.error("This is an error message.")
logger.critical("This is a critical message.")

YAML Configuration

Follow the steps to create logging_config.yaml file:

version: 1
disable_existing_loggers: false
formatters:
  standard:
    format: '%(asctime)s - %(levelname)s - %(name)s - %(message)s'
handlers:
  console:
    class: logging.StreamHandler
    formatter: standard
    level: DEBUG
    stream: ext://sys.stdout
loggers:
  '':
    handlers: [console]
    level: DEBUG
    propagate: true

Next, use this configuration in your Python script:

import logging
import logging.config
import yaml

# Load logging configuration from YAML file
with open('logging_config.yaml', 'r') as f:
    config = yaml.safe_load(f)

# Configure logging
logging.config.dictConfig(config)

# Create a logger
logger = logging.getLogger(__name__)

# Example log messages
logger.debug("This is a debug message.")
logger.info("This is an info message.")
logger.warning("This is a warning message.")
logger.error("This is an error message.")
logger.critical("This is a critical message.")

With the use of JSON or YAML configurations for logging, you can enhance the flexibility and manageability of your logging setups, leading to more maintainable and adaptable tools.

Avoid Logging Sensitive Information

Ensuring security and privacy is one of the major concerns in software development. The best practice to address this critical issue is by avoiding logging sensitive information. If sensitive data such as passwords, credit card numbers, personal identification information, and other confidential details are logged and then accessed by unauthorized individuals it will lead to significant security threat.

Implementation in Python

Strategies to Avoid Logging Sensitive Information

1. Redaction: You should hide sensitive information before logging.
2. Conditional Logging: Use conditions to avoid logging sensitive information.
3. Custom Logging Filters: Create custom logging filters to remove sensitive information.

Let us now look at the implementation of these strategies:

Redaction

import logging
import re

class RedactingFormatter(logging.Formatter):
    def format(self, record):
        message = super().format(record)
        # Redact sensitive information using regex
        redacted_message = re.sub(r'\bpassword\b', '**', message, flags=re.IGNORECASE)
        return redacted_message

# Setup logger
logger = logging.getLogger('secureLogger')
logger.setLevel(logging.DEBUG)

# Console handler
console_handler = logging.StreamHandler()
formatter = RedactingFormatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
console_handler.setFormatter(formatter)
logger.addHandler(console_handler)

# Example usage
logger.debug("User logged in with password: 12345")
logger.info("This is an info message without sensitive data.")

Output:

2024-07-12 09:37:06,037 - secureLogger - DEBUG - User logged in with **: 12345
2024-07-12 09:37:06,037 - secureLogger - INFO - This is an info message without sensitive data.

In this example, the RedactingFormatter replaces any instance of the password with asterisks.

Conditional Logging

import logging

# Setup logger
logger = logging.getLogger('conditionalLogger')
logger.setLevel(logging.DEBUG)

# Console handler
console_handler = logging.StreamHandler()
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
console_handler.setFormatter(formatter)
logger.addHandler(console_handler)

# Function to log conditionally
def log_info(message):
    if "password" not in message:
        logger.info(message)
    else:
        logger.warning("Attempted to log sensitive information!")

# Example usage
log_info("This is an info message without sensitive data.")
log_info("User logged in with password: 12345")

Output:

2024-07-08 00:21:01,526 - conditionalLogger - INFO - This is an info message without sensitive data.
2024-07-08 00:21:01,526 - conditionalLogger - WARNING - Attempted to log sensitive information!

In this example, the log_info function checks the message for sensitive information before logging it.

Custom Logging Filters

import logging

class SensitiveInfoFilter(logging.Filter):
    def filter(self, record):
        # Remove sensitive information
        if "password" in record.msg:
            return False
        return True

# Setup logger
logger = logging.getLogger('filterLogger')
logger.setLevel(logging.DEBUG)

# Console handler
console_handler = logging.StreamHandler()
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
console_handler.setFormatter(formatter)
logger.addHandler(console_handler)

# Add filter to logger
sensitive_info_filter = SensitiveInfoFilter()
logger.addFilter(sensitive_info_filter)

# Example usage
logger.info("This is an info message without sensitive data.")
logger.info("User logged in with password: 12345")

Output:

2024-07-12 09:31:13,672 - filterLogger - INFO - This is an info message without sensitive data.

In this example, the SensitiveInfoFilter prevents any log message containing the word "password" from being logged.

Avoid Root Logger

One of the best practices while logging in python is to avoid root logger. Root logger is the default logger instance created by the logging module. It is a convenient method, though it reduces code flexibility. It is better to create and configure named loggers for your application. The use of appropriate named loggers provide more granularity, isolation and clarity to the code.

Named loggers in Python's logging module are instances of the Logger class that are given specific names. These names help to identify and categorize log messages, making it easier to control logging behavior and manage log output from different parts of an application. Named loggers can be hierarchical, which means they can have parent-child relationships. This allows for fine-grained control over logging configuration.

Implementation in Python

Creating and Configuring Named Loggers

Let us learn to create and configure named loggers in Python.

import logging

# Configure the root logger minimally or not at all
logging.basicConfig(level=logging.WARNING)  # Optional: Minimal configuration for the root logger

# Create and configure a named logger
def setup_logger(name, log_file, level=logging.INFO):
    logger = logging.getLogger(name)
    logger.setLevel(level)

    # Create handlers
    file_handler = logging.FileHandler(log_file)
    console_handler = logging.StreamHandler()

    # Create formatters and add them to handlers
    formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
    file_handler.setFormatter(formatter)
    console_handler.setFormatter(formatter)

    # Add handlers to the logger
    logger.addHandler(file_handler)
    logger.addHandler(console_handler)

    return logger

# Set up different named loggers
app_logger = setup_logger('appLogger', 'app.log')
db_logger = setup_logger('dbLogger', 'db.log', level=logging.DEBUG)

# Example usage
app_logger.info("This is an info message from the application.")
app_logger.error("This is an error message from the application.")

db_logger.debug("This is a debug message from the database module.")
db_logger.warning("This is a warning message from the database module.")

Output:

2024-07-08 00:23:04,943 - appLogger - INFO - This is an info message from the application.
INFO:appLogger:This is an info message from the application.
2024-07-08 00:23:04,943 - appLogger - ERROR - This is an error message from the application.
ERROR:appLogger:This is an error message from the application.
2024-07-08 00:23:04,943 - dbLogger - DEBUG - This is a debug message from the database module.
DEBUG:dbLogger:This is a debug message from the database module.
2024-07-08 00:23:04,943 - dbLogger - WARNING - This is a warning message from the database module.
WARNING:dbLogger:This is a warning message from the database module.

Write Meaningful Logs

For an enhanced ability to monitor, debug, and maintain an application one should always write meaningful logs. They give clear and useful information that helps developers and system administrators understand the workings of the application, identify issues, and track application behavior over time.

Principles for Writing Meaningful Logs

1. Clarity and Conciseness: Logs need to be clear and concise that provide the required context only.
2. Contextual Information: Logs should only include relevant information to provide proper context.
3. Consistency: A consistent format for all log messages needs to be followed throughout an application.
4. Appropriate Log Levels: The correct log level needs to be used to reflect the severity and nature of the event being logged.
5. Actionable Information: Logs should provide information that can be acted upon.

Examples of Meaningful Logs

Example 1: Logging a Function Entry with Parameters

import logging

logger = logging.getLogger('exampleLogger')
logger.setLevel(logging.DEBUG)

console_handler = logging.StreamHandler()
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
console_handler.setFormatter(formatter)
logger.addHandler(console_handler)

def process_order(order_id, user_id):
    logger.debug(f"Entered process_order function with order_id={order_id}, user_id={user_id}")
    # Processing logic
    logger.info(f"Order {order_id} processed successfully for user {user_id}")

# Example usage
process_order(123, 456)

Output:

2024-07-08 00:24:04,342 - exampleLogger - DEBUG - Entered process_order function with order_id=123, user_id=456
2024-07-08 00:24:04,342 - exampleLogger - INFO - Order 123 processed successfully for user 456

Example 2: Logging an Error with Context

import logging

logger = logging.getLogger('exampleLogger')
logger.setLevel(logging.DEBUG)

console_handler = logging.StreamHandler()
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
console_handler.setFormatter(formatter)
logger.addHandler(console_handler)

def divide_numbers(a, b):
    try:
        result = a / b
        logger.info(f"Division successful: {a} / {b} = {result}")
    except ZeroDivisionError:
        logger.error(f"Failed to divide {a} by {b}: Division by zero", exc_info=True)

# Example usage
divide_numbers(10, 0)

Output:

2024-07-08 00:24:47,032 - exampleLogger - ERROR - Failed to divide 10 by 0: Division by zero
Traceback (most recent call last):
  File "/Users/falcon/Desktop/signoz-python-logging/main.py", line 13, in divide_numbers
    result = a / b
ZeroDivisionError: division by zero

Example 3: Logging a Warning with a Specific Condition

import logging

# Create a logger with the name 'exampleLogger'
logger = logging.getLogger('exampleLogger')
# Set the logging level to DEBUG, so all messages of this level and above will be logged
logger.setLevel(logging.DEBUG)

# Create a console handler to output logs to the console
console_handler = logging.StreamHandler()
# Define a formatter to specify the format of the log messages
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
# Set the formatter for the console handler
console_handler.setFormatter(formatter)
# Add the console handler to the logger
logger.addHandler(console_handler)

def check_inventory(item_id, quantity):
    """
    Check the inventory for the given item and quantity.
    Log a warning if the inventory is low, otherwise log an info message.

    :param item_id: The ID of the item to check
    :param quantity: The quantity of the item requested
    """
    # Example inventory dictionary with item IDs and their available quantities
    inventory = {"item1": 10, "item2": 5}

    # Check if the requested quantity is greater than the available quantity in the inventory
    if inventory.get(item_id, 0) < quantity:
        # Log a warning if the inventory is low
        logger.warning(f"Low inventory for item {item_id}: requested {quantity}, available {inventory.get(item_id, 0)}")
    else:
        # Log an info message if the inventory check passes
        logger.info(f"Inventory check passed for item {item_id}: requested {quantity}, available {inventory.get(item_id, 0)}")

# Example usage of the check_inventory function
check_inventory("item1", 15)

Output:

2024-07-08 00:25:17,706 - exampleLogger - WARNING - Low inventory for item item1: requested 15, available 10

Use Rotating Logs

One of the best python practices in logging is the use of rotating logs. The use of rotating logs helps to prevent log files from growing indefinitely, which can consume maximum disk space making log management difficult. They improve performance and organization of log files. The logging module in Python provides handlers such as RotatingFileHandler and TimedRotatingFileHandler to facilitate log rotation.

Implementation in Python

Using RotatingFileHandler

RotatingFileHandler rotates logs based on file size. When the log file reaches a certain size, it is renamed, and a new log file is created.

import logging
from logging.handlers import RotatingFileHandler

# Create a logger
logger = logging.getLogger('rotatingLogger')
logger.setLevel(logging.DEBUG)

# Create a handler for rotating logs
handler = RotatingFileHandler(
    'app.log', maxBytes=1024*5, backupCount=3  # 5 KB per file, 3 backup files
)

# Create a formatter and set it for the handler
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)

# Add the handler to the logger
logger.addHandler(handler)

# Example usage
for i in range(100):
    logger.debug(f"This is log message {i}")

In this example:

• maxBytes specifies the maximum file size (5 KB in this case) before rotation occurs.
• backupCount specifies the number of backup files to keep. Old files will be deleted when this limit is reached.

Using TimedRotatingFileHandler

TimedRotatingFileHandler rotates logs based on time intervals (e.g., daily, hourly).

import logging
from logging.handlers import TimedRotatingFileHandler

# Create a logger
logger = logging.getLogger('timedRotatingLogger')
logger.setLevel(logging.DEBUG)

# Create a handler for timed rotating logs
handler = TimedRotatingFileHandler(
    'timed_app.log', when='midnight', interval=1, backupCount=7
)

# Create a formatter and set it for the handler
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)

# Add the handler to the logger
logger.addHandler(handler)

# Example usage
for i in range(100):
    logger.debug(f"This is log message {i}")

Explanation:

• when='midnight' specifies that log rotation should occur at midnight.
• interval=1 indicates the frequency of rotation (every 1 day in this case).
• backupCount=7 specifies the number of backup files to keep (one week of daily logs).

Output:

When using RotatingFileHandler with the above configuration, you might see log files like:

app.log
app.log.1
app.log.2
app.log.3

When using TimedRotatingFileHandler, you shall see log files like:

timed_app.log
timed_app.log.2024-07-01
timed_app.log.2024-07-02
...

Scaling logs

As the complexity of an application increases, scaling log becomes significant in monitoring, debugging, and analyzing application behavior. Several strategies for scaling logs are as follows:

1. Distributed Logging: Implement a distributed logging architecture where logs are collected from multiple sources and centralized for storage and analysis. This approach helps distribute the load and ensures that logs are accessible from a central location.
2. Log Aggregation: Use log aggregation tools or services (e.g., Elasticsearch, Splunk, Sumo Logic) to consolidate logs from various sources into a centralized repository. These tools often provide advanced querying and visualization capabilities for analyzing log data.
3. Asynchronous Logging: Use asynchronous logging to improve application performance. Asynchronous logging allows the application to continue running without waiting for the log messages to be written to disk or sent over the network.
4. Batch Processing: Batch log messages to reduce the overhead of logging operations. Instead of writing each log message individually, batch them together and write them in larger chunks.
5. Log Compression: Compress log files to reduce storage costs and improve transfer speeds, especially when shipping logs over the network to a centralized location.
6. Horizontal Scaling: Scale your logging infrastructure horizontally by adding more logging nodes or instances. This approach ensures that the logging system can handle increased load and redundancy in case of failures.
7. Load Balancing: Use load balancers to distribute incoming log messages across multiple logging nodes or services. This helps evenly distribute the workload and ensures high availability.
8. Monitoring and Alerting: Implement monitoring and alerting mechanisms to proactively monitor the health and performance of your logging infrastructure. Set up alerts for high log volumes, errors in logging, or infrastructure failures.

Handle Exceptions Gracefully

Proper handling of exceptions is essential while logging in Python. It ensures that the application remains stable, providing meaningful feedback during failures and helps in troubleshooting issues. Let us see how to handle exceptions gracefully and integrate logs effectively.

Best Practices for Handling Exceptions and Logging

1. Use Try-Except Blocks:

   • Wrap code likely to raise exceptions in try-except blocks.
   • Log exceptions with relevant context to aid in debugging.

   import logging
   
   # Configure logging
   logging.basicConfig(level=logging.INFO)
   logger = logging.getLogger(__name__)
   
   try:
       # Code that may raise exceptions
       result = 10 / 0  # Example division by zero to trigger an exception
   except ZeroDivisionError as e:
       logger.error("Division by zero occurred", exc_info=True)
       # Optionally, handle the exception (e.g., notify, recover, fallback)
   
   

   Output :

   	ERROR:__main__:Division by zero occurred
   	Traceback (most recent call last):
   	  File "/Users/falcon/Desktop/signoz-python-logging/main.py", line 9, in <module>
   	    result = 10 / 0  # Example division by zero to trigger an exception
   	ZeroDivisionError: division by zero
   
   

2. Log Contextual Information:

   • Include additional context in log messages, such as variable values or the current state of the application.

   try:
       # Code that may raise exceptions
       item = inventory[item_id]
   except KeyError as e:
       logger.error(f"Item {item_id} not found in inventory", exc_info=True)
   
   

3. Use Exception Hierarchy:

   • Handle specific exceptions separately to provide tailored error messages and actions.

   try:
       # Code that may raise exceptions
       file = open('file.txt', 'r')
   except FileNotFoundError as e:
       logger.error("File not found: file.txt", exc_info=True)
   except IOError as e:
       logger.error("IOError occurred while opening file", exc_info=True)
   
   

4. Ensure Logging Configuration:

   • Configure logging settings, such as log level and output format, to suit your application's needs.

   import logging
   
   logging.basicConfig(
       filename='app.log',
       format='%(asctime)s - %(levelname)s - %(message)s',
       level=logging.INFO
   )
   
   

5. Centralize Exception Handling:

   • Use a centralized mechanism for logging exceptions across your application to maintain consistency and simplify maintenance.

   def process_data(data):
       try:
           # Code that may raise exceptions
           result = data / 0
       except Exception as e:
           logger.error("An error occurred during data processing", exc_info=True)
           # Optionally, raise or handle the exception appropriately
   
   def main():
       try:
           # Main application logic
           process_data(some_data)
       except Exception as e:
           logger.error("Unexpected error occurred", exc_info=True)
           # Optionally, handle or re-raise the exception
   
   

Test and Monitor Logs

Testing and monitoring logs are important practices that ensure the working of logging system correctly. It helps to capture relevant information and ensure reliability.

Testing Logs

1. Unit Testing:

   • Write unit tests to validate that log messages are generated correctly under different scenarios.
   • Use mocking to simulate log events and verify the behavior of loggers and handlers.

   import unittest
   import logging
   from unittest.mock import patch
   
   class TestLogging(unittest.TestCase):
   
       def test_logging(self):
           with patch('logging.Logger.error') as mock_error:
               logging.error('Error message')
               mock_error.assert_called_once_with('Error message')
   
   

2. Integration Testing:

   • Include tests that verify the integration of logging with other components of your application, such as error handling and data processing.

   class TestIntegration(unittest.TestCase):
   
       def test_data_processing_with_logging(self):
           # Simulate data processing
           try:
               result = 10 / 0  # Example to trigger an exception
           except ZeroDivisionError:
               logging.error('Division by zero occurred')
               # Assert log message or expected behavior
               self.assertTrue(logging.error.called)
   
   

3. Coverage:

   • Ensure that log messages cover various scenarios, including edge cases, error conditions, and expected outputs.
   • Use code coverage tools to verify that logging statements are executed during tests.

   class TestCoverage(unittest.TestCase):
   
       def test_critical_error_handling(self):
           try:
               # Trigger critical error
               raise RuntimeError('Critical error')
           except RuntimeError as e:
               logging.critical('Critical error occurred: %s', e)
               # Assert log message
               self.assertTrue(logging.critical.called)
   
   

Asynchronous Logging

Asynchronous logging is used to improve performance of logging operations in Python applications. It is used when logging synchronously would introduce delays and affect the overall application speed. In this section, you will learn about the implementation of asynchronous logging. Asynchronous logging provides improved performance, non blocking feature and helps maintain application performance under heavy loads.

Implementation Using JsonLogger (aiologger)

The aiologger library provides a JsonLogger specifically designed for asynchronous logging using Python's asyncio framework.

1. Installation: Install aiologger using pip:

   pip install aiologger
   
       ```
   
   

2. Example Implementation:

    import asyncio
    from aiologger.loggers.json import JsonLogger
   
    async def main():
        # Create a logger instance with default handlers
        logger = JsonLogger.with_default_handlers(name="exampleLogger", level="DEBUG")
   
        # Generate log messages asynchronously
        for i in range(10):
            await logger.debug(f"This is log message {i}")
   
    if __name__ == "__main__":
        asyncio.run(main())
   
   

Output:

{"logged_at": "2024-07-08T00:31:46.472045+05:30", "line_number": 10, "function": "main", "level": "DEBUG", "file_path": "/Users/falcon/Desktop/signoz-python-logging/main.py", "msg": "This is log message 0"}
{"logged_at": "2024-07-08T00:31:46.472372+05:30", "line_number": 10, "function": "main", "level": "DEBUG", "file_path": "/Users/falcon/Desktop/signoz-python-logging/main.py", "msg": "This is log message 1"}
{"logged_at": "2024-07-08T00:31:46.472465+05:30", "line_number": 10, "function": "main", "level": "DEBUG", "file_path": "/Users/falcon/Desktop/signoz-python-logging/main.py", "msg": "This is log message 2"}
{"logged_at": "2024-07-08T00:31:46.472537+05:30", "line_number": 10, "function": "main", "level": "DEBUG", "file_path": "/Users/falcon/Desktop/signoz-python-logging/main.py", "msg": "This is log message 3"}
{"logged_at": "2024-07-08T00:31:46.472603+05:30", "line_number": 10, "function": "main", "level": "DEBUG", "file_path": "/Users/falcon/Desktop/signoz-python-logging/main.py", "msg": "This is log message 4"}
{"logged_at": "2024-07-08T00:31:46.472666+05:30", "line_number": 10, "function": "main", "level": "DEBUG", "file_path": "/Users/falcon/Desktop/signoz-python-logging/main.py", "msg": "This is log message 5"}
{"logged_at": "2024-07-08T00:31:46.472732+05:30", "line_number": 10, "function": "main", "level": "DEBUG", "file_path": "/Users/falcon/Desktop/signoz-python-logging/main.py", "msg": "This is log message 6"}
{"logged_at": "2024-07-08T00:31:46.472825+05:30", "line_number": 10, "function": "main", "level": "DEBUG", "file_path": "/Users/falcon/Desktop/signoz-python-logging/main.py", "msg": "This is log message 7"}
{"logged_at": "2024-07-08T00:31:46.472892+05:30", "line_number": 10, "function": "main", "level": "DEBUG", "file_path": "/Users/falcon/Desktop/signoz-python-logging/main.py", "msg": "This is log message 8"}
{"logged_at": "2024-07-08T00:31:46.472959+05:30", "line_number": 10, "function": "main", "level": "DEBUG", "file_path": "/Users/falcon/Desktop/signoz-python-logging/main.py", "msg": "This is log message 9"}

Explanation:

• JsonLogger: JsonLogger.with_default_handlers() is used here as an example to configure logging with a JSON format, suitable for structured logging.
• Logging Levels: Adjust the level parameter as needed (DEBUG, INFO, WARNING, ERROR, CRITICAL) to control which log messages are processed by the handler.

Structured Logging

Structured logging involves logging data in a structured format generally using JSON, key- value pairs, or other structured data formats. It offers many advantages over traditional logging method, It provides easier parsing, improved searchability, and better integration with log aggregation and analysis tools.

Implementation Using logging Module in Python

Python’s built-in logging module can be configured to output structured log messages using a custom formatter. Here’s an example of structured logging using JSON format:

import logging
import json

# Configure logging to output JSON format
logging.basicConfig(level=logging.DEBUG, format='%(message)s')
logger = logging.getLogger(__name__)

class StructuredMessage:
    def __init__(self, message, **kwargs):
        self.message = message
        self.kwargs = kwargs

    def __str__(self):
        return json.dumps({
            'message': self.message,
            **self.kwargs
        })

    def __repr__(self):
        return self.__str__()

    def as_dict(self):
        return {
            'message': self.message,
            **self.kwargs
        }

# Example usage
structured_log = StructuredMessage('User login', user_id=123, status='success')
logger.debug(structured_log)

Centralize Log Management

One of the best practices for handling logs in complex applications is to use a centralized log management system. SigNoz is an excellent tool for this purpose, offering powerful features for collecting, analyzing, and visualizing logs from your Python applications.

Why Centralize Logs with SigNoz?

• Unified view: See logs from all your services in one place • Easy search and analysis: Quickly find relevant logs using powerful search capabilities • Performance insights: Correlate logs with traces and metrics for better debugging • Scalability: Handle large volumes of logs efficiently

Implementing SigNoz with Python and OpenTelemetry

To send Python logs to SigNoz:

1. SetUp SigNoz

   SigNoz cloud is the easiest way to run SigNoz. Sign up for a free account and get 30 days of unlimited access to all features. You can also install and self-host SigNoz yourself since it is open-source. With 18,000+ GitHub stars, open-source SigNoz is loved by developers. Find the instructions to self-host SigNoz.

2. Setup OpenTelemetry: Install the necessary libraries and configure the exporter.
3. Configure Logging: Use OpenTelemetry's Python SDK to set up logging.
4. Integrate with SigNoz: Configure the logging exporter to send logs to SigNoz's backend.
5. Verify and Test: Monitor SigNoz's dashboard for incoming logs from your Python application.

For detailed implementation steps, refer to SigNoz's guide on logging in Python with OpenTelemetry here. This guide will provide specific instructions tailored to integrate Python logging with SigNoz's observability platform using OpenTelemetry.

Analysis of different log tools and their placement

Let us look into some popular log tools and their features with the help of a table.

Here's a table comparing different logging tools in Python, including their features, strengths, and typical use cases:

Conclusion

• Effective logging in Python is essential for debugging and maintaining applications.
• Practices like structured logging, appropriate log levels, and custom loggers improve clarity and debugging efficiency.
• Centralizing logging configuration ensures consistency across the application.
• Handling exceptions gracefully and avoiding sensitive information in logs enhances security.
• Using tools like rotating logs and asynchronous logging supports scalability and performance.
• Regular testing and monitoring of logs ensure they accurately reflect application behavior, aiding in troubleshooting and optimization efforts.

FAQs

What is the best way to log in Python?

The best way to log in Python is by using the built-in logging module. It will log messages to the console with a timestamp, logger name, log level and the message.

How to improve logging in Python?

The guidelines to follow for improved logging in Python are as follows:

1. Use appropriate log levels
2. Add context
3. Use handlers
4. Format logs
5. Rotate log files
6. External Libraries

What is the fastest way to log data in Python?

The fastest way to log data in Python is with the built-in 'logging' module with asynchronous logging.

What is the lowest level of logging in Python?

The lowest level of logging in Python is DEBUG.

What is the highest logging level in Python?

The highest logging level in Python is CRITICAL.

What are the different types of logging in Python?

There are five different types of logging in Python which are as follows:

1. Debug
2. Info
3. Warning
4. Error
5. Critical

What is the most popular logger?

The most popular logger is the built-in logging module. It's extensively used due to its versatility, ease of use, and comprehensive features for logging messages across applications.

Why do we use logging in Python?

Logging is used in Python to record useful information, errors, and debugging messages during the execution of a program. It helps to understand the behavior of a program, diagnose issues, and monitor applications.

How to write good logs?

In order to write good logs one must ensure that the log messages are clear and concise, relevant, consistent, structured and handle errors gracefully.