It is essential to configure log rotation for Docker containers. Log rotation is not performed by default, and if it’s not configured, logs on the Docker host can build up and eat up disk space. This guide will teach us how to set up Docker log rotation.
Logs are an essential piece of telemetry data. Logs can be used to debug performance issues in applications. They produce a thorough list of the events that take place in your application, which can be used to troubleshoot issues and identify the root cause of a problem.
With containerization, it is easier to scale applications. But the operation complexity has increased manifolds. Containers are ephemeral. Frequently changing container-based environments are challenging to monitor. Docker logs can help debug performance issues. Applications in Docker containers emit logs through stdout and stderr output streams. The logging driver that you choose can access these streams. Based on your logging driver, you can configure the format and storage of Docker logs. You can also send the emitted logs to a central log management system.
Before deep-diving into configuring Docker log rotation, let's briefly overview Docker logs.
A brief on Docker Logs
Logging in Docker is different from applications hosted on physical or virtual hosts. Container environment is dynamic, and troubleshooting is more complex.
In Docker, primarily, there are two types of log files.
Docker daemon logs
These logs are generated by the Docker daemon and located on the host. It provides insights into the state of the Docker platform.Docker container logs
Docker container logs cover all the logs related to a particular application running in a container.
Docker does not impose a size restriction on logging files. Hence they will inevitably increase over time and consume storage if left unchecked. You can imagine the growth of log files over time and the amount of storage they would require in a scenario where you have numerous containers running.
Limiting the size and quantity of locally stored log files is the main goal of log rotation. Docker logs must be cycled at predetermined intervals because manually deleting logs is a laborious task. But the first question is, where are Docker logs stored?
Where are Docker logs stored?
Logs are often kept on the Docker host because containers are stateless (failing to remember or save data from previous actions). Docker uses the JSON-file logging driver by default, and it records all stdout and stderr output in JSON format.
Log files are created for each container and are generally stored at:
/var/lib/docker/containers/[container-id]/[container-id]-JSON.log
Docker provides support for multiple logging drivers. You can check out other logging drivers available.
To know your current logging driver for Docker Daemon, run the following command:
docker info --format '{{.LoggingDriver}}'
Why is Docker Log Rotation needed?
Docker log rotation is critical for several reasons:
Resource Management: Without log rotation, logs can consume significant disk space, especially in high-traffic environments, leading to system slowdown or failure.
Performance Optimization: Regularly rotating logs helps maintain optimal performance of Docker containers and the host system.
Security and Compliance: For security-sensitive applications, rotating logs can help manage sensitive data and comply with data retention policies.
Simplified Troubleshooting: It makes it easier to analyze recent logs without wading through old, irrelevant data.
Configuring Docker Log Rotation
Let’s first configure the Docker daemon to a particular log driver.
To configure the Docker Daemon to a particular log driver:
Step 1: Go to the Docker daemon configuration file location:
On Linux: /etc/docker/ directory
On Windows: C:\ProgramData\docker\config\daemon.json
Step 2: If your logging driver hasn’t been set yet, use the following command to set it up:
{
"log-driver": "local"
}
or
{
"log-driver": "json-file"
}
You can specify any log driver of your choice but Docker recommends using the local logging driver to prevent disk exhaustion because it performs log rotation by default.
Step 3: Add configuration for log rotation
Add the following code to the daemon.json file after editing or creating it to rotate the log.
The final file, when using json-file driver looks like this:
{
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "5"
}
}
You can also update the parameters for local logging driver using the same format:
{
"log-driver": "local",
"log-opts": {
"max-size": "10m",
"max-file": "5"
}
}
Step 4: Save the file and restart docker.
systemctl restart docker
The change in the default logging driver impacts only the containers created after modifying the daemon configuration. The existing containers carry the initial configuration of the logging driver. In order to update their logging driver for existing containers, they must be recreated with the preferred options.
Configuring Log Drivers and Rotation for specific containers
To configure a container to use a different logging driver than the Docker daemon’s default, use the --log-driver flag. You need to include the container ID or container name alongside the command.
docker run -it --log-driver json-file nginx
The --log-driver flag specifies the log driver you are assigning to that container. In the above example, the log driver assigned to the container is the json-file.
To find the current logging driver for a running container, if the daemon is using the json-file logging driver, run the following command:
docker inspect -f '{{.HostConfig.LogConfig.Type}}' <CONTAINER>
Then to setup log rotation for the specified container, run the below commands:
docker run \
--log-driver json-file \
--log-opt max-size=15m \
--log-opt max-file=5 \
nginx echo hello world
Another way to do this is by running the below command
docker run --log-opt max-size=15m --log-opt max-file=5 nginx:latest
Simplifying the above command:
docker run --log-opt max-size=15m
It tells Docker to limit the size of your Docker log file.
--log-opt max-file=5
It instructs Docker to store no more than 5 log files. When the limit is met, Docker is instructed to destroy the older files.
nginx:latest
The container image name
You can also have a look at the logging options the JSON-file accepts.
Final Thoughts
In this guide, we learned how to set up Docker log rotation. Container environments are highly dynamic, with multiple layers of abstraction. As a result, it's hard to debug such environments, and logs can play a critical role in providing much-needed visibility into performance issues.
Docker provides various terminal commands to interact with Docker logs. You can use your console to view logs while developing your application or debugging a specific scenario. But in production environments, it’s advisable to centralize your logs for storage and analysis. That’s where a log management solution comes into the picture.
The first step for efficient log management is log aggregation. Once the logs are aggregated, you need to store them and set them up for analysis. There are multiple log management solutions out there. But which solution is best suited for modern distributed cloud-native applications?
SigNoz, a full-stack open-source APM, can help you to store, manage, and analyze logs at scale. It correlates all your telemetry data(logs, metrics, and traces) into a single suite of monitoring. Apart from logs, SigNoz can also help you monitor metrics and traces.
SigNoz is built to support OpenTelemetry natively. OpenTelemetry, an open-source project backed by Cloud Native Computing Foundation, can be used to aggregate and process logs from multiple sources. OpenTelemetry is quietly becoming the world standard for instrumenting cloud-native applications to generate telemetry data.
Getting started with SigNoz
SigNoz cloud is the easiest way to run SigNoz. Sign up for a free account and get 30 days of unlimited access to all features.
You can also install and self-host SigNoz yourself since it is open-source. With 19,000+ GitHub stars, open-source SigNoz is loved by developers. Find the instructions to self-host SigNoz.
Further Reading
