How to Monitor Docker Containers With Grafana Loki and Promtail

In this article, we’ll walk through the process of monitoring Docker containers using Grafana's Loki data source. I’m sharing this because I faced several challenges when setting up container monitoring with Grafana on my server, and I thought it might help others. Let's get started!

What is Grafana Loki?

Grafana Loki is a log aggregation system designed to store and query logs in a cost-effective, user-friendly way. It is commonly used with Grafana to visualize logs and troubleshoot issues across containerized applications. To use the Grafana data source with Loki, we first need to install Grafana on our host. We’ll do this by pulling the Grafana Docker container from Docker Hub and then add Grafana Loki and Promtail by pulling images from a container registry.

Setting Up Loki and Promtail Configuration Files

Before pulling the containers, we need to create configuration files for Loki and Promtail on the host. These files contain the settings for how Loki and Promtail will run, where they will store data, and what they will monitor.

Step 1: Create the Loki Configuration File

The Loki configuration file specifies how Loki will store and manage logs. Here’s how to set it up:

1. Create a directory for Grafana to keep related files organized:

mkdir grafana

2. Open a new file for the Loki configuration:

   vi loki-local-config.yaml

3. Paste the following configuration code:

   auth_enabled: false

   server:

   http_listen_port: 3100

   grpc_listen_port: 9096

   common:

   instance_addr: 127.0.0.1

   path_prefix: /tmp/loki

   storage:

   filesystem:

   chunks_directory: /tmp/loki/chunks

   rules_directory: /tmp/loki/rules

   replication_factor: 1

   ring:

   kvstore:

   store: inmemory

   query_range:

   results_cache:

   cache:

   embedded_cache:

   enabled: true

   max_size_mb: 100

   schema_config:

   configs:

   - from: 2020-10-24

   store: tsdb

   object_store: filesystem

   schema: v13

   index:

   prefix: index_

   period: 24h

   ruler:

   alertmanager_url: http://localhost:9093

Explanation:

• auth_enabled: false disables authentication, simplifying access.

• server specifies the ports for HTTP and gRPC communication, enabling Loki to listen on port 3100 for queries and on 9096 for gRPC requests.

• storage specifies that logs will be stored in the local filesystem, defining directories for chunks and rules.

• schema_config defines how logs are indexed and stored in Loki over time.

• ruler provides settings for alerting, though you can modify it as needed.

4. Save and close the file.

Step 2: Create the Promtail Configuration File

Promtail acts as a log collector, scraping log data and sending it to Loki. Here’s how to set up its configuration file:

1. Open a new file for the Promtail configuration:

   vi promtail-config.yaml

2. Paste the following configuration code:

   server:

   http_listen_port: 9080

   grpc_listen_port: 0

   positions: filename: /tmp/positions.yaml

   clients:

   - url: http://loki:3100/loki/api/v1/push

   scrape_configs:

   - job_name: system

   static_configs:

   - targets:

   - localhost

   labels:

   job: varlogs

   path: /var/log/*log

   Explanation:

   • server defines the ports for HTTP access.

   • positions keeps track of how far Promtail has read in the log files, preventing it from re-sending logs on restart.

   • clients specifies where Promtail should send the logs; here, it’s configured to push logs to Loki on port 3100.

   • scrape_configs defines the targets and file paths for Promtail to monitor—in this case, it’s set to monitor /var/log for any log files.

3. Save and close the file.

Creating a Docker Compose File for Grafana, Loki, and Promtail

Now that we have the configuration files, we need to create a Docker Compose file to define and manage the Grafana, Loki, and Promtail services. This file will allow us to easily start, stop, and manage the services.

1. Open or create a docker-compose.yml file:

   vi docker-compose.yml

2. Paste the following configuration code:

   version: '3'

   services:

   grafana:

   image: grafana/grafana

   ports: - "3000:3000"

   volumes:

   - grafana-storage:/var/lib/grafana

   loki:

   image: grafana/loki

   ports: - "3100:3100"

   volumes:

   - ./loki-local-config.yaml:/etc/loki/local-config.yaml

   command: -config.file=/etc/loki/local-config.yaml

   promtail:

   image: grafana/promtail

   volumes:

   - /var/log:/var/log

   - ./promtail-config.yaml:/etc/promtail/config.yaml

   command: -config.file=/etc/promtail/config.yaml

   Explanation:

   • The grafana service pulls the Grafana image, exposing it on port 3000 for the web interface.

   • The loki service uses the loki-local-config.yaml file to configure its settings and makes it accessible on port 3100.

   • The promtail service mounts the /var/log directory to read system logs and uses the configuration file we created earlier to define log collection and push settings.

3. Save and close the file.

Configuring Docker to Send Logs to Loki

We now need to set up Docker to use the Loki logging driver, enabling it to send container logs directly to Loki.

1. Open the Docker daemon configuration file:

   vi /etc/docker/daemon.json

2. Add the following configuration code:

   { "debug": true,

   "log-driver": "loki",

   "log-opts": {

   "loki-url": "http://localhost:3100/loki/api/v1/push",

   "loki-batch-size": "400"

   }

   }

   Explanation:

   • "log-driver": "loki" configures Docker to use Loki as the log driver.

   • "loki-url" specifies the URL where Loki will receive the log data.

   • "loki-batch-size" controls the size of log batches sent to Loki, allowing for better performance by reducing the frequency of network requests.

3. Save and close the file.

Installing the Docker Driver for Loki

Now, let’s install the Docker driver to send logs to Loki. This step enables Docker to directly forward logs from all containers to Loki.

docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

After installation, restart Docker to apply the changes:

systemctl restart docker

Once the services are up, open your browser and navigate to your server's IP address with port 3000 (e.g., http://<your-ip>:3000). The default username and password for Grafana is admin.

Adding Loki as a Data Source in Grafana

With Grafana running, let’s add Loki as a data source so we can view logs from our containers.

1. Go to Connections in Grafana.

2. Click on Add Data Source.

3. Search for Loki, click on it, and enter the following URL:

   http://loki:3100

4. Save and set up the data source

   Finally, go to Explore in Grafana, select Loki, and start viewing your logs.

In this article, we configured Loki and Promtail to monitor Docker container logs. Each step contributes to capturing and visualizing logs in Grafana:

• Loki serves as the central log storage and retrieval system.

• Promtail collects logs from the host and pushes them to Loki.

• Docker Configuration allows all container logs to be directly sent to Loki.

• Grafana Integration provides a user-friendly dashboard for visualizing and exploring logs.

This setup will help you monitor container logs in real-time, enabling effective troubleshooting and system oversight.