Monitoring Nginx Logs with ClickStack

TL;DR

This guide shows you how to monitor nginx with ClickStack by configuring the OpenTelemetry collector to ingest nginx access logs. You'll learn how to:

• Configure nginx to output JSON-formatted logs
• Create a custom OTel collector configuration for log ingestion
• Deploy ClickStack with your custom configuration
• Use a pre-built dashboard to visualize nginx metrics (requests, errors, latency)

A demo dataset with 10,000 sample logs is provided to test the integration before connecting your production nginx instances.

Time Required: 5-10 minutes.

Prerequisites

• ClickStack instance running
• Existing nginx installation
• Access to modify nginx configuration files

Integration with existing nginx

This section covers configuring your existing nginx installation to send logs to ClickStack by modifying the ClickStack OTel collector configuration.

Configure nginx log format

First, configure nginx to output logs in JSON format for easier parsing. Add this log format definition to your nginx.conf:

The nginx.conf file is typically located at:

• Linux (apt/yum): /etc/nginx/nginx.conf
• macOS (Homebrew): /usr/local/etc/nginx/nginx.conf or /opt/homebrew/etc/nginx/nginx.conf
• Docker: Configuration is usually mounted as a volume

Add this log format definition to the http block:

http {
    log_format json_combined escape=json
    '{'
      '"time_local":"$time_local",'
      '"remote_addr":"$remote_addr",'
      '"request_method":"$request_method",'
      '"request_uri":"$request_uri",'
      '"status":$status,'
      '"body_bytes_sent":$body_bytes_sent,'
      '"request_time":$request_time,'
      '"upstream_response_time":"$upstream_response_time",'
      '"http_referer":"$http_referer",'
      '"http_user_agent":"$http_user_agent"'
    '}';

    access_log /var/log/nginx/access.log json_combined;
    error_log /var/log/nginx/error.log warn;
}

After making this change, reload nginx.

Create custom otel collector configuration

ClickStack allows you to extend the base OpenTelemetry Collector configuration by mounting a custom configuration file and setting an environment variable. The custom configuration is merged with the base configuration managed by HyperDX via OpAMP.

Create a file named nginx-monitoring.yaml with the following configuration:

receivers:
  filelog:
    include:
      - /var/log/nginx/access.log
      - /var/log/nginx/error.log
    start_at: end 
    operators:
      - type: json_parser
        parse_from: body
        parse_to: attributes
      - type: time_parser
        parse_from: attributes.time_local
        layout: '%d/%b/%Y:%H:%M:%S %z'
      - type: add
        field: attributes.source
        value: "nginx"

service:
  pipelines:
    logs/nginx:
      receivers: [filelog]
      processors:
        - memory_limiter
        - transform
        - batch
      exporters:
        - clickhouse

This configuration:

• Reads nginx logs from their standard locations
• Parses JSON log entries
• Extracts and preserves the original log timestamps
• Adds source: nginx attribute for filtering in HyperDX
• Routes logs to the ClickHouse exporter via a dedicated pipeline

Note

• You only define new receivers and pipelines in the custom config
• The processors (memory_limiter, transform, batch) and exporters (clickhouse) are already defined in the base ClickStack configuration - you just reference them by name
• The time_parser operator extracts timestamps from nginx's time_local field to preserve original log timing
• The pipelines route data from your receivers to the ClickHouse exporter via the existing processors

Configure ClickStack to load custom configuration

To enable custom collector configuration in your existing ClickStack deployment, you must:

1. Mount the custom config file at /etc/otelcol-contrib/custom.config.yaml
2. Set the environment variable CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml
3. Mount your nginx log directories so the collector can read them

Update your ClickStack deployment configuration to include these settings, this example uses docker compose.

services:
  clickstack:
    # ... existing configuration ...
    environment:
      - CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml
      # ... other environment variables ...
    volumes:
      - ./nginx-monitoring.yaml:/etc/otelcol-contrib/custom.config.yaml:ro
      - /var/log/nginx:/var/log/nginx:ro
      # ... other volumes ...

Note

Ensure the ClickStack collector has appropriate permissions to read the nginx log files. In production, use read-only mounts (:ro) and follow the principle of least privilege.

Verifying Logs in ClickStack

Once configured, log into HyperDX and verify logs are flowing:

1. Navigate to the Logs view
2. Verify you see JSON-parsed log entries with fields like request, request_time, upstream_response_time, etc.

Demo dataset

For users who want to test the nginx integration before configuring their production systems, we provide a sample dataset of pre-generated nginx access logs with realistic traffic patterns.

Download the sample dataset

Download the sample log file and update timestamps to the current time:

# Download the logs
curl -O https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/access.log

# Update timestamps to current time while preserving traffic patterns
python3 << 'EOF'
import json
from datetime import datetime, timedelta

# Read all log lines
with open('access.log', 'r') as f:
    logs = [json.loads(line) for line in f]

# Parse timestamps and find the newest
def parse_time(time_str):
    return datetime.strptime(time_str, "%d/%b/%Y:%H:%M:%S %z")

original_times = [parse_time(log['time_local']) for log in logs]
newest_time = max(original_times)
now = datetime.now(newest_time.tzinfo)
time_shift = now - newest_time

# Update all timestamps
for log in logs:
    original_time = parse_time(log['time_local'])
    new_time = original_time + time_shift
    log['time_local'] = new_time.strftime("%d/%b/%Y:%H:%M:%S %z")

# Write back as newline-delimited JSON
with open('access.log', 'w') as f:
    for log in logs:
        f.write(json.dumps(log) + '\n')

print('✅ Log timestamps updated to current time')
EOF

The dataset includes:

• 10,000 log entries with realistic traffic patterns
• Various endpoints and HTTP methods
• Mix of successful requests and errors
• Realistic response times and byte counts
• Timestamps now distributed over recent time

Create test collector configuration

Create a file named nginx-demo.yaml with the following configuration:

receivers:
  filelog:
    include:
      - /tmp/nginx-demo/access.log
    start_at: beginning  # Read from beginning for demo data
    operators:
      - type: json_parser
        parse_from: body
        parse_to: attributes
      - type: time_parser
        parse_from: attributes.time_local
        layout: '%d/%b/%Y:%H:%M:%S %z'
      - type: add
        field: attributes.source
        value: "nginx-demo"

service:
  pipelines:
    logs/nginx-demo:
      receivers: [filelog]
      processors:
        - memory_limiter
        - transform
        - batch
      exporters:
        - clickhouse

Run ClickStack with demo configuration

Run ClickStack with the demo logs and configuration:

docker run --name clickstack-demo \
  -p 8080:8080 -p 4317:4317 -p 4318:4318 \
  -e CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml \
  -v "$(pwd)/nginx-demo.yaml:/etc/otelcol-contrib/custom.config.yaml:ro" \
  -v "$(pwd)/access.log:/tmp/nginx-demo/access.log:ro" \
  docker.hyperdx.io/hyperdx/hyperdx-all-in-one:latest

Verify logs in HyperDX

Once ClickStack is running:

1. Open HyperDX at http://localhost:8080
2. Navigate to the Logs view
3. Set time range to "Last 1 Day"
4. You should see ~10,000 log entries

Dashboards and visualization

To help you get started monitoring nginx with ClickStack, we provide essential visualizations for nginx logs.

Download the dashboard configuration.

Import Pre-built Dashboard

1. Open HyperDX and navigate to the Dashboards section.
2. Click "Import Dashboard" in the upper right corner under the ellipses.

3. Upload the nginx-logs-dashboard.json file and click finish import.

The dashboard will be created with all visualizations pre-configured.

Troubleshooting

Custom config not loading

• Verify the environment variable CUSTOM_OTELCOL_CONFIG_FILE is set correctly

docker exec <container-name> printenv CUSTOM_OTELCOL_CONFIG_FILE

• Check that the custom config file is mounted at /etc/otelcol-contrib/custom.config.yaml

docker exec <container-name> ls -lh /etc/otelcol-contrib/custom.config.yaml

• View the custom config content to verify it's readable

docker exec <container-name> cat /etc/otelcol-contrib/custom.config.yaml

No logs appearing in HyperDX

• Ensure nginx is writing JSON logs

tail -f /var/log/nginx/access.log

• Check the collector can read the logs

docker exec `<container>` cat /var/log/nginx/access.log

• Verify the effective config includes your filelog receiver

docker exec `<container>` cat /etc/otel/supervisor-data/effective.yaml | grep filelog

• Check for errors in the collector logs

docker exec `<container>` cat /etc/otel/supervisor-data/agent.log

Next Steps

If you want to explore further, here are some next steps to experiment with your dashboard

• Set up alerts for critical metrics (error rates, latency thresholds)
• Create additional dashboards for specific use cases (API monitoring, security events)