Nginx访问日志分析与实时监测配置指南

一、Nginx日志基础配置

1. 日志格式配置

在nginx.conf中配置自定义日志格式：

http {
    log_format main '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent" '
                    '$request_time $upstream_response_time';

    access_log /var/log/nginx/access.log main;
}

2. 日志分割配置

使用logrotate实现日志分割：

# /etc/logrotate.d/nginx
/var/log/nginx/*.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 `cat /var/run/nginx.pid`
    endscript
}

二、实时日志分析工具

1. GoAccess (实时终端分析)

安装与使用：

# 安装
apt-get install goaccess  # Debian/Ubuntu
yum install goaccess      # CentOS/RHEL

# 实时分析
goaccess /var/log/nginx/access.log -a -o /var/www/html/report.html --real-time-html

2. ELK Stack (完整日志分析方案)

2.1 Filebeat配置

# /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  fields:
    type: nginx-access

output.elasticsearch:
  hosts: ["localhost:9200"]

2.2 Logstash配置 (可选)

# /etc/logstash/conf.d/nginx.conf
input {
  beats {
    port => 5044
  }
}

filter {
  if [fields][type] == "nginx-access" {
    grok {
      match => { "message" => "%{IPORHOST:remote_ip} - %{DATA:user_name} \[%{HTTPDATE:access_time}\] \"%{WORD:http_method} %{DATA:url} HTTP/%{NUMBER:http_version}\" %{NUMBER:response_code} %{NUMBER:body_sent_bytes} \"%{DATA:referrer}\" \"%{DATA:user_agent}\"
      }
    }
    date {
      match => [ "access_time", "dd/MMM/YYYY:HH:mm:ss Z" ]
      target => "@timestamp"
    }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "nginx-access-%{+YYYY.MM.dd}"
  }
}

三、实时监控与告警

1. Prometheus + Grafana方案

1.1 Nginx Prometheus exporter

docker run -d -p 9113:9113 nginx/nginx-prometheus-exporter -nginx.scrape-uri=http://nginx-status

在Nginx中启用stub_status：

server {
    listen 8080;
    server_name localhost;
    location /nginx-status {
        stub_status on;
        access_log off;
        allow 127.0.0.1;
        deny all;
    }
}

1.2 Grafana仪表板

导入Nginx仪表板 (ID: 12708)

2. 简单实时监控脚本

#!/bin/bash
# 实时监控HTTP状态码
tail -f /var/log/nginx/access.log | awk '
{
    status[$9]++
    total++
}
END {
    print "HTTP Status Code Distribution:"
    for (code in status) {
        printf "%s: %d (%.2f%%)\n", code, status[code], (status[code]/total)*100
    }
}'

四、高级分析配置

1. GeoIP分析

http {
    geoip_country /usr/share/GeoIP/GeoIP.dat;
    geoip_city /usr/share/GeoIP/GeoIPCity.dat;

    log_format main_geo '$remote_addr - $remote_user [$time_local] '
                       '"$request" $status $body_bytes_sent '
                       '"$http_referer" "$http_user_agent" '
                       '$geoip_country_name $geoip_city '
                       '$request_time $upstream_response_time';
}

2. 实时异常检测

使用Fluentd + Elasticsearch + Alertmanager：

# fluentd配置
<source>
  @type tail
  path /var/log/nginx/access.log
  pos_file /var/log/fluentd/nginx-access.log.pos
  tag nginx.access
  format /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" "(?<forwarder>[^\"]*)")?$/
  time_format %d/%b/%Y:%H:%M:%S %z
</source>

<match nginx.access>
  @type elasticsearch
  host localhost
  port 9200
  logstash_format true
  logstash_prefix nginx-access
</match>

<match nginx.access>
  @type prometheus
  <metric>
    name nginx_http_requests_total
    type counter
    desc The total number of HTTP requests
    <labels>
      method ${method}
      status ${code}
      path ${path}
    </labels>
  </metric>
</match>

五、安全监控配置

1. 实时拦截恶意请求

map $status $is_attack {
    ~^4     1;
    ~^5     1;
    default 0;
}

server {
    # ...其他配置...

    if ($is_attack) {
        set $block_attack 1;
    }

    if ($http_user_agent ~* (nmap|nikto|wget|curl|libwww-perl)) {
        set $block_attack 1;
    }

    if ($block_attack = 1) {
        access_log /var/log/nginx/security.log;
        return 444;
    }
}

2. Fail2Ban集成

# /etc/fail2ban/jail.d/nginx.conf
[nginx-http-auth]
enabled = true
filter = nginx-http-auth
port = http,https
logpath = /var/log/nginx/error.log

[nginx-badbots]
enabled = true
port = http,https
filter = nginx-badbots
logpath = /var/log/nginx/access.log
maxretry = 2

通过以上配置，您可以构建一个完整的Nginx访问日志分析和实时监测系统，从基础日志收集到高级安全监控，全方位掌握网站访问情况。