前言

ELK日志管理系统是Elasticsearch + Logstash + Kibana的简称,加上Filebeat组成一整套日志收集,过滤,处理,数据存储和前端展示系统.博主经过一段时间的安装,配置,测试也基本满足了个人以及中小企业团队对于日志管理的需求.在研究的这段时间里参考了网上各种教程,整合了相对详细的部署和配置教程分享给大家.

本教程使用docker-compose整合部署,参考 deviantony/docker-elk 7.13.2的TLS加密版.

项目流程

服务端:部署ELK即 Elasticsearch + Logstash + Kibana 到日志服务器.
客户端:部署Filebeat到需要收集日志的服务器.

常规流程

客户端部署Filebeat监控日志,发送至日志服务器ELK中的logstash做进一步的过滤和整理,在发送至数据库elasticsearch存储,通过kibana前端展示.

本文流程

由于个人使用,并没有过多的日志过滤和整理需求,加上logstash本身非常耗费服务器资源.所以本文教程将不使用logstash功能,直接在客户端部署Filebeat监控日志,发送至日志服务器ELK中的elasticsearch存储,通过kibana前端展示.

部署服务端 ELK

部署 TLS 版 ELK

拉取代码

git clone -b tls https://github.com/deviantony/docker-elk elk
# 拉取代码

reset 到指定版本

git reset c03a1d06e2e1fc2c62b399059bf4762bc7661db4 --hard
# reset 到 7.13.2 with tls 版本

修改配置

修改docker-compose.yml中顶级卷配置到指定目录用于存储数据,并创建data目录.

volumes:
  elasticsearch:
    driver: local
    driver_opts:
      type: 'none'
      o: 'bind'
      device: '/root/elk/data'

在elk根目录创建data目录

cd elk
mkdir data

启动 elk 并重置密码

docker-compose up -d
# 启动 elk
docker-compose exec -T elasticsearch bin/elasticsearch-setup-passwords auto --batch -u https://localhost:9200
# 重置并保存好密码
docker-compose down
# 停止 elk

批量修改密码

推荐使用vscode等软件搜索整个elk目录,将changeme修改为上文生成elastic的新密码.

创建证书

docker run -it \
  -v ${PWD}:/usr/share/elasticsearch/tls \
  docker.elastic.co/elasticsearch/elasticsearch:7.13.2 \
  bin/elasticsearch-certutil cert \
    --days 3650 \
    --keep-ca-key \
    --in tls/instances.yml \
    --out tls/certificate-bundle.zip

解压缩,删除压缩文件,选择 A 覆盖.

unzip certificate-bundle.zip
# 选择 A 覆盖
rm -rf certificate-bundle.zip
# 删除 zip

再次参考以下提示,执行命令并选择.

docker run -it \
  -v ${PWD}:/usr/share/elasticsearch/tls \
  docker.elastic.co/elasticsearch/elasticsearch:7.13.2 \
  bin/elasticsearch-certutil http

参考选择

Generate a CSR? [y/N] n
Use an existing CA? [y/N] y
CA Path: /usr/share/elasticsearch/tls/ca/ca.p12
Password for ca.p12: <none>
For how long should your certificate be valid? [5y] 10y
Generate a certificate per node? [y/N] n
(Enter all the hostnames that you need, one per line.)
elasticsearch
localhost
Is this correct [Y/n] y
(Enter all the IP addresses that you need, one per line.)
<none>
Is this correct [Y/n] y
Do you wish to change any of these options? [y/N] n
Provide a password for the "http.p12" file: <none>
What filename should be used for the output zip file? tls/elasticsearch-ssl-http.zip

解压缩,删除压缩文件,选择 A 覆盖.

unzip elasticsearch-ssl-http.zip
# 选择 A 覆盖
rm -rf elasticsearch-ssl-http.zip
# 删除 zip

创建 logstash ca 证书

openssl pkcs12 -clcerts -nokeys -in ca/ca.p12 -out ca/ca.pem

创建 logstash 证书

docker run -it --rm \
  -v ${PWD}:/usr/share/elasticsearch/tls \
  docker.elastic.co/elasticsearch/elasticsearch:7.13.2 \
    bin/elasticsearch-certutil cert \
    --ca tls/ca/ca.p12 \
    --name logstash \
    --dns logstash,yourdomain.com \
    --pem \
    --out tls/logstash.zip

解压缩,删除压缩文件,生成 p8 格式及配置权限.

unzip logstash.zip
rm -rf logstash.zip
openssl pkcs8 -in logstash/logstash.key -topk8 -nocrypt -out logstash/logstash.pkcs8.key
chmod 755 logstash/*

修改配置

修改./elasticsearch/config/elasticsearch.yml文件:
将xpack.license.self_generated.type: trial
修改为xpack.license.self_generated.type: basic

修改docker-compose.yml中elasticsearch和logstash的内存配置:
ES_JAVA_OPTS: "-Xmx2g -Xms2g"
LS_JAVA_OPTS: "-Xmx1g -Xms1g"

启动 elk

防火墙开放9200及5601端口并启动 elk

docker-compose up -d

使用http://IP:5601可访问kibana前端,使用用户名elastic和上文重置的PASSWORD elastic密码登陆.同时推荐使用nginx配置域名反向代理和HTTPS支持.

客户端 Filebeat

服务端依旧推荐使用docker compose来部署filebeat.将上文生成的证书logstash.crt上传至此,并参考下文创建docker-compose.yml和filebeat.yml.

docker-compose.yml

自行修改挂载目录,将需要发送的日志目录挂载至容器内.

version: '3.2'
services:
  filebeat:
    image: docker.elastic.co/beats/filebeat:7.13.2
    container_name: filebeat
    user: root
    volumes:
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml
      - ./logstash.crt:/logstash.crt
      # 挂载本地日志路径
      - /var/log/nginx:/logs/nginx
      - /var/lib/docker/containers:/var/lib/docker/containers
    environment:
      TZ: Asia/Shanghai
    restart: always

filebeat.yml 配置示例

注意修改ELK服务器地址,端口,密码等信息.

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false


# 定义本服务器位置字段
fields:
  location: hk


processors:
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
# 根据 docker 日志传递的 docker.attrs.tag 容器名,筛选并丢弃无需的容器日志.
  - drop_event:
      when:
        contains:
          docker.attrs.tag: filebeat


filebeat.inputs:
# 注意 nginx 或其他日志需要在 docker-compose.yml 挂载日志目录到容器内.

  - type: log
    enabled: true
    paths: /logs/nginx/*.log
    fields:
      source: nginx

  - type: docker
    containers:
      path: "/var/lib/docker/containers"
      json.keys_under_root: true
      json.add_error_key: true
      json.message_key: log
      tail_files: true
      multiline.pattern: '^([0-9]{4}|[0-9]{2})-[0-9]{2}'
      multiline.negate: true
      multiline.match: after
      multiline.timeout: 10s
      ids:
        - "*"
    fields:
      source: docker


setup.template.name: "IOIOX"
setup.template.pattern: "IOIOX-*"
setup.template.overwrite: true
setup.template.enabled: true


output.elasticsearch:
  hosts: ['https://elasticsearch:9200']
  username: elastic
  password: xxxxxxxxxxxxxx
  ssl.certificate_authorities: ["/logstash.crt"]
  ssl.verification_mode: none
  indices:
    - index: "IOIOX-%{[fields.location]}-%{[fields.source]}-%{+yyyy.MM.dd}"

容器日志监控

配置docker的全局日志,编辑/etc/docker/daemon.json,其中tag标签将传递docker.attrs.tag字段到filebeat.更多tag参考 官方文档 .

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "200k",
    "max-file": "100",
    "labels": "production_status",
    "env": "os,customer",
    "tag": "{{.Name}}"
  }
}

执行以下命令生效,并且需将现有容器完全停止,删除,再次启动才能将日志按照配置传入ELK.

systemctl daemon-reload
systemctl restart docker

结语

ELK的安装部署教程已经分享给大家,欢迎大家留言交流.同时博主也可以提供部署服务,有需求的可以联系我.