【云原生】日志收集方案（二）：基于ECK部署生产级ElasticSearch + Kibana + Logstash

介绍

之前文章写过基于ECK部署生产级ElasticSearch + Kibana + FileBeat，采用filebeat轻量级采集器，本文使用Logstash采集器。

ELK组件关系

官方在 2018 年后把“ELK”改名 Elastic Stack，成员扩充为：
Elastic Stack = Elasticsearch + Kibana + Logstash + Beats + 其他（如 Elastic Agent、Fleet、APM、Security 等）

早期的 ELK 只有 Elasticsearch、Logstash、Kibana，后来官方推出 Beats 家族作为轻量采集器，Filebeat 的前身叫 logstash-forwarder，由 Logstash 作者 Jordan Sissel 用 Go 写成，之后被 Elastic 团队重构并纳入 Beats；Logstash 与 Beats 是同级兄弟，而非 Beats 成员。

虽然 Logstash 因 JVM 吃资源被吐槽笨重，但它拥有 200+ 插件、强大 ETL 能力、持久化缓冲以及与 Kafka/Redis 的高可靠链路。

二者在现代架构中互补共存：边缘用 Beats 采集，必要时经 Logstash 深加工后再进 Elasticsearch。

产品支持列表

https://www.elastic.co/cn/support/matrix/#matrix_kubernetes

部署ECK crd

官方地址：https://www.elastic.co/docs/deploy-manage/deploy/cloud-on-k8s/install-using-yaml-manifest-quickstart

1	kubectl create -f https://download.elastic.co/downloads/eck/3.0.0/crds.yaml

1	kubectl apply -f https://download.elastic.co/downloads/eck/3.0.0/operator.yaml

部署elasticsearch

部署前应设置虚拟内存：https://www.elastic.co/docs/deploy-manage/deploy/cloud-on-k8s/virtual-memory

elasticsearch.yaml

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
  namespace: elk
spec:
  version: 9.0.3
  nodeSets:
  - name: elk
    count: 3
    #config:
    #  node.store.allow_mmap: false
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
        storageClassName: nfs-storage
    podTemplate:
      spec:
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
            runAsUser: 0
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']

部署kibana

kibana.yaml

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: elk
spec:
  version: 9.0.3
  count: 3
  elasticsearchRef:
    name: elasticsearch
  config:
    # 更多kibana设置：https://www.elastic.co/docs/reference/kibana/configuration-reference/general-settings
    elasticsearch.requestHeadersWhitelist:
    - authorization
  podTemplate:
    spec:
      containers:
      - name: kibana
        env:
        - name: I18N_LOCALE
          value: "zh-CN"
  http:
    service:
      spec:
        type: NodePort

部署Logstash

logstash.yaml

apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: logstash
  namespace: elk
spec:
  count: 3
  elasticsearchRefs:
    - name: elasticsearch
      clusterName: elasticsearch
  version: 9.0.3
  pipelines:
    - pipeline.id: main
      config.string: |
        input {
          beats {
            port => 5044
          }
        }
        output {
          elasticsearch {
            hosts => [ "${ELASTICSEARCH_ES_HOSTS}" ]
            user => "${ELASTICSEARCH_ES_USER}"
            password => "${ELASTICSEARCH_ES_PASSWORD}"
            ssl_certificate_authorities => "${ELASTICSEARCH_ES_SSL_CERTIFICATE_AUTHORITY}"
          }
        }
  services:
    - name: beats
      service:
        spec:
          type: NodePort
          ports:
            - port: 5044
              name: "filebeat"
              protocol: TCP
              targetPort: 5044

Elijah