Как правильно передать логи через Fluentd в Elasticsearch 8.X.X

Я пытаюсь передавать логи K8s в Elasticsearch 8.2.0 используя Fluentd. Его конфиг передаётся через configmap.yaml и выглядит так:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    stack: app
  name: fluentd-config
  namespace: kube-system
data:
fluent.conf: |-
      @include kubernetes.conf
      @include elasticsearch.conf
      @include pods.conf
kubernetes.conf |-
      <filter kubernetes.**>
        @type kubernetes_metadata
      </filter>
      <match fluent.**>
        @type null
      </match>
      <match kubernetes.var.log.containers.**kube-logging**.log>
        @type null
      </match>
      <match kubernetes.var.log.containers.**kube-system**.log>
        @type null
      </match>
      <match kubernetes.var.log.containers.**monitoring**.log>
        @type null
      </match>
      <match kubernetes.var.log.containers.**infra**.log>
        @type null
      </match>
      <match kubernetes.var.log.containers.**fluentd**.log>
        @type null
      </match>
      <match ignore.**>
        @type null
      </match>
     <filter kubernetes.**>
        @id filter_concat
        @type concat
        key log
        multiline_end_regexp /\n$/
        separator ""
      </filter>
      <filter kubernetes.**>
        @type parser
        key_name log
        reserve_data true
        remove_key_name_field true
        <parse>
          @type multi_format
          <pattern>
            format json
          </pattern>
          <pattern>
            format none
          </pattern>
        </parse>
       </filter>
elasticsearch.conf: |-
      <match **>
        @type elasticsearch
        host 10.11.111.160
        port 9100
        scheme http
        ssl_verify false
        logstash_format false
        include_timestamp true
        flush_interval 5s
        index_name fluentd-main
        <buffer>
          @type "file"
          path "/var/log/fluentd/buffers/elasticsearch"
          flush_interval 10s
          retry_max_interval 30
          chunk_limit_size 2M
          queue_limit_length 8
          flush_thread_count 2
       </buffer>
      </match>
...

Настройки fluentd и плагины установлены под 8.X.X версию Elasticsearch:

elastic-transport (8.4.1)
elasticsearch (8.19.2)
elasticsearch-api (8.19.2)
fluent-plugin-elasticsearch (5.4.4)

логи k8s собираются правильно и как мне нужно, но в elasticsearch они не передаются. Телнет до elasticsearch есть. Могу вручную создавать индексы командой curl -X PUT "10.11..." В логах fluentd я вижу сообщения:

Но при старте fluentd я получаю ошибку:
[debug]: #0 'host 10.11.111.160' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: 10.11.111.160' doesn't have tag placeholder
[debug]: #0 'index_name fluentd-main' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: fluentd-main' doesn't have tag placeholder
[debug]: #0 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
[debug]: #0 'logstash_prefix kubernetes' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: kubernetes' doesn't have tag placeholder
[debug]: #0 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
[debug]: #0 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
[debug]: #0 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
[debug]: #0 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
[debug]: #0 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
[debug]: #0 Need substitution: false
[debug]: #0 'host_placeholder 10.11.111.160' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: 10.11.111.160' doesn't have tag placeholder
[debug]: #0 Detected ES 8.x: This parameter has no effect

Почему я получаю эти ошибки? Как это исправить можно?


Ответы (0 шт):