Fluentd не может обработать файл лога
Пробую настроить с нуля Fluentd в кластере Kubernetes для стака FLG. Мне нужно захватывать файлы логов по шаблону имени, обрабатывать их содержимое и пересылать полученные записи в Loki. Однако при обработке файла логов я получаю ошибку:
2024-05-07 16:22:58 +0300 [warn]: #0 got unrecoverable error in primary and no secondary error_class=NoMethodError error="undefined method `each' for nil:NilClass"
2024-05-07 16:22:58 +0300 [warn]: #0 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-loki-0.3.0/lib/fluent/plugin/out_loki.rb:184:in `format_labels'
2024-05-07 16:22:58 +0300 [warn]: #0 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-loki-0.3.0/lib/fluent/plugin/out_loki.rb:191:in `handle_record'
2024-05-07 16:22:58 +0300 [warn]: #0 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-loki-0.3.0/lib/fluent/plugin/out_loki.rb:223:in `block in write'
2024-05-07 16:22:58 +0300 [warn]: #0 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/event.rb:319:in `each'
2024-05-07 16:22:58 +0300 [warn]: #0 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/event.rb:319:in `block in each'
2024-05-07 16:22:58 +0300 [warn]: #0 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/plugin/buffer/memory_chunk.rb:81:in `open'
2024-05-07 16:22:58 +0300 [warn]: #0 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/plugin/buffer/memory_chunk.rb:81:in `open'
2024-05-07 16:22:58 +0300 [warn]: #0 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/event.rb:318:in `each'
2024-05-07 16:22:58 +0300 [warn]: #0 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-loki-0.3.0/lib/fluent/plugin/out_loki.rb:222:in `write'
2024-05-07 16:22:58 +0300 [warn]: #0 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/plugin/output.rb:1225:in `try_flush'
2024-05-07 16:22:58 +0300 [warn]: #0 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/plugin/output.rb:1538:in `flush_thread_run'
2024-05-07 16:22:58 +0300 [warn]: #0 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/plugin/output.rb:510:in `block (2 levels) in start'
2024-05-07 16:22:58 +0300 [warn]: #0 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2024-05-07 16:22:58 +0300 [warn]: #0 bad chunk is moved to /tmp/fluent/backup/worker0/object_80c/617dd150cde458ebebfae8b4f4b0af7f.log
Мой конфиг для Fluentd:
apiVersion: v1
kind: ConfigMap
metadata:
name: fluentd-logging
namespace: monitoring
labels:
app: fluentd-logging
data:
fluent.conf: |
<system>
log_level warn
</system>
<source>
@type tail
path /tmp/logs/srv-kube-master-01/commands.log*
pos_file /var/log/fluentd/tmp/access.log.pos
tag kubernetes.master01.system.commands
read_from_head true
<parse>
@type regexp
expression /(?<time>[a-zA-Z]+\s{1,2}\d{1,2}\s\d{2}:\d{2}:\d{2})\s(?<host>[\w-]+)\s(?<user>[\w]+):\s{2}\[(?<pid>\d+)\]\s(?<command>.*)/
time_format %b %d %H:%M:%S
</parse>
</source>
<match kubernetes.master01.system.commands>
@type loki
endpoint_url "http://loki:3100/loki/api/v1/push"
# label_keys srv-kube-master-01
# buffer_type memory
# buffer_chunk_limit 1m
<buffer>
flush_interval 10s
flush_at_shutdown true
</buffer>
# label {env: "dev"}
# <line_format>
# key_name "command"
# </line_format>
</match>
Содержимое обрабатываемого файла логов:
Apr 8 15:28:44 srv-kube-master-01 root: [1396657] exit
Apr 8 15:28:47 srv-kube-master-01 root: [1396657] export
Apr 8 15:29:04 srv-kube-master-01 root: [1396657] cat /etc/bash.bashrc
Apr 8 15:29:26 srv-kube-master-01 root: [1396657] ll
Apr 8 15:29:33 srv-kube-master-01 root: [1396657] cat commands.log
Apr 9 10:29:53 srv-kube-master-01 kubemaster: [717392] exit
Apr 9 10:31:54 srv-kube-master-01 kubemaster: [717392] sudo crontab -e
Я использую образ fluent/fluentd-kubernetes-daemonset:v1.16.5-debian-forward-1.0
и grafana/loki:latest
.
Что мне нужно исправить для того, чтобы обработка логов работала корректно?