We run docker images on AWS ECS. These ECS clusters are backed by EC2 machines. Each EC2 machine has a fluentbit daemon service running.
This is our current fluentbit config:
[SERVICE]
Flush 5
Daemon Off
Log_Level info
Parsers_File parsers.conf
[INPUT]
Name Forward
Port 24224
Buffer_Max_Size 256KB
[FILTER]
Name parser
Parser docker
Match *
Key_Name log
Reserve_Data On
Preserve_Key On
[OUTPUT]
Name stdout
Match *flog*
[OUTPUT]
Name es
Match *
Buffer_Size 128KB
Host ${ES_HOST}
Port ${ES_PORT}
Logstash_Format On
Logstash_Prefix logs-
Include_Tag_Key On
Tag_Key image
Type _doc
Since we have different services running on the same ec2 machine. Is there a way to change the config so that if the log is from one docker service using X index, otherwise use Y index?
As I typed it, I realised we need to use a second output and Match.
I'm hoping we can use but still trying to piece it together.