We are pulling data like Red hat logs, Apigee, Ansible etc. from AWS through fluentd plugin which is forwarding data to Heavy Forwarder in AWS, and then from that HF to another HF in a DMZ to another HF outside of DMZ.
The data is passing through and getting indexed so the firewall rules and ports are established properly.
However, when trying to transform the data so that we can split it into numerous sourcetypes it will not work.
It’s still applies the original sourcetype applied from fluentd plugin.
In fluentd plugin we are defining index name , sourcetype and the default format is JSON, when we are trying to override this index and sourcetype at the destination for differentiating types of data with different sourcetypes by defining inputs.conf, props.conf, transforms.conf.
It is not applying the values what we define here at the destination, it is only taking the values what the source is defining in the fluentd plugin config file.
So the question is can we add a props and transforms config in fluentd plugin in AWS to differentiate the logs with sourcetypes.
Can anyone suggest a possible solution for this kind of problem.
FLuentd plugin is ----k24d/fluent-plugin-splunkapi
We are using splunk 6.2.2 in all Indexers, Forwarders etc
Here are the configs that we defined at the destination.
Please help us.
Inputs.conf
[splunktcp://1600]
connection_host = ip
sourcetype = journald
index = aws_fluentd_index
Props config
[source::poc.aws.system.journald]
KV_MODE = json
TIME_PREFIX=^
TIME_FORMAT=%Y-%m-%d %T %z
SHOULD_LINEMERGE=false
MAX_TIMESTAMP_LOOKAHEAD=30
NO_BINARY_CHECK = 1
pulldown_type = 1
[source::poc.aws.system.journald]
TRANSFORMS-override=override_ST_journald,override_IDX_journald
Transforms config
[override_ST_journald]
SOURCE_KEY=_raw
REGEX=.*
FORMAT = sourcetype::journald
DEST_KEY = MetaData:Sourcetype
[override_IDX_journald]
SOURCE_KEY=_raw
REGEX=.*
FORMAT = aws_fluentd_index
DEST_KEY = _MetaData:Index
↧