Hey everyone,
Summary of the long post:
On universal forwarders, I need to add some kind of identifier like a tag or metadata value to all data before it is sent to distinguish the environment it is coming from, allow it to be searchable based on the value, and a heavy/intermediate forwarder using props/transforms to change and forward data based on this value.
I'm currently working on a large environment that will have multiple environment's universal forwarders reporting to my environment. The way we are setup:
- Have about 10 customers with their own environments
- Each environment will have roughly 10-50 servers in AWS
- Each server will have a universal forwarder installed to point to my splunk environment
- The universal forwarders will use data cloning to send data to my indexers and to an intermediate forwarder on the edge of my environment.
- All of the data will be indexed on my indexers and certain inputs that are sent to the intermediate forwarder will be sent to another environment for security monitoring.
- The intermediate forwarder has props and transforms setup to forward data to the external splunk environment based off of sourcetype, but now that we are adding multiple customer environments that want the security monitoring, and will use different indexes, the transforms need to be modified.
So here is my question,
Is there a way to tag data or add an identifier within the universal forwarders in an environment so the intermediate forwarder can forward to the external splunk environment to a specific index?
The intermediate is a heavy forwarder without local indexing and is the only connection that has routes to the external splunk environment. The reason for all of this and they way it's constructed, is the level of security requirements from our primary.
For example:
- Customer A has 20 servers with universal forwarder installed. Universal forwarders add an identifier to all data as it is sent that matches the customer's environment name like CustA.
- Customer B has 40 servers and much like Customer A, the forwarders add an identifier to all data, CustB.
- The inputs for both environments are configured to go to their respective indexes on my indexers; Customer A to customerA_data and Customer B to customerB_data. The data is then forwarded to both my indexers and the intermediate forwarder.
- The indexes customerA_data and customerB_data exist on my indexers and receive the data, but the external splunk security environment has custA_security and custA_application, and custB_security and custB application.
- The intermediate forwarder would use props and transforms. When it receives data with sourcetype=linux_audit and the identifier is CustA, it sends that data to the external environment's custA_security index and when receives sourcetype=nginx (or any application source) and the identifier is CustB, it sends that data to the custB_application index in the external environment.
- While this is all occurring on the intermediate, all data is being sent from the universal forwarders to my indexers with and being indexed and now be searchable using CustA or CustB.
Thanks in advance, it's a lot of information.
↧