I am new to Splunk and so far I find that the real difficulty is not learning Splunk itself but understanding my organisation's data and the way they set up Splunk. I wish Splunk would make this easier, but it doesn't. Allow me to use an example to explain.
So I see the "New Attacks - Last 30 Days" table in the "Intrusion Center" dashboard in Splunk Enterprise Security. I want to see where it gets its data from, so I click on "Open in Search" in the bottom left corner.
First hurdle: the search in question starts with a macro. So I need to do "Settings" > "Advanced Search" > "Search macros", copy-paste the macro name to see what the macro does. I have to do that every single time I meet a macro, which is painful.
Second hurdle: the macro is actually simply inputting a lookup. So I do "Settings" > "Lookups" > "Lookup definitions", search for the lookup name, find out what lookup file it uses, search for it under "Settings" > "Lookups" > "Lookup table files" and... find there's no way of knowing where this lookup comes from.
The "App" field does mention "SA-NetworkProtection" and after searching in the "Content Management" of ES, I did find a saved search called "Network - IDS Attack Tracker - Lookup Gen". I don't have permissions to see it (yet), but that's another issue.
This is just an example but my point is: Splunk is like an onion - when trying to understand where things come from, one painstakingly goes through lots of layers, whether these are macros, lookups, data models, etc... and that is frustrating. I understand that complexity is partially unavoidable because of the flexibility of Splunk and the need for abstracting and normalising the data. I just wonder if there is any tips or tricks people have found useful to cut through all these layers and gain insight faster?
One such trick I'm considering using, when I have my admin account, is to log on the linux box and grep through the configuration files...
↧