Hello, I am looking for a way to parse the JSON data that exists in the "Message" body of a set of Windows Events. Ideally I would like it such that my team only has to put in search terms for the sourcetype and the fields will be extracted and formatted appropriately. However, I would settle for simply creating a bunch of saved searches\reports and instructing my team to use those.
Here is an example record:
09/19/2017 11:42:20 AM
LogName=PowerShell-Endpoint-IMS-APISession
SourceName=PowerShell-Endpoint-IMS-APISession-Source
EventCode=1000
EventType=4
Type=Information
ComputerName=SOME_MACHINE.some.domain.tld
TaskCategory=None
OpCode=Info
RecordNumber=2275
Keywords=Classic
Message={
"Message": "User, jdoe, is already Lync-enabled.",
"CorrelationId": "38d97480-08a0-4e81-971c-8ab3f68747bc",
"SessionInfo": {
"SessionConfigurationName": "IMS-APISession",
"SessionConnectionString": "http://some_machine:5985/wsman?PSVersion=5.1.14393.1715",
"RunspaceID": "044d7c40-1de2-4c20-ad74-3745c3d99ac3",
"ProcessID": 2412,
"ClientIP": "169.68.128.128",
"SessionUser": "DOMAIN\\sessionuser",
"RunAsUser": "DOMAIN\\runasuser"
},
"CmdInvocationInfo": {
"InvocationName": "Enable-CCILyncUser",
"BoundParameters": {
"Username": "jdoe"
},
"UnboundArguments": [
],
"ScriptLineNumber": 0,
"OffsetInLine": 0,
"HistoryId": 5,
"ScriptName": "",
"Line": "",
"PositionMessage": "",
"PSScriptRoot": "",
"PSCommandPath": null,
"PipelineLength": 2,
"PipelinePosition": 1,
"ExpectingInput": false,
"CommandOrigin": 0,
"DisplayScriptPosition": null
},
"LogInvocationInfo": {
"InvocationName": "Add-EndpointLogEntry",
"ScriptLineNumber": 294,
"OffsetInLine": 25,
"HistoryId": 5,
"ScriptName": "C:\\some_path\\Functions\\Lync.ps1",
"Line": " Add-EndpointLogEntry -WriteDebug -Message \"User, $Username, is already Lync-enabled.\"\r\n",
"PositionMessage": "At C:\\some_path\\Functions\\Lync.ps1:294 char:25\r\n+ ... Add-EndpointLogEntry -WriteDebug -Message \"User, $Usernam ...\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~",
"PSScriptRoot": "C:\\some_path\\Functions",
"PSCommandPath": "C:\\some_path\\Functions\\Lync.ps1",
"PipelineLength": 1,
"PipelinePosition": 1,
"ExpectingInput": false,
"CommandOrigin": 1,
"DisplayScriptPosition": null
}
}
As you can see this is a standard windows event but the Message body is all JSON. Automatic Field Discovery is capable of pulling out many of these fields automatically but the values for the fields typically include the quotes and commas that are a part of the JSON syntax (i.e. ClientIP = **"169.68.128.128",**).
I am able to successfully create search time field extractions using regex but as I understand it the only way you can see those is if you are using Smart/Verbose mode which will in turn cause automatic field discovery to occur which means I will get duplicate values, one formatted correctly and one incorrectly. If I use the same ClientIP field name, those two values both show up under ClientIP which is just as confusing as using a different name for the field as I will then have a incorrectly formatted ClientIP and a correctly formatted ClientIPAddress.
So as I see it I need to figure out how to do one of two things. Either I need to find a way to do search time field extractions while preventing automatic field discovery displaying the fields I have custom extractions for or I need to find a way to get automatic field discovery to properly parse the nested JSON. (Or just figure out how to manipulate the data in a search and save the searches, again though that is not ideal.)
I would also be interested in a solution that involves index time field extractions but that of course is only recommended as a last resort due to the performance impact. That said, I don't know that this system would generate enough logs for that performance impact to be noticeable in any way.
Please note that I do not have Splunk admin access, but I do have admin access to the machine the forwarder is on and can modify the .conf files if needed. Also, I'm a bit of a noob to Splunk. All I've really done is take Power Users course and have been given access to Splunk accordingly. So apologies if I am missing something basic here.
Thanks for you time,
↧
