Hi Team,
Need your help/suggestion on what is the best way to handle below scenario.
I am using field extractor screen from search head GUI to extract fields from below proxy log patterns. For example am using below pattern and extracting below 5 fields[filed1 to filed5]
`[06/Sep/2017:17:02:29] failure (13878): for host 141.139.7.270 trying to GET /ba_staticfiles/LeakingNews/LrnJsonRsp.txt, service-http reports: COBRE7740: unable to contact wa01.abc.xyz.com:9102 (Directory lookup error)`
field1 -141.139.7.270
field2 -/ba_staticfiles/LeakingNews/LrnJsonRsp
field3 -COBRE7740
field4 -wa01.abc.xyz.com:9102
field5 -Directory lookup error
Challenge that i am running into is that not all events in the proxy logs are of same length , some log events have all the field information while others do not have them,also if one of the fields is missing in the event none of the fields get extracted.Due to this when i do stats on any of the extracted fields i miss out on the events[total evets=100 , my stats show only 70]
Below are my possible proxy log statements [3rd and 4th event is same as 1 and 2 except that it has additional space after (]
**Patterns**
1st event ->[06/Sep/2017:16:38:23] security (13878): for host 141.139.7.270 trying to GET /favicon.ico, deny-service reports: denying service of /favicon.icon
2nd event ->[06/Sep/2017:17:02:29] failure (13878): for host 141.139.7.270 trying to GET /ba_staticfiles/LeakingNews/LrnJsonRsp.txt, service-http reports: COBRE7740: unable to contact wa01.abc.xyz.com:9102 (Directory lookup error)
3rd event ->[06/Sep/2017:16:38:23] security ( 13878): for host 141.139.7.270 trying to GET /favicon.ico, deny-service reports: denying service of /favicon.icon
4th event ->[06/Sep/2017:17:02:29] failure ( 13878): for host 141.139.7.270 trying to GET /ba_staticfiles/LeakingNews/LrnJsonRsp.txt, service-http reports: COBRE7740: unable to contact wa01.abc.xyz.com:9102 (Directory lookup error)
Any suggestion on best way to handle them [ use regex to handle it during indexing by modifying props/tranforms.conf] or handle it through field extraction in the GUI. If using GUI how this can be achieved using field extraction OR regex from the search head, or if there is another better way.
Also is there a dedicated inbuilt sourcetype for proxy error logs [ i am using access_combined sourcetype]
Thank you in advance.
Splunk Version - 6.5.1
↧