I have reviewed and tried most ever suggestion that I have seen on this site but still no luck. I am trying to filter out, pre-index, all java stack traces containing lines like robots.txt, favicon.ico, etc. These are WebSphere 8 Application Server logs and I am currently testing this in my sandbox. I am using the Splunk_TA_ibm-was which has a sourcetype of ibm:was:systemOutLog for the SystemOut.log
As I mentioned I have tried several variations that all work on the search command line like:
sourcetype="ibm:was:systemOutLog" | REGEX _raw != "(/apple.+png|/favicon.ico|/robots.txt|/yahoo-dom-event.js)"
Which reduces the total number of events from 58,785 to 33,303. Below are my last attempt's configuration:
props.conf
[sourcetype::ibm:was:systemOutLog]
TRANSFORMS-null = null_queue_filter
transforms.conf
[null_queue_filter]
REGEX=(/apple.+png|/favicon.ico|/robots.txt|/yahoo-dom-event.js)
DEST_KEY=queue
FORMAT=nullQueue
I have tried these in several places, but I believe that /opt/splunk/etc/apps/Splunk_TA_ibm-was/local/ is the correct location. I leave these in the web server logs, but do not need the stack traces that java dumps on everything. All applications are running under RHEL 6 if that makes a difference.
Just in case;
ibm_was.conf
(one of the four entries)
[monitor:///opt/IBM/WebSphere/AppServers/profiles/DMT-AS8P03/logs]
whitelist = SystemOut.log
crcSalt =
disabled = false
followTail = false
index = cfnc_appsrv
host =
host_segment = 6
sourcetype = ibm:was:systemOutLog
TIA as I am sure it is something simple I am overlooking.