I had a forwarder on an AIX server sending a number of log files to my Splunk Indexer and all was working well.
Then debugging got turned on on the application producing the log files, my Splunk license got blown out of the window so I had to stop the forwarder.
Since then whenever I turn on the forwarder again Splunk only creates an event for the first (multi) line in the logfiles, giving it a timestamp of the system time as there is no date or time against the first line in the logs. Also it creates an event if the logfile rolls over, again taking the first line in it.
The log files are Maximo WebSphere UI logs.
The event that is being recorded is like this;
************ Start Display Current Environment ************
WebSphere Platform 6.1 [ND 6.1.0.47 cf471333.02] running with process name ctgCell01\ctgNode01\XXXXXXXServer and process id 426116
Detailed IFix information: Please use the versionInfo command to view this information
Host Operating System is AIX, version 5.3
Java version = 1.5.0, Java Compiler = NONE, Java VM name = IBM J9 VM
was.install.root = /hostname/IBM/WebSphere/AppServer
user.install.root = /hostname/IBM/WebSphere/AppServer/profiles/ctgAppSrv01
Java Home = /hostname/IBM/WebSphere/AppServer/java/jre
ws.ext.dirs = /hostname/IBM/WebSphere/AppServer/java/lib:/hostname/IBM/WebSphere/AppServer/profiles/ctgAppSrv01/classes:/hostname/IBM/WebSphere/AppServer/classes:/hostname/IBM/WebSphere/AppServer/lib:/hostname/IBM/WebSphere/AppServer/installedChannels:/hostname/IBM/WebSphere/AppServer/lib/ext:/hostname/IBM/WebSphere/AppServer/web/help:/hostname/IBM/WebSphere/AppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime
Classpath = /hostname/IBM/WebSphere/AppServer/profiles/ctgAppSrv01/properties:/hostname/IBM/WebSphere/AppServer/properties:/hostname/IBM/WebSphere/AppServer/lib/startup.jar:/hostname/IBM/WebSphere/AppServer/lib/bootstrap.jar:/hostname/IBM/WebSphere/AppServer/lib/j2ee.jar:/hostname/IBM/WebSphere/AppServer/lib/lmproxy.jar:/hostname/IBM/WebSphere/AppServer/lib/urlprotocols.jar:/hostname/IBM/WebSphere/AppServer/deploytool/itp/batchboot.jar:/hostname/IBM/WebSphere/AppServer/deploytool/itp/batch2.jar:/hostname/IBM/WebSphere/AppServer/java/lib/tools.jar
Java Library path = /hostname/IBM/WebSphere/AppServer/java/jre/bin:/hostname/IBM/WebSphere/AppServer/java/jre/bin:/hostname/IBM/WebSphere/AppServer/java/jre/bin/classic:/hostname/IBM/WebSphere/AppServer/java/jre/bin:/hostname/IBM/WebSphere/AppServer/bin:/hostname/IBM/WebSphere/AppServer/java/jre/bin/j9vm:/hostname/IBM/WebSphere/AppServer/java/jre/bin/j9vm:/hostname/IBM/WebSphere/AppServer/java/jre/bin//headless:/hostname/IBM/WebSphere/AppServer/java/jre/bin/j9vm:/usr/lib:/hostname/IBM/WebSphere/AppServer/lib/WMQ/java/lib
************* End Display Current Environment *************
Subsequent lines are like this, but not appearing in Splunk;
[13/11/15 08:24:01:218 GMT] 0000002e SystemOut O 13 Nov 2015 08:24:01:218 [INFO] BMXAA6370I - Total number of users connected to the system: 0
[13/11/15 08:24:01:219 GMT] 0000002e SystemOut O 13 Nov 2015 08:24:01:219 [INFO] BMXAA7019I - The total memory is 2147483648 and the memory available is 1897199760.
All I did was stop the forwarder, then restart it again a few days later.
↧