Hi,
I know there are lot of questions under the same topic,but i am stuck.i have an application server which forwards the logs to splunk.The way logs are written is that are on random selection.i will share that information as well. So, when there is a process and being written into the log it picks a random one from all the logs and appends to it.even the log date modified is ,lets say today, when i open up the log it might start with a date and a process written onto that log from 3 months ago and at the end of that log i can see the latest process from today ,and when another process happens it writes it to another log and that is the cycle.
here is my inputs.conf
[default]
host = xxxxxx
[monitor://D:\y\Log Files\]
disabled = 0
index=z
followTail = 0
sourcetype=Data Import
ignoreOlderThan = 30d
Here are the screenshots
alt text
could post the last screenshot but it is showing the end of the same log i posted with today's date.
My question is,i am not getting all the log files form that location.not sure how long this has been happening for but i jut found out about this couple days ago.Lets say i have 15 log files from yesterday,i only got 3 of them.To troubleshoot the issue i tried looking at the splunkd but that did not give me much.
this is the latest entry on splunkd
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-wmi.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 10000000000 ms
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-MonitorNoHandle.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 60000 ms
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-admon.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 60000 ms
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-netmon.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 60000 ms
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-perfmon.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: run once
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-powershell.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 60000 ms
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-powershell.exe --ps2
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 60000 ms
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-regmon.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 60000 ms
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-winevtlog.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 60000 ms
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - New scheduled exec process: D:\splunk\bin\splunk-winprintmon.exe
01-09-2018 12:21:38.010 -0500 INFO ExecProcessor - interval: 60000 ms
01-09-2018 12:21:38.041 -0500 INFO PipelineComponent - Launching the pipelines for set 0.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - TailWatcher initializing...
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME\var\spool\splunk.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME\var\spool\splunk\...stash_new.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME\etc\splunk.version.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME\var\log\splunk.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME\var\log\splunk\license_usage_summary.log.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME\var\log\splunk\metrics.log.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME\var\log\splunk\splunkd.log.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://D:\y\Log Files\.
01-09-2018 12:21:38.088 -0500 INFO TailReader - State transitioning from 1 to 0 (initOrResume).
01-09-2018 12:21:38.088 -0500 INFO TailReader - State transitioning from 1 to 0 (initOrResume).
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Adding watch on path: D:\y\Log Files.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Adding watch on path: D:\splunk\etc\splunk.version.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Adding watch on path: D:\splunk\var\log\splunk.
01-09-2018 12:21:38.088 -0500 INFO TailingProcessor - Adding watch on path: D:\splunk\var\spool\splunk.
01-09-2018 12:21:38.088 -0500 INFO TailReader - Registering metrics callback for: tailreader0
01-09-2018 12:21:38.088 -0500 INFO TailReader - Starting tailreader0 thread
01-09-2018 12:21:38.088 -0500 INFO TailReader - Registering metrics callback for: batchreader0
01-09-2018 12:21:38.088 -0500 INFO TailReader - Starting batchreader0 thread
01-09-2018 12:21:38.088 -0500 INFO loader - Limiting REST HTTP server to 3333 sockets
01-09-2018 12:21:38.088 -0500 INFO loader - Limiting REST HTTP server to 1365 threads
01-09-2018 12:21:39.710 -0500 INFO WatchedFile - Will begin reading at offset=988394 for file='D:\y\Log Files\DataImport-62-[2384].log'.
01-09-2018 12:21:39.726 -0500 INFO WatchedFile - Will begin reading at offset=3402522 for file=''D:\y\Log Files\DataImport-62-[2364].log'.
01-09-2018 12:21:39.804 -0500 INFO TcpOutputProc - Connected to idx=10.14.0.246:9997, pset=0, reuse=0.
01-09-2018 12:21:52.876 -0500 INFO WatchedFile - Will begin reading at offset=344718 for file=''D:\y\Log Files\DataImport-62-[5712].log'.
01-09-2018 12:22:12.220 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='D:\splunk\var\log\splunk\splunkd_ui_access.log'.
01-09-2018 12:22:12.220 -0500 INFO WatchedFile - Will begin reading at offset=50885 for file='D:\splunk\var\log\splunk\splunkd-utility.log'.
01-09-2018 12:22:12.220 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='D:\splunk\var\log\splunk\searchhistory.log'.
01-09-2018 12:22:12.220 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='D:\splunk\var\log\splunk\scheduler.log'.
01-09-2018 12:22:12.236 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='D:\splunk\var\log\splunk\remote_searches.log'.
01-09-2018 12:22:12.236 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='D:\splunk\var\log\splunk\mongod.log'.
01-09-2018 12:22:12.314 -0500 INFO WatchedFile - Will begin reading at offset=12261005 for file='D:\splunk\var\log\splunk\metrics.log'.
01-09-2018 12:22:12.314 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='D:\splunk\var\log\splunk\license_usage_summary.log'.
01-09-2018 12:22:12.314 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='D:\splunk\var\log\splunk\license_usage.log'.
01-09-2018 12:22:12.314 -0500 INFO WatchedFile - Will begin reading at offset=11480 for file='D:\splunk\var\log\splunk\conf.log'.
01-09-2018 12:22:12.314 -0500 INFO WatchedFile - Will begin reading at offset=77366 for file='D:\splunk\var\log\splunk\audit.log'.
01-09-2018 12:50:02.481 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file=''D:\y\Log Files\DataImport-62-[2384].log'.
01-09-2018 12:50:02.481 -0500 INFO WatchedFile - Will begin reading at offset=0 for file=''D:\y\Log Files\DataImport-62-[2384].log'.
01-09-2018 12:50:03.495 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file=''D:\y\Log Files\DataImport-62-[2364].log'.
01-09-2018 12:50:03.495 -0500 INFO WatchedFile - Will begin reading at offset=0 for file=''D:\y\Log Files\DataImport-62-[2364].log'.
01-10-2018 03:29:25.021 -0500 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='D:\splunk\var\log\splunk\metrics.log'.
01-10-2018 03:29:25.021 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='D:\splunk\var\log\splunk\metrics.log'.
01-10-2018 03:29:25.099 -0500 INFO WatchedFile - Will begin reading at offset=24999075 for file='D:\splunk\var\log\splunk\metrics.log.1'.
i deleted the splunkd and restarted the splunk service and check to see if i was getting the missing logs and that worked for a day.and whenever i made a change to the log it was being captured and sent to indexer.But today,it is the same behavior.i am missing log files in splunk.
i hope this is not too complicated .i am kind of stuck and need second set of eyes to tell me that i missing something.Any help is appreciated.
Thanks
↧
