Hi,
We have a Universal Forwarder on our Linux rSyslog server. It was working fine until two weeks ago. The problem was it would stop sending data to the indexer, but showed no errors in the splunkd.log. When we would restart it, it would send a burst of information over the course of 4-5 minutes then stop sending data again.
Over the past two weeks we have replaced the rSyslog server with a new server. The new server has 8 cores, tons of memory, and a 10GB network connection to the Splunk indexer. Once we installed the forwarder it ran for two days non-stop catching up on the data that had been missed over the two week period. At 6pm last night it stopped forwarding data again. We're now back to the same problem we started with. We get a burst of log data on restart, but then it just stops. No errors, nothing to suggest we've hit any limits. The splunkforwarder.server process is still running. What we DO notice is that Splunkd holds the files open, and the number of open files continues to climb once it stops forwarding data. Some of these files are large, but we don't get any error messages about batch
in limits.conf we have this set
maxKBps = 0
max_fd = 10240
The ulimits on the server are set to 100000 - we're averaging about 4500-5000 before the forwarder stops running.
Indexer 7.3.1
Universal Forwarder 7.3.1
We have about 35 windows forwarders working on servers with no issues at all. It's this one Linux forwarder that's not working correctly.
Any help you can give would be appreciate. Let me know if there is any additional information needed.
↧