Has anyone ever run into a situation where the forwarder opened hundreds of sockets on a system?
Here is what we have configured on the system's output.conf:
[tcpout]
maxQueueSize = auto
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_internal|_introspection)
forwardedindex.filter.disable = false
indexAndForward = false
autoLBFrequency = 30
blockOnCloning = true
compressed = false
disabled = false
dropClonedEventsOnQueueFull = 5
dropEventsOnQueueFull = -1
heartbeatFrequency = 30
maxFailuresPerInterval = 2
secsInFailureInterval = 1
maxConnectionsPerIndexer = 2
forceTimebasedAutoLB = false
sendCookedData = true
connectionTimeout = 20
readTimeout = 300
writeTimeout = 300
tcpSendBufSz = 0
useACK = false
blockWarnThreshold = 100
sslQuietShutdown = false
[syslog]
type = udp
priority = <13>
dropEventsOnQueueFull = -1
maxEventSize = 1024
Admittedly, the admin before me who set the forwarder up had the forwarder pulling logs from a directory. The directory has over 2000 log files that are updated regularly. To elaborate, they are all flatfiles that contain metric data from an application performance monitoring tool.
I know that this is poor practice and I am working to correct this moving forwarder, but I wanted to use this as a learning opportunity. Would the consumption of a directory containing thousands of log files cause a forwarder to do something like that?
They were all stuck in a CLOSE_WAIT state as well
tcp 1294 0 myappserver:tproxy indexerServerOne:25022 CLOSE_WAIT
tcp 1218 0 myappserver:tproxy indexerServerOne:25023 CLOSE_WAIT
tcp 1246 0 myappserver:tproxy indexerServerOne:25020 CLOSE_WAIT
tcp 1269 0 myappserver:tproxy indexerServerOne:25021 CLOSE_WAIT
tcp 1218 0 myappserver:tproxy indexerServerOne:icl-twobase9 CLOSE_WAIT
tcp 1207 0 myappserver:tproxy indexerServerOne:icl-twobase3 CLOSE_WAIT
tcp 1226 0 myappserver:tproxy indexerServerOne:icl-twobase4 CLOSE_WAIT
tcp 1269 0 myappserver:tproxy indexerServerOne:icl-twobase7 CLOSE_WAIT
tcp 1294 0 myappserver:tproxy indexerServerOne:icl-twobase8 CLOSE_WAIT
tcp 1194 0 myappserver:tproxy indexerServerOne:icl-twobase5 CLOSE_WAIT
tcp 1246 0 myappserver:tproxy indexerServerOne:icl-twobase6 CLOSE_WAIT
tcp 1218 0 myappserver:tproxy indexerServerOne:24992 CLOSE_WAIT
tcp 1207 0 myappserver:tproxy indexerServerOne:24986 CLOSE_WAIT
tcp 1226 0 myappserver:tproxy indexerServerOne:24987 CLOSE_WAIT
tcp 1269 0 myappserver:tproxy indexerServerOne:24990 CLOSE_WAIT
tcp 1294 0 myappserver:tproxy indexerServerOne:24991 CLOSE_WAIT
tcp 1194 0 myappserver:tproxy indexerServerOne:24988 CLOSE_WAIT
tcp 1246 0 myappserver:tproxy indexerServerOne:24989 CLOSE_WAIT
tcp 1294 0 myappserver:tproxy indexerServerOne:24978 CLOSE_WAIT
tcp 1218 0 myappserver:tproxy indexerServerOne:24979 CLOSE_WAIT
tcp 1246 0 myappserver:tproxy indexerServerOne:24976 CLOSE_WAIT
tcp 1269 0 myappserver:tproxy indexerServerOne:24977 CLOSE_WAIT
tcp 1226 0 myappserver:tproxy indexerServerOne:24974 CLOSE_WAIT
tcp 1194 0 myappserver:tproxy indexerServerOne:24975 CLOSE_WAIT
tcp 1207 0 myappserver:tproxy indexerServerOne:24973 CLOSE_WAIT
tcp 1269 0 myappserver:tproxy indexerServerOne:24962 CLOSE_WAIT
tcp 1294 0 myappserver:tproxy indexerServerOne:24960 CLOSE_WAIT
tcp 1218 0 myappserver:tproxy indexerServerOne:24961 CLOSE_WAIT
↧