Hello everyone,
Currently the following error occurs with a group of universal forward that should send their logs to a Splunk Indexer by TCP/9997 port.
**Stage**
Splunk Universal Fowrard Version ( 6.3.12 and 6.2.14) OS Windows server 2008 X64 and windows server 2003 X64
Splunk Enterprise Version: 7.1.0 (Role Heavy forward) OS Debian 9.5
**Troubleshooting 1**
Since I do not see logs indexed from this source, the following troubleshooting tests were performed:
WARN TcpOutputFd - Connect to 10.3.7.127:9997 failed. No connection could be made because the target machine actively refused it.
For this test the connectivity tests are carried out, a telnet is made from the universal to the indexer by port 9997 it opens, on the indexer there is a tcpdump and the package is seen to arrive.
tcp ESTAB 0 0 10.3.7.127:8089 10.3.5.145:52522 users:(("splunkd",pid=32523,fd=127))
tcp ESTAB 0 0 10.3.7.127:9997 10.3.5.145:65480 users:(("splunkd",pid=32523,fd=12
As you can see the connections between the origin and the destination are established, it is ruled out that the issue is due to a connectivity problem.
**Troubleshooting 2**
We proceed to review the splunk.log files of the universal forwarder:
08-16-2018 11:23:43.268 -0500 ERROR TcpOutputFd - Read error. An existing connection was forcibly closed by the remote host.
08-16-2018 11:23:43.273 -0500 INFO TcpOutputProc - Connection to 10.3.7.127:9997 closed. Read error. An existing connection was forcibly closed by the remote host.
08-16-2018 11:23:45.102 -0500 WARN TcpOutputFd - Connect to 10.3.7.127:9997 failed. No connection could be made because the target machine actively refused it.
08-16-2018 11:23:45.105 -0500 ERROR TcpOutputFd - Connection to host=10.3.7.127:9997 failed
08-16-2018 11:23:46.105 -0500 WARN TcpOutputFd - Connect to 10.3.7.127:9997 failed. No connection could be made because the target machine actively refused it.
08-16-2018 11:23:46.105 -0500 ERROR TcpOutputFd - Connection to host=10.3.7.127:9997 failed
08-16-2018 11:23:46.105 -0500 WARN TcpOutputProc - Applying quarantine to ip=10.3.7.127 port=9997 _numberOfFailures=2
08-16-2018 11:24:25.726 -0500 WARN HttpPubSubConnection - Unable to parse message from PubSubSvr:
08-16-2018 11:24:25.726 -0500 INFO HttpPubSubConnection - Could not obtain connection, will retry after=50.990 seconds.
As we can see the main error is the following:
ERROR TcpOutputFd - Read error. An existing connection was forcibly closed by the remote host.
And after this we see that the connection was rejected:
08-16-2018 11:23:45.105 -0500 ERROR TcpOutputFd - Connection to host=10.3.7.127:9997 failed
08-16-2018 11:23:46.105 -0500 WARN TcpOutputFd - Connect to 10.3.7.127:9997 failed. No connection could be made because the target machine actively refused it.
The splunkd service has been rebooted, both the universal and the heavy forward and the same ERROR is still obtained.
I have a suspicion that the error is presented by the issue of the versions that runs the universal foward since it is lower than the Heavy, regarding this failure I do not find technical documentation of the cmunication between these, what solutions could I give for the case taking into account that splunk uf is no longer supported by server 2008 and 2003 until version 6.3.2.
Thanks, I remain attentive to any contribution.
↧