My indexer cluster is down except for 1 out of 6. 8089 is suddenly not working for indexers and CM<>indexer comms and i get the below error messages. Its a multi site indexer cluste. I have ran telnet and curl commands on 8089 & indexers but still unable to connect to all but 1/6 indexers. Also, deployment server is not accessible. CM is unable to connect to 8089 for the indexers, the indexers cannot talk to each other on port 8089 either and the DS is not able to connect to my indexers at 9996.
----------------------------------------
FYI custom SSL is enabled at 8089 but i don't see as the cause for this connectivity issue.
I have checked with networking team who are saying its an application issue and not iptables/routing issue on the server like i suspected. Please help.
IDX:
02-10-2020 03:19:20.324 +0000 WARN CMSlave - Failed to register with cluster master reason: failed method=POST path=/services/cluster/master/peers/?output_mode=json master=myCM:8089 rv=0 gotConnectionError=0 gotUnexpectedStatusCode=1 actual_response_code=500 expected_response_code=2xx status_line="Internal Server Error" socket_error="No error" remote_error=Cannot add peer=myidx mgmtport=8089 (reason: http client error=No route to host, while trying to reach https://myidx:8089/services/cluster/config). [ event=addPeer status=retrying AddPeerRequest: { _id= active_bundle_id=EF3B7708025567663732F8D6B146A83 add_type=Clear-Masks-And-ReAdd base_generation_id=2063 batch_serialno=1 batch_size=1 forwarderdata_rcv_port=9996 forwarderdata_use_ssl=1 last_complete_generation_id=0 latest_bundle_id=EF3B77080255676637732F8D6B146A83 mgmt_port=8089 name=EEC311D7-7778-44FA-B31D-E66672C1D568 register_forwarder_address= register_replication_address= register_search_address= replication_port=9100 replication_use_ssl=0 replications= server_name=myidx site=site3 splunk_version=7.2.6 splunkd_build_number=c0bf0f679ce9 status=Up } ].
CM:
02-07-2020 18:00:41.497 +0000 WARN CMMaster - event=heartbeat guid=BDD6A029-2082-48ED-96F3-21BD624D94CD msg='signaling Clear-Masks-And-ReAdd' (unknown peer and master initialized=1
02-07-2020 18:00:41.911 +0000 WARN TcpOutputFd - Connect to myidx:9996 failed. No route to host
02-07-2020 18:00:41.912 +0000 WARN TcpOutputProc - Applying quarantine to ip=myidx port=9996 _numberOfFailures=2
02-07-2020 18:00:42.013 +0000 WARN TcpOutputFd - Connect to myidx:9996 failed. No route to host
02-07-2020 18:00:42.323 +0000 WARN CMMaster - event=heartbeat guid=44AF1666-AB56-4CC1-8F01-842AD327CF79 msg='signaling Clear-Masks-And-ReAdd' (unknown peer and master initialized=1
02-07-2020 10:36:54.650 +0000 WARN CMRepJob - _rc=0 statusCode=502 transErr="No route to host" peerErr=""
02-07-2020 10:36:54.650 +0000 WARN CMRepJob - _rc=0 statusCode=502 transErr="No route to host" peerErr=""
DS trying to connect to indexers:
02-07-2020 11:56:12.097 +0000 WARN TcpOutputFd - Connect to idx2:9996 failed. No route to host
02-07-2020 11:56:12.098 +0000 WARN TcpOutputFd - Connect to idx3:9996 failed. No route to host
02-07-2020 11:56:13.804 +0000 WARN TcpOutputFd - Connect to idx1:9996 failed. No route to host
↧