I have a set of four nodes on site (site1) local L2 network to the splunk cluster master.
I changed cluster to have second site defined, now adding the four remote index nodes. They are on seperate L3 routed network (in a hosted secondary site). The hosts can ping forward and reverse (DNS and NTP are checked). I have setup full SSH shared keys so ssh works and file transfering fine (did a couple of large file copies to test. Network communication ~20ms so not too bad considering distance.
The join command hangs and never completes. Not getting any output bad / good but for one entry in the `/var/log/messages`:
Jun 17 09:45:32 splunkindex11 smbd[4243]: failed to retrieve printer list: NT_STATUS_UNSUCCESSFUL
Jun 17 09:50:01 splunkindex11 winbindd[2125]: [2016/06/17 09:50:01.232278, 0] libads/kerberos_util.c:101(ads_kinit_password)
Jun 17 09:50:01 splunkindex11 winbindd[2125]: kerberos_kinit_password SPLUNKINDEX11$@AESSCLD.ARROW.COM failed: Client not found in Kerberos database
Jun 17 09:51:29 splunkindex11 winbindd[2125]: [2016/06/17 09:51:29.349709, 0] libads/kerberos_util.c:101(ads_kinit_password)
Jun 17 09:51:29 splunkindex11 winbindd[2125]: kerberos_kinit_password SPLUNKINDEX11$@AESSCLD.ARROW.COM failed: Client not found in Kerberos database
Jun 17 09:58:33 splunkindex11 smbd[9068]: [2016/06/17 09:58:33.750926, 0] printing/print_cups.c:151(cups_connect)
Jun 17 09:58:33 splunkindex11 smbd[9068]: Unable to connect to CUPS server localhost:631 - Connection refused
Jun 17 09:58:33 splunkindex11 smbd[4243]: [2016/06/17 09:58:33.751998, 0] printing/print_cups.c:528(cups_async_callback)
Jun 17 09:58:33 splunkindex11 smbd[4243]: failed to retrieve printer list: NT_STATUS_UNSUCCESSFUL
Jun 17 10:00:01 splunkindex11 winbindd[2125]: [2016/06/17 10:00:01.322778, 0] libads/kerberos_util.c:101(ads_kinit_password)
Jun 17 10:00:01 splunkindex11 winbindd[2125]: kerberos_kinit_password SPLUNKINDEX11$@AESSCLD.ARROW.COM failed: Client not found in Kerberos database
Shell command to join index that hangs:
Last login: Fri Jun 17 08:51:21 2016 from 172.20.14.79
[root@splunkindex14 ~]# ping splunkcmaster01.ibm.aessatl.arrow.com
PING splunkcmaster01.ibm.aessatl.arrow.com (172.20.14.85) 56(84) bytes of data.
64 bytes from 172.20.14.85: icmp_seq=1 ttl=60 time=26.0 ms
64 bytes from 172.20.14.85: icmp_seq=2 ttl=60 time=26.5 ms
^C
--- splunkcmaster01.ibm.aessatl.arrow.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1500ms
rtt min/avg/max/mdev = 26.098/26.343/26.588/0.245 ms
[root@splunkindex14 ~]# ping splunkdeploy01.ibm.aessatl.arrow.com
PING splunkdeploy01.ibm.aessatl.arrow.com (172.20.14.83) 56(84) bytes of data.
64 bytes from 172.20.14.83: icmp_seq=1 ttl=60 time=25.9 ms
^C
--- splunkdeploy01.ibm.aessatl.arrow.com ping statistics ---
2 packets transmitted, 1 received, 50% packet loss, time 1024ms
rtt min/avg/max/mdev = 25.984/25.984/25.984/0.000 ms
[root@splunkindex14 ~]# cd /opt/splunk/bin/
[root@splunkindex14 bin]# ./splunk edit cluster-config -master_uri https://splunkcmaster01.ibm.aessatl.arrow.com:8089 -mode slave -site site2 -replication_port 9000 -secret Splunkd4ta
Your session is invalid. Please login.
Splunk username: admin
Password:
Googling around this notes DNS and such, this is why I double checked DNS /NTP. They are in different sites, so of course L3 and DNS zones are in different context, but from cluster master to the remote index and reverse back ... forward and reverse work.
Any suggestions?
↧