Hey All,
We were seeing this message in Splunk Web:
Search peer *hostname* has the following message: Failed to synchronize configuration with KVStore cluster. replSetReconfig should only be run on PRIMARY, but my state is SECONDARY; use the "force" argument to override
During the course of the investigation and after running a show kvstore-status command from the KV store captain I had noticed that a particular search head was listed under the "Enabled KV store members:" section, however it was not listed under the "KV store members:" section.
Out of curiosity, I ran the show kvstore-status command from the search head that was missing from the "KV store members" and all the only output I got back was the "Enabled KV store members" section, so it appeared that something was off with this particular search head.
I did some searching online and although I didn't find anything specifically related to this, I did read some similar articles that suggested cleaning the local kvstore on that specific search head, so I did that: splunk clean kvstore --local
Unfortunately, now when running the show kvstore-status command it's showing a status of "starting" and it seems to be stuck.
I did a restart of splunk just to see if anything would change, but the kvstore status is still stuck on "starting"
I had a look in the splunkd and mongod logs and did find a couple of errors that seem to be pointing to the issue, however am unable to find anything online that has helped in resolving, so I am looking for some assistance and/or guidance with this.
Errors from the logs:
splunkd log error:
05-31-2017 13:10:30.594 -0400 ERROR KVStorageProvider - An error occurred during the last operation ('replSetGetStatus', domain: '5', code: '94'): no replset config has been received
05-31-2017 13:10:30.594 -0400 ERROR KVStoreAdminHandler - An error occurred.
05-31-2017 13:10:30.619 -0400 ERROR KVStorageProvider - An error occurred during the last operation ('replSetGetStatus', domain: '5', code: '94'): no replset config has been received
05-31-2017 13:10:30.619 -0400 ERROR KVStoreAdminHandler - An error occurred.
mongod log error:
2017-05-31T17:09:11.419Z I REPL [initandlisten] Did not find local replica set configuration document at startup; NoMatchingDocument Did not find replica set configuration document in local.system.replset
2017-05-31T17:09:11.421Z I NETWORK [initandlisten] waiting for connections on port 8191 ssl
Any help with this would be greatly appreciated.
Thanks in advance!
↧