We are having an issue with our Splunk installation not being able to run any searches on our Test environment. We see the following message on the search head:
Search peer xxxindex01xxx has the following message: Failed to contact license master: reason='WARN: path=/masterlm/usage: invalid signature on request from ip=xx.xx.x.xxx' first failure time=1463689647
Our Splunk deployment consists of two environments: Test and Production. Each environment has 1 search head, 1 master server, and 3 indexers. The Production Master Server is also the Master License Server for both environments (Test and Production). All systems/environments are running Splunk 6.3.3. The problem being reported is only on the 3 indexers in our Test environment.
All Splunk nodes on VM's.
We currently have a 2GB license and have been set up on a license pool; 250MB allocated to the Test environment; 1.75 GB allocated to Production. According to the license manager, the Test pool has used up 0% of its license.
The problem also appears to have started after patching the Linux servers with security fixes, this past Sat 5/14.
Could use some guidance on how to diagnose the problem and how to fix it.
Splunk> answers indicates looking at pass4SymmKey. That’s fine, but why would a OS patch affect this?
We ran the licenser command on each affected indexer as this fixed a previous issue, but this did not work this time.
./splunk edit licenser-localslave -master_uri 'https://master:port'
Also, if it were the pass key, I see that one exists in [clustering] stanza, and did not see one in the [general] of one of our Prod indexers. Our Test indexer does seem to have the pass key in both stanza's.
If we were to add a pass key to all [general] stanzas on all nodes, could I just copy the pass key from the license master node server.conf file to all others OR should it be created at each node and entered in plain text and restarted and let splunk encrypt on restart for each node?
↧