So I am working on handling variable situations in a deployed environment so the way I was solving this issue was to use Splunk's scripted input capability to run an initial script (powershell) on the host, find some data about the host, update my "ClientName" field in the deploymentclient.conf, and then restart Splunk.
Everything is going great up until the "restart Splunk" requirement. Which must happen because otherwise it won't update the ClientName until something else causes Splunk to restart.
In powershell you can call a "Restart-Service" command which will restart the service. The problem is, Splunk is what executed the underlying script, so I am essentially telling it to restart itself. This is a problem, because it then kills the script when Splunk stops, and nothing controls it to turn it back on again.
So I thought I would look into spawning a background process through powershell to force it to be an independant process and it seemed like it going to work, until it stopped in the same place... and my background task closed out as well only I can't see what happened because it is in the background...
The relevant part of my powershell script is below:
Start-Process powershell -ArgumentList "-NonInteractive -command & ({Restart-Service SplunkForwarder})"
I checked this command does work by running it manually on the host. It pops up a new powershell window and you can see it restart the service. And I even tried modifying the command to instead add the "-NoExit" parameter. This time, it runs, stops the splunk service and the powershell.exe stays alive... but it never turns splunk back on again...
In the splunkd.log I still get the same:
01-28-2016 18:57:57.234 -0500 WARN ProcessRunner - Process with pid 8900 did not exit within a given grace period after being signaled to exit. Will have to forcibly terminate.
Anyone else try using Splunk Scripts to make a change to your Splunk environment and then kick a restart? If so how did you handle it?
↧