Hi, I'm in a Search Head Cluster environment and while looking at our scheduling load, I found some references to schedule ID's (seemingly from Unix/Linux app) that don't seem to exist.
The report below displays upcoming scheduled searches based on their next execution time.
| rest /servicesNS/-/-/saved/searches
| search disabled=0 is_scheduled=1 next_scheduled_time!=""
| dedup title,next_scheduled_time
| table title cron_schedule next_scheduled_time id | sort next_scheduled_time
This led me to some saved searches that run on cron schedules but cannot be found via .conf files or the REST API. In particular, there are 2 searches from SA-nix "app" that I can't seem to find.
I've tried "grep -R /opt/splunk" on both the deployer and the cluster member nodes. I've also looked all over the API and can't find a reference. The exact ID's are below.
https://127.0.0.1:8089/servicesNS/nobody/SA-nix/saved/searches/Alert%20-%20syslog%20errors%20last%20hour
https://127.0.0.1:8089/servicesNS/nobody/SA-nix/saved/searches/fired_alerts
And can be easily found by adding
id="https://127.0.0.1*" to the above search.
Has anyone experienced these "orphaned" searches before? As you can guess, I used to have SA-Unix (part of [this app][1]), but it was removed (maybe improperly) as we migrated from a single-host doing everything to a true multi-host cluster.
[1]: https://splunkbase.splunk.com/app/273/#/overview