TL;DR - **Is there a way (without custom scripts or commands) to run a command from a string in the format of a `union` that contains a dynamic number of subsearches?**
I have quite a few heavy dashboards that get hit by many users, quite frequently. In order to save both processing power and user time, I've been caching search results (such as in a `loadjob` or `outputlookup`) and stitching them together dynamically. These dashboards all use a single `savedsearch`, and an instance of the search will cache data hourly, usually saving a day's worth of back-data. The dashboards then take the saved data and combine it with a call of the same search from the start of the hour until now(), and it's significantly faster.
I'm trying to create a macro that can do this intelligently so I don't have to re-build it per dashboard/savedsearch etc. I've created a search that can generate a string that, when used as a command, searches fine. However, I've been unable to find a way to get it to actually run - as far as I or anyone in my team can tell, anything that's a string can only be run as a `search`, which is insufficient for my needs as it ends up literally searching for a string talking about unions.
The example string that's being generated by the query below reads nicely as: (though NB, the output string could just as easily be a single line with no `union` or `sort`, or a `union` of two subsearches)
| union
[ | savedsearch my_savedsearch earliest=1568343703.000000 latest=1568462400.000000 ]
[ | inputlookup mycsv.csv ]
[ | savedsearch my_savedsearch earliest=1568602800.000000 latest=1568602843.000000 ]
| sort - _time
The generating query has an initial `eval` statement that I intended to use when moving it into a macro, which is stubbing the various args. Modifying these will dynamically change the number of subsearches/times (as the user might be requesting a time period relative to the cached data that is before, after, during, or some combination of each). Variants of this have been attempted using things like `append`, `appendpipe`, `multisearch`, but all seem to fail due to various reasons except the above (circular dependencies, streaming commands etc). The string generated, when run against a valid dataset, works. When the command is passed as the search itself using subsearches/`return` etc all fail - as far as we can tell, it's trying to run it as a `search`, or otherwise is unable to handle it.
Generating query: (replace contents in the initial eval for different dataset examples)
| makeresults
| eval
timerange_string="| inputlookup mycsv.csv",
timerange_earliest=relative_time(now(), "-1d@d"),
timerange_latest=relative_time(now(), "@h"),
time_string="| savedsearch my_savedsearch",
time_earliest=relative_time(now(), "-3d"),
time_latest=relative_time(now(), "-1m")
| eval
earliest_period=case(
time_earliest < timerange_earliest,-1,
(timerange_earliest <= time_earliest AND time_earliest <= timerange_latest),0,
timerange_latest < time_earliest,1,
true()=true(),"null"
),
latest_period=case(
time_latest < timerange_earliest,-1,
(timerange_earliest <= time_latest AND time_latest <= timerange_latest),0,
timerange_latest < time_latest,1,
true()=true(),"null"
)
| fields - _time
| eval
union_string=if(earliest_period!=latest_period,"| union [ ",""),
search=
if(earliest_period=0 OR latest_period=0,"| search","")
.if(earliest_period=0," earliest=".time_earliest,"")
.if(latest_period=0," latest=".time_latest,""),
inputlookup=timerange_string." ".search,
inputlookup_string=if(earliest_period*latest_period<1,inputlookup,""),
ss1=time_string." earliest=".time_earliest." latest=".if(latest_period=-1,time_latest,timerange_earliest),
ss1_string=if(earliest_period=-1,ss1,""),
ss2=time_string." earliest=".if(earliest_period=1,time_earliest,timerange_latest)." latest=".time_latest,
ss2_string=if(latest_period=1,ss2,""),
sort_string=if(earliest_period!=latest_period," ] | sort - _time","")
| eval search=union_string.ss1_string.if(earliest_period=-1 AND latest_period!=-1," ] [ ","").inputlookup_string.if(latest_period=1 AND earliest_period!=1," ] [ ","").ss2_string.sort_string
| fields search
There are some similar questions (don't have enough karma to post links), but none directly tackle this use case, as `map` doesn't seem to be a viable solution here (triggers circular dependencies) and the general advice given with `return` doesn't work (as mentioned above).
Is there an obvious option I'm missing here? The next step I'd be looking at would be writing a custom command, which is less portable than a macro. I'll be working on that at some point, but if there's a way to do this with a macro using a basic Splunk query, it'd be my preference.
↧