I am trying to create a dashboard that graphs the parsing queue size for a HF by `ingest_pipe`. I noticed that most of these logs have that field but some don't (i'm not sure why).
**sample logs**
06-03-2020 12:21:30.964 -0400 INFO Metrics - group=queue, name=parsingqueue, max_size_kb=512, current_size_kb=0, current_size=0, largest_size=2, smallest_size=0
06-03-2020 12:21:27.144 -0400 INFO Metrics - group=queue, ingest_pipe=3, name=parsingqueue, max_size_kb=6144, current_size_kb=0, current_size=0, largest_size=2, smallest_size=0
06-03-2020 12:21:27.142 -0400 INFO Metrics - group=queue, ingest_pipe=2, name=parsingqueue, max_size_kb=6144, current_size_kb=0, current_size=0, largest_size=11778, smallest_size=0
**current SPL**
index=_internal host=$hostToken$ group=queue name=parsingqueue
| timechart avg(current_size_kb) by ingest_pipe
I can't modify the search with `ingest_pipe=*` because I have tokenized the host field in the search and some of my HF's only have 1 ingest pipe. In that scenario, there is no `ingest_pipe` field at all so hardcoding that into the search will result in 0 results when the HF only has 1 pipeline.
The solution I came up with is to count the # of events where ingest_pipe exists (yesPipe), count the # of events where it does not exist (noPipe), and assign my `count by foo` value to the field that is greater. If yesPipe is greater, `count by ingest_pipe`, else `count by host`. I don't have the query for these counts and checks.
Alternatively, I thought I could use a lookup table that has a "count by field" column, where per host I simply specify either `ingest_pipe` or `host` to count by.
I feel like there is an easy solution and I'm overthinking it. Any ideas?
↧