I have data which contain a field with a lot of values and has duplicates on almost every one - a barcode, scanned in more than one place. In addition to being a part of a long field (a kind of a combined tag) like `<%03d length>`, that barcode does appear on its own in a couple of places in the same event, which hopefully means I can search on it quickly enough - though what format of the search string to use to optimize it, I'm not sure. The event might contain several such barcodes, and the "main" one is actually the one from that combined tag I mentioned earlier. I can't predict whether that barcode will appear as barcode1, barcode2 or barcode3 (the actual field names are different, but you get the idea), so even that is not so straightforward.
However, there is a bigger problem: one of the search terms our application must have is a track ID, which is a trailing part of the barcode, extracted by some special rules (which I can easily put into `eval` command, if necessary) and never appearing directly in the raw event. For example, we might have a barcode 310200549315, which will appear in barcode2 (barcode1 and barcode3 will have different values and be of no interest to our application) and be a part of the combined_tag=PRD015310200549315. The track ID, as a suffix of the barcode, might have a value of 49315, which is always only a part of a token inside of the event.
Even if I succeed in creating a calculated field track_id, searching on it will be excruciatingly slow. And searching on *49315 (continuing my previous example) will be no better.
My question is: how would you attack this problem? I'm ready to create an index-time track_id field, but can I put an EVAL somewhere in props/transforms to achieve that? Alternatively, is there a way to optimize the search for such situations?
↧