Digging through the new stuff in 6.3 in preparation for some upgrades, I see LZ4 compression is available for bucket rawdata journal compression in indexes.conf. Awesome! I'm excited. Splunk bucket data seems like it should be a great fit for LZ4's strengths.
But LZ4 should also incur a measurable hit on storage needs over gzip, and algorithm benchmarks often focus on specific interesting data cases or a broad set of varying data types. Splunk's intake focus is pretty narrow by comparison, so I'm curious to see if anyone has any real-world numbers to throw down yet, since changing to LZ4 should change the calculations for capacity planning.
↧