We have about a 3 TB/day ingest rate, spread across about 20 indexes, and we have a 2 to 5 year retention time depending on the index. 4 of these indexes account for 90% of the data, and we are expecting to approximately double or triple our ingest rate eventually. We currently have out maxHotSpanSecs = 86400, maxHotBuckets = 10, and quarantine(Past|Future)Secs = 86400. I have some questions regarding the best practices for setting up buckets and wanted to get feedback on our current settings.
1. Is it better to have each maxHotSpanSecs for the large indexes be something like 24 hours, or should they be larger, such as 7 days or more? What is the reasoning for this?
2. How many hot buckets should be set for a large index? Does it make a difference vs. a small index?
3. What are good settings for quarantine buckets? What could be the negative impacts of setting this to 86400?
4. How are quarantine buckets rolled over to warm/cold? Is it based on the earliest event in the index?
↧