Just recently setup smartstore in a test environment using a single index and I'm trying to figure out some details on the bucket lifecycle.
So far I know that hot buckets are stored locally and when they roll to warm, some that are in use or reserved stay local, and then warm buckets are uploaded to the S3 remote storage (AWS in my case). When a search is performed and the data needed is in a warm bucket in remote storage, it is copied from S3 to the indexer's cache manager and gets evicted when no longer needed.
From what I've seen in their documentation, smartstore warm buckets in S3 will be frozen and if they are not being locally archived with the coldToFrozenDir setting, then they will be deleted. As I got further into the documentation, they say contradicting things like the process is only hot to warm, or the warm buckets will roll to frozen to be deleted locally and from remote storage, or warm buckets on remote storage will roll from warm to frozen and archived on the indexer if archiving is setup.
I don't want to send data to remote storage and then bring it back onto the indexer for archiving locally. To prevent freezing/deletion, I haven't configured any frozen time or max size settings.
In the following page for smartstore indexing, it says Hot and Warm only and then randomly says, "Buckets roll to frozen directly from warm" without mentioning anything else in the page about freezing buckets which made me think that it's automatic for smartstore. https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/SmartStoreindexing
1. With the following example indexes.conf, when the buckets roll from hot to warm and are uploaded to AWS S3 , will the splunk ever automatically roll it to frozen?
[default]
#daily hot to warm index transition; seconds
maxHotSpanSecs=86400
[volume:s3]
storageType = remote
path = s3://
[smartstore_test]
repFactor=auto
homePath = $SPLUNK_DB/$_index_name/db
coldPath = $SPLUNK_DB/$_index_name/colddb
thawedPath = $SPLUNK_DB/$_index_name/thaweddb
remotePath = volume:s3/$_index_name
2. If warm to frozen is inevitable, can smartstore manage archiving to the S3 bucket?
If you have SmartStore configured and have any suggestions or examples of things that you wish you knew when you first set it up that you would like to share, I would greatly appreciate the input.
Thank you ahead of time and sorry for the long post.
↧