Hi Splunkers,
We are going to migrate our current single-site indexer cluster (running 4 nodes, with replication factor: 2 and search factor: 2, multiple TB or raw data) to new multi-site cluster on 2 data centers.
Currently, the cluster is running fine on these 4 nodes, and a we have a new set of 4 physical servers ready, so we will migrate from single-site to multi-site and also migrate from old servers to new servers.
The current cluster has several TB of raw data from almost every possible type, structured, unstructured, with a low retention (from 1 month to several months) and very high retention (from 1 year to several years)
The final multi-site configuration must imperatively be able to respect an high SLA level, with data center recovery plan compliance.
I am looking for advices, real world feedbacks to build the better scenario possible for this migration to be a success with the lowest level possible of operation, which is why i am asking today.
First, we are aware that it is (unfortunately) not possible to migrate single-site buckets that have non site origin guId to multi-site clusters. (http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Migratetomultisite)
That is something really missing, i hope Splunk will one day implement this...
"The cluster will never create a new copy of the bucket on a non-origin site."
What could be the better approach for the migration scenario ?
**We have some ideas, but i would very appreciate any help and interesting advises:**
**Scenario 1: Perform a standard migration, let the natural retention solves the single-site bucket and manually export / import data for indexes with an high retention**
*The first scenario would be the following:*
- Integrate the 4 new physical servers to the current single-site cluster
- Move indexing flow from old nodes to the new multi-site cluster
- One by one, decomission old indexer nodes and remove them for the cluster
- Convert the single site-cluster to a multi-site cluster with 2 x nodes on each site
- Export data for each index with an high retention policy, and re-index these data sets in a new index of the cluster (such that buckets be multi-sites)
- Remove migrated indexes
- Wait the required time to have the retenton policy deletes buckets with non origin identification
- Ensure we have node mode single site buckets
- Issue data disaster recovery plan tests
Manually exporting data, and re-indexing it is a pain, and will costs time and money...
But this looks like the better approach, any idea ?
**Scenario 2:**
A other migration scenario would be:
- Build the new multi-site cluster
- Set our search head cluster to address both single-site cluster and multi-site cluster (not even sure it is possible ?)
- Move indexing flow from old nodes to the new multi-site cluster
- Back up unique buckets from the single-site cluster
- Restore in 1 node of the Site 1, and 1 node of the Site 2 (search that we have at least one copy of every data on each site, including non origin buckets)
- Set the search head cluster to search only on the new multi-site cluster
- Decommission the old single-site cluster
This scenario could be reliable too (if we success in backing up / restoring !), but the bad thing here is that non origin buckets will be never be managed by the cluster if a am not wrong.
If we someday need to migrate on of nodes that were restored with non origin buckets, we will have manually migrates these buckets which is not really compliant with a cluster philosophy...
And finally, in case of unavailability of non restored nodes on each sites, past data would not be available
**Scenario 3:**
*Finally we could export / re-index every piece of data from the old single-site cluster to the new cluster.*
This looks the "cleaner" solution as every bucket will be a multi-site bucket.
- Build the new multi-site cluster
- Move indexing flow from old nodes to the new multi-site cluster
- Set the search head cluster to address the new multi-site cluster
- Export every piece of data
- Re-index
- Decommission the old single-site cluster
Et voila :-)
This would be magic... but i have serious doubts about exporting multiple TB of heterogeneous raw data, and re-indexing it with respect of every application specificity like metadata rewriting and so on...
I do not personally know strong and industrial ways to export / re-index data for high volume and high data complexity...
Thank you in advance for sharing your advice, opinion, feedbacks, and even better solutions !
I may be wrong in some points, don't hesitate to tell :-)
Thanks
Guilhem
↧