We are currently running out of space in one of the splunk indexer instance, out of 5 indexer instances and ours is a distributed environment. Using Splunk 6.2.1 Version.
Total size of the indexer volume is about **5.2TB** and currently we are left out with less then 100 GB of space and everyday on an average of 10GB of space is occupied The data which are occupying space are almost **3.5 year old data**. and most of the data are present under the **colddb storage** unit under the mount point /splogs.
**Disk Usage status**
df -h /splogs
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_splunk03_san-splunk_logs
5.6T 5.3T 93G 99% /splogs
we could find most of the space is occupied by these indexes.
[net_proxy], [net_fw], [unix_svrs] & [unix_bsm]
example :-
[root@splunk03 splogs]# cd unix_svrs
[root@splunk03 unix_svrs]# ls -ltr
total 416
drwx------ 2 splunk splunk 4096 Apr 19 2012 thaweddb
drwx------ 1590 splunk splunk 102400 Aug 6 09:18 colddb
drwx------ 1890 splunk splunk 131072 Aug 6 12:51 summary
drwx------ 1893 splunk splunk 143360 Aug 6 12:53 datamodel_summary
drwx------ 307 splunk splunk 28672 Aug 6 12:54 db
[root@splunk03 unix_svrs]# du -sh *
**1007G colddb**
1.6G datamodel_summary
229G db
366M summary
4.0K thaweddb
[root@splunk03 splogs]# cd net_fw
[root@splunk03 net_fw]# ls -ltr
total 612
drwx------ 2 splunk splunk 4096 Apr 19 2012 thaweddb
drwx------ 1358 splunk splunk 131072 Sep 27 2015 summary
drwx------ 2956 splunk splunk 180224 Aug 6 12:17 colddb
drwx------ 3258 splunk splunk 266240 Aug 6 12:55 datamodel_summary
drwx------ 313 splunk splunk 28672 Aug 6 12:55 db
[root@splunk03 net_fw]# du -sh *
**1.3T** colddb
76G datamodel_summary
147G db
24M summary
4.0K thaweddb
Indexes.conf details for these indexers
[volume:Hot]
path = /splogs
[volume:Cold]
path = /splogs
[volume:Base]
path = /splogs
[default]
frozenTimePeriodInSecs = 31536000
[net_fw]
homePath = volume:Hot/net_fw/db
coldPath = volume:Cold/net_fw/colddb
tstatsHomePath = volume:Hot/net_fw/datamodel_summary
thawedPath = $SPLUNK_DB/net_fw/thaweddb
maxTotalDataSizeMB = 250000
[unix_svrs]
homePath = volume:Hot/unix_svrs/db
coldPath = volume:Cold/unix_svrs/colddb
tstatsHomePath = volume:Hot/unix_svrs/datamodel_summary
thawedPath = $SPLUNK_DB/unix_svrs/thaweddb
maxTotalDataSizeMB = 250000
[summary]
frozenTimePeriodInSecs = 188697600
There are other indexers, configured in the same manner as shown above in Indexes.conf.
Kindly let me know whether we can delete the data that are present under the **colddb directory** for the indexer occupying more than 1TB, by doing this what will be the impact. Or is there any other method we can prevent the failure of splunk service due to low disk space.
↧