Following a server crash (unknown reason) the collection of S3 logs has partially broken down. Seeing the following in _internal (after enabling DEBUG logging for the S3 generic part).
Any ideas on how to get it working again?
2018-09-03 14:38:22,122 level=INFO pid=15104 tid=Thread-4 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:104
| datainput="securitylogs_blahblah_staging" bucket_name="blahblah-staging-ext-securitylogs-20180215145qqqqqqq00000001", job_uid="757azzzz-xxxx-4a5b-b735-yyyyy31754e" start_time=1535978302
| message="Start processing."
2018-09-03 14:38:22,123 level=DEBUG pid=15104 tid=Thread-4 logger=splunksdc.checkpoint pos=checkpoint.py:build_indexes:169
| datainput="securitylogs_blahblah_staging" bucket_name="blahblah-staging-ext-securitylogs-20180215145qqqqqqq00000001", job_uid="757azzzz-xxxx-4a5b-b735-yyyyye31754e" start_time=1535978302
| message="Key was set." pos=221 key="securitylogs/ls.s3.2cf5f2c6-xxxx-4947-bb01-yyyyy96c7e22.2018-08-01T22.30.part11921.txt.gz"
some more of the above DEBUG message - different key, but same Job_uid, and then it ends with:
2018-09-03 14:38:22,123 level=ERROR pid=15104 tid=Thread-4 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:index_data:90
| datainput="securitylogs_blahblah_staging" bucket_name="blahblah-staging-ext-securitylogs-20180215145qqqqqqq00000001"
| message="Failed to collect data through generic S3." job_uid="757azzz-xxxx-4a5b-b735-yyyyy31754e" start_time=1535978302 Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_data_loader.py", line 85, in index_data self._do_index_data()
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_data_loader.py", line 106, in _do_index_data self.collect_data()
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_data_loader.py", line 139, in collect_data index_store = s3ckpt.S3IndexCheckpointer(self._config)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_checkpointer.py", line 141, in __init__ config[asc.data_input], config[tac.checkpoint_dir]
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_checkpointer.py", line 80, in get S3CkptPool.S3CkptItem(ckpt_name, ckpt_dir)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_checkpointer.py", line 65, in __init__ self.idx_ckpt = LocalKVStore.open_always(ckpt_file_idx)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunksdc/checkpoint.py", line 156, in open_always indexes = cls.build_indexes(fp)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunksdc/checkpoint.py", line 163, in build_indexes for flag, key, pos in cls._replay(fp):
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunksdc/checkpoint.py", line 92, in _replay flag, key, _ = umsgpack.unpack(fp) TypeError: 'int' object is not iterable
Something seems to have gotten corrupted. Have tried to disable/enable the inputs from within the AWS-addon-app-GUI, but with little success.
↧