I created a custom namespace in Amazon CloudWatch and have data in it that I want to get into Splunk via the Splunk add-on for Amazon Web Services. I managed to do so, but Splunk only got new data from CloudWatch (generated after the Splunk input was running) and none of the historical data from the cloudwatch bucket (existing prior to using Splunk). I have also noticed that it is very flakey and sometimes gets new data, but not always.
For example, I've been waiting 2 hours and Splunk hasn't picked up the new data I generated. The data appeared in cloudwatch correctly so I know it is available. Splunk was getting data from the namespace previously (last week), and I haven't changed anything in the input or configuration of the Splunk Add-on. The input also collects data from the AWS/EC2 namespace in Splunk and during the two hours that I have been waiting, the input has ran, and picked up data for the AWS/EC2 bin from cloudwatch, so I know that the input is running.
I was under the impression that Splunk would gather all of the information from AWS Cloudwatch, historical and new, and that Splunk could be used for real-time monitoring. So I have 2 questions:
1. Does Splunk / Splunk add-on for Amazon Web Services normally retrieve historical data that existed before using Splunk services? If so, is there something that needs to be configured or enabled to do this?
2. Is there a reason that Splunk add-on for Amazon Web Services would only **sometimes** pick up data and ignore others? What can I do to resolve this and why isn't it real time?
↧