I setup 3 DB inputs as part of our requirement to ingest DB logs. Problem is we're encountering duplicate events upon ingesting data to splunk. I can observe the duplicate events when there's no available data to be fetched from DB, Splunk DB connect up still use the last checkpoint value and keeps re-ingesting the data until it fetch new data in DB.
Here's sample internal logs for how checkpoint value generates in 1 of the DB setup. Any idea where the issue and how to fix it.
$SPLUNK_HOME/var/lib/splunk/modinputs/server/splunk_app_db_connect
{"value":"2020-03-02 00:42:39.307","appVersion":"3.1.4","columnType":93,"timestamp":"2020-03-02T11:45:00.019+11:00"}
{"value":"2020-03-02 00:47:17.17","appVersion":"3.1.4","columnType":93,"timestamp":"2020-03-02T11:50:00.084+11:00"}
{"value":"2020-03-02 00:58:00.783","appVersion":"3.1.4","columnType":93,"timestamp":"2020-03-02T12:05:00.164+11:00"}
{"value":"2020-03-02 00:58:00.783","appVersion":"3.1.4","columnType":93,"timestamp":"2020-03-02T12:15:00.018+11:00"}
{"value":"2020-03-02 00:58:00.783","appVersion":"3.1.4","columnType":93,"timestamp":"2020-03-02T12:20:00.624+11:00"}
{"value":"2020-03-02 00:58:00.783","appVersion":"3.1.4","columnType":93,"timestamp":"2020-03-02T12:25:00.017+11:00"}
{"value":"2020-03-02 00:58:00.783","appVersion":"3.1.4","columnType":93,"timestamp":"2020-03-02T12:35:00.423+11:00"}
{"value":"2020-03-02 01:37:59.38","appVersion":"3.1.4","columnType":93,"timestamp":"2020-03-02T12:50:00.108+11:00"}
Thanks.
↧