Hi Splunkers,
I am monitoring a folder (/opt/pvlogs/QUT-GP-P10) with a collection of CSV text files, as follows:
....
int_magnetek_151019.txt
int_magnetek_151020.txt
int_magnetek_151021.txt
int_magnetek_151022.txt
int_magnetek_151023.txt
int_magnetek_151024.txt
int_magnetek_151025.txt
int_magnetek_151026.txt
int_magnetek_151027.txt
....
Log file format example:
[info]
Anlage=QUT P BLOCK LEVEL 10
Datum=151027
[messung]
;s;Adr;WR;MPC;S_GL;S_WR;S_DC1;S_DC2;S_AL;UDC1;IDC1;UDC2;IDC2;UAC;IAC;PAC;FAC;T_WR1;T_WR2;I_LC;R_ISO;E_TOTAL;E_INT;P_LIMIT;COS_PHI
;s;;;%;;;;;;V;A;V;A;V;A;W;Hz;°C;°C;A;MOhm;kWh;Wh;;
[Start]
00:15:00;900;2;PVI-10.0-OUTD;;;;;;;;;;;;;;;;;;;;;;
00:15:00;900;3;PVI-10.0-OUTD;;;;;;;;;;;;;;;;;;;;;;
00:15:00;900;4;PVI-6000-OUTD;;;;;;;;;;;;;;;;;;;;;;
....
The configuration settings:
**inputs.conf**
[monitor:///opt/pvlogs/QUT-GP-P10/*.txt]
disabled = false
index = test
sourcetype = sec_pv_data
host = QUT-GP-P10
crcSalt =
**props.conf**
[sec_pv_data]
SHOULD_LINEMERGE=false
HEADER_FIELD_LINE_NUMBER=5
HEADER_FIELD_DELIMITER=;
SEDCMD-null=s/\[Start\]|\[info\]|\[messung\]|Anlage.*|Datum.*|Info.*|;Time.*|;s;.*//
FIELD_DELIMITER=;
The log files are created by a solar PV logger that updates the file (located on the actual logger device) with a new entry at 15 minute intervals. At midnight every night, the log file for that day is copied from the logger device, to the monitored folder. The forwarder should then detect the new file and forward the data to the indexer.
The file copy is achieved by a cron job that uses curl to connect to the logger device URL and pull down the new log file into the folder monitored by splunk. The problem is that the forwarder does not detect this new file written into the monitored folder. However, if I restart the Splunk service (with no other changes), the forwarder then ingests the previously unindexed log file/s.
In **splunkd.log**, there is an ERROR that seems to be related to this issue:
10-27-2015 00:15:04.961 +1000 ERROR WatchedFile - Bug during applyPendingMetadata, header processor does not own the indexed extractions confs.
10-27-2015 00:15:04.962 +1000 ERROR TailReader - Ignoring path="/opt/pvlogs/QUT-GP-Y11/int_magnetek_151027.txt" due to: Bug during applyPendingMetadata, header processor does not own the indexed extractions confs.
Does anyone have any information about this error, or any advice on why my forwarder does not automatically detect new files, copied into the monitored folder (using curl)?
Also, why would restarting the service then allow the indexer to ingest the file?