I recently added several servers to our splunk system, and they are all reporting as `sourcetype=linux_audit` (Which I do not believe is overridden from something else)
Looking at the logs, I am pretty sure they are from redhat (or similar), as the log looks like [(this)][1]:
type=SYSCALL msg=audit(1364481363.243:24287): arch=c000003e syscall=2 success=no exit=-13 a0=7fffd19c5592 a1=0 a2=7fffd19c4b50 a3=a items=1 ppid=2686 pid=3538 auid=500 uid=500 gid=500 euid=500 suid=500 fsuid=500 egid=500 sgid=500 fsgid=500 tty=pts0 ses=1 comm="cat" exe="/bin/cat" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="sshd_config"
But, when I go through my logs, I see that any log from this sourcetype is using a default timestamp generated by splunk, and I have about a million logs indexing in a single second at the beginning of a minute.
Looking into it further, I see that splunk is not even trying to parse the timestamp `msg=audit(1364481363.243:24287)` (No "failed to parse timestamp" errors.
The rest of the message seems to be parsing correctly -- all of the "key=value" pairs are showing up in verbose mode. But the msg=audit is showing up in "msg" and not as a timestamp. Being a RHEL log, it seems to be something that splunk would automatically identify, but I don't even see a "linux_audit" sourcetype in the pretrained sourcetypes.
http://docs.splunk.com/Documentation/Splunk/7.1.2/Data/Listofpretrainedsourcetypes
What can I do from here to nudge these logs back into automatically parsing? Is this a situation where I need to override the sourcetype with some other syslog? (Again, I see no "msg=audit(unixtimestamp)" in the pretrained sources)
[1]: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security_guide/sec-understanding_audit_log_files
***Further investigation shows:***
A few logs are coming in just fine. Same host, same source, same sourcetype. But them BAM. A huge influx of non-parsed timestamps. I see nothing different.
↧