First off I have looked over the instructions contained here:
http://docs.splunk.com/Documentation/AddOns/latest/BlueCoatProxySG/About
That being said, the biggest issue is that I have been working with the understanding that feeding syslog directly into Splunk (e.g. monitor entry on port 514) is not the way to go since every time you do a restart of the Splunk service you are going to lose logs, and the lost is generally however long of a time period that the service is restarting (anywhere from 1 minute to 10 minutes... yeah... it happens).
So I have rsyslog (version 5, thanks Red Hat for lacking in the latest versions) and have set up the instructions on that to handle the logs. Because of the way that BlueCoat dumps the logs as one giant flat file (essentially) and doesn't really export their access logs in a traditional syslog fashion, this required the following template on my rsyslog:
$template bluecoat, "/opt/log/%fromhost%/syslog.log"
Just a side bar here, since rsyslog documentation isn't the greatest, and it took a very long time to figure some things out, here is all the relevant pieces from my configuration to get rsyslog to work:
#### MODULES ####
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog # provides kernel logging support (previously done by rklogd)
#$ModLoad immark # provides --MARK-- message capability
$ModLoad imtcp
#### GLOBAL DIRECTIVES ####
# Needed to allow proper permissions for creating files
$umask 0000
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
#### Templates ####
# log every host in its own directory
$template splunklog, "/opt/log/%HOSTNAME%/%PROGRAMNAME%.log"
$template splunkmsg, "%rawmsg%\n"
#bluecoats aren't cooperating at all using this in the meantime
$template bluecoatlog, "/opt/log/%fromhost%/syslog.log"
# ## BlueCoat Ruleset ##
$RuleSet bluecoat
# Force permissions to work with Splunk as files are created
$DirOwner root
$DirGroup splunk
$FileOwner root
$FileGroup splunk
$DirCreateMode 0750
$FileCreateMode 0640
# Log everything coming in through the network (see above template)
*.* ?bluecoatlog;splunkmsg
# ## End BlueCoat Ruleset ##
# BlueCoat
$InputTCPServerBindRuleset bluecoat
$InputTCPServerRun 1514
I then set a monitor path
[monitor:///opt/log/bluecoat.mydomain.com/]
disabled = 0
host_segment = 3
index = bluecoat
sourcetype = bluecoat:proxysg:access:syslog
whitelist = syslog\.log$
Now, you would think this minor change, writing to a file before reading it into Splunk would cause a problem, but lo and behold I'm having to major issues:
1: My log rotation script breaks the log writing. No it isn't a permission issue (or shouldn't be) because it works on my other syslog data types. I am currently running the following logrotate every hour:
/opt/log/*/*.log {
missingok
create 0640 root splunk
rotate 3
sharedscripts
compress
postrotate
reload rsyslog >/dev/null 2>&1 || true
endscript
}
This is controlled by an hourly cron job such that I can keep around 3 hours of logs in case something happens. The end state would be to eventually up this to around 2 days. Anyway, the issue is that it compresses the file, and then successfully makes a new file "syslog.log" like it is supposed to, but it seems like the reload isn't quite working since at that point I just stop getting logs unless I manually do a "service rsyslog restart". Note that a service restart is much more harsh than a reload and you run the risk of losing a little bit of logs during the period that it blows away your connections (if TCP) or that logs just went to the abyss (UDP).
2: Splunk is not decoding the file correctly at all. It is just breaking each event on the line break instead of breaking them where they should be. Note that each line break is basically coming in as each packet, so the way this dumps you get multiple events per packet transmitted and it doesn't break each packet into individual events.
Anyone else have any luck setting BlueCoat up this way?
↧