A little background:
- We have a single instance install now and are moving to a multisite index cluster and multisite SH cluster.
- Our dashboards on the current single instance are 95% inline searches, with a few summary index panels and saved searches in the mix. Very small amount of those.
- No real-time searches.
- The dashboards have on average 15 panels that are refreshed every 2 minutes (and yes we need it refreshed that quickly).
I have been thinking of rewriting the searches to be base and post-processes. However as I have been reading about the changes needed in the dashboards when in the new environment, I have read some comments and articles where post-processing may not be suitable for multi-index sites. Doing that might be slower than saved searches.
Does anybody have any real-world experiences that they can share? I don't want to proceed down one path when the other might be better in the long run.
↧
Which is faster on a dashboard, post processing or using saved searches
↧
redirect logs to seperate index based on Host
Hi.. I have a question on
From a heavy forwarder , based on the incoming host, I like to send the logs into a separate index.
Currently I configured to send the logs from a single heavy forwarder into a separate indexer based on the Environment variable we are passing with the logs(eg environment = cert/Dev/test/prod),Which is working fine.
On the top of that, I like to send the logs into a separate index within the indexer based on the host. here is Our conf files
**Inputs.conf**
-------------
[default]
host=Heavyfw.aaaa.com
[udp://514]
index=main
sourcetype=syslog_log
connection_host=ip
**props**.conf
----------------------------
[syslog_log]
LINE_BREAKER = (\s+)\{|\n
SHOULD_LINEMERGE = false
MUST_BREAK_AFTER = ([\r\n]+)[A-Z]+\s+\|\s+\d+
TRANSFORMS-test=test
TRANSFORMS-cert=cert
TRANSFORMS-dev=dev
TRANSFORMS-prod=prod
TRANSFORMS-local=local
TRANSFORMS-null=null
[host::(198.*)]
TRANSFORMS = rewrite-DPindex
**transforms.conf**
_____________________
[cert]
REGEX=(?i)["]?environment["]?\s*[:=]\s*["]?(cert)["]?
FORMAT=cert
DEST_KEY=_TCP_ROUTING
[local]
REGEX=(?i)["]?environment["]?\s*[:=]\s*["]?(local)["]?
DEST_KEY=_TCP_ROUTING
FORMAT=local
[test]
REGEX=(?i)["]?environment["]?\s*[:=]\s*["]?(test)["]?
DEST_KEY=_TCP_ROUTING
FORMAT=test
[null]
REGEX=^default send string$
FORMAT=nullQueue
DEST_KEY=queue
[dev]
DEST_KEY=_TCP_ROUTING
FORMAT=dev
REGEX=(?i)["]?environment["]?\s*[:=]\s*["]?(dev)["]?
[prod]
FORMAT=prod
DEST_KEY=_TCP_ROUTING
REGEX=(?i)["]?environment["]?\s*[:=]\s*["]?(prod)["]?
[rewrite-DPindex]
REGEX = .*
DEST_KEY = _MetaData:Index
FORMAT = Newindex
**outputs.conf**
__________________
[tcpout:Prodserv1]
server=ProdServer.aaaa.com:9997
[tcpout:cert]
server= CertServer.aaaa.com:9997
[default]
defaultGroup=default
[tcpout:test]
server= testServer.aaaa.com:9997
[tcpout:dev]
server= dev.Server.aaaa.com:9997
[tcpout:default]
server= testServer.aaaa.com:9997
[tcpout:prod]
server= ProdServer.aaaa.com:9997
[tcpout:local]
server= ProdServer.aaaa.com:9997
Now all the logs coming from the host 198.* going to Newindex and also to main index. How can I stop going to main index and send only to Newindex .
↧
↧
transaction grouping with startwith and endwith
log format:
start: A
End: A
start: B
End: B
Start: C
Start: D
End: C
End: D
Start:E
End:F
Query I am using:
| rex field=_raw "END: (?.*)"
| transaction processEndName startswith="Start:" endswith="[stdout] END:" keeporphans=false keepevicted=false
| sort -duration
| head 10
| stats avg(duration) by processEndName
In the return:
A and B can be group together correctly as:
startA
endA
startB
endB
But C and D are messed up:
startD
endC
also E only has starts, F only has ends, but their process name is different, they got grouped together as well:
startE
endF
Is there a way to properly group C/D and exclude E and F?
Thanks in advance!
↧
How to list only distinct values from the listed results
Hi I have a query which runs and results me the list of Ip's in a table format grouped by username.
In my table of results there might be different IP's for the same username which are listed down in the single IP cell. Please find below the example of my result table:
Username-------------------------------------------Ipaddress------------------------application---------------------------city-----------------------------country
1) abcd--------------------------------------------------123.123.123.12---------------------xyz---------------------------------asdf-----------------------------zxcvb
123.123.123.12 xyz asdf zxcvb
234.456.677.22 ghj ghjk fghjk
2) dfgh--------------------------------------------------234.123.12.345----------------------ssss------------------------------dfggh----------------------------ghjhjkk
As shown above for one username there will be list of ip's and corresponding city and country info are displayed. What i want to achieve here is that I need to display only distinct ip's for each username. How can I do it?
To display my results in above table I am using the following search:
mysearch
| iplocation clientip1
| streamstats count as occuranceCount list(clientip1) as client_IP, list(applicationid) as application list(Country) as Country, list(City) as City by subject
| sort - occuranceCount
| dedup subject
| table subject occuranceCount client_IP connectionid City Country
Please help!
Thanks in advance
↧
Delete PII from Splunk
We're using Splunk for logging from multiple applications. Some of these applications deal with PII data.
If one of those applications puts the PII data in Splunk, how can it be removed or deleted?
I need to remove the data from the machine and Splunk.
↧
↧
Java API query syntax failure
I''m using Splunk 6.6.3, Java API 1.6.4.0, Java 1.8.0_45, IntelliJ IDE.
I'm making part of a simple application that checks that a given system is actively logging, where the sourceType, hostname, and minutes back from present are being read from a database and become part of the query.
An equivalent search query that works as expected in Splunk GUI, with time set as "Last 60 minutes" would be:
sourcetype=WinEventLog:Security host=abcxyz | head 1
I'm working from the examples provided, but none seem to show multiple arguments i.e. sourcetype, host, time range. In the code below, if I set:
String mySearch = "search host="+ lsb.getSystem() + " "; // just a host String
It will work for at least some hosts.
If I try to add the sourcetype, all will fail:
String mySearch = "search sourcetype=WinEventLog:Security host="+ lsb.getSystem() + " ";
Note: In the code below, the method minutesBackString() returns a String like: "2018-03-27T12:53:46.626-04:00"
Can someone suggest a combination that will give the equivalent result of the GUI search? Ideally I would specify the field list, but I can get by without that. Any suggestions very much appreciated. Please Ignore the boolean return for now - it will be dependent on the content returned by the query.
private boolean oneSystem(LoggingSystemBean lsb) {
boolean retval = false;
String mySearch = "search sourcetype=WinEventLog:Security host="+ lsb.getSystem() + " "; // lsb.system is String
JobArgs jobargs = new JobArgs();
jobargs.setExecutionMode(JobArgs.ExecutionMode.NORMAL);
jobargs.setEarliestTime(minutesBackString(60));
jobargs.setLatestTime(minutesBackString(0));
jobargs.setMaximumCount(1);
Job job = service.getJobs().create(mySearch, jobargs);
try {
while ( !job.isDone() ) {
Thread.sleep(500);
}
} catch (InterruptedException ie) {
}
// Display results
InputStream results = job.getResults();
String line = null;
try {
BufferedReader br = new BufferedReader(new InputStreamReader(results, "UTF-8"));
while ( (line = br.readLine()) != null ) {
System.out.println(line);
}
br.close();
} catch (Exception ex) {
errLog.severe(ex.getMessage() + "\n" + ExceptionUtils.getStackTrace(ex));
}
return (retval);
}
↧
Filtering on multiple times
I have a search with two indexes.
The first index needs to be run on all time, but I have a field within that index that I would like to filter on (it contains a date/time field as a string), I have another field within that index that I would also like to filter on (it also contains a date/time field as a string).
index=BASE earliest=0
| eval LPR = strptime(LastPolicyRequest, "%m/%d/%Y %I:%M:%S")
| where LPR >= "-30d@d" AND LPR <= "now"
| eval now_epoch = now()
| eval valid_to_epoch = strptime(ValidTo, "%m/%d/%Y %I:%M:%S %p")
| where valid_to_epoch < now_epoch
| table "WiFiMAC","LastPolicyRequest","ValidFrom","ValidTo"
| rename WiFiMAC as hdr.macaddress
![alt text][1]
Today is 3/27/18, yet getting results for 2/24/18
The query that I am joining needs to be set for 2 days
index=BASE earliest =-48h
| eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S")
| eval Indextime =strptime(indextime,"%Y-%m-%d %H:%M:%S.%N")
| eval Time =strptime(_time,"%s")
| eval Minutes_Diff = round((Indextime - Time)/60,2)
| stats avg(Minutes_Diff) as Avg_Minutes_Diff stdev(Minutes_Diff) as StDev_Minutes_Diff min(Minutes_Diff) as Min_Minutes_Diff max(Minutes_Diff) as Max_Minutes_Diff count as count by hdr.macaddress hdr.locId hdr.deviceModel hdr.firmwareVersion
| eval Avg_Minutes_Diff = round(Avg_Minutes_Diff,2)
| rename count as "Sample Size"
| table "hdr.macaddress", "Avg_Minutes_Diff", "StDev_Minutes_Diff", "Min_Minutes_Diff", "Max_Minutes_Diff", "Sample Size"
![alt text][2]
Here is the joined search:
index=BASE earliest=0
| eval LPR = strptime(LastPolicyRequest, "%m/%d/%Y %I:%M:%S %p")
| where LPR >= "-30d@d" AND LPR <= "now"
| eval now_epoch = now()
| eval valid_to_epoch = strptime(ValidTo, "%m/%d/%Y %I:%M:%S %p")
| eval twenty_days_epoch=relative_time(now(), "-20d@d")
| where valid_to_epoch < now_epoch
| table "WiFiMAC","LastPolicyRequest","ValidFrom","ValidTo"
| rename WiFiMAC as hdr.macaddress
| join type=left hdr.macaddress
[ search index=BASE earliest =-48h
| eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S")
| eval Indextime =strptime(indextime,"%Y-%m-%d %H:%M:%S.%N")
| eval Time =strptime(_time,"%s")
| eval Minutes_Diff = round((Indextime - Time)/60,2)
| stats avg(Minutes_Diff) as Avg_Minutes_Diff stdev(Minutes_Diff) as StDev_Minutes_Diff min(Minutes_Diff) as Min_Minutes_Diff max(Minutes_Diff) as Max_Minutes_Diff count as count by hdr.macaddress hdr.locId hdr.deviceModel hdr.firmwareVersion
| eval Avg_Minutes_Diff = round(Avg_Minutes_Diff,2)
| rename count as "Sample Size"
| table "hdr.macaddress", "Avg_Minutes_Diff", "StDev_Minutes_Diff", "Min_Minutes_Diff", "Max_Minutes_Diff", "Sample Size" ]
| table "hdr.macaddress","Avg_Minutes_Diff", "StDev_Minutes_Diff", "Min_Minutes_Diff", "Max_Minutes_Diff","ValidFrom","ValidTo","LastPolicyRequest", "Sample Size"
| dedup hdr.macaddress
![alt text][2]
Any Ideas as to what I am doing wrong?
[1]: /storage/temp/236573-screen-shot-2018-03-27-at-12659-pm.png
[2]: /storage/temp/236576-screen-shot-2018-03-27-at-14754-pm.png
↧
useing metadata commend to display sourcetype host and sources at the same time
HI
I want to use | metadata commend to display sourcetype host and sources at the same time, so far I cant make connection between them.
As we know when I run | metadata type=sourcetypes search it will return me sourcetype information,like below
firstTime lastTime recentTime sourcetype totalCount type
151572 1515399 152170 RT2RO 108 sourcetypes
the output I am looking for is
firstTime lastTime recentTime sourcetype totalCount source host
151572 1515399 152170 RT2RO 108 \var\log\a rt2.server.com
Can this be done using | metadata command?
The reason I want to use it is just because it give result fast :)
Thanks in advance
↧
Splunk CyberArk and Syslog-NG
We're sending CyberArk Vault data to Splunk via a syslog-ng server. We have a number of sources going to the syslog server. CyberArk logs are delayed from writing to the directory by over an hour. The other issue is Splunk_TA_CyberArk doesn't appear to be splitting up the vault log files like it should. CyberArk is running 9.10. Splunk is running 7.0. Any help would be appreciated.
↧
↧
monitoring freeradius raddact files
Hello,
I'd like to monitor raddact files. I have the following config in inputs.conf.:
[monitor:///var/log/freeradius/radacct]
recursive = true
sourcetype = syslog
The main problem is about line-breaking. I'd like to the whole event, but I get one event per line. How can i configure props.conf to see the whole event? It is possible recursive on the whole directory as well ?
Thanks,
↧
getting Splunk, CyberArk, and Syslog-ng working together
We setup a centralized syslog-ng server. We are receiving logs from multiple different sources without issue. The logs from CyberArk Vault server are the only logs delayed an hour or more with no line breaks for new lines. The Splunk CyberArk TA is installed on the syslog server and the search head. It does not appear logs are being parsed when sent to Splunk as expected. Any help would be appreciated.
↧
splunk universal forwarder batch input forwarding but not deleting
Hi, we have an indexer cluster, to which we index many many small files.
we have about a few hundreds thousand files.
we run a universal forwarder on a strong machine(130GB 24CPU) and have a batch input on local directory.
our problem is as follows:
the data is indexed very slowly, and also the batch input is freaking a little....
it used to write logs about every indexed file("Batch input finished reading file..."), but now it writes a few, than stops, than continue to forward data but doesn't delete the files.
the only log we can see is when we turn on DEBUG level logging.
I have checked the logs and I dont have any blocked queues.
We would really appreciate if anyone would either have a reasonable explanation for the problem i'm having, or if someone will be able to suggest another way of indexing this immense amount of files.
↧
Symantec Cloud Scripted Input
Evaluating Symantec EndPoint Protection Cloud product which has a technote for getting events into Splunk Enterprise running on a Windows Server.
Created a scripted input per the Symantec Technote
[Symantec Technote][1]
however I get the following error in SPLUNKD.log
______________
ERROR ExecProcessor - Couldn't start command ""C:\Program Files\Splunk\bin\scripts\wrapper.sh"": FormatMessage was unable to decode error (193), (0xc1)
_______________
The scripted input uses a wrapper (wrapper.sh) for calling a python script. Contents of the wrapper.sh file are
`#!/bin/bash
/usr/bin/python /Applications/Splunk/bin/scripts/ExportClient.py`
If I try and execute the actual python script (ExportClient.py) from the command line I get the following error:
____________
C:\Program Files\Splunk\bin>splunk cmd python scripts\ExportClient.py
Traceback (most recent call last):
File "scripts\ExportClient.py", line 8, in
import dateutil.parser
ImportError: No module named dateutil.parser
______________
Any help is appreciated.
[1]: https://support.symantec.com/en_US/article.HOWTO128173.html#v126173001
↧
↧
"No SSL Cert Validation can be performed since no CA File has been provided"? KV Store issue
Hello All,
I'm working within the windows environment. I'm getting 3 error messages in my Splunk Web Browser that probably related to an expired cert on our indexer (Only on our splunk indexer). I believe at one point the previous person in my position setup the Web Server to use SSL. However....... That local Cert CA authority is no longer in existence within our subnet AND the Cert has expired..... we'd simply like to return splunk back to it's out of the box settings (No SSL Cert used in the web browser).
What did I do so far?:
From the browser I switched it from SSL Enabled to non SSL. I believe i need to make some more changes within the configurations though.... I believe somewhere it's still referencing the Local SSL Cert Auth instead of the previous Original SSL/ self signed CA Root cert. How do I get things back to square one/ OOTB?
The Bad:
1) From the Web Browser i keep getting this message pop up: " Search peer "Server A" has the following message: KV Store changed status to failed. KV Store process terminated".
2) Search peer "Server A" has the following message: Failed to start KV Store Process. See mongod.log and splunkd.log
When I navigate to mongo db i see since the 19th of March i've been getting this message:
CONTROL: "No SSL Certificate validation can be performed since no CA File has been provided; please specify an sslCAFile parameter"
NETWORK: "The provided SSL Certificate is expired or not yet valid"
I: Fatal Assertion 28652
I'm on a windows box and I' would like to know the steps required to correctly point my SSL/ sslCAFile. Once again we do not use SSL from the web browser so the out of the box self signed certs would work fine for me. Thanks for your assistance.
↧
SSO login banner when using a CAC
I've been able to configure SSO for CAC via Apache proxy and everything works fine. I'm trying to figure out how to display a login banner like if you were to use username and password on the default Splunk login page where the user has to "accept" or at least be prompted fr some tpe of input before the user is allowed into Splunk. Has anyone been able to get this configured or point me in the right direction? Google searches didn't really help a lot. Thanks in advance.
↧
Windows Configuration Splunk Stream
Hi!
Having some trouble configuring windows to collect data from a Windows forwarder(UF). I have a heavy forwarder configured with token where I also have Splunk_TA_stream installed.
On the searchhead I have both TA_stream and the actual stream app.
On the Windows forwarder I have pushed out the TA_stream app with inputs.conf pointing towards the searchhead. I have also made sure the FW openings has been made so that not an issue.
However cant seem to get a connection for the Windows server. I have it configured on a Linux host which works fine.
Read something about WinPcap. I found the docs a bit confusing here though. Something I need to manually install?
Does anyone have other tips or "good to know" knowledge when it comes to stream and windowsforwarder?
Perhaps to get some help I need to specify more info. Let me know in that case!
Thanks!
↧
How to parse/format json at search-time?
New in handling json files in splunk with little knowledge in rex. Need help on parsing/formatting json logs in search time. Recently collected auth0 json logs. Every event starts with different character and ends with different character. Looks like Splunk break based on the max limit of the characters per event. I am trying to read the data in Search with proper formatting in table form with all fields.
props.conf stanza:
[auth0]
INDEXED_EXTRACTIONS = json
KV_MODE = none
logs looks like:
ol: ...: {}\n...., ...."}, hostname":"...","user_id":"...", user_name":"...", "strategy":"...", "strategy_type":"enterprise","_id":"...", "log_id":"...",
"isMobile":...},{"date":"2018-03-21T23:28:59.296Z","type":"...","connection_id":"","client_id":"...","client_name":"...", "ip":"...","user_agent":"...",
"details":{"return_to":"...","allowed_logout_url":[..."]},"hostname":"...","user_id":"...","user_name":"...","_id":"...","log_id":"...","isMobile":...},
{"date":"2018-03-21T23:28:59.286Z","type":"...","connection":"...","connection_id":"...","client_id":"...","client_name":"...","ip":"...","user_agent":"...",
"details":{"prompts":[{"name":"....","initiatedAt":...,
....several line from json file
Sometimes the event starts with `[{"date",` sometimes with something else and ends before closing `}`.
I went through the logs and understood that, it should be something like below in each event.
{ "date":"2018-03-09T23:41:05.437Z",
"type":"..",
"connection":"...
"connection_id":"...",
"client_id":"...",
"client_name":"...",
"ip":"...",
"user_agent":"...",
"details":{
"timings":{
"connection":...,
"all_rules":...,
"internal":...,
"total":...
},
"stats":{
"loginsCount":...
},
"consoleOut":"..."
},
"hostname":"...",
"user_id":"...",
"user_name":"...",
"strategy":"...",
"strategy_type":"...",
"_id":"...",
"log_id":"...",
"isMobile":...
},
ends up comma and then another one starts. Is there any way, I can extract log in search. Please advise.
↧
↧
Are there examples for creating a custom settings page for add-ons made by the add-on builder?
Hi,
I'm trying to create a custom settings page for an add-on that I made using the add-on builder. I need to allow the users to configure a REST endpoint where they can send data to on a button click.
I've tried following [this example][1] to create a custom setup.xml, endpoint, and python script; but when I try to configure my add-on, I receive a 404 error.
A follow on question will be how do I get that configuration value back out of the config file and into the dashboard's JS so I can actually use it.
Thanks
[1]: http://dev.splunk.com/view/SP-CAAAE9B#endpt
↧
How to redirect logs to separate indexes based on host?
Hi.. I have a question
From a heavy forwarder , based on the incoming host, I like to send the logs into a separate index.
Currently I configured to send the logs from a single heavy forwarder into a separate indexer based on the Environment variable we are passing with the logs(eg environment = cert/Dev/test/prod),Which is working fine.
On the top of that, I like to send the logs into a separate index within the indexer based on the host. here is Our conf files
**Inputs.conf**
-------------
[default]
host=Heavyfw.aaaa.com
[udp://514]
index=main
sourcetype=syslog_log
connection_host=ip
**props.conf**
----------------------------
[syslog_log]
LINE_BREAKER = (\s+)\{|\n
SHOULD_LINEMERGE = false
MUST_BREAK_AFTER = ([\r\n]+)[A-Z]+\s+\|\s+\d+
TRANSFORMS-test=test
TRANSFORMS-cert=cert
TRANSFORMS-dev=dev
TRANSFORMS-prod=prod
TRANSFORMS-local=local
TRANSFORMS-null=null
[host::(198.*)]
TRANSFORMS = rewrite-DPindex
**transforms.conf**
_____________________
[cert]
REGEX=(?i)["]?environment["]?\s*[:=]\s*["]?(cert)["]?
FORMAT=cert
DEST_KEY=_TCP_ROUTING
[local]
REGEX=(?i)["]?environment["]?\s*[:=]\s*["]?(local)["]?
DEST_KEY=_TCP_ROUTING
FORMAT=local
[test]
REGEX=(?i)["]?environment["]?\s*[:=]\s*["]?(test)["]?
DEST_KEY=_TCP_ROUTING
FORMAT=test
[null]
REGEX=^default send string$
FORMAT=nullQueue
DEST_KEY=queue
[dev]
DEST_KEY=_TCP_ROUTING
FORMAT=dev
REGEX=(?i)["]?environment["]?\s*[:=]\s*["]?(dev)["]?
[prod]
FORMAT=prod
DEST_KEY=_TCP_ROUTING
REGEX=(?i)["]?environment["]?\s*[:=]\s*["]?(prod)["]?
[rewrite-DPindex]
REGEX = .*
DEST_KEY = _MetaData:Index
FORMAT = Newindex
**outputs.conf**
__________________
[tcpout:Prodserv1]
server=ProdServer.aaaa.com:9997
[tcpout:cert]
server= CertServer.aaaa.com:9997
[default]
defaultGroup=default
[tcpout:test]
server= testServer.aaaa.com:9997
[tcpout:dev]
server= dev.Server.aaaa.com:9997
[tcpout:default]
server= testServer.aaaa.com:9997
[tcpout:prod]
server= ProdServer.aaaa.com:9997
[tcpout:local]
server= ProdServer.aaaa.com:9997
Now all the logs coming from the host 198.* going to Newindex and also to main index. How can I stop going to main index and send only to Newindex .
↧
REST endpoint for datasets?
I'm trying to get the REST endpoint for datasets of a data model. I can find the endpoint for the data model, but I'm having no luck finding the endpoint for the datasets. Can I get some help on this?
↧