Hi all,
We enlarged RAM and CPU capabilites on the indexer and search head. After that we had a problem on these servers: the free disk space decreases dramatically on the search head (in a few minuts to 0%), and the free swap space decreases dramatically on the indexer.
After servers reboot the problem temporarily disappears. The problem repeats in a few days.
There was no such problem before enlarging RAM and CPU capabilites.
In the logs splunk did not find anytheng.
Can this problem be related to the enlarging RAM and CPU capabilites? Where else to look for the cause of this problem?
Guys could you comment on this?
↧
Not enough free disk space on the search head, and not enough free swap space on the indexer
↧
use of fillnull displays wrong color in 'single value'
I am using '| fillnull totalCount' in my search so i get an 0 when ther is no result. The colorrange i use is from min to 0 is green, from 0 to max is red. Somehow the '0' is still showing red. Is there anyway to change this?
↧
↧
Is it possible to build a "everyday table" even if there is no events?
I know is kind of weird the title, but I hope the explanation helps...
I need to run a report for some months back, but I need to know by everyday if there where events or not and how many. E.g:
+-------+------------+------------------+
| Date | Changes | Count of changes |
+-------+------------+------------------+
| 1-Apr | No changes | 0 |
| 2-Apr | No changes | 0 |
| 3-Apr | Change 1 | 2 |
| 3-Apr | Change 2 | 2 |
| 4-Apr | No Changes | 0 |
| 5-Apr | Change 1 | 3 |
| 5-Apr | Change 2 | 3 |
| 5-Apr | Change 3 | 3 |
+-------+------------+------------------+
I have the search to build the table if earliest/latest is in between 1 day -> (earliest=Apr 1 00:00 to latest=Apr 1 24:00), but I have no clue how to change it to have the feel that it have been run daily within a given period.
Any help is very welcome. Thanks.
↧
events with future timestamp
Hello
this is my event:
> Jun 19 12:31:44 : Info:copyconfig.cpp:319: copyConfig: copy configuration to /tmp/t5871.cfg
this is the source:
> s3://ssyssplunk/AMER/FDM/F123/D/D02/2019-05-31T13:17:14.002Z_1.91.0.192_1.85.0.0_2.0.5608.0/75fbcf50-a6a4-4520-aa58-f63498a9c265_System> Log
this is my sourcetype configuration :
[fdm_f123_systemLog]
BREAK_ONLY_BEFORE = ^\w\s\d+\s\d{2}:\d{2}:\d{2}
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 15
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %b %d %H:%M:%S
TIME_PREFIX = ^
TRUNCATE = 0
category = Custom
disabled = false
pulldown_type = 1
and the timestamp for this event (and many others) looks like :
> 2019-06-07T12:41:08.000+00:00
how come i got future date and also not the correct one ?
↧
Condition in xml for charting.fieldscolors
Hi !
I have my chart in function of the time and I have ``
How, in my xml, can I change the color of OK between a period ( ex: 0h - 1h) ???
Thx
↧
↧
Field can only be used while also refreshing
Hi, I have a totally weird situation.
The field list on the left shows me the stuff I have defined.
When I click on one of them, I see the field values. But when I then select one, the search does not show anything:
index=amp_sal message_id=AU1
Delivers no results even hough Splunk just told me there are AU1 message_ids...
But when I exclude the field I see results:
index=amp_sal message_id!=AU1
And I also see results when I perform a reload in the query:
index=amp_sal
| extract reload=t
| search message_id=AU1
So what the ... is going on?
Of course there have been plenty of restarts...
This is how the fields are defined:
EXTRACT-sal = ^(?.{3})(?.{8})(?
↧
How to see recently released apps and add-ons on SplunkBase
See accepted answer below for a search to show what apps and add-ons have recently been released on Splunkbase. It depends on the [Analysis Of SplunkBase Apps for Splunk][1] app and is intended as an example that will need to be modified (see line 11).
Please let me know if this is helpful or improvements you would suggest.
Thanks!
↧
Show IP addresses not matching CIDR ranges in lookup table
I have a list of CIDR ranges in a single column with name Prefix in a csv file. I only want to show events with source IPs (sIP) that are not in any of those ranges. My lookup definition for cidr_lookup is as follows:
minimum matches: 1
default matches: "NONE"
Match type: match_type = CIDR(Prefix)
I tried this search and lots of others I found online:
| lookup cidr_lookup Prefix as sIP OUTPUT Prefix as cidr_range
| where cidr_range= "NONE"
I get an error saying:
basic_string::erase: __pos (which is 18446744073709551615) > this->size() (which is 0)
I know that most events contain IPs that are in one of the ranges in the lookup file.
Can you help me use my lookup file correctly?
↧
Cisco Firepower Estreamer Questions
Hello,
We want to onboard Cisco firepower devices and we can't decide between estreamer and syslog input.
I would be grateful if you could help me to answer the questions below:
1) Is it possible to connect 1 heavy forwarder to more than 1 FMC?
2) Is there a difference in what kind of data we can receive ( ex. is syslog able to send ips data, and estreamer firewall data?) ?
3) Are there any issues with using one or the other method?
Thank you,
Dawid
↧
↧
What is an alert error code 3?
Sendalert is returning `error code 3`.
Where are these error codes documented?
What does 'error code 3' mean?
↧
Splunk add-on for SQL Server, No templates for db inputs
As per the documentation of Splunk add-on for SQL server below
------------------------------------------------------
Configure database inputs using the Splunk DB Connect GUI
If you want to create a MS SQL Server database input, choose the template created for the Splunk Add-on for Microsoft SQL Server under the Template field of DB Connect.
------------------------------------------------------
we can create a db input based on the template. However, I have installed the Splunk add-on for SQL server on HF. But when I am trying to create a new db input using dbx3 app, I do not see any template in the drop down. How to get the SQL templates for dbinputs?
↧
How to convert result of search to collection for supporting IN ?
Because I get error
Error in 'search' command: Unable to parse the search: Right hand side of IN must be a collection of literals. '(thread = "Thread-32")' is not a literal.
index=xx sourcetype=xxx "sent" thread IN [search index=ibul sourcetype=acct-reporting " sent" | table thread] | stats values(thread)
This is short example for my case, please tell me how to make convertion, theres no need for fixing case logic.
↧
Sharing a License
I am looking to share a license between two splunk servers, and I am unable to allow these two servers to communicate. Is this possible or would I need to purchase a second license?
↧
↧
Linebreaking problem on a small amount of events
Having issue with a linebreak. Seems most events are breaking properly, but a small number are not. I think this may be due to the ',' following the capture, which is inconsistent in the logs. Any suggestions to break on this for the remaining events not working?
Existing props.conf
[ ]
SHOULD_LINEMERGE=true
LINE_BREAKER=StandartDeviation=NaN
NO_BINARY_CHECK=true
CHARSET=UTF-8
disabled=false
Raw Data: (**bolded** instances not breaking properly)
2019-06-05 09:35:33,617 - AOB:com.fnf.xes.PRF:IntRateInqRq=ServiceName=IntRateInqRq, ChannelName=AOB, SPName=com.fnf.xes.PRF, successCount=0, successTotalTime=0, successMinTime=0, sucssessMaxTime=0, sqrSuccessTotalTime=0, Id=IntRateInqRq, TotalCount=1, TotalTime=84, TotalMinTime=84, TotalMaxTime=84, TotalAvgTime=84.0, **StandartDeviation=NaN,** null:com.fnis.xes.PRF:com.fnis.ifx.xbo.v1_1.commondata.CommonDataXQO=ServiceName=com.fnis.ifx.xbo.v1_1.commondata.CommonDataXQO, ChannelName=null, SPName=com.fnis.xes.PRF, successCount=0, successTotalTime=0, successMinTime=0, sucssessMaxTime=9, sqrSuccessTotalTime=0, Id=com.fnis.ifx.xbo.v1_1.commondata.CommonDataXQO, TotalCount=2, TotalTime=20, TotalMinTime=9, TotalMaxTime=11, TotalAvgTime=10.0, StandartDeviation=NaN
2019-06-05 09:54:23,352 - null:null:IFXServiceBase=EndPointName=IFXServiceBase, Protocol=null, ReqMsgTotalLength=0, ReqMinLength=0, ReqMaxLength=0, ResMsgTotalLength=0, ResMinLength=0, ResMaxLength=0, ServiceName=IFXServiceBase, ChannelName=null, SPName=null, successCount=0, successTotalTime=0, successMinTime=0, sucssessMaxTime=68, sqrSuccessTotalTime=0, Id=IFXServiceBase, TotalCount=3, TotalTime=217, TotalMinTime=68, TotalMaxTime=81, TotalAvgTime=72.0, StandartDeviation=NaN
2019-06-05 09:48:51,991 - AOB:com.fnf.xes.PRF:IntRateInqRq=ServiceName=IntRateInqRq, ChannelName=AOB, SPName=com.fnf.xes.PRF, successCount=0, successTotalTime=0, successMinTime=0, sucssessMaxTime=0, sqrSuccessTotalTime=0, Id=IntRateInqRq, TotalCount=1, TotalTime=63, TotalMinTime=63, TotalMaxTime=63, TotalAvgTime=63.0, **StandartDeviation=NaN,** null:com.fnis.xes.PRF:com.fnis.ifx.xbo.v1_1.commondata.CommonDataXQO=ServiceName=com.fnis.ifx.xbo.v1_1.commondata.CommonDataXQO, ChannelName=null, SPName=com.fnis.xes.PRF, successCount=0, successTotalTime=0, successMinTime=0, sucssessMaxTime=35, sqrSuccessTotalTime=0, Id=com.fnis.ifx.xbo.v1_1.commondata.CommonDataXQO, TotalCount=2, TotalTime=68, TotalMinTime=33, TotalMaxTime=35, TotalAvgTime=34.0, StandartDeviation=NaN
2019-06-05 09:48:51,991 - null:null:SOAP_IFX20=EndPointName=SOAP_IFX20, Protocol=null, ReqMsgTotalLength=0, ReqMinLength=0, ReqMaxLength=0, ResMsgTotalLength=0, ResMinLength=0, ResMaxLength=0, ServiceName=SOAP_IFX20, ChannelName=null, SPName=null, successCount=0, successTotalTime=0, successMinTime=0, sucssessMaxTime=38, sqrSuccessTotalTime=0, Id=SOAP_IFX20, TotalCount=2, TotalTime=74, TotalMinTime=36, TotalMaxTime=38, TotalAvgTime=37.0, **StandartDeviation=NaN,** null:null:IFXServiceBase=EndPointName=IFXServiceBase, Protocol=null, ReqMsgTotalLength=0, ReqMinLength=0, ReqMaxLength=0, ResMsgTotalLength=0, ResMinLength=0, ResMaxLength=0, ServiceName=IFXServiceBase, ChannelName=null, SPName=null, successCount=0, successTotalTime=0, successMinTime=0, sucssessMaxTime=0, sqrSuccessTotalTime=0, Id=IFXServiceBase, TotalCount=1, TotalTime=67, TotalMinTime=67, TotalMaxTime=67, TotalAvgTime=67.0, StandartDeviation=NaN
2019-06-05 09:45:33,617 - AOB:com.fnf.xes.PRF:IntRateInqRq=ServiceName=IntRateInqRq, ChannelName=AOB, SPName=com.fnf.xes.PRF, successCount=0, successTotalTime=0, successMinTime=0, sucssessMaxTime=81, sqrSuccessTotalTime=0, Id=IntRateInqRq, TotalCount=2, TotalTime=141, TotalMinTime=60, TotalMaxTime=81, TotalAvgTime=70.0, StandartDeviation=NaN
2019-06-05 09:45:33,617 - null:null:IFXServiceBase=EndPointName=IFXServiceBase, Protocol=null, ReqMsgTotalLength=0, ReqMinLength=0, ReqMaxLength=0, ResMsgTotalLength=0, ResMinLength=0, ResMaxLength=0, ServiceName=IFXServiceBase, ChannelName=null, SPName=null, successCount=0, successTotalTime=0, successMinTime=0, sucssessMaxTime=86, sqrSuccessTotalTime=0, Id=IFXServiceBase, TotalCount=2, TotalTime=150, TotalMinTime=64, TotalMaxTime=86, TotalAvgTime=75.0, StandartDeviation=NaN
2019-06-05 09:44:23,352 - null:com.fnis.xes.PRF:com.fnis.ifx.xbo.v1_1.commondata.CommonDataXQO=ServiceName=com.fnis.ifx.xbo.v1_1.commondata.CommonDataXQO, ChannelName=null, SPName=com.fnis.xes.PRF, successCount=0, successTotalTime=0, successMinTime=0, sucssessMaxTime=10, sqrSuccessTotalTime=0, Id=com.fnis.ifx.xbo.v1_1.commondata.CommonDataXQO, TotalCount=2, TotalTime=20, TotalMinTime=10, TotalMaxTime=10, TotalAvgTime=10.0, StandartDeviation=NaN
2019-06-05 09:44:23,352 - null:null:SOAP_IFX20=EndPointName=SOAP_IFX20, Protocol=null, ReqMsgTotalLength=0, ReqMinLength=0, ReqMaxLength=0, ResMsgTotalLength=0, ResMinLength=0, ResMaxLength=0, ServiceName=SOAP_IFX20, ChannelName=null, SPName=null, successCount=0, successTotalTime=0, successMinTime=0, sucssessMaxTime=12, sqrSuccessTotalTime=0, Id=SOAP_IFX20, TotalCount=2, TotalTime=24, TotalMinTime=12, TotalMaxTime=12, TotalAvgTime=12.0, StandartDeviation=NaN
2019-06-05 09:38:51,991 - AOB:com.fnf.xes.PRF:IntRateInqRq=ServiceName=IntRateInqRq, ChannelName=AOB, SPName=com.fnf.xes.PRF, successCount=0, successTotalTime=0, successMinTime=0, sucssessMaxTime=72, sqrSuccessTotalTime=0, Id=IntRateInqRq, TotalCount=2, TotalTime=142, TotalMinTime=70, TotalMaxTime=72, TotalAvgTime=71.0, StandartDeviation=NaN
↧
How to extract response time
I am trying to create a graph with the top 10 longest response times by host.
An example is: 200 0 0 78 where the last set of numbers represents the time taken in milliseconds which is what im trying to extract to make my graph.
↧
dropdown based table search query
Hi, I am working on a splunk roster. I have a drop down box with below values and all the teams have 3 members
Team = "Team A","Team B","TeamB"
if the input is "Team A", then the search query should return "Team A" roster as a table (i.e | table Date member1, member2, member3)
I have used "where" (below query) to get table roster output for only one member, but i need to create a table for all team members in that particular team.
sourcetype="sample_roster_june" | table Date member1 | where like ("$MemName$",member1)
↧
Deployment Server seen as Unavailable/Offline by other members
Hi,
I'm currently stuck on an issue where my Deployment Server seems to not be communicating with the same group as the other members. The server is listed as "Offline" or "Unavailable" when accessing the Monitoring Console. I believe this may be effecting clients communicating with the deployment server receiving apps and config information. I tried to see if any services are currently not running on the Deployment Server but nothing has stood out. The Deployment Server only lists itself when I access the Monitoring Console (it's in Distributed Mode). Not sure what other details are needed as I am kind of new at this but included some pictures to help describe my situation.
Any assistance is much appreciated. Thank you.
![alt text][1]
![alt text][2]
[1]: /storage/temp/272885-splunk-deploymentserver-dmc.png
[2]: /storage/temp/272887-splunk-deploymentserver-dmc2.png
↧
↧
Splunk add-on for SQL Server: No templates for db inputs
As per the documentation of Splunk add-on for SQL server below
------------------------------------------------------
Configure database inputs using the Splunk DB Connect GUI
If you want to create a MS SQL Server database input, choose the template created for the Splunk Add-on for Microsoft SQL Server under the Template field of DB Connect.
------------------------------------------------------
We can create a db input based on the template. However, I have installed the Splunk add-on for SQL server on HF. But when I am trying to create a new db input using dbx3 app, I do not see any template in the drop down. How to get the SQL templates for dbinputs?
↧
Sending splunk logs to third party server
I need to send Windows Event logs to the third party syslog solutions. Logs from Windows UF is sent to HFWD and from there it is routed both Splunk IDX and Syslog Aggregator. For some reasons its not hitting the syslog server. I have checked btool for input,output,props and transforms and couldnt find anything there..
Config on the HFWD to accept logs from the Windows server and to send it to syslog
=========================================================================
props.conf
[host::10.20.10.10]
TRANSFORMS-routing_syslog = fwd_data_to_syslog
transforms.conf
[fwd_data_to_syslog]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = to_syslog
outputs.conf
#Sent to Indexer
[syslog:to_syslog]
server = 10.172.148.186:1514
#type = udp
inputs.conf
[splunktcp://10.20.10.10:9997]
#machine not part of the domain so need to use the IP address
#_SYSLOG_ROUTING = to_syslog
=========================================================================
Config on Windows UF
[tcpout]
defaultGroup = send_to_syslog
maxQueueSize = 7MB
autoLBFrequency=15
[tcpout:send_to_syslog]
server = 10.175.108.40:9997
#sendCookedData = false
=========================================================================
One of the base app to send logs from Heavy FWD to INDX
[tcpout]
default_group = indexer_fwd
axQueueSize = 7MB
autoLBFrequency=15
[tcpout:indexer_fwd]
server = IDX1.abcd.com:9997, IDX2.abcd.com:9997, IDX3.abcd.com:9997
↧
kvstore replication issues
Hi guys.
Please help me with replicating kvstore to new cluster member. I already try resync kvstore but still have replication error. In mongo log file i've some errors with short read statuses.
2019-06-05T15:49:12.460Z I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to splunk-sh02:8191
2019-06-05T15:49:12.462Z I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to splunk-sh02:8191 - HostUnreachable: short read
2019-06-05T15:49:12.462Z I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to splunk-sh02:8191 due to failed operation on a connection
2019-06-05T15:50:00.638Z I REPL_HB [replexec-214] Error in heartbeat (requestId: 1721194) to splunk-sh02:8191, response status: HostUnreachable: short read
bash-4.2$ /opt/splunk/bin/splunk show kvstore-status
This member:
backupRestoreStatus : Ready
disabled : 0
guid : D9F9CB60-E02D-4B3A-8376-58E2A5DC8F02
port : 8191
standalone : 0
status : starting
Enabled KV store members:
splunk-sh11.sec.rambler.tech:8191
guid : 6C58D8D2-2C37-492F-A14C-E32B7D55B040
hostAndPort : splunk-sh11:8191
splunk-sh21:8191
guid : 90730C74-4A10-423E-919B-6F88F01F3DF5
hostAndPort : splunk-sh21:8191
splunk-sh02:8191
guid : D9F9CB60-E02D-4B3A-8376-58E2A5DC8F02
hostAndPort : splunk-sh02:8191
↧
