Looking for an example Azure webhook alert from Application Insights into a Splunk HTTP Event Collector.
↧
Are there any Azure alert webhooks into Splunk HTTP event collector?
↧
How to extract a field using regex at indexing time?
Hi,
I'm ingesting the data in JSON format. we have a field event.user, which is auto extracted. is there a way to extract the new field user from event.user filed at indexing time?
for example:
event.user :
kiran331@SPl,
splunk@ADDS
I need to extract:
user:
kiran331
splunk
↧
↧
Installing Splunk App for Microsoft Exchange on standalone installation
Hi,
I was wondering is it possible to install Splunk App for Microsoft Exchange on standalone Splunk instance. I have followed the guide. After creating the "sent to indexer" app instance started not to index any data. I have get the below error.
ERROR:
"Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 10 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data"
What could be the possible reason for such an issue? Should I have to deploy this app to indexer?
Best Regards,
↧
How to schedule delivery for some app that have the option grayed out ?
I have some Splunk apps like Cisco, Exchange, A.D, Clearpass that have the "Schedule delivery" option grayed out but the Export PDF option available.
I have a requirement from client to schedule delivery of these reports everyday at 9am ? Please advise how can I achieve this ?
↧
How do you schedule delivery for app that has the option grayed out?
I have some Splunk apps like Cisco, Exchange, A.D, Clearpass that have the "Schedule delivery" option grayed out but the Export PDF option available.
I have a requirement from client to schedule delivery of these reports every day at 9am? Please advise how can I achieve this ?
↧
↧
JMX monitoring stopped working on a few machines(VMs) from the past few days
JMX monitoring stopped working on 4 of our VMs, where as the other servers (around 100) are still working. There was an upgrade of OS on all of these machines along with Java upgrade. Nothing seem different from working ones to non-working ones.
We are using config file & this is how our **inputs.conf** looks like
[jmx://jmx]
config_file = config.xml
_TCP_ROUTING = myindexset
polling_frequency = 60
sourcetype = jmx
index = my_index_jmx
disabled = 0
interval = 60
Below are the errors/messages, we are getting. Did anyone face similar issue?
Taking one non-working host as as example, it stopped receiving data since "9/30/17 1:25:11.542 AM".
This is from "/opt/splunkforwarder/var/log/splunk/**jmx.log**"
2017-09-30 01:25:49,924 - com.splunk.modinput.ModularInput -159035 [Thread-1] ERROR - Probing socket connection to SplunkD failed.Eith
er SplunkD has exited ,or if not, check that your DNS configuration is resolving your system's hostname (127.0.0.1) correctly : Connection refused
2017-09-30 01:25:57,735 - com.splunk.modinput.ModularInput -166846 [main] ERROR - Error executing modular input : Connection refused : java.lang.RuntimeException: Connection refused
at com.splunk.HttpService.send(Unknown Source)
at com.splunk.Service.send(Unknown Source)
at com.splunk.HttpService.get(Unknown Source)
at com.splunk.ResourceCollection.list(Unknown Source)
at com.splunk.ResourceCollection.refresh(Unknown Source)
at com.splunk.ResourceCollection.refresh(Unknown Source)
at com.splunk.Resource.validate(Unknown Source)
at com.splunk.ResourceCollection.validate(Unknown Source)
at com.splunk.ResourceCollection.values(Unknown Source)
at com.splunk.jmx.InfoManager.getAccounts(Unknown Source)
at com.splunk.jmx.JMXModularInputV3.doRun(Unknown Source)
at com.splunk.modinput.ModularInput.init(Unknown Source)
at com.splunk.jmx.JMXModularInputV3.main(Unknown Source)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:625)
at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:160)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:264)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:933)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
... 13 more
Below is from **splunkd.log**
09-30-2017 01:25:44.479 -0700 INFO WatchedFile - Will use tracking rule=modtime for file='/etc/alternatives/java_sdk_oracle/lib/missioncontrol/plugins/com.jrockit.mc.rjmx_5.5.1.172852/plugin.properties'.
09-30-2017 01:25:57.735 -0700 ERROR ExecProcessor - message from "python /opt/splunkforwarder/etc/apps/Splunk_TA_jmx/bin/jmx.py" Error executing modular input : Connection refused : java.lang.RuntimeException: Connection refused
09-30-2017 01:25:57.735 -0700 ERROR ExecProcessor - message from "python /opt/splunkforwarder/etc/apps/Splunk_TA_jmx/bin/jmx.py"
at com.splunk.HttpService.send(Unknown Source)
09-30-2017 01:25:57.735 -0700 ERROR ExecProcessor - message from "python /opt/splunkforwarder/etc/apps/Splunk_TA_jmx/bin/jmx.py"
at com.splunk.Service.send(Unknown Source)
09-30-2017 01:25:57.735 -0700 ERROR ExecProcessor - message from "python /opt/splunkforwarder/etc/apps/Splunk_TA_jmx/bin/jmx.py"
at com.splunk.HttpService.get(Unknown Source)
09-30-2017 01:25:57.735 -0700 ERROR ExecProcessor - message from "python /opt/splunkforwarder/etc/apps/Splunk_TA_jmx/bin/jmx.py"
at com.splunk.ResourceCollection.list(Unknown Source)
java -version on both WORKING & NON-WORKING VM
java version "1.8.0_141"
Java(TM) SE Runtime Environment (build 1.8.0_141-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.141-b15, mixed mode)
We tried to compare everything, tried re-push the app from deployment server. No luck.
A few things we validated on non-working VM
1) port is open to the localhost
2) port is configured in JVM/JMX
3) JMX metrics are coming fine in the JConsole
Please let us know if anyone faced similar issue and were able to fix.
↧
question on sourcetype override differentiating based on hosts
I have WLC and Equallogic sending logs on port udp 514.
Currently, only cisco sourcetype is configured and hence all data is getting parsed as cisco:ios sourcetype. I want to parse data sent by 6 Equallogic hosts to customised "equal_log" sourcetype.
I followed this documentation, http://docs.splunk.com/Documentation/SplunkCloud/6.6.1/Data/Advancedsourcetypeoverrides
and created the following transforms.conf and props.conf in $SPLUNK_HOME/etc/system/local/,
[set_sourcetype_syslog_for_Equallogic_hosts]
REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(host1|host2|host3|host4|host5|host6)[\w\.\-]*\]?\s
FORMAT = sourcetype::equal_log
DEST_KEY = MetaData:Sourcetype
[source::udp:514]
TRANSFORMS-changesourcetype = set_sourcetype_equal_log_for_Equallogic_hosts
Please note, above regex is to extract the host field for syslog events . I am using the same regex because data coming from Equallogic is syslog which is getting parsed into cisco:ios sourcetype.
Should I be using regex to extract host field for cisco:ios events ?
Also, should I make the above props+transforms changes in Cisco app's props+transforms.conf file as well ?
↧
Showing current logged in VPN users
Hi,
I wanted to display in a form of a table the current logged in VPN users.
my search command is this
host="" user=* | stats count by user
![alt text][1]
However, i do not want it to show the count and i want to see the time logged in as well, how can i improve my search to show that?
I am new to Splunk and from what i understand, if I am using openvpn logs i should have the PF-sense app downloaded for the CIM compliant field extractions?
I have downloaded the add-on to my Splunk but have problems understanding how i should be configuring the PF-sense app to support the field extractions for openvpn logs?
Any help would be appreciated! Thank you!
this is something that i would like:
user | ip address | Connected Time
=======================
student01 | 10.0.0.80 | 02:50:51
[1]: /storage/temp/217863-capture3.png
↧
Sending conditional alerts based on previous search result
Following is the json log format being stored in Splunk.
{ data:[
{
"endpoint":"ep_1",
"service":"service_1",
"http_status_code":"500"
},
{
"endpoint":"ep_2",
"service":"service_1",
"http_status_code":"500"
},
{
"endpoint":"ep_3",
"service":"service_2",
"http_status_code":"503"
} ] }
Currently, an alert is setup to check once every 5 mins with the following search query and send an email if results>0.
host=something "data{}.http_status_code"= 5*| eval endpoint='data{}.endpoint' | Stats Count("data{}.http_status_code") as Count by endpoint | where Error_Count > 0
The generated result is :
![alt text][1]
Now, the ask is, once an alert is generated (which contains a particular endpoin, say ep_1), in the next run, if the search result contains the same endpoint, it should NOT send an email for another hour.
So in this case, when the search query runs after 5 mins and contains ep_1 in the result, it should not send an email. So effectively, I need a condition for this endpoint's occurrence in the previously executed search result.
Is there a way this can be accomplished? Please help.
[1]: /storage/temp/216823-216795-table1.png
↧
↧
Sort the number of hits according to the number of hits.
For the query :
host=aeperf01api02 Level="INFO" | stats count by AppDomain
I have following output
Web 4504
WebApi 180240
ComplianceWeb 9384
ReportingWeb 34152
ReferenceDataWeb 161710
SecurityMasterWeb 78878
Login 38514
I have 6 host like above, how to sort them in one query where I can present these output in rows and hosts in the column so that I can check what is the number of services hit by the particular host.
↧
Import CSV and column as "%" percent symbol in it
Hello,
I have been importing a csv that has a column that has a percent symbol in it.
How do I search on this particular field?
The name of the column in the csv is: "Change %".
Thanks!
↧
Change / Delete Tags via Search
Hi,
anyone can tell me if it is possible to change and delete tags by splunk search. Let me tell you why. I import data from a database. Each time a record is updated, I receive a new event in my index at the same time. Therefore, I am forced to sort all events before I can "dedup". My idea is the following:
When a new event occurs, I give it the tag "latest". When this event receives an update, I want to remove the tag "latest" of the older event and pass it to the new version. Therefore I don't have to sort the data anymore and can use "tag=latest" instead.
I know it isn't the right way to use tags because usually you would tag a field=value expression.
↧
Why are my logs being pulled periodically?
My access_logs files are not being pulled constantly. There are large gaps between the pulling of logs.
The logs are being updated within the server path (timestamp shows this), but they are not all being pulled. The source files being pulled tends to be inconsistent; on some occasions it will pull all the proper source files, on other occasions, it will pull 1-2 source files (all from same path).
![![alt text][1]][1]
I attached an image as well (for visual reference).
By the way, our inputs.conf file has a very basic setup:
[monitor:path*.log]
sourcetype =log
index = name
Any help would be greatly appreciated.
[1]: /storage/temp/217857-screenshot-2.png
↧
↧
how to make visio icons appear correctly in Visio ?
When I drag an Indexer or Heavy Forwarder icon in Visio from Splunk stencil, it is not displayed properly. Image attached.
![alt text][1]
[1]: /storage/temp/217862-c.png
How to make this image display correctly ?
↧
Splunk Systemd Service
Hello,
has anyone a working systemd script for Redhat/SUSE?
If I using the script from https://answers.splunk.com/answers/59662/is-there-a-systemd-unit-file-for-splunk.html
I get some error at the HTTP-Listener
10-17-2017 09:07:36.017 +0200 ERROR DispatchProcess - Failed to start the search process. 10-17-2017 09:07:36.032 +0200 ERROR SearchProcessRunner - Error reading from preforked process=0/25: Connection reset by peer 10-17-2017 09:07:36.123 +0200 WARN Thread - HTTPDispatch: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 75 threads active 10-17-2017 09:07:36.123
+0200 ERROR HttpListener - Error spawning thread: HTTPDispatch: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 75 threads active 10-17-2017 09:07:45.273 +0200 ERROR SearchProcessRunner - preforked search=0/32 on process=0/31 caught exception. completed_searches=0, process_started=1508224065.223881, search_started=1508224065.228171, search_ended=1508224065.273768, total_usage_time=0.046 10-17-2017 09:07:45.273 +0200 ERROR SearchProcessRunner - preforked process=0/31 died on exception: Main Thread: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 3 threads active 10-17-2017 09:07:50.688
+0200 WARN ProcessTracker - executable=splunk-optimize failed to start reason='': Resource temporarily unavailable 10-17-2017 09:07:50.692
+0200 WARN ProcessTracker - executable=splunk-optimize failed to start reason='': Resource temporarily unavailable 10-17-2017 09:07:50.693
+0200 WARN ProcessTracker - executable=splunk-optimize failed to start reason='': Resource temporarily unavailable 10-17-2017 09:07:50.693
+0200 WARN ProcessTracker - executable=splunk-optimize failed to start reason='': Resource temporarily unavailable
↧
How to index the log data only from a single server when the log path is in shared drive
Hi All, I am facing the below issue:
I am reading few log sources (monitor) from the 3 servers, Server1, Server2 and Server3.
Along with that, I am also reading a log source (test1.txt) from a shared path (This path is shared across all 3 servers).
Now, the issue is: the same log source (test1.txt) is indexed twice on Splunk against the host Server2 and Server3.
Whereas, I want to index this source only once against the server Server1 and not to index for Server2 and Server3.
Is there a way in config file where I can specify that test1.txt should be monitored only from Server1.
How can I achieve this? Please help me. regards, Santosh
↧
Sort Source based on its earliest event indexing
I am in the log sources provisioning phase.
I examine the "data summary" frequently to see the change in number of hosts/sources/sourcetypes to determine from which log sources, Splunk has started collecting/receving data
However, now I have noticed jump in no. of sources but same no of hosts and sourcetypes. Hence, I want to be able to find out which was that new source that has newly emerged in Splunk.
In order to do this, I am looking for a search command that will give me a list of all sources with it's first event displayed which, I guess, can be achieved by using the earliest event command.
Can someone please advise how I can achieve this ?
↧
↧
use eval in xml
Hi,
I have a dashboard with a timechart, and I have created a drilldown for the timechart. the click uses the time clicked on, and passes it to another dashboard as a token. how do I change the click value before I pass the token to the next drilldown. I don't want the users to see the epoch time, I want them to see a regular date.
Thanks
↧
Single Value individual color for trend and value
A single value in Splunk has the following simple xml code:
↧
props.conf how to break event after every new line?
As stated in the question, my props.conf has the following settings:
[daemonforCent]
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE=false
And as you can see, the result is still the same, not breaking anything. I've tried BREAK_ONLY_BEFORE=\r\n too but also will not work.
![alt text][1]
[1]: /storage/temp/216825-capture.png
↧