We have 3 Node SHC pool and the SHC is still frequently gets out-of-synch and keeps throwing the following UI banner message: "Error pulling configurations from the search head cluster captain; consider performing a destructive configuration resync on this search head cluster member."
These are the recommended setting changes implemented:
scheduling_heuristic = round_robin
captain_is_adhoc_searchhead = true
replication_factor = 1
12-14-2015 17:22:54.072 +0000 WARN ConfReplication - installed_snapshot="/ngs/app/splunkt/SHC/splunk/var/run/splunk/snapshot/1450111567-b0d62539eea238d3c00ccbe9f81601fd6675f5d9.bundle" has earlier timestamp than existing snapshot="/ngs/app/splunkt/SHC/splunk/var/run/splunk/snapshot/1450113494-f4754dfa40753dbce4014552b9f64dbc6c00844d.bundle"; check for clock skew
What does this error message mean? Could this be the cause of the issue?
↧
Search Head Cluster: Members in SHC pool get out of synch and error in log files check for clock skew
↧
Spec for reading DAT files
Hi,
I have configured an app being pushed from deployment server to a remote Windows host to read DAT files.
Links already refrred :
http://splunk-base.splunk.com/answers/60643/archiveprocessor-bypassing-normal-systemlocalpropsconf-processing-for-dat-files-inside-archives-434
https://answers.splunk.com/answers/55279/handling-text-dat-files-how-can-i-override-splunks-system-default-props-conf-configuration-for-just-a-single-app.html?utm_source=typeahead&utm_medium=newquestion&utm_campaign=no_votes_sort_relev
The configuration looks like this :
props.conf
[source::....(dat)]
sourcetype = mysourcetype
inputs.conf
[default]
index = app
sourcetype = mysourcetype
[monitor://D:\folder\folder\Server34\encyc\status\*\*]
[monitor://C:\Anupama\status\...\...]
[monitor://C:\folder\status\*\*]
[monitor://C:\folder\status\*.dat]
It is weird that all the files in the folder getting read, except for the required DAT files.
Can someone help with the best configurations, please ?
↧
↧
Regular Expression to Extract Values From a Field
Hello Ninjas,
Am having some trouble trying to figure out how to use regex to perform a simple action.
So I have a field called Caller_Process_Name which has the value of `C:\Windows\System32\explorer.exe`
I want to take the "explorer.exe" part out of this field and place it in a new field (called process_name_short). So I see regex as the solution here.
I have been trying the following but I do not believe I am using regex correctly in Splunk and the documentation isn't very helpful.
| rex field=Caller_Process_Name (?/(\w+)\.(\w+)$/)
I'm sure my regex is solid as it pulls out only the explorer.exe part of the string in the online regex testers.
Would anyone be willing to show me what I'm not doing right here please.
Thanks :)
↧
Using REST API search endpoints to retrieve a saved search SID and search results, why are no results returned?
Hi All,
I'm trying to build a mini SDK for the REST API using Golang (focusing on the search/saved search endpoints at the moment). I've got alot of the endpoints working individually where I can create saved search, dispatch, delete, etc. Same with searching where I can search jobs, get search results of a job, etc.
However,
When I try to run multiple methods in succession (mainly Dispatch Saved Search to get SID and then get Search Results for given SID), it fails. The Dispatch method returns the SID.
When I try to run the Search Results GET with the SID though, it returns nothing.
Any one have any suggetsions on this?
Thanks
↧
Does the Cisco eStreamer for Splunk app support retrieving payload for intrusion events?
Does the Cisco eStreamer for Splunk app support retrieving payload for intrusion events?
↧
↧
Can Splunk integrate with Lansweeper if the data is stored in an MSSQL database? Is it possible using Splunk DB Connect?
Is it possible to Integrate Lansweeper with Splunk. Since Lansweeper stores all the logs and inventory information in a Microsoft SQL Server Database, it is possible to query this data using DB Connect or something? Does anyone here have any experience with this?
Thanks
↧
Can Splunk Integrate with Lansweeper?
Is there a way to integrate Splunk with Lansweeper? Since Lansweeper stores all configuration and inventory information in a Microsoft SQL Database, can we query this database for information using Splunk DBConnect or something similar? I wasn't able to find any useful documents or procedures online. Does anyone here have any previous experience with this?
Thanks
↧
Palo Alto Networks App for Splunk 5.0.0: Why are some dashboards showing 'tstats' and not displaying any data?
Some dashboards (Traffic, WebActivity) showing 'tstats' and not displaying any data. I recently upgraded to 5.0. Regular (not accelerated) searches work fine. All troubleshooting steps were followed. Data model shows 100% acceleration on all three models.
What am I missing ?
↧
Splunk DB Connect 2: Why am I getting error "Cannot get a connection, pool error Timeout waiting for idle object"?
Hello Splunkers.
I have 2 connections with a Microsoft DB using Splunk DB Connect 2.
For some time, everything was OK, but all of sudden Splunk stopped indexing new data.
Looking at _internal, I saw these following errors:
01/04/2016 17:19:44 [ERROR] [websocket.py] ERROR: java.sql.SQLException: java.sql.SQLException:java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object.
01/04/2016 17:19:44 [ERROR] [ws.py] [DBInput Service] Exception encountered from server on_message for entity-name = mi_input://Input_PT and type = input with error = ERROR: java.sql.SQLException: java.sql.SQLException:java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object.
I've read that maybe DB Connect 1 could solve this issue.
Have anyone seen this before?
Thanks in advance
↧
↧
Is there a way to display a different name in a drop-down list, but use the original string value in the search using the chart replace function?
Hi
I have a drop-down and Chart/List. The chart should show the event on the item selected from list.
Is there a way display the ProcessContext_ProjectName in the drop-down list removing Java, but while searching, it should use original string?
The replace function is working `replace "Java*" with "*" IN ProcessContext_ProjectName`, but while doing the search below on another chart from token_projectname2 , it should pick up from the original string including Java.
Right now, the chart is always giving a blank result if I do a filter by Replace.
index=u2 sourcetype=jms_body_header_txt | dedup ProcessContext_ProjectName | table ProcessContext_ProjectName 0 ProcessContext_ProjectName ProcessContext_ProjectName
↧
How can I troubleshoot why suddenly 8 of 10 subfolders with proxy logs have stopped being indexed?
I've been sending proxy logs to the FTP server and from there I installed an universal forwarder to send the logs to the Splunk indexers. They are all in a gz format. Everything was working fine until a day when I've noticed that proxy logs stopped getting indexed. There are about 10 subfolders and only 2 of them are still getting indexed, and the rest of the proxy logs had stopped getting indexed on the same day. How should I troubleshoot this?
Not sure why some of the subfolders with gz files (proxy logs from each site) has stopped getting indexed and the rest is still going.
↧
Why are my nested subsearches failing?
Hello,
I'm running into a problem where if I nest subsearches too far, I start to return no results. I'm unable to find a published limit of nesting, though. Is there one?
The search I'm trying to run can be paraphrased like so:
sourcetype=weblogs status=200 OR status=410 [search sourcetype=weblogs status=200 earliest=-3d@d latest=now request="GET /path/*" [search sourcetype=weblogs request="GET /path/*" status=410 earliest=-3d@d latest=now [search sourcetype=firewalllogs "/blocked/" earliest=-3d@d latest=now | dedup ip | table ip] | dedup ip | table ip] | dedup ip | table ip] | chart count over ip by status
In plainspeak: I have a firewall listing of IP addresses that have been blocked and I put that into a table. I use that table to find web requests that have a status=410 and put all those IP addresses into a table. I then use that table to find web requests that have a status=200 and put all those IP addresses into a table.
This table now has the IP addresses of people who have been blocked by the firewall and also seen status=200 AND status=456 at some point in the last 3 days. And this works (returning about 40 addresses) until I put that final wrapper on it to show the counts by status, at which point I get no results.
↧
HTML Drop-down Search not populating
I have a KVstore and created a drop-down input filter. I can't seem to get it to filter my data. The drop-down lists all the correct data, but I can't seem to filter out information.
For example:
Sample Statistics Table has the fields: NAME, ID, FavoriteColor
Data/Inputs in the collection:
Abby, 01, Blue
Bill, 02, Green
Chris, 03, Purple
My drop-down is searching based on the ID. So when I click my drop-down, it lists: 01,02,03. However, if I select 03, it does nothing. Same with all other IDs. Is there something I'm missing?
I want to be able to list all data based on the ID, even if there are duplicates. For example:
Abby, 01, Blue
Bill, 02, Green
Chris, 03, Purple
Daniella, 03, Black
If I search on all IDs with 03, I'll get:
Chris, 03, Purple
Daniella, 03, Black
↧
↧
How to write the regex to extract a field from XML data if the field is not completely XML?
Hi
I have a field which I would like to extract a field from the XML being displayed. The only problem is the field is not completely XML. I am not allowed to post an example, but basically I want to extract something that looks like:
Event xml
0 0055 3 000121481 1 TransferStarted
And I would like to grab **TransferStarted** in between the two tags <bos:implementationName> and </bos:implementationName>.
I have worked with regex in the past, but am still not confident. Any help would be much appreciated and Happy New Year!
↧
Splunk DB Connect and Oracle VPD (Virtual Private Databases): Is there a way to execute a stored proc to set a user context?
Hello,
In my gathering of how Splunk DB Connect works, it appears it's not possible to execute a stored proc to set a user context.
Is there a way to support this in DB Connect?
I want to execute the following to set VPD context:
DBMS_SESSION.SET_CONTEXT('CLIENT_CTX', 'CLIENT_ID', V_CLIENT_ID);
Any ideas how to get that working?
Thank you
↧
Why am I unable to install a Splunk Forwarder on Windows 2008 64 bit (non domain controller) with error "Faulting application openssl.exe.."?
My attempts to install a Splunk forwarder on Windows 2008 fails and is rolled back.
In this case, the application event logs show:
Faulting application openssl.exe, version 0.0.0.0, time stamp 0x5666e2ba, faulting module LIBEAY32.dll, version 1.0.2.4, time stamp 0x5666e274, exception code 0xc0000409, fault offset 0x0000000000064b0e, process id 0x684, application start time 0x01d14748dac0bc14.
Anyone ever encounter this before?
↧
Splunk DB Connect 1.2.2: Java bridge server is loading, but why does the app start page keep refreshing continuously?
When accessing Splunk DB Connect, the start page for the app just keeps refreshing continuously. There are db connections configured and they do work, the bridge does appear to be running, I can run queries.
If I check the bridge status from the menu in run query, it does a similar behavior, just says loading and keeps refreshing. It's not possible to edit database inputs as a result of this, portions of those pages keep refreshing.
Any help much appreciated. Here are the particulars of the environment.
Thanks
-Doug
dbx 1.2.2
splunk enterprise 6.3.0
/usr/lib/jvm/jre-1.7.0-openjdk.x86_64
mysql-connector-java-5.1.18-bin.jar
ps output for bridge:
root 27507 27005 1 18:58 ? 00:00:02 /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java -cp /opt/splunk/etc/apps/dbx/bin/lib/hsqldb.jar:/opt/splunk/etc/apps/dbx/bin/lib/jtds-1.2.6.jar:/opt/splunk/etc/apps/dbx/bin/lib/commons-pool-1.5.6.jar:/opt/splunk/etc/apps/dbx/bin/lib/log4j-1.2.15.jar:/opt/splunk/etc/apps/dbx/bin/lib/stringtemplate-3.2.1.jar:/opt/splunk/etc/apps/dbx/bin/lib/dbx.jar:/opt/splunk/etc/apps/dbx/bin/lib/antlr-2.7.7.jar:/opt/splunk/etc/apps/dbx/bin/lib/xstream-1.4.1.jar:/opt/splunk/etc/apps/dbx/bin/lib/h2-1.3.162.jar:/opt/splunk/etc/apps/dbx/bin/lib/jdbm-2.2.jar:/opt/splunk/etc/apps/dbx/bin/lib/commons-logging-1.0.4.jar:/opt/splunk/etc/apps/dbx/bin/lib/mysql-connector-java-5.1.18-bin.jar:/opt/splunk/etc/apps/dbx/bin/lib/postgresql-9.0-801.jdbc3.jar:/opt/splunk/etc/apps/dbx/bin/lib/sqlite-jdbc-3.7.2.jar -Xmx512m -Dfile.encoding=UTF-8 -server -Duser.language=en -Duser.region= -Dsplunk.app.ctx=dbx com.splunk.bridge.JavaBridgeServer 27005
↧
↧
What is the process to downgrade the NMON Performance Monitor for Unix and Linux Systems app from 1.6.13 to 1.6.12?
I just upgraded to 1.6.13.
The nmon data for various panels has changed from line to scatter graphs. I'd like to downgrade back to 1.6.12 ( I'll put in another ticket about the scatter graph issue )
Version of the Nmon core application deployed - 1.6.13
Versions of addon packages (TA-nmon and PA-nmon) deployed, if any - None
Versions of Splunk in use - 6.3.0
Brief description of your deployment: One server running all services
Type of Operating Systems - AIX and Linux
Mention any local configuration in use within your deployment - running scp command to gather the nmon files and monitoring the repository directory for new files.
↧
Why is my indexer randomly indexing old logs?
I have noticed that at random times my indexer is indexing old data logs from days, and sometimes even months in the past. I have no clue as to why this is happening. The logs are formatted like this:
1452006410 January 5, 2016 9:06:50 AM CST NOTIFICATION-Proview_A1827-2100_ATM_20-_20A1827-2100_20-_20SERVICEMODE_20ENTERED Proview A1827-2100 ATM - A1827-2100 - SERVICEMODE ENTERED 11 DXA CLEAR Server: INCHARGE-OI
1452006410 January 5, 2016 9:06:50 AM CST NOTIFICATION-Proview_A1827-2100_ATM_20-_20A1827-2100_20-_20SERVICEMODE_20ENTERED Proview A1827-2100 ATM - A1827-2100 - SERVICEMODE ENTERED 12 SYSTEM ESCALATION MATCHED: Proview2/ArchiveInActiveTraps
1452006410 January 5, 2016 9:06:50 AM CST NOTIFICATION-Proview_A1827-2100_ATM_20-_20A1827-2100_20-_20SERVICEMODE_20ENTERED Proview A1827-2100 ATM - A1827-2100 - SERVICEMODE ENTERED 13 SYSTEM ESCALATION MATCHED: Notification Clear/Archive - InActive/Archive Inactive Resolved Notifications
1452006410 January 5, 2016 9:06:50 AM CST NOTIFICATION-Proview_A1827-2100_ATM_20-_20A1827-2100_20-_20SERVICEMODE_20ENTERED Proview A1827-2100 ATM - A1827-2100 - SERVICEMODE ENTERED 14 SYSTEM ESCALATION REACHED: Proview2/ArchiveInActiveTraps, Level-0
At times, I see in the searched logs the date from the indexer will say, this:
**1/5/16
9:06:50.000 AM**
1448550410 **November 26, 2015 9:06:50 AM** CST NOTIFICATION-CPU__Performance__CiscoSystem_I-CPU__Performance__CiscoSystem-PSR-ALBMDSP301/0_HighUtilization CPU_Performance_CiscoSystem I-CPU_Performance_CiscoSystem-PSR-ALBMDSP301/0 HighUtilization 8 SYSTEM ESCALATION SCHEDULED: Resources/ResoursesClearEvent for Level-1 due at November 26, 2015 9:11:51 AM CST
1448550416 November 26, 2015 9:06:56 AM CST NOTIFICATION-Memory__Performance__HostResources_I-Memory__Performance__HostResources-MEM-ALVPHASE3UI/6_InsufficientFreeMemory Memory_Performance_HostResources I-Memory_Performance_HostResources-MEM-ALVPHASE3UI/6 InsufficientFreeMemory 22 SYSTEM ESCALATION REACHED: Resources/ResoursesClearEvent, Level-1
1448550416 November 26, 2015 9:06:56 AM CST NOTIFICATION-Memory__Performance__HostResources_I-Memory__Performance__HostResources-MEM-ALVPHASE3UI/6_InsufficientFreeMemory Memory_Performance_HostResources I-Memory_Performance_HostResources-MEM-ALVPHASE3UI/6 InsufficientFreeMemory 23 SYSTEM Action invoked... ClearEvent
1448550417 November 26, 2015 9:06:57 AM CST NOTIFICATION-Memory__Performance__HostResources_I-Memory__Performance__HostResources-MEM-ALVPHASE3UI/6_InsufficientFreeMemory Memory_Performance_HostResources I-Memory_Performance_HostResources-MEM-ALVPHASE3UI/6 InsufficientFreeMemory 24 SYSTEM Action completed successfully... ClearEvent
1448550417 November 26, 2015 9:06:57 AM CST NOTIFICATION-Memory__Performance__HostResources_I-Memory__Performance__HostResources-MEM-ALVPHASE3UI/6_InsufficientFreeMemory Memory_Performance_HostResources I-Memory_Performance_HostResources-MEM-ALVPHASE3UI/6 InsufficientFreeMemory 25 SYSTEM Action invoked... zArchiveEvent
Show all 257 lines
ClassName = CPU_Performance_CiscoSystem Escalations = SCHEDULED: Resources/ResoursesClearEvent for Level-1 due at November 26 EventName = HighUtilization InstanceName = I-CPU_Performance_CiscoSystem-PSR-ALBMDSP301/0 SourceEsc = Server: INCHARGE-AM-PM-GA-FL eventtype = ActionSuccess eventtype = Escalations Scheduled eventtype = Notification Clear eventtype = Notification Notify host = ALVIONIX01 source = \\ALVIONIX01\d\InCharge\SAM\smarts\local\logs\INCHARGE-SA.audit sourcetype = SAM_Audit
So as you can see, the indexer is picking up older log entries and indexing them as a group as one date.
What can be done?
Any help would be appreciated.
↧
How to troubleshoot why my Splunk DB Connect 2 app does not load?
My app Splunk DB connect 2 does not load and or display on WEB. Plugin is configured default. Javahome (inputs.conf) and java_home (settings.conf) is true. Dbx2 and rpc logs is without Errors. Some Errors are in web_service.log
OS - Oracle EL 6.7
Splunk is running from user root privileges and SElinux is disabled.
Please help me to resolve this issue!
Thank you!
Logs from my Server (Were generated when opening the page ):
[root@splunk splunk]# tail -f web_access.log web_service.log dbx2.log rpc.log
==> web_access.log <==
127.0.0.1 - infosecadmin [05/Jan/2016:17:21:39.601 +0200] "GET /en-US/app/search/license?hideEdit=true&hideTitle=true&hideSplunkBar=true&hideAppBar=true&hideFooter=true&targetTop=true HTTP/1.1" 200 3513 "https://xxx.xxx.xxx.xx:8000/en-US/app/launcher/home" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0" - 568bdf839a7f16c06f3790 1883ms
127.0.0.1 - infosecadmin [05/Jan/2016:17:21:41.533 +0200] "GET /en-US/config?autoload=1 HTTP/1.1" 304 - "https://xxx.xxx.xxx.xx:8000/en-US/app/search/license?hideEdit=true&hideTitle=true&hideSplunkBar=true&hideAppBar=true&hideFooter=true&targetTop=true" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0" - 568bdf85887f16c06f3ad0 9ms
==> web_service.log <==
2016-01-05 17:21:52,346 INFO [568bdf90527f16c05d7d50] cached:77 - memoized decorator used on function with non hashable arguments
==> web_access.log <==
127.0.0.1 - infosecadmin [05/Jan/2016:17:21:52.321 +0200] "GET /en-US/app/splunk_app_db_connect/ HTTP/1.1" 303 128 "https://xxx.xxx.xxx.xx:8000/en-US/app/launcher/home" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0" - 568bdf90527f16c05d7d50 408ms
==> web_service.log <==
2016-01-05 17:21:52,758 INFO [568bdf90be7f16c06a6450] cached:77 - memoized decorator used on function with non hashable arguments
2016-01-05 17:21:52,976 INFO [568bdf90be7f16c06a6450] view:1034 - bypass module system fast path
2016-01-05 17:21:55,076 INFO [568bdf90be7f16c06a6450] view:1098 - PERF - viewType=fastpath viewTime=1.9873s templateTime=0.331s
==> web_access.log <==
127.0.0.1 - infosecadmin [05/Jan/2016:17:21:52.744 +0200] "GET /en-US/app/splunk_app_db_connect/explorer HTTP/1.1" 200 1559 "https://xxx.xxx.xxx.xx:8000/en-US/app/launcher/home" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0" - 568bdf90be7f16c06a6450 2344ms
127.0.0.1 - infosecadmin [05/Jan/2016:17:21:55.153 +0200] "GET /en-US/config?autoload=1 HTTP/1.1" 304 - "https://xxx.xxx.xxx.xx:8000/en-US/app/splunk_app_db_connect/explorer" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0" - 568bdf93277f16c07833d0 7ms
==> web_service.log <==
2016-01-05 17:21:56,196 INFO [568bdf94227f16c0783e10] error:129 - Masking the original 404 message: 'The path '/en-US/static/@f3e41e4b37b2/js/views/shared/documentcontrols/dialogs/permissions_dialog/ACL.js' was not found.' with 'Page not found!' for security reasons
==> web_access.log <==
127.0.0.1 - - [05/Jan/2016:17:21:56.134 +0200] "GET /en-US/static/@f3e41e4b37b2/js/views/shared/documentcontrols/dialogs/permissions_dialog/ACL.js HTTP/1.1" 404 3115 "https://xxx.xxx.xxx.xx:8000/en-US/app/splunk_app_db_connect/explorer" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0" - 568bdf94227f16c0783e10 63ms
==> web_service.log <==
2016-01-05 17:21:57,057 ERROR [568bdf950c7f16c017a450] utility:49 - name=javascript, class=Splunk.Error, lineNumber=166, message=Error: Script error for: views/shared/documentcontrols/dialogs/permissions_dialog/ACL
http://requirejs.org/docs/errors.html#scripterror, fileName=https://xxx.xxx.xxx.xx:8000/en-US/static/@f3e41e4b37b2/js/contrib/require.js
==> web_access.log <==
127.0.0.1 - infosecadmin [05/Jan/2016:17:21:57.049 +0200] "POST /en-US/util/log/js HTTP/1.1" 200 279 "https://xxx.xxx.xxx.xx:8000/en-US/app/splunk_app_db_connect/explorer" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0" - 568bdf950c7f16c017a450 9ms
↧