JMX monitoring stopped working on 4 of our VMs, where as the other servers (around 100) are still working. There was an upgrade of OS on all of these machines along with Java upgrade. Nothing seem different from working ones to non-working ones.
We are using config file & this is how our **inputs.conf** looks like
[jmx://jmx]
config_file = config.xml
_TCP_ROUTING = myindexset
polling_frequency = 60
sourcetype = jmx
index = my_index_jmx
disabled = 0
interval = 60
Below are the errors/messages, we are getting. Did anyone face similar issue?
Taking one non-working host as as example, it stopped receiving data since "9/30/17 1:25:11.542 AM".
This is from "/opt/splunkforwarder/var/log/splunk/**jmx.log**"
2017-09-30 01:25:49,924 - com.splunk.modinput.ModularInput -159035 [Thread-1] ERROR - Probing socket connection to SplunkD failed.Eith
er SplunkD has exited ,or if not, check that your DNS configuration is resolving your system's hostname (127.0.0.1) correctly : Connection refused
2017-09-30 01:25:57,735 - com.splunk.modinput.ModularInput -166846 [main] ERROR - Error executing modular input : Connection refused : java.lang.RuntimeException: Connection refused
at com.splunk.HttpService.send(Unknown Source)
at com.splunk.Service.send(Unknown Source)
at com.splunk.HttpService.get(Unknown Source)
at com.splunk.ResourceCollection.list(Unknown Source)
at com.splunk.ResourceCollection.refresh(Unknown Source)
at com.splunk.ResourceCollection.refresh(Unknown Source)
at com.splunk.Resource.validate(Unknown Source)
at com.splunk.ResourceCollection.validate(Unknown Source)
at com.splunk.ResourceCollection.values(Unknown Source)
at com.splunk.jmx.InfoManager.getAccounts(Unknown Source)
at com.splunk.jmx.JMXModularInputV3.doRun(Unknown Source)
at com.splunk.modinput.ModularInput.init(Unknown Source)
at com.splunk.jmx.JMXModularInputV3.main(Unknown Source)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:625)
at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:160)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:264)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:933)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
... 13 more
Below is from **splunkd.log**
09-30-2017 01:25:44.479 -0700 INFO WatchedFile - Will use tracking rule=modtime for file='/etc/alternatives/java_sdk_oracle/lib/missioncontrol/plugins/com.jrockit.mc.rjmx_5.5.1.172852/plugin.properties'.
09-30-2017 01:25:57.735 -0700 ERROR ExecProcessor - message from "python /opt/splunkforwarder/etc/apps/Splunk_TA_jmx/bin/jmx.py" Error executing modular input : Connection refused : java.lang.RuntimeException: Connection refused
09-30-2017 01:25:57.735 -0700 ERROR ExecProcessor - message from "python /opt/splunkforwarder/etc/apps/Splunk_TA_jmx/bin/jmx.py"
at com.splunk.HttpService.send(Unknown Source)
09-30-2017 01:25:57.735 -0700 ERROR ExecProcessor - message from "python /opt/splunkforwarder/etc/apps/Splunk_TA_jmx/bin/jmx.py"
at com.splunk.Service.send(Unknown Source)
09-30-2017 01:25:57.735 -0700 ERROR ExecProcessor - message from "python /opt/splunkforwarder/etc/apps/Splunk_TA_jmx/bin/jmx.py"
at com.splunk.HttpService.get(Unknown Source)
09-30-2017 01:25:57.735 -0700 ERROR ExecProcessor - message from "python /opt/splunkforwarder/etc/apps/Splunk_TA_jmx/bin/jmx.py"
at com.splunk.ResourceCollection.list(Unknown Source)
java -version on both WORKING & NON-WORKING VM
java version "1.8.0_141"
Java(TM) SE Runtime Environment (build 1.8.0_141-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.141-b15, mixed mode)
We tried to compare everything, tried re-push the app from deployment server. No luck.
A few things we validated on non-working VM
1) port is open to the localhost
2) port is configured in JVM/JMX
3) JMX metrics are coming fine in the JConsole
Please let us know if anyone faced similar issue and were able to fix.
↧