Suggestions for improvement to the Python SDK script implementation are being requested. Would modifying the EXEC_MODE or OUTPUT_MODE to another value help?
I'm am using a Python SDK (splunk-sdk-python-1.6.2) script in the examples directory (search.py) on a heavy forwarder to collect search results from a Splunk Enterprise server, writing them to file, monitoring the file and forwarding to Splunk Cloud.
I've wrapped the search.py script it in a BASH shell script and it is somewhat successfully executing from the splunk user crontab every minute. Initially, it appears data is collected and everything is working fine. However, after a few minutes, I start to receive HTTP Error 503 (too many HTTP threads) and start to get socket timeout errors (errno 110).
Eventually, the host's memory utilization is so high that it is no longer reachable and needs to be rebooted. I can see there a variety of processes spawned, like: kthreadd, ksoftirqd/0, kworker/0:0H and the like.
I know the one minute, repeated execution is a lot and am working with the requestors to change that requirement. In addition, I have asked them to consider forwarding the data directly to Splunk Cloud. In the meantime, I am trying to get a stable implementation working.
***The BASH wrapper:***
-#Modify this file if you need to change PYTHONPATH, host, port, username or password
SCRIPT_HOME=/opt/splunk/etc/apps/gcs-shippingapi/bin
source $SCRIPT_HOME/gcs-shippingapi-hostcred.cfg
-#The time string can be either a UTC time (with fractional seconds), a relative time specifier (to now) or a formatted time string.
EARLIEST='-2m@m'
LATEST='-1m@m'
-#Execution mode valid values: (blocking | oneshot | normal); default=normal
-#Refer to the following for more information: http://dev.splunk.com/view/python-sdk/SP-CAAAEE5
EXEC_MODE='oneshot'
-#Output mode valid values: (atom | csv | json | json_cols | json_rows | raw | xml); default=xml
OUTPUT_MODE='raw'
SEARCH='search Org=pitneybowes AND Env=prod AND EndpointName= AND responseStatus='
/opt/splunk/bin/python $SCRIPT_HOME/search.py "$SEARCH" --host=$SPLUNK_HOST --port=$PORT --username=$SPLUNK_USERNAME --password=$SPLUNK_PASSWORD --output_mode=$
OUTPUT_MODE --earliest_time=$EARLIEST --latest_time=$LATEST
***Cron Error Message #1:***
Traceback (most recent call last):
File "/opt/splunk/etc/apps/gcs-shippingapi/bin/search.py", line 115, in main(sys.argv[1:]) File "/opt/splunk/etc/apps/gcs-shippingapi/bin/search.py", line 72, in main service = client.connect(**kwargs_splunk) File "/opt/splunk-sdk-python-1.6.2/splunklib/client.py", line 321, in connect s.login() File "/opt/splunk-sdk-python-1.6.2/splunklib/binding.py", line 857, in login cookie="1") # In Splunk 6.2+, passing "cookie=1" will return the "set-cookie" header File "/opt/splunk-sdk-python-1.6.2/splunklib/binding.py", line 1201, in post return self.request(url, message) File "/opt/splunk-sdk-python-1.6.2/splunklib/binding.py", line 1221, in request raise HTTPError(response) splunklib.binding.HTTPError: HTTP 503 Too many HTTP threads (628) already running, try again later --
Too many HTTP threads (628) already running, try again later
The server can not presently handle the given request.
***Cron Error Message #2:***
Traceback (most recent call last):
File "/opt/splunk/etc/apps/gcs-shippingapi/bin/search.py", line 115, in
main(sys.argv[1:])
File "/opt/splunk/etc/apps/gcs-shippingapi/bin/search.py", line 72, in main
service = client.connect(**kwargs_splunk)
File "/opt/splunk-sdk-python-1.6.2/splunklib/client.py", line 321, in connect
s.login()
File "/opt/splunk-sdk-python-1.6.2/splunklib/binding.py", line 857, in login
cookie="1") # In Splunk 6.2+, passing "cookie=1" will return the "set-cookie" header
File "/opt/splunk-sdk-python-1.6.2/splunklib/binding.py", line 1201, in post
return self.request(url, message)
File "/opt/splunk-sdk-python-1.6.2/splunklib/binding.py", line 1218, in request
response = self.handler(url, message, **kwargs)
File "/opt/splunk-sdk-python-1.6.2/splunklib/binding.py", line 1357, in request
connection.request(method, path, body, head)
File "/opt/splunk/lib/python2.7/httplib.py", line 1042, in request
self._send_request(method, url, body, headers)
File "/opt/splunk/lib/python2.7/httplib.py", line 1082, in _send_request
self.endheaders(body)
File "/opt/splunk/lib/python2.7/httplib.py", line 1038, in endheaders
self._send_output(message_body)
File "/opt/splunk/lib/python2.7/httplib.py", line 882, in _send_output
self.send(msg)
File "/opt/splunk/lib/python2.7/httplib.py", line 844, in send
self.connect()
File "/opt/splunk/lib/python2.7/httplib.py", line 1255, in connect
HTTPConnection.connect(self)
File "/opt/splunk/lib/python2.7/httplib.py", line 821, in connect
self.timeout, self.source_address)
File "/opt/splunk/lib/python2.7/socket.py", line 575, in create_connection
raise err
socket.error: [Errno 110] Connection timed out
↧