I have a really old 4.1.5 instance on Ubuntu 8.04. I am finally upgrading to 6.3.1 (through 4.3.7 first). I provisioned a new Ubuntu 14.04 server of the same bit-ness as the old server (32-bit) and copied the entire /opt/splunk folder to the new server. I downloaded and installed the Debian 4.3.7 package using dpkg -i. The install recognized that this is an upgrade and proceeded without issue. When I start Splunk with /opt/splunk/bin/splunk start, splunkd starts fine, but splunkweb fails to start. Here is the startup process:
root@fvw-syslog:/opt/splunk# /opt/splunk/bin/splunk start
Splunk> 4TW
Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking configuration... Done.
Checking index directory...
Validated databases: _audit _blocksignature _thefishbucket history main summary
Done
Success
Checking conf files for typos...
Possible typo in stanza [sampledata-1] in /opt/splunk/etc/apps/search/local/savedsearches.conf, line 32: name = sampledata-1
Possible typo in stanza [sampledata-1] in /opt/splunk/etc/apps/search/local/savedsearches.conf, line 36: _actions = new,edit,delete
Possible typo in stanza [unix-all-logs] in /opt/splunk/etc/apps/unix/default/eventtypes.conf, line 7: dispatch.earliest_time = -15m
Possible typo in stanza [Failed_SU] in /opt/splunk/etc/apps/unix/default/eventtypes.conf, line 177: tag = application authentication verify failure
Possible typo in stanza [ssh-invalid-user] in /opt/splunk/etc/apps/unix/default/eventtypes.conf, line 317: example = Dec 17 18:31:42 domU-12-31-39-03-01-11 sshd[31787]: input_userauth_request: invalid user php
Possible typo in stanza [ssh-close] in /opt/splunk/etc/apps/unix/default/eventtypes.conf, line 327: Example = Dec 17 15:15:12 domU-12-31-39-03-01-11 sshd[24912]: Connection closed by 195.43.9.246
Possible typo in stanza [ssh-disconnect] in /opt/splunk/etc/apps/unix/default/eventtypes.conf, line 331: example = Dec 17 18:31:44 domU-12-31-39-03-01-11 sshd[31792]: Received disconnect from 74.53.187.50: 11: Bye Bye
Possible typo in stanza [vmstat] in /opt/splunk/etc/apps/unix/default/eventtypes.conf, line 374: sourcetype = vmstat
Possible typo in stanza [iostat] in /opt/splunk/etc/apps/unix/default/eventtypes.conf, line 377: sourcetype = iostat
Possible typo in stanza [ps] in /opt/splunk/etc/apps/unix/default/eventtypes.conf, line 380: sourcetype = ps
Possible typo in stanza [top] in /opt/splunk/etc/apps/unix/default/eventtypes.conf, line 383: sourcetype = top
Possible typo in stanza [netstat] in /opt/splunk/etc/apps/unix/default/eventtypes.conf, line 386: sourcetype = netstat
Possible typo in stanza [protocol] in /opt/splunk/etc/apps/unix/default/eventtypes.conf, line 389: sourcetype = protocol
Possible typo in stanza [openPorts] in /opt/splunk/etc/apps/unix/default/eventtypes.conf, line 392: sourcetype = openPorts
Possible typo in stanza [time] in /opt/splunk/etc/apps/unix/default/eventtypes.conf, line 395: sourcetype = time
Possible typo in stanza [lsof] in /opt/splunk/etc/apps/unix/default/eventtypes.conf, line 398: sourcetype = lsof
Possible typo in stanza [df] in /opt/splunk/etc/apps/unix/default/eventtypes.conf, line 401: sourcetype = df
Possible typo in stanza [who] in /opt/splunk/etc/apps/unix/default/eventtypes.conf, line 404: sourcetype = who
Possible typo in stanza [usersWithLoginPrivs] in /opt/splunk/etc/apps/unix/default/eventtypes.conf, line 407: sourcetype = usersWithLoginPrivs
Possible typo in stanza [lastlog] in /opt/splunk/etc/apps/unix/default/eventtypes.conf, line 410: sourcetype = lastlog
Possible typo in stanza [interfaces] in /opt/splunk/etc/apps/unix/default/eventtypes.conf, line 413: sourcetype = interfaces
Possible typo in stanza [cpu] in /opt/splunk/etc/apps/unix/default/eventtypes.conf, line 416: sourcetype = cpu
Possible typo in stanza [auditd] in /opt/splunk/etc/apps/unix/default/eventtypes.conf, line 419: sourcetype = auditd
Possible typo in stanza [package] in /opt/splunk/etc/apps/unix/default/eventtypes.conf, line 422: sourcetype = package
Possible typo in stanza [hardware] in /opt/splunk/etc/apps/unix/default/eventtypes.conf, line 425: sourcetype = hardware
Possible typo in stanza [sslConfig] in /opt/splunk/etc/system/local/server.conf, line 18: _actions = new,edit,delete
There might be typos in your conf files. For more information, run 'splunk btool check --debug'
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
Done.
Starting splunkweb... Generating certs for splunkweb server
Generating a 1024 bit RSA private key
...........++++++
..............................++++++
writing new private key to 'privKeySecure.pem'
-----
Signature ok
subject=/CN=fvw-syslog/O=SplunkUser
/opt/splunk//certs/cert.pem: No such file or directory
Command failed (ret=1), exiting.
The /opt/splunk/certs folder indeed does not exist. I tried symlinking from /opt/splunk/share/splunk/certs (the location of my old certs) to /opt/splunk/certs. This allows the process to get further, but startup now says the /opt/splunk/etc/auth/splunkweb/privkey.pem file can't be found. That folder has only a README file in it. I copied the privkey.pem file from my old certs folder to /opt/splunk/etc/auth/splunkweb and again that allows the process to get further. However, now in the startup process I just see "Unable to start splunkweb.". The web_service.log file shows this:
2015-12-16 10:29:58,772 ERROR [56718375dc95100cc] root:554 - Unable to start splunkweb
2015-12-16 10:29:58,772 ERROR [56718375dc95100cc] root:555 - /certs/cert.pem Not Found
If the /certs folder is indeed relative to /opt/splunk/share/splunk as I saw in other posts then the file is there with 644 permissions. This is where I stop and ask for help :-)
↧