I have updated my splunk infra from 6.2 to 6.3
1) Deployment server -1
2) indexer cluster servers -2
2) sh head cluster servers -2
Update process was successful on all nodes but I am unable to start splunk services on my both indexers.
Please help
error :
[root@CDCDSPLKNDX1 bin]# ./splunk start
Splunk> The IT Search Engine.
Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: _audit _internal _introspection _thefishbucket collaboration_db firedalerts ghd history linux main os summary unix windows
Done
Bypassing local license checks since this instance is configured with a remote license master.
Checking filesystem compatibility... Done
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/splunk/splunk/splunk-6.3.0-aa7d4b1ccb80-linux-2.6-x86_64-manifest'
All installed files intact.
Done
Checking replication_port port [8080]: open
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
Done
[ OK ]
Waiting for web server at http://127.0.0.1:8000 to be availablesplunkd 3362 was not running.
Stopping splunk helpers...
[ OK ]
Done.
Stopped helpers.
Removing stale pid file... done.
WARNING: web interface does not seem to be available!
############
Crash file:
[root@CDCDSPLKNDX1 splunk]# cat "crash-2015-11-27-18:26:28.log"
[build aa7d4b1ccb80] 2015-11-27 18:26:28
Received fatal signal 6 (Aborted).
Cause:
Signal sent by PID 3362 running under UID 0.
Crashing thread: IdataDO_Collector
Registers:
RIP: [0x0000003938C32625] gsignal + 53 (/lib64/libc.so.6)
RDI: [0x0000000000000D22]
RSI: [0x0000000000000D35]
RBP: [0x0000000001893940]
RSP: [0x00007FD620FFC9F8]
RAX: [0x0000000000000000]
RBX: [0x00007FD62A1AA000]
RCX: [0xFFFFFFFFFFFFFFFF]
RDX: [0x0000000000000006]
R8: [0xFEFEFEFEFEFEFEFF]
R9: [0x00007FD62A1FDF60]
R10: [0x0000000000000008]
R11: [0x0000000000000206]
R12: [0x0000000001816C94]
R13: [0x0000000001894130]
R14: [0x0000000000000000]
R15: [0x0000000000000003]
EFL: [0x0000000000000206]
TRAPNO: [0x0000000000000000]
ERR: [0x0000000000000000]
CSGSFS: [0x0000000000000033]
OLDMASK: [0x0000000000000000]
OS: Linux
Arch: x86-64
Backtrace:
[0x0000003938C32625] gsignal + 53 (/lib64/libc.so.6)
[0x0000003938C33E05] abort + 373 (/lib64/libc.so.6)
[0x0000003938C2B74E] ? (/lib64/libc.so.6)
[0x0000003938C2B810] __assert_perror_fail + 0 (/lib64/libc.so.6)
[0x0000000000AEA0DD] ? (splunkd)
[0x0000000000AECACD] _ZN22IdataCollectorCallback4tickEv + 157 (splunkd)
[0x0000000000AECBE3] _ZN17IdataDO_Collector4mainEv + 83 (splunkd)
[0x000000000109F0EE] _ZN6Thread8callMainEPv + 62 (splunkd)
[0x00000039394079D1] ? (/lib64/libpthread.so.0)
[0x0000003938CE88FD] clone + 109 (/lib64/libc.so.6)
Linux / CDCDSPLKNDX1 / 2.6.32-504.12.2.el6.x86_64 / #1 SMP Sun Feb 1 12:14:02 EST 2015 / x86_64
Last few lines of stderr (may contain info on assertion failure, but also could be old):
splunkd: /home/build/build-src/ember/src/pipeline/indexer/IdataDO_Collector.cpp:272: void collect__indexes(): Assertion `! name.empty()' failed.
2015-11-27 18:01:06.097 +0530 splunkd started (build aa7d4b1ccb80)
splunkd: /home/build/build-src/ember/src/pipeline/indexer/IdataDO_Collector.cpp:272: void collect__indexes(): Assertion `! name.empty()' failed.
2015-11-27 18:03:37.368 +0530 splunkd started (build aa7d4b1ccb80)
splunkd: /home/build/build-src/ember/src/pipeline/indexer/IdataDO_Collector.cpp:272: void collect__indexes(): Assertion `! name.empty()' failed.
2015-11-27 18:26:27.740 +0530 splunkd started (build aa7d4b1ccb80)
splunkd: /home/build/build-src/ember/src/pipeline/indexer/IdataDO_Collector.cpp:272: void collect__indexes(): Assertion `! name.empty()' failed.
/etc/redhat-release: Red Hat Enterprise Linux Server release 6.6 (Santiago)
glibc version: 2.12
glibc release: stable
Last errno: 0
Threads running: 19
Runtime: 0.954489s
argv: [splunkd -p 8089 start]
Thread: "IdataDO_Collector", did_join=0, ready_to_run=Y, main_thread=N
First 8 bytes of Thread token @0x7fd622820610:
00000000 00 d7 ff 20 d6 7f 00 00 |... ....|
00000008
x86 CPUID registers:
0: 0000000B 756E6547 6C65746E 49656E69
1: 000106A4 06010800 80982201 0FABFBFF
2: 55035A01 00F0B2E4 00000000 09CA212C
3: 00000000 00000000 00000000 00000000
4: 00000000 00000000 00000000 00000000
5: 00000000 00000000 00000000 00000000
6: 00000003 00000002 00000001 00000000
7: 00000000 00000000 00000000 00000000
8: 00000000 00000000 00000000 00000000
9: 00000000 00000000 00000000 00000000
A: 07300401 0000007F 00000000 00000000
B: 00000000 00000000 000000FD 00000006
80000000: 80000008 00000000 00000000 00000000
80000001: 00000000 00000000 00000001 28100800
80000002: 65746E49 2952286C 6F655820 2952286E
80000003: 55504320 20202020 20202020 58202020
80000004: 30353535 20402020 37362E32 007A4847
80000005: 00000000 00000000 00000000 00000000
80000006: 00000000 00000000 01006040 00000000
80000007: 00000000 00000000 00000000 00000100
80000008: 00003028 00000000 00000000 00000000
terminating...
↧