We have simple csv lookup like:
network,descr
192.168.0.0/24,network_name
Lookup description in transforms.conf:
[networklist_allocs_all]
filename = networklist_allocs_all.csv
max_matches = 1
min_matches = 1
default_match = OK
match_type = CIDR(network)
Any search command like:
...
| lookup networklist_allocs_all network AS src_ip
...
too often crashing splunk on the indexers. It is about 5-10 crashlogs on the each indexers per day.
Part of crashlog at the end of question.
How can we resolve this situation? Seems that splunk began to crash after update from 7 to 8 version. We did't any changes in lookup format or definition.
Unfortunately we can't open support case for some reason, so ask for community help.
crash-xx.log:
[build 6db836e2fb9e] 2020-02-13 17:00:56
Received fatal signal 11 (Segmentation fault).
Cause:
No memory mapped at address [0x0000000000000058].
Crashing thread: BatchSearch
Registers:
RIP: [0x0000564E2F44470F] _ZN14LookupMatchMap16mergeDestructiveERS_ + 31 (splunkd + 0x223670F)
RDI: [0x00007FC9C01FCA80]
RSI: [0x0000000000000000]
RBP: [0x0000000000000000]
RSP: [0x00007FC9C01FC9A0]
RAX: [0x00007FC9CDB3AE08]
RBX: [0x0000000000000000]
RCX: [0x0000000000000001]
RDX: [0x00007FC9BFB7F608]
R8: [0x00007FC9C01FC9C0]
R9: [0x00007FC9C01FC9BF]
R10: [0x0000000000000010]
R11: [0x0000000000000080]
R12: [0x00007FC9928E33C0]
R13: [0x00007FC9C01FCA80]
R14: [0xAAAAAAAAAAAAAAAB]
R15: [0x00007FC9BFB7F600]
EFL: [0x0000000000010246]
TRAPNO: [0x000000000000000E]
ERR: [0x0000000000000004]
CSGSFS: [0x0000000000000033]
OLDMASK: [0x0000000000000000]
OS: Linux
Arch: x86-64
Backtrace (PIC build):
[0x0000564E2F44470F] _ZN14LookupMatchMap16mergeDestructiveERS_ + 31 (splunkd + 0x223670F)
[0x0000564E2F444EE8] _ZN14UnpackedResult8finalizeEv + 168 (splunkd + 0x2236EE8)
[0x0000564E2F445E26] _ZN18LookupDataProvider6lookupERSt6vectorIP15SearchResultMemSaIS2_EERK17SearchResultsInfoR16LookupDefinitionPK22LookupProcessorOptions + 2118 (splunkd + 0x2237E26)
[0x0000564E2F451B7F] _ZN12LookupDriver13executeLookupEP29IFieldAwareLookupDataProviderP15SearchResultMemR17SearchResultsInfoPK22LookupProcessorOptions + 367 (splunkd + 0x2243B7F)
[0x0000564E2F451C22] _ZN18SingleLookupDriver7executeER18SearchResultsFilesR17SearchResultsInfoPK22LookupProcessorOptions + 98 (splunkd + 0x2243C22)
[0x0000564E2F43BDFF] _ZN15LookupProcessor7executeER18SearchResultsFilesR17SearchResultsInfo + 79 (splunkd + 0x222DDFF)
[0x0000564E2F08FC7D] _ZN15SearchProcessor16execute_dispatchER18SearchResultsFilesR17SearchResultsInfoRK3Str + 749 (splunkd + 0x1E81C7D)
[0x0000564E2F07F528] _ZN14SearchPipeline7executeER18SearchResultsFilesR17SearchResultsInfo + 344 (splunkd + 0x1E71528)
[0x0000564E2F1931D9] _ZN16MapPhaseExecutor15executePipelineER18SearchResultsFilesb + 153 (splunkd + 0x1F851D9)
[0x0000564E2F193901] _ZN25BatchSearchExecutorThread13executeSearchEv + 385 (splunkd + 0x1F85901)
[0x0000564E2DEDCB8F] _ZN20SearchExecutorThread4mainEv + 47 (splunkd + 0xCCEB8F)
[0x0000564E2EC41AC8] _ZN6Thread8callMainEPv + 120 (splunkd + 0x1A33AC8)
[0x00007FC9CC7B76BA] ? (libpthread.so.0 + 0x76BA)
[0x00007FC9CC4EC41D] clone + 109 (libc.so.6 + 0x10741D)
Linux / c-6.index.splunk / 4.4.0-21-generic / #37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 / x86_64
/etc/debian_version: stretch/sid
...
Last errno: 0
Threads running: 13
Runtime: 436566.363455s
argv: [splunkd -p 8089 start]
Process renamed: [splunkd pid=8371] splunkd -p 8089 start [process-runner]
Process renamed: [splunkd pid=8371] search --id=remote_d-0.search.splunk_scheduler__gots_eWFuZGV4LWFsZXJ0cw__RMD5d361bb2fcae608a1_at_1581602460_31361_374F9CE8-43DB-48C4-9F22-7982EC4B6AD5 --maxbuckets=0 --ttl=60 --maxout=0 --maxtime=0 --lookups=1 --streaming --s
idtype=normal --outCsv=true --acceptSrsLevel=1 --user=gots --pro --roles=admin:power:user
Regex JIT enabled
RE2 regex engine enabled
using CLOCK_MONOTONIC
Preforked process=0/59436: process_runtime_msec=54987, search=0/188869, search_runtime_msec=1240, new_user=Y, export_search=Y, args_size=356, completed_searches=3, user_changes=2, cache_rotations=3
Thread: "BatchSearch", did_join=0, ready_to_run=Y, main_thread=N
First 8 bytes of Thread token @0x7fc9c70845e8:
00000000 00 e7 1f c0 c9 7f 00 00 |........|
00000008
SearchExecutor Thread ID: 1
Search Result Work Unit Queue: 0x7fc9b77ff000
Search Result Work Unit Queue Crash Reporting
Type: NON_REDISTRIBUTE
Number of Active Pipelines: 2
Max Count of Results: 0
Current Results Count: 352
Queue Current Size: 0
Queue Max Size: 200, Queue Drain Size: 120
FoundLast=N
Terminate=N
Total Bucket Finished: 1.0012863152522, Total Bucket Count: 16
===============Search Processor Information===============
Search Processor: "lookup"
type="SP_STREAM"
search_string=" networklist_allocs_all network AS dest_ip "
normalized_search_string="networklist_allocs_all network AS dest_ip"
litsearch="networklist_allocs_all network AS dest_ip "
raw_search_string_set=1 raw_search_string=" networklist_allocs_all network AS dest_ip "
args={StringArg: {string="networklist_allocs_all" raw="networklist_allocs_all" quoted=0 parsed=1 isopt=0 optname="" optval=""},StringArg: {string="network" raw="network" quoted=0 parsed=1 isopt=0 optname="" optval=""},StringArg: {string="AS" raw="AS"
quoted=0 parsed=1 isopt=0 optname="" optval=""},StringArg: {string="dest_ip" raw="dest_ip" quoted=0 parsed=1 isopt=0 optname="" optval=""}}
input_count=885
output_count=0
directive_args=
==========================================================
↧