I have a REST API which returns a very large, but valid, JSON payload. The structure of this JSON is a single array of many objects. Last I checked the response is around 1.2mb or roughly 1million chars. Here is a sample of the JSON, pretty printed (the actual response contains no newlines):
[
{
"barcode": "10010208",
"comment": null,
"flagged": 1,
"fromCode": "war_rep",
"fromStation": "Warehouse Repair",
"lastTrackScan": "12/10/2015 12:31:48 AM",
"muted": true,
"priority": "RED",
"reservations": 1,
"sku": "TB44_10",
"toCode": "war_rep",
"toStation": "Warehouse Repair"
},
{
"barcode": "10011135",
"comment": null,
"flagged": 1,
"fromCode": "cus_rec",
"fromStation": "Customer Receiving",
"lastTrackScan": "12/09/2015 10:17:12 AM",
"muted": true,
"priority": "RED",
"reservations": 2,
"sku": "RR52_8",
"toCode": "ins",
"toStation": "Pre-Inspection"
},
... many more
]
After adding a REST data input that made an HTTP GET req once every 60s in Splunk, we were able to successfully have this JSON broken into events, with one event per object. But following an upgrade, this stopped working. Now, the payload isn't parsed as JSON but appears to be treated as a single event, and is truncated at 10,000 chars. We're still using `sourcetype="_json"` but somehow this isn't working.
We're using Splunk Enterprise 6.3.1
↧