However, multiple trials confirm that when given 1 million events to process, the custom search command takes an average of 00:09:10, while the external lookup can do it in 00:07:06. which means the geoip database gets reloaded and caching is wiped out, leading one to hypothesize that the external lookup version should perform much worse.
Note that both command versions are written in Python, use the same geoip lookup library with the same caching flag, and make the same lookup function calls.Īlso note that while the custom streaming command is only dispatched/invoked once and events are passed in chunks, Splunk seems to re-invoke the external lookup script every 255 events. Geoip_db = pygeoip.GeoIP(ISP_DB_PATH, ipasnCommand(StreamingCommand):ĭispatch(ipasnCommand, sys.argv, sys.stdin, sys.stdout, _name_) Loads geoip database and defines custom streaming command like so: #create GeoIP instance with Memory Cache Writes each line (event) back to stdout using csv moduleĢ. Performs a geoip lookup on each event's ip field, stores result in new field Uses the csv module to read events from std_in Loads geoip database into memory with MEMORY_CACHE flag Here is a break-down of the two command versions. Why might this be? Is there a reason custom search commands could actually be slower than equivalent external lookup scripts? I tested the performance of both commands, and found that the external lookup script was much faster.
Ip2location of my laptop full#
I wrote two versions of the same Python streaming command: one as a simple external lookup script, and one as a full custom search command (using V2 of the custom search command protocol).
0 kommentar(er)
