Splunk DB Connect limits

I wanted to use Splunk DB Connect to automatically incrementally query data from a database server based on data found in another database server.

So the idea was this:

Use DB Query to get the most recent date found in the destination database table, pipe that to a “map” command that allows me to specify the second dbquery on the source database table (specifying the results of the first dbquery as the starting date), and then doing some filtering and transforms, and then using dboutput to send the results to the destination table.

Well, that doesn’t work. It errors out with some null pointer excetption. It all works fine as soon as I remove the “map” command segment, which means I have no way to perfectly get only those records that are new since the last extraction.

My guess is that since the “map” command fires of one query per result row from the previous search, it means the dboutput can’t reference a single search to use as the source of data it will insert. I suspect it may be possible they could fix this so it looks at whatever the results were that are being piped into it. Currently, I am guessing it is looking for the base search.

I expect the current behavior is actually expected, so I bet the behavior I’m hoping for would be a feature request.

Here’s the Splunk Answers question I submitted for this:

http://ift.tt/1udaB7N