Exporting results to 3rd party data lakes or other destinations is straightforward with the help of the
freeze command. With the
freeze command, we can easily push the results of our query directly to a preconfigured 3rd party store, or save the results locally to a file.
Results that are saved locally or in read/write stores (ex. Amazon S3, Google Cloud Storage, Azure Blob Storage) can be retrieved and pushed into a query pipeline using the
Example: Freezing to a local file
|| freeze mylocalfile
This query will save results from the previous stage to a local file set by the first argument. If results have already been written to a local file with the same name, the results will be overwritten, otherwise a new file will be created. You can use tokens in constructing names for uniqueness or filtering.
freezing, the stored results can accessed using the
All locally frozen results can be found in the cellar.
Example: Freezing to a Kafka topic
|| freeze --store "kafka-prod" --kafka.topic "mytopic"
This query shows how to export results to a (preconfigured) 3rd party store. Unlike in the local file query, there is no argument provided to the
freeze command, instead a
--store flag is set to a preconfigured store.
In this query the
--store was configured with the name
kafka-prod and is a Kafka
type store. The freeze command also sets the
--kafka.topic flag which is a store specific flag (
splunk.*, etc.) to control destination settings, the
--kafka.topic flag will override the default configured Kafka topic.