Stores

You may be looking to further process your crul query results in a 3rd party tool. Maybe you want to create some dashboards, run some machine learning, create alerts, or another use case. This is where the concept of stores comes in.

crul has ~30 different available destinations. These can be configured in the stores page of crul web.

The crul query language includes a dedicated freeze command for writing query results to a configured 3rd party store (or storing locally to a file).

Using the freeze command

Freeze to a local file

The simplest way to freeze results is to a local file.

Example

api get https://pokeapi.co/api/v2/pokemon
|| freeze pokemon

These results can be "thawed" using the thaw command, thawed results will be immediately available, rather than having to rerun the query that generated them. You can think of results frozen locally as a permanent cache that can be read from using the thaw command. The thaw command is further described below.

Freeze to 3rd Party

Example (Kafka)

open https://...
|| filter "attributes.class == 'headline'"
|| freeze --store "kafka"

Example (Custom HTTP Endpoint)

open https://...
|| filter "attributes.class == 'headline'"
|| freeze --store "custom-http"

Thawing results

The crul query language also includes a thaw command for bringing previously frozen results directly into a query. The thaw command only works with results frozen to a local file, an Amazon S3 bucket, an Azure Blob store, or a Google Cloud Storage bucket.

Example

api get https://pokeapi.co/api/v2/pokemon
|| freeze pokemon

This query will get a list of pokemon form an api, then freeze to a local file named "pokemon".

thaw pokemon

This query will thaw the previously frozen "pokemon" file, those results are now in our query to be processed by additional stages, etc.