Skip to main content

A command line ETL

extract, transform and load operations in concise easy to read command line chains for humans and machines.

replace complex code with a few one-liners.

A mech suit for generative AI. 🕹ī¸

Dispatch prompts as a part of a crul query pipeline. Use expansion to asynchronously run the same or a tokenized prompt across a data set. Include live API/Webpage data in your prompts and chain prompts together.

|| filter "(nodeName == 'A' and parentElement.attributes.class == 'titleline')"
|| rename innerText headline
|| prompt "Write a haiku about the following headline: $headline$"

Lights, camera, 'n IT! ✨

|| find comments
|| filter "(nodeName == 'A') and (parentElement.attributes.class == 'subline')"
|| open $attributes.href$
|| filter "(attributes.class == 'comment')"

Dreamy Webpage and API data feeds. 🌜

|| freeze --store "kafka-prod" --kafka.topic "hn_comments"

Yeah, can do a whole lot... 🎨

Push incremental changes

Compare results from previous results and only retain what is new.

Learn more ↗

Schedule queries

Schedule queries to be run on a set interval.

Learn more ↗

Send data to 30+ stores

Send your data to Amazon S3, Kafka, Splunk and a whole lot more.

Learn more ↗

Client authentication for OAuth

Authenticate with multiple OAuth providers in order to access protected API data.

Learn more ↗

Credentials vault for secrets

Store custom credentials for use in queries.

Learn more ↗

Expand queries

Expand API/web links and aggregate results.

Learn more ↗

Synthesize data

Generate fully and partially synthetic data.

Learn more ↗

GPT prompting

Prompt GPT models with external data.

Learn more ↗

Vectorize data

Generate vector embeddings from data, upload to vector databases, and semantically query.

Learn more ↗

Domain throttling

Control the interval of accessing a domain based on custom prescribed rate limit policies.

Learn more ↗


Take webpage screenshots

Take full size and/or element screenshots from a webpage.

Learn more ↗


Run queries with your favorite language

Access the query results with auth keys and the REST API.

Learn more ↗

The web is yours.