Skip to main content

Structure any Web/API data

Enterprise data exists in difficult-to-access APIs and Web sources. Crul effortlessly extracts and transforms data at the edge for use with every major data pipeline and data lake.

It's all we do, and we do it really well.

Lights, camera, 'n IT! ✨

open https://news.ycombinator.com/news
|| find comments
|| filter "(nodeName == 'A') and (parentElement.attributes.class == 'subline')"
|| open $attributes.href$
|| filter "(attributes.class == 'comment')"

Dreamy Webpage and API data feeds. 🌜

...
|| freeze --store "kafka-prod" --kafka.topic "hn_comments"

A mech suit for generative AI. 🕹ī¸

Dispatch prompts as a part of a crul query pipeline. Use expansion to asynchronously run the same or a tokenized prompt across a data set. Include live API/Webpage data in your prompts and chain prompts together.

open https://news.ycombinator.com/news
|| filter "(nodeName == 'A' and parentElement.attributes.class == 'titleline')"
|| rename innerText headline
|| prompt "Write a haiku about the following headline: $headline$"

Yeah, can do a whole lot... 🎨

Push incremental changes

Compare results from previous results and only retain what is new.

Learn more ↗

Schedule queries

Schedule queries to be run on a set interval.

Learn more ↗

Send data to 30+ stores

Send your data to Amazon S3, Kafka, Splunk and a whole lot more.

Learn more ↗

Client authentication for OAuth

Authenticate with multiple OAuth providers in order to access protected API data.

Learn more ↗

Credentials vault for secrets

Store custom credentials for use in queries.

Learn more ↗

Expand queries

Expand API/web links and aggregate results.

Learn more ↗

Synthesize data

Generate fully and partially synthetic data.

Learn more ↗

GPT prompting

Prompt GPT models with external data.

Learn more ↗

Vectorize data

Generate vector embeddings from data, upload to vector databases, and semantically query.

Learn more ↗

Domain throttling

Control the interval of accessing a domain based on custom prescribed rate limit policies.

Learn more ↗

screenshot

Take webpage screenshots

Take full size and/or element screenshots from a webpage.

Learn more ↗

api

Run queries with your favorite language

Access the query results with auth keys and the REST API.

Learn more ↗

The web is yours.