Pipelines

Inputs, transforms,
outputs.

A named graph of nodes and edges. Read a Dataset, extract from Connect, transform with SQL, land a curated table. Batch or stream. One hosted engine.

Graph

Two reads, one transform, one write

sark.pipe.pipeline · curated_orders

Input

orders

iceberg.read

Input

customers

iceberg.read

Transform

join_orders

transform_sql

Output

curated_orders

iceberg.write

The engine never special-cases a one-off SQL job. Even a single transform is stored as nodes and edges.

Model

The builder contract

Input

Schema plus data. Bind an Iceberg Dataset, or extract rows from Connect.

Transform

SQL, join, union, split, or a branch. Input schema in, output schema out.

Output

Land a Dataset or a warehouse table. Adopt the upstream schema, or set it yourself.

Nodes

Kinds, not deploys

New capability is a new node kind on the same engine — not a new deploy.

Input

iceberg.read

Bind a Dataset as a tabular source for transforms.

Extract

connection.extract

Pull rows from a team connector — reports, lists, live ops.

HTTP

http

Call an absolute URL. Optional connection auth later.

Transform

transform_sql

Athena SQL over upstream tables. Steps or a SELECT.

Join

transform.join

Join two upstream tables into one result.

Union

transform.union

Union two or more upstream tables.

Split

transform.split

Split a table into match and remainder.

Branch

control.branch

Fork control flow. Untaken arms are skipped.

Output

iceberg.write

Land an Iceberg Dataset. Write mode chooses replace vs append.

Subgraph

graph.subgraph

Nest a graph inside a graph. Same engine.

Run

Batch, stream, and how it fires

Default

Batch

Each Submit walks the full graph. Reads bind the current tables. Incrementality lives on the write — snapshot, merge, or append.

Continuous

Stream

A long-lived deployment. Start, stop, sync. Not a run row. The graph stays the same contract.

Submit

Run the draft from the editor, or the published graph from the console, CLI, or API.

Schedule

A 5-field cron and an IANA timezone. Jobs checks due pipelines about once a minute.

Upstream

Run when a Dataset this graph reads is written or ingested.

Are

What pipelines are

  • A named graph: nodes, edges, runs, triggers
  • Hosted by Sark — web builder plus one shared engine
  • Able to read lakes, warehouses, and Connect
  • Defined in the repo. Publish pins the graph. Submit runs it

Are not

What they are not

  • One function or container per pipeline
  • Ingest (table.send) on a Dataset — that is not a pipeline
  • A required write into ontology. The graph ends at the output
  • Jobs. Jobs is the scheduler. Runs belong to the pipeline

FAQ

Frequently asked questions

Get started

Author the graph. Publish it. Submit a run.

Get started