Get it — container
No install, nothing on the host.
$ docker run --rm -v "$PWD:/data" \
mancube/shapeshift \
infer -i /data/events.jsonl
Apache-2.0 · one static binary · v0.1.2
Ragged JSON in.
Typed columns out.
shapeshift streams JSON and JSONL into Parquet or a self-contained Apache Iceberg v2 table under one small declarative spec. No JVM, no catalog server, no Python, no row cap, no telemetry — and peak memory that does not move when the input grows eight hundred times.
events.jsonl — nested, ragged, untyped
{"id": 1, "user": {"name": "Ada", "plan": "pro"},
"amount": 12.50, "day": "2026-07-13",
"tags": ["a","b"], "active": true}
{"id": 3, "user": {"name": "Linus"},
"amount": 99.99, "tags": ["x"], ...}
not-json-a-bad-line
{"id": 4, "user": {"name": "Edsger", ...},
"amount": 0, "active": true}
events.parquet — flat, aligned, typed
active Boolean amount Float64 day Date32 event_at Timestamp(Microsecond, None) id Int64 tags Utf8 user.name Utf8 user.plan Utf8
Nested objects flatten to dotted columns. YYYY-MM-DD is detected as
DATE, RFC3339 as TIMESTAMP(µs), arrays land as a
JSON-encoded leaf. The un-parseable line is counted and written to a reject
sidecar — it does not abort the run and it is not silently dropped.
§01 — how to use it
Four commands, and you are done
Every block below is real output from the shipped
examples/events.jsonl — four good records and one line
that is deliberately not JSON.
01
Infer a spec from a sample
Point it at a file. It prints a ready-to-edit YAML spec — the columns, types and coercion policy it would use. This is the governance surface: write it once per source shape, review it, diff it in a pull request.
$ shapeshift infer -i examples/events.jsonl
dataset: events
columns:
- {name: amount, type: float64, required: false}
- {name: day, type: date, required: false} # YYYY-MM-DD detected
- {name: event_at, type: timestamp, required: false} # RFC3339 detected
- {name: tags, type: json, required: false} # arrays -> JSON leaf
- {name: user.name, type: string, required: false} # nested -> dotted
# ...eight columns in total
02
Shape it
With no --spec it infers on the fly. The bad line is counted, not fatal, and its raw text is preserved in a sidecar with the line number and the parser's error.
$ shapeshift shape --input examples/events.jsonl --output events.parquet
shaped `events` → events.parquet (Parquet)
rows_in=4 rows_out=4 rejected=0 parse_errors=1 row_groups=1 bytes=2588
malformed / rejected rows → events.parquet.rejects.jsonl (1 parse, 0 shaped-out)
$ cat events.parquet.rejects.jsonl
{"error":"ExpectedNull at character 0 ('n')","line":4,"raw":"not-json-a-bad-line"}
- is standard input, and the source is read once, so cat x.jsonl | shapeshift shape -i - -o out.parquet shapes exactly what the file would.
03
Inspect it — footer only, no data scan
$ shapeshift inspect events.parquet
parquet file: events.parquet
rows: 4
row-groups: 1
columns (8):
active: Boolean amount: Float64
day: Date32 event_at: Timestamp(Microsecond, None)
id: Int64 tags: Utf8
user.name: Utf8 user.plan: Utf8
04
Or land an Iceberg v2 table instead
Same input, same spec, one flag. The table directory is self-contained — data, manifest, manifest list, table metadata and a version hint. No catalog server is needed to write it or to read it.
$ shapeshift shape -i examples/events.jsonl -o events_tbl --to iceberg
shaped `events` → events_tbl (Iceberg)
rows_in=4 rows_out=4 rejected=0 parse_errors=1 row_groups=1 bytes=3212
iceberg table metadata → events_tbl/metadata/v1.metadata.json
events_tbl/
data/<uuid>.parquet # carries PARQUET:field_id 1..N
metadata/<uuid>-m0.avro # manifest, field-ids intact
metadata/snap-<snapshot-id>-1-<uuid>.avro # manifest list
metadata/v1.metadata.json
metadata/version-hint.text
Get it — binary
A shapeshift-v* tag builds an x86_64-unknown-linux-musl binary, asserts it is statically linked and boots, and attaches it with a .sha256 to the GitHub release.
$ sha256sum -c shapeshift-<ver>-\
x86_64-unknown-linux-musl.sha256
Get it — source
$ rustup target add x86_64-unknown-linux-musl
$ cargo build --release \
--target x86_64-unknown-linux-musl \
-p shapeshift-cli
One file, no runtime deps.
§02 — measured, not claimed
Peak memory does not move
Every number on this page comes from
benchmarks/bench.py — python3 stdlib and cargo, no other
dependencies. Datasets are generated by a deterministic formula, and every
child process is measured with per-child ru_maxrss. Run it on
your hardware and compare.
14.1 MiB
Peak RSS shaping a 1,260 MB input — 8 million rows.
~820×
Input growth across the sweep, for a 1.4× move in peak memory.
273k rows/s
Steady from 1M to 8M rows on the shipped musl binary.
3.0 ms
Median cold start — exec to four rows shaped to exit.
| bench | rows | input | wall | rows/s | peak RSS |
|---|---|---|---|---|---|
| jsonl→parquet | 10,000 | 1.5 MB | 0.06s | 157,980 | 9.9 MiB |
| jsonl→iceberg | 10,000 | 1.5 MB | 0.06s | 163,567 | 9.9 MiB |
| jsonl→parquet | 1,000,000 | 157 MB | 3.85s | 259,769 | 12.3 MiB |
| jsonl→iceberg | 1,000,000 | 157 MB | 3.84s | 260,646 | 12.4 MiB |
| jsonl→parquet | 8,000,000 | 1,260 MB | 29.3s | 273,495 | 14.1 MiB |
| jsonl→iceberg | 8,000,000 | 1,260 MB | 29.7s | 269,506 | 14.1 MiB |
Machine: Intel Xeon @ 2.80 GHz (4 cores), 15.7 GiB RAM · rustc 1.94.1 · release build · shipped musl-static binary (~4.0 MB, stripped). One stated machine — evidence that the design property holds, not a promise about yours.
The flat curve is what the row-group == batch design promises: the
shaper flushes a RecordBatch every row_group_rows
(default 50,000) and the sink closes that row group immediately, so memory
is bounded by the row group rather than the input. The Iceberg path costs
almost nothing over Parquet, because it is the Parquet writer plus a
metadata tail.
The unflattering half
The shipped musl binary is about 1.34× slower than the same engine
built against glibc — 273k rows/s against 367k at 8M rows. In exchange,
glibc's peak RSS appears to grow with the input, reaching 67 MiB
where musl stays flat at 14. That is allocator retention, not a leak:
MALLOC_MMAP_THRESHOLD_=65536 on the same glibc binary is flat at
16 MiB. Pick your binary by what you are optimising for. The harness
measures both.
§03 — the work behind the number
273k rows/s was 121k, in three steps
The same sweep on the same machine, at 8 million rows. Each step is A/B'd in the benchmark document, and the earlier rows are kept as measured rather than quietly restated.
| step | what changed | rows/s, musl, 8M |
|---|---|---|
| baseline | allocating three times per row | 121,185 |
| step 1 | removed the per-row allocations | 149,649 |
| step 2 | overlapping the writer — no gain on musl at the time | 149,649 |
| step 3 | parsing onto a reusable tape | 273,495 |
The record model used to be a serde_json::Value — a tree of
small heap allocations, one map per object and a string per key, built and
thrown away for every record. It now parses onto a reusable simd-json
tape: a flat vector of nodes whose strings point back into the read
buffer. In the steady state nothing is allocated per record.
Read the gap as well as the rows. Parsing 1M records, the owned model is 2.1× slower on musl than on glibc; the tape only 1.16×. With nearly nothing to allocate, the allocator nearly stops mattering — which is why this, and not a different allocator, was the answer.
| parse only, 1M records | glibc | musl |
|---|---|---|
owned serde_json::Value | 540k rec/s | 256k rec/s |
| borrowed value | 855k rec/s | 488k rec/s |
| tape, reused buffers | 1,849k rec/s | 1,592k rec/s |
The negative result is the useful half
The first attempt at threading put the source on the far thread and
handed
Values across. It measured about 2× slower than
single-threaded on both targets — 0.47× on glibc, 0.54× on
musl. A Value is a tree of small allocations, so shipping one
across means every record is allocated on one core and freed on another. An
Arrow RecordBatch is the opposite shape: a few large contiguous
buffers behind an Arc. What crossed the boundary mattered
far more than whether anything did.
Output never changed through any of this: the shaped Parquet is byte-identical
across the pipeline flag and across both record models, on both targets, and
tape_and_serde_agree_on_every_accessor asserts the two readers
answer identically. It caught two real divergences — simd-json's
as_f64 refuses an integer node, so 3 in a
float64 column would have been written as null.
§04 — the correctness edge
A plausible Iceberg table can be silently unreadable
Iceberg has a sharp edge: the field-id chain. A table can look
completely well-formed and still fail to open, and that failure is the whole
reason this write path exists.
The general Rust Avro crate drops the field-id attributes on the
manifest schema. A reader then hits
No default expression in FieldId Map and the table is dead. So
shapeshift emits the Avro Object Container File itself — a roughly
150-line hand-rolled encoder — specifically to preserve them, and the
data Parquet carries PARQUET:field_id 1..N to match. There is no
Avro crate in the dependency tree at all.
The acceptance gate is an engine we did not write. DuckDB 1.5.4 reads
both paths and returns the same rows and the same aggregates — four
rows, sum(amount) = 115.49, identical from
read_parquet() and iceberg_scan().
Self-contained
Each run writes the data files, manifest, manifest list, v{N}.metadata.json and a version hint. No catalog service to produce it or to read it.
Multi-snapshot
--append adds a snapshot to an existing table, with per-column manifest statistics.
Partitioned
--partition-by gives identity partitioning — one data file per value under data/<col>=<value>/.
C-free by default
parquet is built Snappy-only, so no zstd or brotli C codecs are linked. That is what keeps the binary musl-static. --features zstd is the one sanctioned exception — it writes 11 MB against Snappy's 20 MB on the same 1M rows, for about 2% more wall time.
§05 — where it loses
shapeshift has zero connectors, and that is on purpose
It is the transform-and-land layer, not the movement layer. It owns one job: turn semi-structured JSON into a governed columnar table from a spec you can read and diff. Here is where the incumbents are simply better.
| capability | shapeshift | Fivetran / Airbyte | dlt | Spark / PyIceberg |
|---|---|---|---|---|
| Runtime | one static binary | managed service | Python library | JVM / Python |
| Connector breadth | none | hundreds — best | growing | ecosystem-wide |
| Incremental / CDC today | no | yes | yes | yes |
| Managed catalog | no | yes | via engine | best |
| Billing model | no row cap, no telemetry | MAR / credit | self-run | self-run |
| Iceberg v2 write, field-ids | yes, no catalog | managed dest | emerging | best |
| Declarative transform spec | yes — the boundary | partial (SQL) | code | code |
| Bounded-RAM streaming | yes | managed | yes | cluster |
If your problem is “reliably pull from 40 SaaS APIs”, a managed ELT vendor wins outright and shapeshift is not in the running. If your team lives in Python and wants pipelines-as-code, dlt is a joy. If you need a managed catalog across an estate, that is Spark and PyIceberg's home ground.
Be precise about schema drift too. shapeshift detects drift within a run — undeclared fields, values that stopped coercing — and gives you five ways to handle it. What it does not do is manage the schema over time: propagate an added column on the next scheduled sync, keep a drift history, alert on it. That is stateful cross-run work. The honest summary is that shapeshift makes drift visible and survivable; a managed vendor makes it someone else's problem.
§06 — the economics, without the sales pitch
The cost command will tell you not to bother
Managed ELT bills by Monthly Active Rows: every distinct key inserted, updated
or deleted, on a tiered curve. Re-touching the same rows pays again.
shapeshift cost makes the trade arithmetic instead of rhetoric
— and it never guesses a vendor price. You supply your plan's
effective rate.
# a realistic monthly volume, where self-hosting amortises
$ shapeshift cost --rows 50000000 --vendor-per-million 0.75 --self-host-cost 12.00
billable rows (MAR-equivalent): 50000000
vendor: $37.50 (@ $0.75/million MAR)
self-host: $12.00
saved: $25.50 (68.0%)
# and the four-row sample, which does not amortise its own compute
$ shapeshift cost -i examples/events.jsonl --vendor-per-million 0.75 --self-host-cost 0.01
billable rows (MAR-equivalent): 4
vendor: $0.00 (@ $0.75/million MAR)
self-host: $0.01
saved: $-0.01 (-333233.3%)
The negative number is not a bug. It is the point: the tool shows you when self-hosting does not pay, rather than selling you a figure.
Apache-2.0
Shape unbounded rows for the cost of the CPU that does it
The engine is open source, complete on its own, and has no row cap and no phone-home. Read it before you run it.
Not an orchestrator — that is dagron. Not a connector catalog. Not a query engine: it writes Parquet and Iceberg for DuckDB, Spark and Trino to read. It is the shaping data-plane, and it does that one job completely.