Benchmarks
dsh-dedup
LayerFS against Btrfs
vs Btrfs- LayerFS
- core at main e5c7aaae05, through the JS SDK
- Against
- Btrfs snapshots of the same trees (2026-09-09); Git 2.47.1 pack
- Store
- Disposable · depth 16/8/8 · one construction worker
- Host
- arm64 Mac · Docker Desktop, linux/arm64, 8 CPUs, 8.3 GB
- Load
- 6.0–12.6 (1 min)
All 157 snapshots
100.1 MB
every snapshot a Commit · 156/156 read back from history
19.9×
less space
1,994.6 MB
a read-only snapshot per checkpoint · 157/157 verified
- LayerFS v1.0.0100.1 MB156/156 Commits read back
- Btrfs + zstd1,994.6 MB19.9×157/157 verified
- Btrfs + zstd + bees2,068.9 MB20.7×incl. 33.6 MB bees state · 157/157 verified
- Btrfs, no compression3,209.1 MB32.1×157/157 verified
Same 157 trees and the same rule (a changed file is rewritten whole). LayerFS: the closed Store, allocated. Btrfs: btrfs filesystem usage Used, plus the dedupe tool's own state.
Why snapshots cost Btrfs so much
A rewrite is new data
Snapshots share blocks that stay where they are. A file rewritten in full is new data to Btrfs, so each version takes its own space. LayerFS sees the new version against the old one at Commit and keeps only what changed.
Dedupe after the fact moves the cost
bees merged 373 MB of duplicate data, and the metadata that records the sharing across 157 snapshots grew by 414 MB. Net: more space, not less.
Small files, doubled metadata
About two thirds of the distinct files are under 8 KiB, and Btrfs keeps two copies of its metadata by default. LayerFS packs the history structure itself: the namespace is content too.
Every stride
Every 10th, every 3rd and every snapshot. The closed Store, split by its own totals; bars share one scale.
Stride 10
17 snapshots · 0.56 GB53.1 MB
10.6× smaller than the trees
- payload / metadata / other
- 41.5 / 1.3 / 10.3 MB
- Σ Commit
- 22.4 s for 16
- history proof
- 16/16 Commits
Stride 3
53 snapshots · 1.68 GB69.5 MB
24.1× smaller than the trees
- payload / metadata / other
- 51.9 / 2.5 / 15.1 MB
- Σ Commit
- 45.1 s for 52
- history proof
- 52/52 Commits
Stride 1
157 snapshots · 4.94 GB100.1 MB
49.3× smaller than the trees
- payload / metadata / other
- 67.5 / 5.5 / 27.1 MB
- Σ Commit
- 97.9 s for 156
- history proof
- 156/156 Commits
How history grows
Stored payload added state by state. Closer snapshots each add less: stride 1 keeps 9× as many trees as stride 10 for 1.6× the payload.
snapshot (1 = Commit 100, 157 = Commit 15,632) →
How long a Commit takes
Each Commit at stride 1, in order (tallest 2.8 s)
A Commit costs what its change costs: 628 ms on average per snapshot step, about 2–3 s for the largest (thousands of changed files). Each is the client's wall time around sandbox.commit.
LayerFS and Git
Git packs the same trees smaller, and it gets there a different way: an offline pass over a finished history with deep delta chains. LayerFS keeps history live.
LayerFS: history you can work in
- Each snapshot is committed the moment the Workspace writes it. No repack, no second pass.
- Every delta chain is capped (16 whole-file, 8 chunk, 8 metadata), so reading any version back is bounded work, however long history gets.
- Every Commit stays a mountable, writable filesystem: all 156 were forked, mounted and read back here.
Git: an archive of a finished history
- Packed after the fact:
git repack -a -d -f -F --window=10 --depth=50over all the trees at once. - Deltas chain up to 50 deep, so an old file is rebuilt through up to 50 steps.
- A pack is not a filesystem: to use a version you check it out into a working tree first.
How it ran
Only through the JS SDK
Every step is a public @layerfs/sdk call, as an agent would make it; every command in the Sandbox goes through sandbox.exec.
- project.init(state 1)Store policy 16/8/8
- sandbox.create("dedup", project, image)the corpus is in the image
- branch.fork("dedup", "main")its own Branch
- sandbox.mount("dedup", "dedup")once per segment
- sandbox.exec(apply.py prev k)rewrite changed files in place
- sandbox.exec(verify.py k)every path against the oracle
- sandbox.commit("dedup", path, message)timed as Commit
- branch.fork("at-k", commit)mount + verify every Commit
Checked, not assumed
The 157 trees (75,929 distinct files, 892 MB) are built into the Sandbox image. Each step rewrites changed files and is checked against an independent oracle (every path's type, mode, size and SHA-256) before it is committed. After the run, every Commit is forked, mounted and checked again.
Btrfs kept the same trees as a read-only snapshot per checkpoint on a fresh loop device, with the same importer rule, and read all 157 back the same way.
Read with care
One sample each
Each stride ran once on Docker Desktop on an arm64 Mac at 1-minute load 6–13. Times move between runs; bytes much less.
Separate Btrfs run
The Btrfs numbers come from a September run on the same trees with the same importer, not from a paired run with v1.0.0. Btrfs Used includes its doubled metadata.
Warm by its own writes
Each Commit reads what the step just wrote through the mount; nothing else is pre-touched. Mount, unmount and the checks are outside every timer.
Open item
v1.0.0 Stores are 5–18% larger than an earlier in-process prototype of ours; the gap is recorded and being traced.
Data: benchmark/data/dsh-dedup/v1.0.0/ in the LayerFS repository (results, environment, every run's receipts and record,btrfs-reference.csv, the storage regression) and the Btrfs report in docs/research/layerfs-vs-btrfs-157/.