Stop paying to store the same bytes twice.
A content-addressed storage substrate that deduplicates and compresses everything you write — transparently. Same bytes in, far fewer bytes (and dollars) out.
Modern data workloads bury you in storage bills
AI pipelines, document processing, backups, and data lakes write enormous volumes of highly redundant data. Stored raw, the bill compounds every single month — for bytes you already have.
of tensors + page imagery per ingested page in our own document pipeline — the workload that made us build this. Multiply by millions of pages.
is how much of your corpus is duplicate or near-duplicate — until you measure it. Re-uploaded files, shared boilerplate, versioned snapshots. We ship a free tool to find out.
is what you pay every month, on premium object storage, for bytes you already have — and again for every replica and downstream copy.
You are paying to store the same bytes over and over, uncompressed, on premium object storage. MemoryLayer Storage stops that at the substrate.
How much would you save?
Drag the sliders to match your workload. The math updates live — and stays entirely in your browser.
Defaults reflect mixed real-world corpora. Dedup is applied first (chunk-level, cross-object), then compression on the compressible fraction — exactly how the substrate works.
Estimate for planning only. Measure your real data →
Three ideas, compounding
The savings aren't a feature you toggle. They're how the storage layer is built.
Content-addressed
Every chunk is named by the hash of its bytes. Identical content is stored exactly once — automatically, forever, across every object and every tenant boundary you allow.
Cross-object dedup
Content-defined chunking (BuzHash) finds shared regions even when files differ. A global chunk→pack index dedups across your entire dataset, not just versions of one file.
Transparent compression
Per-pack zstd compression, chosen by content type — text and JSON shrink hard, media and tensors are stored raw (no wasted CPU). Dedup happens on uncompressed content, so you never trade one win for the other.
Meet your data where it already lives
The same content-addressed core powers an S3 endpoint, a simple object API, and a real POSIX filesystem. Adopt whichever fits — they all dedup and compress against the same pool.
S3-compatible gateway
Drop-in SigV4 endpoint. Your existing tools, SDKs, and pipelines get dedup + compression with zero code changes.
Object API + SDK
Simple put/get/head/delete + presigned uploads over HTTP, with a dependency-free Python client that drops into your storage seam.
POSIX filesystem
Mount it. Region-aware metadata, durable write-back cache, copy-on-write — a real filesystem on a deduped, compressed core.
Layered architecture
One namespace. Every office. Your control.
Content addressing makes a shared global filesystem tractable: identical bytes are identical everywhere, so a unique byte only ever needs to cross the ocean once. Concurrent multi-node access ships today. Region-aware placement and cross-region caching are what we're building toward — a self-hosted alternative to SharePoint, Dropbox, and NetApp that you own end to end.
PostgreSQL metadata system-of-record · durable write-back disk cache · ownership-lease fencing for safe concurrent writes across nodes. Per-region placement and geo-partitioned metadata are on the roadmap.
Cheaper at scale
No per-seat licensing. You pay for storage you actually use — already shrunk by dedup + compression — on infrastructure you choose.
Total control
Self-hosted, your keys, your audit trail, your data-residency rules. No third-party tenant holding your corporate files.
Fast & familiar
It's a real POSIX filesystem with a local cache — open files at LAN speed, not through a sync client or a browser.
Correctness first, and we hold ourselves to it.
Parallel-safe, crash-recoverable, integrity-checked end to end — and measured against the standard POSIX conformance suite rather than asserted.
Content-defined chunking
Variable-size chunks keyed by their own content, grouped into ~16 MiB packs; shared regions dedup at chunk granularity across the whole domain.
Smart compression policy
zstd for text/JSON, raw for .safetensors/.pt/.npy/images. Self-describing pack headers — backward compatible, no re-encode.
Mark-and-sweep GC
Safe reclamation with a configurable safety window; nothing young is collected, so live writes are never at risk.
S3-compatible gateway
SigV4, multipart, streaming aws-chunked. Point any S3 client at it and gain dedup + compression transparently.
Capability edge
Ed25519 capability tokens, per-tenant dedup domains, quotas, rate limits, jti revocation, and audit for external traffic.
Integrity by construction
Content hashes verify every read. Presigned uploads are bound to an expected hash + size — no silent corruption.
POSIX filesystem
Mountable FUSE filesystem (mlfs) on the same substrate: flock/plock, xattrs, POSIX ACLs, copy-on-write via copy_file_range, Postgres metadata.
Kubernetes-native
A CSI driver provisions PVCs on the same substrate — shared-mount, multi-node access modes, verified against the Kubernetes CSI sanity suite.
Not the only option — just the right-shaped one
JuiceFS, Ceph, SeaweedFS, and others are all solid systems. We've put together an honest side-by-side so you can decide where MemoryLayer Storage fits your stack.
Don't trust the slider. Measure your data.
The calculator guesses. This measures. Point our free, open-source analyzer at a folder or a sample of your real data — it reports the dedup ratio, compression ratio, and projected monthly savings for your bytes. Nothing leaves your machine.
- Runs locally — your data never leaves your infrastructure
- One container, zero dependencies: docker run and go
- Reads your files read-only and makes no network calls
- Human-readable summary, or JSON to drop into a spreadsheet
# analyze a folder of your real data
$ docker run --rm -v /data:/in:ro \
ghcr.io/scitrera/mlstore-savings /in
Storage savings analysis
(local, read-only — nothing left this machine)
Logical size: 1.92 TiB
After dedup: 0.80 TiB (dedup ratio 2.40x)
After compression: 0.45 TiB (compression ratio 1.78x)
Overall reduction: 76.6% (4.27x smaller)
Storage savings: $34.62 /month $415.46 /year If you can talk to S3, you're done
No new SDK to learn. Repoint your endpoint and keep your code.
import boto3
# Same boto3 you already use — just a new endpoint.
s3 = boto3.client("s3", endpoint_url="https://storage.internal:8080")
# Upload 10,000 model checkpoints. Duplicates collapse,
# compressible bytes shrink — automatically, server-side.
for ckpt in checkpoints:
s3.upload_file(ckpt.path, "models", ckpt.key)
# You stored 40 TB. You're billed for ~12 TB. See what it does to your bill
Run the open-source analyzer on your real data, then deploy MemoryLayer Storage yourself — in your cloud or on-prem. A local object gateway is one command away:
# from a checkout of scitrera/memorylayer-storage
cd blobgw && go run ./cmd/blobgw -addr 127.0.0.1:8080 -domain demo \
-backend local -data-dir /tmp/memorylayer-demo \
-index memory -staging memory -gc-interval 0 Open source (AGPL-3.0) · powers MemoryLayer Enterprise · available standalone · self-host anywhere
Scitrera has closed its doors and opened the source. There is no commercial licensing or paid support; questions and contributions go through GitHub.