MemoryLayer Storage · Open Source · AGPL-3.0

Stop paying to store the same bytes twice.

A content-addressed storage substrate that deduplicates and compresses everything you write — transparently. Same bytes in, far fewer bytes (and dollars) out.

Content-addressed dedup Transparent compression S3-compatible & POSIX Open source · self-host anywhere
The cost problem

Modern data workloads bury you in storage bills

AI pipelines, document processing, backups, and data lakes write enormous volumes of highly redundant data. Stored raw, the bill compounds every single month — for bytes you already have.

~41 MB

of tensors + page imagery per ingested page in our own document pipeline — the workload that made us build this. Multiply by millions of pages.

Unknown

is how much of your corpus is duplicate or near-duplicate — until you measure it. Re-uploaded files, shared boilerplate, versioned snapshots. We ship a free tool to find out.

Again

is what you pay every month, on premium object storage, for bytes you already have — and again for every replica and downstream copy.

You are paying to store the same bytes over and over, uncompressed, on premium object storage. MemoryLayer Storage stops that at the substrate.

Savings calculator

How much would you save?

Drag the sliders to match your workload. The math updates live — and stays entirely in your browser.

50 TB
10 %
10 %
1.3 ×
0.023 $/GB·mo
2 TB
0.09 $/GB

Defaults reflect mixed real-world corpora. Dedup is applied first (chunk-level, cross-object), then compression on the compressible fraction — exactly how the substrate works.

Estimated savings / year
$0
0% of your storage & bandwidth spend
1.0× less storage · 0% smaller
Stored raw100 TB
With MemoryLayer Storage100 TB
Effective footprint 100 TB
Storage saved / mo $0 (0%)
Bandwidth saved / mo $0 (0%)
Total saved / mo $0 (0%)

Estimate for planning only. Measure your real data →

How it works

Three ideas, compounding

The savings aren't a feature you toggle. They're how the storage layer is built.

Content-addressed

Every chunk is named by the hash of its bytes. Identical content is stored exactly once — automatically, forever, across every object and every tenant boundary you allow.

Cross-object dedup

Content-defined chunking (BuzHash) finds shared regions even when files differ. A global chunk→pack index dedups across your entire dataset, not just versions of one file.

Transparent compression

Per-pack zstd compression, chosen by content type — text and JSON shrink hard, media and tensors are stored raw (no wasted CPU). Dedup happens on uncompressed content, so you never trade one win for the other.

One substrate, three interfaces

Meet your data where it already lives

The same content-addressed core powers an S3 endpoint, a simple object API, and a real POSIX filesystem. Adopt whichever fits — they all dedup and compress against the same pool.

Any S3 client, unmodified

S3-compatible gateway

Drop-in SigV4 endpoint. Your existing tools, SDKs, and pipelines get dedup + compression with zero code changes.

Apps & services

Object API + SDK

Simple put/get/head/delete + presigned uploads over HTTP, with a dependency-free Python client that drops into your storage seam.

Workloads that need files

POSIX filesystem

Mount it. Region-aware metadata, durable write-back cache, copy-on-write — a real filesystem on a deduped, compressed core.

Layered architecture

L2 mlfs
POSIX filesystem (FUSE) · Postgres metadata · CoW
L1 blobgw + edge
Object gateway · S3-compatible front end · capability tokens
L0 casstore
Content-defined chunking · dedup · compression · GC
S3 · RustFS · local filesystem
Built for global Multi-region in progress

One namespace. Every office. Your control.

Content addressing makes a shared global filesystem tractable: identical bytes are identical everywhere, so a unique byte only ever needs to cross the ocean once. Concurrent multi-node access ships today. Region-aware placement and cross-region caching are what we're building toward — a self-hosted alternative to SharePoint, Dropbox, and NetApp that you own end to end.

New York local cache
London local cache
Singapore local cache
Single global namespace
deduped · compressed · content-addressed

PostgreSQL metadata system-of-record · durable write-back disk cache · ownership-lease fencing for safe concurrent writes across nodes. Per-region placement and geo-partitioned metadata are on the roadmap.

Cheaper at scale

No per-seat licensing. You pay for storage you actually use — already shrunk by dedup + compression — on infrastructure you choose.

Total control

Self-hosted, your keys, your audit trail, your data-residency rules. No third-party tenant holding your corporate files.

Fast & familiar

It's a real POSIX filesystem with a local cache — open files at LAN speed, not through a sync client or a browser.

Built for production

Correctness first, and we hold ourselves to it.

Parallel-safe, crash-recoverable, integrity-checked end to end — and measured against the standard POSIX conformance suite rather than asserted.

~99.7% pjdfstest pass rate Kubernetes CSI sanity suite green Runs in production behind MemoryLayer Enterprise

Content-defined chunking

Variable-size chunks keyed by their own content, grouped into ~16 MiB packs; shared regions dedup at chunk granularity across the whole domain.

Smart compression policy

zstd for text/JSON, raw for .safetensors/.pt/.npy/images. Self-describing pack headers — backward compatible, no re-encode.

Mark-and-sweep GC

Safe reclamation with a configurable safety window; nothing young is collected, so live writes are never at risk.

S3-compatible gateway

SigV4, multipart, streaming aws-chunked. Point any S3 client at it and gain dedup + compression transparently.

Capability edge

Ed25519 capability tokens, per-tenant dedup domains, quotas, rate limits, jti revocation, and audit for external traffic.

Integrity by construction

Content hashes verify every read. Presigned uploads are bound to an expected hash + size — no silent corruption.

POSIX filesystem

Mountable FUSE filesystem (mlfs) on the same substrate: flock/plock, xattrs, POSIX ACLs, copy-on-write via copy_file_range, Postgres metadata.

Kubernetes-native

A CSI driver provisions PVCs on the same substrate — shared-mount, multi-node access modes, verified against the Kubernetes CSI sanity suite.

Compared to

Not the only option — just the right-shaped one

JuiceFS, Ceph, SeaweedFS, and others are all solid systems. We've put together an honest side-by-side so you can decide where MemoryLayer Storage fits your stack.

JuiceFS Ceph SeaweedFS MooseFS 3FS Alluxio GlusterFS Lustre s3fs s3ql
Free analyzer · Apache-2.0

Don't trust the slider. Measure your data.

The calculator guesses. This measures. Point our free, open-source analyzer at a folder or a sample of your real data — it reports the dedup ratio, compression ratio, and projected monthly savings for your bytes. Nothing leaves your machine.

  • Runs locally — your data never leaves your infrastructure
  • One container, zero dependencies: docker run and go
  • Reads your files read-only and makes no network calls
  • Human-readable summary, or JSON to drop into a spreadsheet
Get the analyzer
terminal
# analyze a folder of your real data
$ docker run --rm -v /data:/in:ro \
    ghcr.io/scitrera/mlstore-savings /in

  Storage savings analysis
  (local, read-only — nothing left this machine)

  Logical size:         1.92 TiB
  After dedup:          0.80 TiB   (dedup ratio 2.40x)
  After compression:    0.45 TiB   (compression ratio 1.78x)
  Overall reduction:    76.6%   (4.27x smaller)

  Storage savings:      $34.62 /month   $415.46 /year
Drop-in

If you can talk to S3, you're done

No new SDK to learn. Repoint your endpoint and keep your code.

upload.py
import boto3

# Same boto3 you already use — just a new endpoint.
s3 = boto3.client("s3", endpoint_url="https://storage.internal:8080")

# Upload 10,000 model checkpoints. Duplicates collapse,
# compressible bytes shrink — automatically, server-side.
for ckpt in checkpoints:
    s3.upload_file(ckpt.path, "models", ckpt.key)

# You stored 40 TB. You're billed for ~12 TB.

See what it does to your bill

Run the open-source analyzer on your real data, then deploy MemoryLayer Storage yourself — in your cloud or on-prem. A local object gateway is one command away:

# from a checkout of scitrera/memorylayer-storage
cd blobgw && go run ./cmd/blobgw -addr 127.0.0.1:8080 -domain demo \
  -backend local -data-dir /tmp/memorylayer-demo \
  -index memory -staging memory -gc-interval 0

Open source (AGPL-3.0) · powers MemoryLayer Enterprise · available standalone · self-host anywhere

Scitrera has closed its doors and opened the source. There is no commercial licensing or paid support; questions and contributions go through GitHub.