mongo-stats

A swamp repo that measures and garbage-collects a shared @keeb/mongodb-datastore. It is also published as the @keeb/mongodb-stats extension — swamp extension pull @keeb/mongodb-stats adds both methods to any @keeb/mongodb model.

It stores nothing of its own. It exists to report on the other repos that share that datastore — every swamp repo whose data lives in the swamp database as a t_<tenantId>_r_<namespace>_* collection triple.

Layout

Path	What
models/@keeb/mongodb/	The one instance, treehouse — db swamp, URI from vault.get("mongo", "uri")
extensions/models/mongodb_datastore_stats.ts	datastore_stats method
extensions/models/mongodb_datastore_gc.ts	datastore_gc method
workflows/	The datastore-gc workflow (baseline → gc → verify)
bench/	Datastore-unit and end-to-end benchmarks against the real source

Both extensions layer onto the published @keeb/mongodb model, so the base query/find/insert methods are available on treehouse too.

How the datastore is laid out

One namespace = three collections, prefixed t_<tenantId>_r_<namespace>_:

• _paths — { _id: relPath, hash, size, updatedAt, deletedAt }
• _blobs — { _id: sha256, size, ...bytes }, content-addressed, so duplicates collapse server-side
• _locks — distributed lock docs

relPath is rooted at a datastore subdir (data, outputs, workflow-runs, bundles, audit, telemetry, logs, …), which is what the per-subdir byte breakdown keys off.

Usage

How big is it?

swamp model method run treehouse datastore_stats
swamp data query 'modelName == "treehouse" && name == "queryResults"' \
  --select 'attributes.totals'

Aggregates every namespace server-side: active/deleted file counts, logical bytes by subdir, blob store size + count, and lock totals. Add attributes.repos for the per-namespace breakdown, or scope the run with --input namespace=<ns> / --input tenantId=<id>.

What would a GC reclaim?

swamp workflow run datastore-gc --input dryRun=true

Pulls a janitor mirror per namespace and runs swamp data gc --dry-run. Mutates nothing. Report gc.versionsDeleted and gc.bytesReclaimed.

Actually GC

swamp workflow run datastore-gc
swamp data query 'modelName == "treehouse" && name == "gcResults"' \
  --select 'attributes.totals'

Baseline stats → gc every namespace → verify stats, so the before/after delta lands as durable data.

Every method overwrites the instance's queryResults resource: stats land under key queryResults, gc under gcResults.

Things that will bite you

Blob bytes never shrink. The datastore has no blob GC. _blobs is append-only forever; gc only soft-tombstones _paths by setting deletedAt. Roughly 99.5% of the ~388 GB blob store is already orphaned and nothing here reclaims it. When someone asks how much a GC frees, say this out loud.

Sync counters lie about tombstones. filesPushed cheerfully reports 0 while tombstoning 160k paths. Trust the _paths re-count in gcResults (tombstoned / errors), never the CLI counter.

GC needs a complete mirror. The push sweep tombstones every remote path absent from the local mirror, so a partial or lazily-hydrated cache would tombstone everything unhydrated. That is why datastore_gc keeps its own cold pull under ~/.swamp/gc-janitor/<tenant>__<ns>/ (deterministic repoId, so the mirror survives between runs) instead of borrowing a writer repo's cache.

The janitor repo's connection settings are assumed, not derived. datastore_gc reuses only the host and credentials from mongodbUri; the janitor .swamp.yaml it writes hardcodes replicaSet=rs0, authSource=admin, directConnection=true, and passwordEnv: MONGO_PASSWORD. A cluster that differs on any of those needs the janitor config in mongodb_datastore_gc.ts adjusted. datastore_stats has no such assumption — it connects with the URI as given.

Benchmarks

bench/ measures the datastore's real sync paths, not mocks:

• hydration_bench.ts — cold pullChanged(), full vs. metadataOnly lazy hydration, against a real namespace. Read-only.
• pushchanged_bench.ts — pushChanged() timed directly against the real sync service (no swamp-core lock, no model execution), on scratch namespaces that are dropped afterward.
• writeback_bench.py — the end-to-end analogue, driven through the actual swamp CLI and read out of swamp's own ms-resolution logs. Includes core's global lock and model execution, so it is not directly comparable to the datastore-unit numbers.

Reference

• Base model: the mongodb skill (.claude/skills/mongodb/SKILL.md).
• Datastore internals: ~/git/mongodb-swamp-datastore/extensions/datastores/mongodb/.
• Method parameters: swamp model method describe treehouse — fetch them, don't memorize them.