Skip to content
Quartermaster Quartermaster

Collector & hub

The thin collector runs the scan pipeline’s produce half standalone — crawl, classify, parse, redact, snapshot — with no SQLite and no embedding model, and pushes wire-format items to a hub’s POST /api/ingest. The hub runs the same sink (embed + upsert) it uses for local crawls.

This page covers pushing to a self-hosted hub on your own network. To push to a cloud registry instead, see Cloud hub on Cloudflare — and the guided installers for Windows, macOS, and Linux set that collector up with no Node install at all.

Run it

Section titled “Run it”
Terminal window
node dist/collector.js --push http://hub:8765 --all-volumes \
  --token "$QUARTERMASTER_AUTH_TOKEN" --device-id my-laptop
  • --roots a,b scopes the crawl (the sentinel auto works there too).
  • --dry-run crawls and reports without pushing.
  • --no-reconcile skips stale-item reconciliation.

Ship it as a single file

Section titled “Ship it as a single file”
Terminal window
npm run bundle:collector       # -> dist/collector.bundle.cjs (~100 KB, any Node >= 18)
npm run bundle:collector:sea   # -> dist/quartermaster-collector(.exe) (Node SEA, no Node install needed; build per-OS)

Security properties

Section titled “Security properties”
  • Secrets are redacted client-side — records are redacted at parse time and MCP configs sanitized before anything is serialized; raw secret values never leave the source machine. Snapshots carry file hashes, never file content.
  • Remote pushes are always authenticated — the hub must be bound non-loopback (HOST=0.0.0.0), which forces an auth token.
  • Devices can’t collide — the hub registers the sending device and namespaces pushed paths as //<device-id>/<path>, so items from different machines never overwrite each other. A local rescan never purges pushed items.
  • Payloads are bounded — ≤500 items / 8 MB per call, validated at the trust boundary.

Reconciliation

Section titled “Reconciliation”

After a fully successful crawl+push, the collector calls POST /api/ingest/reconcile so the hub drops that device’s items that vanished from the crawled roots — scoped exactly like a local rescan. Its authority is narrow: only under //<device-id>/, and only under the roots the collector reports having crawled. One device can never delete another device’s (or the hub’s local) items, and an aborted run never triggers deletions.

Why this seam matters

Section titled “Why this seam matters”

The collector speaks the same wire format regardless of what’s on the other end — a self-hosted hub on your network or a Cloudflare Worker in your own account — so you can start with one and switch to the other later.