feat(cmd): add retriever graph dump tooling

GOLANG_CODING_STANDARD.md

@@ -0,0 +1,332 @@
+# Go Coding Standard
+
+This standard captures the preferred Go style used in `dawgs`. Follow `gofmt`
+first, then apply the conventions below when choosing code shape.
+
+## Receiver Names
+
+Use `s` as the receiver name for methods, regardless of the concrete receiver
+type. The goal is to remove per-type receiver-name churn and reduce cognitive
+load while reading method bodies.
+
+```go
+func (s *Server) Start() error {
+	go s.loop()
+	return nil
+}
+
+func (s Config) Validate() error {
+	if s.Firewall.Backend != "nftables" {
+		return fmt.Errorf("unsupported firewall backend %q", s.Firewall.Backend)
+	}
+
+	return nil
+}
+```
+
+Avoid type-derived receiver names such as `srv`, `cfg`, `db`, or `wm` unless a
+local collision makes `s` unusable.
+
+## Vertical Spacing
+
+Use single blank lines inside functions and closures to separate logical
+paragraphs. The driver packages are the reference style: setup, validation,
+resource acquisition, side-effecting work, mutation, and final return each get
+their own visual space when they are independent steps.
+
+Keep validation blocks at the top of a function compact when they are a single
+gate into the rest of the operation.
+
+```go
+if len(kindIDs) == 0 {
+	return graph.Kinds{}, true
+}
+
+if mappedKinds, err := kindMapper.MapKindIDs(ctx, kindIDs); err == nil {
+	return mappedKinds, true
+}
+
+return nil, false
+```
+
+Separate a multi-line call from the next independent operation. This includes
+logging, progress emission, database calls, and other side-effecting calls.
+
+```go
+slog.Info("retriever load node phase started",
+	slog.String("graph", graphEntry.Name),
+	slog.Int64("node_count", graphEntry.NodeCount),
+)
+progress.emit(ProgressEvent{
+	Operation: OperationLoad,
+	Message:   "retriever load node phase started",
+	Graph:     graphEntry.Name,
+})
+
+nodeMap, nodeCount, err := loadGraphNodes(ctx, db, graphEntry)
+if err != nil {
+	return 0, 0, err
+}
+```
+
+Within loops and callbacks, give each filter, transformation, mutation, and
+checkpoint its own paragraph when the block does more than a couple of trivial
+statements.
+
+```go
+for _, node := range nodes {
+	kinds := node.Kinds.Strings()
+	sort.Strings(kinds)
+	addKindsToSet(nodeKinds, kinds)
+
+	properties := node.Properties.MapOrEmpty()
+	if activeScrubber != nil {
+		properties = activeScrubber.scrubProperties(properties)
+	}
+
+	item := FragmentNode{
+		ID:         node.ID.String(),
+		Kinds:      kinds,
+		Properties: properties,
+	}
+
+	items = append(items, item)
+	if len(items) >= shardSize {
+		if err := flush(); err != nil {
+			return err
+		}
+	}
+}
+```
+
+Do not add blank lines between statements that are one tight operation, such as
+creating a value and immediately returning it, assigning a field and returning
+the receiver, or a short `if`/`else if` chain that represents one decision.
+
+## Error Handling and Scope
+
+Prefer initializer-backed `if` statements for operations that can fail. Handle
+the error at the point it is raised, and keep successful values scoped to the
+branch that uses them.
+
+```go
+if cfg, err := server.ReadConfiguration(cfgPath); err != nil {
+	log.Fatalf("Error reading config: %v", err)
+} else if dbInst, err := db.NewDatabase(cfg.DBPath); err != nil {
+	log.Fatalf("Error opening database: %v", err)
+} else {
+	// cfg and dbInst are only available where they are valid.
+}
+```
+
+Use `else if` chains when each later step depends on the successful output of
+the previous step. This keeps the happy path close to the failure path and
+prevents partially initialized variables from leaking into wider scopes.
+
+```go
+if content, err := os.ReadFile(path); err != nil {
+	return cfg, err
+} else if err := toml.Unmarshal(content, &cfg); err != nil {
+	return cfg, err
+} else {
+	cfg = cfg.withDefaults()
+	return cfg, nil
+}
+```
+
+When an error requires a special-case branch, keep that branch nested at the
+error site.
+
+```go
+if record, err := s.db.GetHostRecord(match.IPAddress); err != nil {
+	if !errors.Is(err, db.ErrNotFound) {
+		return err
+	}
+
+	// Create the missing record here.
+} else {
+	// Update the existing record here.
+}
+```
+
+Use plain early returns when there is no useful success value to scope or when
+the initializer chain would make the code harder to follow.
+
+## Variable Grouping
+
+Group related local variables aggressively with `var` blocks. This is preferred
+when multiple locals establish the state for the same operation, especially when
+some values are initialized and others are intentionally zero-valued.
+
+```go
+var (
+	records HostRecords
+
+	txn    = s.db.NewTransaction(false)
+	iter   = txn.NewIterator(badger.DefaultIteratorOptions)
+	prefix = []byte(recordKeyPrefix)
+)
+```
+
+Use blank lines inside the group to separate conceptual clusters. Prefer a
+single grouped declaration over several adjacent `var` statements.
+
+For short-lived values used immediately, `:=` is still appropriate.
+
+```go
+line := scanner.Text()
+```
+
+## Logical Spacing
+
+Use blank lines inside functions to separate logical phases. This is guidance,
+not a hard formatting rule: prefer readability over mechanically inserting
+empty lines.
+
+Common phase boundaries include setup before control flow, guard checks before
+work, resource acquisition before deferred cleanup and use, and mutation before
+the final return.
+
+```go
+transaction, err := s.renderFirewallTransaction(state)
+if err != nil {
+	return err
+}
+
+log.Infof("Applying nftables transaction. Banned %d hosts.", state.Count())
+if output, err := s.runNftCommand(s.nftPath, transaction.Payload, "-j", "-f", "-"); err != nil {
+	return fmt.Errorf("backend apply failed running %s: %s: %w", s.nftPath, output, err)
+}
+
+s.firewallState = state
+return nil
+```
+
+Within loops, use blank lines to make each filtering or transformation step
+stand on its own.
+
+```go
+for _, record := range hostRecords {
+	if !now.Before(record.NextRefresh) {
+		continue
+	}
+
+	ipAddress := net.ParseIP(record.IPAddress)
+	if ipAddress == nil || ipAddress.To4() == nil {
+		continue
+	}
+
+	if staticCIDRAllows(cfg.StaticAllowCIDRs, ipAddress) {
+		continue
+	}
+
+	state.BannedIPv4[record.IPAddress] = record
+}
+```
+
+In `select` statements, separate cases with blank lines when each case performs
+distinct work.
+
+```go
+select {
+case <-refreshTicker.C:
+	s.update()
+
+case <-s.updateC:
+	s.update()
+
+case <-s.joiner.StopC:
+	return
+}
+```
+
+Avoid splitting tightly coupled statements when the second line is the immediate
+effect of the first.
+
+## Function Ordering
+
+Prefer ordering functions in the same file so dependencies appear before the
+functions that call them, when that makes the file read naturally. This is
+guidance, not a hard formatting rule: public entry points, framework
+conventions, and established local ordering may take precedence.
+
+## Struct Definitions
+
+Write struct type definitions across multiple lines, with one field per line.
+Align naturally with `gofmt`; do not compress structs onto one line.
+
+```go
+type FirewallConfig struct {
+	Backend           string `toml:"backend"`
+	Table             string `toml:"table"`
+	BanSet            string `toml:"ban_set"`
+	Family            string `toml:"family"`
+	DryRunSummaryOnly bool   `toml:"dry_run_summary_only"`
+}
+```
+
+Use field names in struct literals, especially for exported types, config
+types, tests, and any literal with more than one field.
+
+```go
+return Server{
+	cfg:           cfg,
+	db:            dbInst,
+	firewallState: NewFirewallState(),
+	updateC:       updateC,
+	joiner:        NewJoinChannelPair(),
+	nftPath:       nftPath,
+	nftBackend:    NewNftablesBackend(cfg.Firewall),
+	runNftCommand: defaultNftCommandRunner,
+}, nil
+```
+
+Prefer multi-line keyed literals even when a literal is currently small, because
+they remain stable as fields are added and make diffs easier to read.
+
+```go
+freshWatch := db.File{
+	Path:   event.Name,
+	Offset: 0,
+}
+```
+
+## Practical Defaults
+
+Keep code direct and local. Favor readable control flow over abstractions that
+only hide one or two calls.
+
+Return zero values explicitly on error for multi-return functions.
+
+```go
+if watcher, err := fsnotify.NewWatcher(); err != nil {
+	return FileWatcher{}, err
+} else {
+	return FileWatcher{
+		cfg:     cfg,
+		watcher: watcher,
+	}, nil
+}
+```
+
+Defer cleanup immediately after acquiring a resource.
+
+```go
+fin, err := os.Open(fileWatch.Path)
+if err != nil {
+	return uniqueBanHosts, uniqueAllowHosts
+}
+defer fin.Close()
+```
+
+Use package-level grouped `const` and `var` declarations for related values.
+
+```go
+const (
+	ErrNotFound = errors.New("not found")
+
+	fileWatchKey        KeyFormat = "file_watch.%s"
+	hostRecordKey       KeyFormat = "hosts.%s"
+	hostRecordKeyPrefix KeyFormat = "hosts."
+)
+```

README.md

@@ -95,7 +95,9 @@ make quality FUZZ_REPORT=.coverage/fuzz.json MUTATION_REPORT=.coverage/mutation.
 
 `make quality_backend` captures PostgreSQL and Neo4j integration results for backend equivalence comparison. It requires
 `PG_CONNECTION_STRING` and `NEO4J_CONNECTION_STRING`. `make quality_bench` writes benchmark markdown and JSON captures
-for later baseline comparison.
+for later baseline comparison. Benchmark drift comparison is performed by
+`make quality` through `tools/metrics`; there is no separate benchmark diff
+command package.
 
 `make plan_corpus` captures plan diagnostics for the shared Cypher integration corpus. It accepts either
 `CONNECTION_STRING` for one backend or `PG_CONNECTION_STRING` and `NEO4J_CONNECTION_STRING` for both backends, then
@@ -106,6 +108,21 @@ current modes are `postgres_sql`, `local_traversal`, and `neo4j`; AGE is referen
 comparison mode yet. The command can emit JSONL records plus Markdown and JSON summaries, and can compare current timings
 against a previous JSONL baseline.
 
+`go run ./cmd/retriever` dumps and loads live Dawgs graph databases as
+manifest-based collections of compact OpenGraph-derived fragments. It supports
+PostgreSQL and Neo4j, gzip and zstd compression, checksum validation before
+load, optional deterministic property scrubbing, and a read-throughput benchmark
+mode. It can also package dumps as single HPKE/ML-KEM encrypted TAR archives.
+See [cmd/retriever/README.md](cmd/retriever/README.md) for dump, encrypted
+archive, load, scrubbed dump, metrics verification, and benchmark examples.
+The same import/export functionality is available to library consumers from
+`github.com/specterops/dawgs/retriever`; callers provide an already-open
+`graph.Database`, and archive helpers support both path-based and stream-based
+APIs. The package exposes CLI-matching default option constructors, structured
+progress callbacks, manifest/metrics helpers, HPKE key envelope reader/writer
+helpers, and typed errors for validation, compatibility, checksum, metrics, and
+count mismatches.
+
 PostgreSQL translates exact string property equality with a JSON string type guard and `properties ->>` extraction, so
 indexes created on expressions such as `properties ->> 'objectid'` and `properties ->> 'name'` can be used for selective
 anchors without matching JSON booleans or numbers. Simple relationship count fast paths depend on the schema's