Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions reference/cli/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -250,8 +250,8 @@ harper copy-db <source-database> <target-database-path>

**Parameters**:

- `<source-database>` - Name of the source database
- `<target-database-path>` - Full path to the target database file
- `<source-database>` - Name of the source database (a name, not a file path)
- `<target-database-path>` - Full path to the target database file, which must not already exist

**Example**:

Expand All @@ -261,6 +261,8 @@ harper copy-db data /home/user/hdb/database/copy.mdb

This copies the default `data` database to a new location with compaction applied.

The database's file-backed blobs are copied to `<target-database-path>-blobs/<rootIndex>/`, since blob files live outside the database file and are addressed by database name. The copy cannot be restored without that directory — see [Database Compaction](../database/compaction.md#file-backed-blobs-travel-separately) for the restore steps. LMDB databases only; RocksDB databases compact themselves.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

According to the repository's guidelines, when documenting behavior changes to an existing surface (such as the new blob copying behavior and RocksDB exclusion in copy-db), we should use the <VersionBadge type="changed" version="vX.Y.0" /> format. Please add the appropriate version badge (e.g., v5.3.0 or the target release version) to indicate this change.

Suggested change
The database's file-backed blobs are copied to `<target-database-path>-blobs/<rootIndex>/`, since blob files live outside the database file and are addressed by database name. The copy cannot be restored without that directory — see [Database Compaction](../database/compaction.md#file-backed-blobs-travel-separately) for the restore steps. LMDB databases only; RocksDB databases compact themselves.
<VersionBadge type="changed" version="v5.3.0" /> — The database's file-backed blobs are copied to <target-database-path>-blobs/<rootIndex>/, since blob files live outside the database file and are addressed by database name. The copy cannot be restored without that directory — see [Database Compaction](../database/compaction.md#file-backed-blobs-travel-separately) for the restore steps. LMDB databases only; RocksDB databases compact themselves.
References
  1. Use the format when documenting behavior changes to existing surface, as prescribed in the repository's guidelines.


**Use Cases**:

- Database optimization
Expand Down
25 changes: 24 additions & 1 deletion reference/database/compaction.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Run using the [CLI](../cli/commands.md):
harper copy-db <source-database> <target-database-path>
```

The `source-database` is the database name (not a file path). The target is the full file path where the compacted copy will be written.
The `source-database` is the database name (not a file path). The target is the full file path where the compacted copy will be written, and it must not already exist — `copy-db` refuses to write into an existing file rather than merging the copy into whatever it holds.

To replace the original database with the compacted copy, move or rename the output file to the original database path after Harper is stopped.

Expand All @@ -37,6 +37,29 @@ To replace the original database with the compacted copy, move or rename the out
harper copy-db data /home/user/hdb/database/copy.mdb
```

Copy compaction applies to LMDB databases. RocksDB databases compact themselves and are skipped.

### File-backed blobs travel separately

A database's file-backed blob values (`Blob` and large `Bytes` attributes) are not stored inside the database file. They live in the configured blob roots — `storage.blobPaths[n]`, or `<rootPath>/blobs/<database>` when `blobPaths` is not configured — and are addressed by **database name**, not by the path of the database file.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

According to the repository's guidelines, when documenting behavior changes to an existing surface (such as the new file-backed blobs behavior), we should use the <VersionBadge type="changed" version="vX.Y.0" /> format. Please add the appropriate version badge (e.g., v5.3.0 or the target release version) to indicate this change.

Suggested change
A database's file-backed blob values (`Blob` and large `Bytes` attributes) are not stored inside the database file. They live in the configured blob roots — `storage.blobPaths[n]`, or `<rootPath>/blobs/<database>` when `blobPaths` is not configured — and are addressed by **database name**, not by the path of the database file.
<VersionBadge type="changed" version="v5.3.0" /> — A database's file-backed blob values (Blob and large Bytes attributes) are not stored inside the database file. They live in the configured blob roots — storage.blobPaths[n], or <rootPath>/blobs/<database> when blobPaths is not configured — and are addressed by **database name**, not by the path of the database file.
References
  1. Use the format when documenting behavior changes to existing surface, as prescribed in the repository's guidelines.


`copy-db` therefore writes them alongside the copy:

```
<target-database-path>-blobs/<rootIndex>/…
```

`<rootIndex>` is the position of the source root in the database's blob-root list, preserved so a multi-root database restores each root to its original slot. A `README.md` in that directory records the mapping.

**The copy is not restorable without this directory.** To restore the copy under a database name, put each `<rootIndex>` tree into that name's matching blob root — for example, restoring the copy above as a database named `archive` with no `storage.blobPaths` configured:

```bash
cp -r /home/user/hdb/database/copy.mdb /home/user/hdb/database/archive.mdb
cp -r /home/user/hdb/database/copy.mdb-blobs/0/. /home/user/hdb/blobs/archive/
```

Restoring the copy under its original database name in the same installation needs only the database file, since the blob roots it already references are untouched.

## Compact on Start

Automatically compacts all non-system databases when Harper starts. Harper will not start until compaction is complete. Under the hood, it loops through all user databases, creates a backup of each, compacts it, replaces the original with the compacted copy, and removes the backup.
Expand Down