Garbage Collection

AeorDB is an append-only database: writes, overwrites, and deletes all append new entries without modifying existing ones. Over time, this leaves orphaned entries – old file versions, deleted content, stale directory indexes – consuming disk space. Garbage collection (GC) reclaims that space.

What GC Does

GC uses a mark-and-sweep algorithm:

Mark phase: Starting from HEAD, all snapshots, and all forks, GC walks every reachable directory tree. It marks every entry that is still live: directory indexes, file records, content chunks, system tables, task queue records, and deletion records. GC also treats current path-key FileRecords (file:{path}) as roots for their chunks, so a stale directory HEAD cannot cause a file that is still readable by path to lose its chunk data.
Sweep phase: GC iterates all entries in the key-value store. Any entry whose hash is not in the live set is garbage. Non-live entries are overwritten in-place with DeletionRecord or Void entries, then removed from the KV index.

What Gets Collected

Old file versions that have been overwritten
Content chunks no longer referenced by any file record
Directory indexes from previous tree states
Entries orphaned by deletes

What Does NOT Get Collected

HEAD and all reachable entries from HEAD
All snapshot root trees and their descendants
All fork root trees and their descendants
System table entries (/.system, /.config)
Task queue records (registry + individual task entries)
DeletionRecord entries (needed for KV rebuild from .aeordb scan)
Chunks referenced by a live path-key FileRecord, even if a directory index is temporarily stale

Running GC

CLI

# Run GC
aeordb gc --database data.aeordb

# Dry run -- report what would be collected without deleting
aeordb gc --database data.aeordb --dry-run

Example output:

AeorDB Garbage Collection
Database: data.aeordb

Versions scanned: 3
Live entries:     1247
Garbage entries:  89
Reclaimed:        1.2 MB
Duration:         0.3s

Dry run output:

AeorDB Garbage Collection [DRY RUN]
Database: data.aeordb

[DRY RUN] Would collect 89 garbage entries (1.2 MB)

HTTP API

Synchronous GC (blocks until complete):

# Run GC
curl -X POST http://localhost:6830/system/gc \
  -H "Authorization: Bearer $API_KEY"

# Dry run
curl -X POST http://localhost:6830/system/gc?dry_run=true \
  -H "Authorization: Bearer $API_KEY"

Response:

{
  "versions_scanned": 3,
  "live_entries": 1247,
  "garbage_entries": 89,
  "reclaimed_bytes": 1258291,
  "duration_ms": 312,
  "dry_run": false
}

Background GC (returns immediately, runs as a task):

curl -X POST http://localhost:6830/system/tasks/gc \
  -H "Authorization: Bearer $API_KEY"

This enqueues a GC task that the background task worker will pick up. Track its progress via the task system.

When to Run GC

After bulk deletes: If you delete a large number of files, their content chunks become garbage.
After bulk overwrites: Updating many files leaves old versions behind.
After version cleanup: If you delete old snapshots, the entries they exclusively referenced become garbage.
Periodically: Set up a cron schedule for automatic GC.

Example cron configuration (/.config/cron.json):

{
  "schedules": [
    {
      "id": "nightly-gc",
      "task_type": "gc",
      "schedule": "0 3 * * *",
      "args": {},
      "enabled": true
    }
  ]
}

Concurrency and Safety

GC should not be run concurrently with writes. The sweep phase re-verifies each candidate against the current KV state before overwriting to mitigate races, but for full safety, callers should ensure exclusive access during GC.

Before a non-dry-run GC, AeorDB normally creates an engine-internal _aeordb_pre_gc_* snapshot as a safety net. If lifecycle configuration has "snapshot_writes_enabled": false, GC skips that safety snapshot and continues with the mark-and-sweep run. Existing snapshots are still honored as live roots either way.

Crash safety: If the process crashes mid-sweep, the .aeordb file may contain partially overwritten entries. On restart, the .kv index file will be stale and must be deleted to trigger a full rebuild from the .aeordb file scan. The rebuild replays deletion records and reconstructs the index, so no committed data is lost. Garbage entries that were not yet swept will persist until the next GC run.

Performance

At scale, expect approximately 10K entries/sec sweep throughput. The mark phase is faster since it only walks reachable trees. The sweep phase writes are batched with a single sync at the end for performance.

Diagnostic Environment Variables

The engine exposes a couple of opt-in diagnostic env vars for ops debugging. They are no-ops unless set, so leaving them unset costs nothing.

Variable Effect

AEORDB_GC_TIMING Per-phase wall-clock timing of GC mark + sweep on stderr (per-level BFS times, frontier sizes, dedup/sort/read breakdown, deletion-pass count). Useful for narrowing down where a slow GC is spending time.

AEORDB_GC_MEM_PROFILE Per-phase RSS sampling around mark, mark.bfs, mark.deletion-pass, recheck-drain, and sweep. Each phase emits [gc-mem] <phase>: baseline_rss=X MB peak_rss=Y MB end_rss=Z MB delta_hwm=W MB elapsed=.... Also activates wider PhaseSampler coverage in the file-write paths (store_file_from_reader, store_chunk, finalize_file) and snapshot operations. On macOS the values come from Mach task_info; on Linux from /proc/self/status.

Variable	Effect
`AEORDB_GC_TIMING`	Per-phase wall-clock timing of GC mark + sweep on stderr (per-level BFS times, frontier sizes, dedup/sort/read breakdown, deletion-pass count). Useful for narrowing down where a slow GC is spending time.
`AEORDB_GC_MEM_PROFILE`	Per-phase RSS sampling around mark, mark.bfs, mark.deletion-pass, recheck-drain, and sweep. Each phase emits `[gc-mem] <phase>: baseline_rss=X MB peak_rss=Y MB end_rss=Z MB delta_hwm=W MB elapsed=...`. Also activates wider PhaseSampler coverage in the file-write paths (`store_file_from_reader`, `store_chunk`, `finalize_file`) and snapshot operations. On macOS the values come from Mach `task_info`; on Linux from `/proc/self/status`.

These are intended for ops investigations, not normal operation. Set them only when you’re chasing a specific problem.

Async Index Cleanup

When files are deleted, their index entries are cleaned up asynchronously in the background. Deletions are debounced with a 50ms timeout and batched up to 100 paths per batch. This means index cleanup does not block the delete response, and rapid successive deletes are coalesced into efficient batch operations.

AeorDB Documentation