Introduction

AeorDB is a content-addressed file database that treats your data as a filesystem, not as tables and rows. Store any file at any path, query structured fields with sub-millisecond lookups, and version everything with Git-like snapshots and forks – all from a single binary with zero external dependencies.

What Makes AeorDB Different

Content-addressed storage with BLAKE3. Every piece of data is identified by its cryptographic hash. This gives you built-in deduplication, integrity verification, and a Merkle tree that makes versioning essentially free.

A filesystem, not a schema. Data lives at paths like /users/alice.json and /docs/reports/q1.pdf, organized into directories. No schemas to define, no migrations to run. Store JSON, images, PDFs, or raw bytes – the engine handles them all the same way.

Built-in versioning. Create named snapshots, fork your database into isolated branches, diff between any two versions, and export/import self-contained .aeordb files. The content-addressed Merkle tree means historical reads resolve the exact data at the time of the snapshot, not the latest overwrite.

Native parsers for common formats. AeorDB includes 8 built-in parsers (text, HTML/XML, PDF, images, audio, video, MS Office, ODF) that run automatically during indexing with zero deployment overhead. Common file types are searchable out of the box.

WASM plugin system. Extend the database with WebAssembly plugins for two purposes: parser plugins that extract structured fields from custom file formats for indexing, and query plugins that run custom logic directly at the data layer. Plugins execute in a sandboxed WASM runtime with configurable memory limits. Native parsers handle common formats; WASM plugins extend to anything else.

Native HTTP API. AeorDB exposes its full API over HTTP – no separate proxy, no client library required. Store files with PUT, read them with GET, query with POST /files/query, and manage versions with the /versions/* endpoints. Any HTTP client works.

Embeddable. A single aeordb binary with no external dependencies. Point it at a .aeordb file and you have a running database. Like SQLite, but for files with versioning and a built-in HTTP server.

Lock-free concurrent reads. The engine uses snapshot double-buffering via ArcSwap so readers never block writers and never see partial state. Queries routinely complete in under a millisecond.

Key Features

• Storage: Append-only WAL file, content-addressed BLAKE3 hashing, automatic zstd compression, 256KB chunking for dedup
• Indexing: Scalar bucketing (NVT) with u64, i64, f64, string, timestamp, trigram, phonetic/soundex/dmetaphone index types
• Native parsers: 8 built-in parsers (text, HTML/XML, PDF, images, audio, video, MS Office, ODF) – no WASM deployment needed
• Querying: JSON query API with boolean logic (and, or, not), comparison operators, sorting, pagination, projections, aggregations, and zero-config virtual fields for searching by filename, extension, size, and content type
• Versioning: Snapshots, forks, diff/patch, export/import as self-contained .aeordb files, file-level history and restore
• Plugins: WASM parser plugins for custom formats, WASM query plugins for server-side logic
• Operations: Background task system, cron scheduler (including automated backups), garbage collection, automatic reindexing
• Auth: Self-contained JWT auth, API keys, user/group management with tags, path-level permissions, or --auth false for local use
• TLS: Native HTTPS via rustls with --tls-cert and --tls-key flags
• Configuration: TOML config file support (--config) with 1:1 CLI flag mapping
• Observability: O(1) stats and memory diagnostics at /system/stats, Prometheus metrics at /system/metrics, real-time metrics SSE event, structured logging

Next Steps

• Installation – build from source
• Quick Start – store, query, and version data in 5 minutes
• Architecture – how the engine works under the hood

Bot Quickstart

This page is for automated agents that discover an AeorDB HTTP endpoint and need to understand how to use it safely.

Discovery

• The public health endpoint is GET /system/health.
• The public documentation is served at GET /docs/.
• This raw bot guide is served at GET /docs/SKILL.md.
• The browser portal is served at GET /.

Authentication

Most API routes require an API key exchanged for a bearer token:

curl -X POST "$AEORDB/auth/token" \
  -H "Content-Type: application/json" \
  -d '{"api_key":"<api-key>"}'

Use the returned token as:

Authorization: Bearer <token>

If auth is disabled for a development instance, authenticated routes may work without this header.

High-Value Routes

Search Examples

Search one folder:

{
  "path": "/docs/",
  "query": "how to",
  "limit": 20
}

Structured search:

{
  "path": "/",
  "where": {"field": "@extension", "op": "eq", "value": "md"},
  "limit": 100
}

Search with locators for follow-up range fetch:

{
  "path": "/docs/",
  "query": "database write pattern",
  "include_matches": true,
  "max_matches_per_result": 5,
  "snippet_chars": 240
}

Use returned fetch_hint, ranges, content_hash, and updated_at with POST /files/fetch to retrieve only the relevant parts of large files.

Range Fetch

Use POST /files/fetch for batch reads and partial reads. Prefer range fetch after search hit locators instead of downloading large documents.

Safety Rules

• Treat AeorDB as a database, not a disposable file server.
• Do not run repair, GC, import, export-over, or repeated restarts against a suspected corrupt original until evidence has been preserved.
• If corruption is suspected, preserve the database file, hot files, lock file, and logs before mutation.
• Prefer graceful shutdown with SIGTERM/Ctrl+C and wait for completion.
• Avoid broad unbounded searches or full-file fetches when a scoped search or range fetch will do.

More Detail

Start with:

• GET /docs/api/files.html
• GET /docs/api/querying.html
• GET /docs/api/upload-protocol.html
• GET /docs/api/plugins.html
• GET /docs/operations/threat-model.html

Installation

AeorDB is built from source using the Rust toolchain. There are no external dependencies – the binary is fully self-contained.

Prerequisites

• Rust (stable toolchain, 1.75+)

Install Rust if you don’t have it:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Build from Source

Clone the repository and install the release binary locally:

git clone https://github.com/AeorDB/aeordb.git
cd aeordb
./scripts/install-local.sh

The binary is installed at:

~/.local/bin/aeordb

Verify the Build

aeordb --help

You should see:

AeorDB command-line interface

Usage: aeordb <COMMAND>

Commands:
  start            Start the database server
  stress           Run stress tests against a running instance
  emergency-reset  Emergency reset: revoke the current root API key and generate a new one
  export           Export a version as a self-contained .aeordb file
  diff             Create a patch .aeordb containing only the changeset between two versions
  import           Import an export or patch .aeordb file into a target database
  promote          Promote a version hash to HEAD
  gc               Run garbage collection to reclaim unreachable entries
  help             Print this message or the help of the given subcommand(s)

Optional: Install an Existing Binary

If you already have a built release binary, install it without rebuilding:

./scripts/install-local.sh --from /path/to/aeordb

Optional: Add to PATH

The install script uses ~/.local/bin, which is usually already on PATH. If your shell cannot find aeordb after installation:

export PATH="$HOME/.local/bin:$PATH"

No External Dependencies

AeorDB does not require:

• A separate database process (it IS the process)
• Runtime libraries or shared objects
• Configuration files (sensible defaults for everything)
• Docker, containers, or orchestration

The single binary is all you need. Point it at a file path and it creates the database on first run.

Next Steps

• Quick Start – start the server and store your first file

Quick Start

This tutorial walks you through the core operations: storing files, querying, creating snapshots, and cleaning up. All examples use curl and assume auth is disabled for simplicity.

1. Start the Server

aeordb start --database mydb.aeordb --port 6830 --auth false

You should see log output indicating the server is listening on port 6830. The mydb.aeordb file is created automatically if it does not exist.

2. Store a File

Store a JSON file at the path /users/alice.json:

curl -X PUT http://localhost:6830/files/users/alice.json \
  -H "Content-Type: application/json" \
  -d '{"name":"Alice","age":30,"city":"Portland"}'

Expected response:

{
  "status": "created",
  "path": "/users/alice.json",
  "hash": "a1b2c3d4..."
}

Store a few more files to have data to query:

curl -X PUT http://localhost:6830/files/users/bob.json \
  -H "Content-Type: application/json" \
  -d '{"name":"Bob","age":25,"city":"Seattle"}'

curl -X PUT http://localhost:6830/files/users/carol.json \
  -H "Content-Type: application/json" \
  -d '{"name":"Carol","age":35,"city":"Portland"}'

3. Read a File

curl http://localhost:6830/files/users/alice.json

Expected response:

{"name":"Alice","age":30,"city":"Portland"}

4. List a Directory

Append a trailing slash to list directory contents:

curl http://localhost:6830/files/users/

Expected response:

{
  "items": [
    {"path": "/users/alice.json", "name": "alice.json", "entry_type": 2, "hash": "a1b2c3..."},
    {"path": "/users/bob.json", "name": "bob.json", "entry_type": 2, "hash": "d4e5f6..."},
    {"path": "/users/carol.json", "name": "carol.json", "entry_type": 2, "hash": "a7b8c9..."}
  ]
}

5. Add an Index

To query fields, you need to tell AeorDB which fields to index. Store an index configuration at .aeordb-config/indexes.json inside the directory:

curl -X PUT http://localhost:6830/files/users/.aeordb-config/indexes.json \
  -H "Content-Type: application/json" \
  -d '{
    "indexes": [
      {"name": "age", "type": "u64"},
      {"name": "city", "type": "string"},
      {"name": "name", "type": ["string", "trigram"]}
    ]
  }'

This tells the engine to index the age field as a 64-bit unsigned integer, city as an exact string, and name as both an exact string and a trigram (for fuzzy matching). Existing files in the directory are automatically reindexed in the background.

6. Query

Query for users older than 28 in Portland:

curl -X POST http://localhost:6830/files/query \
  -H "Content-Type: application/json" \
  -d '{
    "path": "/users/",
    "where": {
      "and": [
        {"field": "age", "op": "gt", "value": 28},
        {"field": "city", "op": "eq", "value": "Portland"}
      ]
    }
  }'

Expected response:

{
  "results": [
    {"name": "Alice", "age": 30, "city": "Portland"},
    {"name": "Carol", "age": 35, "city": "Portland"}
  ],
  "total_count": 2
}

Query Operators

7. Create a Snapshot

Save the current state as a named snapshot:

curl -X POST http://localhost:6830/versions/snapshots \
  -H "Content-Type: application/json" \
  -d '{"name": "v1"}'

Expected response:

{
  "status": "created",
  "name": "v1",
  "root_hash": "e5f6a7b8..."
}

You can list all snapshots:

curl http://localhost:6830/versions/snapshots

8. Delete a File

curl -X DELETE http://localhost:6830/files/users/alice.json

Expected response:

{
  "status": "deleted",
  "path": "/users/alice.json"
}

The file is removed from the current state (HEAD), but the snapshot v1 still contains it. You can restore the snapshot to get it back.

9. Run Garbage Collection

Over time, deleted and overwritten data accumulates in the database file. Run GC to reclaim unreachable entries:

curl -X POST http://localhost:6830/system/gc

Expected response:

{
  "versions_scanned": 2,
  "live_entries": 15,
  "garbage_entries": 3,
  "reclaimed_bytes": 1024,
  "duration_ms": 12,
  "dry_run": false
}

To preview what would be collected without actually deleting:

curl -X POST "http://localhost:6830/system/gc?dry_run=true"

Next Steps

• Configuration – CLI flags, auth modes, CORS, index config
• Architecture – understand the storage engine internals
• Versioning – snapshots, forks, diff/patch, export/import
• Indexing – index types, multi-strategy fields, WASM parsers

Configuration

AeorDB is configured through CLI flags, a TOML configuration file, and through configuration files stored inside the database itself. CLI flags always take precedence over config file values.

Configuration File

AeorDB supports a TOML configuration file for all server settings. Pass it with --config:

aeordb start --config aeordb.toml

Every config key has a 1:1 mapping to a CLI flag. CLI flags override config file values. Omit any key to use the built-in default.

[server]
port = 6830
host = "0.0.0.0"
log_format = "pretty"

[server.tls]
cert = "/path/to/cert.pem"
key = "/path/to/key.pem"

[server.cors]
origins = ["https://app.example.com"]

[auth]
mode = "self"
jwt_expiry_seconds = 3600

[storage]
database = "data.aeordb"
chunk_size = 262144
# hot_dir = "./hot"

An example config file is included in the repository as aeordb.example.toml.

CLI Flags

aeordb start [OPTIONS]

TLS

AeorDB supports native HTTPS via rustls. Provide both a certificate and private key:

aeordb start \
  --tls-cert /etc/ssl/certs/aeordb.pem \
  --tls-key /etc/ssl/private/aeordb.key \
  --port 443

Both --tls-cert and --tls-key must be provided together – supplying only one is an error. When TLS is configured, the server listens for HTTPS connections instead of plain HTTP.

Examples

# Minimal: local development with no auth
aeordb start --database dev.aeordb --port 8080 --auth false

# Production: HTTPS with auth, CORS for your frontend
aeordb start \
  --database /var/lib/aeordb/prod.aeordb \
  --port 443 \
  --tls-cert /etc/ssl/certs/aeordb.pem \
  --tls-key /etc/ssl/private/aeordb.key \
  --hot-dir /var/lib/aeordb/hot \
  --cors-origins "https://myapp.com,https://admin.myapp.com" \
  --log-format json

# Using a config file with CLI overrides
aeordb start --config aeordb.toml --port 8080

Auth Modes

The --auth flag controls how authentication works:

Self-Contained Auth (Default)

On first startup, AeorDB creates an internal identity store and prints a root API key:

Root API key: aeor_ak_7f3b2a1c...

Use this key to create additional users and API keys via the admin API. If you lose the root key, use aeordb emergency-reset to generate a new one.

Obtaining a JWT Token

# Exchange API key for a JWT token
curl -X POST http://localhost:6830/auth/token \
  -H "Content-Type: application/json" \
  -d '{"api_key": "aeor_ak_7f3b2a1c..."}'

# Use the token for subsequent requests
curl http://localhost:6830/files/users/ \
  -H "Authorization: Bearer eyJhbG..."

CORS

Global CORS via CLI

The --cors-origins flag sets allowed origins for all routes:

# Allow all origins
aeordb start --cors-origins "*"

# Allow specific origins (comma-separated)
aeordb start --cors-origins "https://myapp.com,https://admin.myapp.com"

Without --cors-origins, no CORS headers are sent and cross-origin browser requests will fail.

Per-Path CORS

For fine-grained control, store a /.config/cors.json file in the database:

curl -X PUT http://localhost:6830/files/.config/cors.json \
  -H "Content-Type: application/json" \
  -d '{
    "rules": [
      {
        "path": "/public/",
        "origins": ["*"],
        "methods": ["GET", "HEAD"],
        "headers": ["Content-Type"]
      },
      {
        "path": "/api/",
        "origins": ["https://myapp.com"],
        "methods": ["GET", "POST", "PUT", "DELETE", "HEAD", "OPTIONS"],
        "headers": ["Content-Type", "Authorization"]
      }
    ]
  }'

Per-path rules are checked first. If no rule matches the request path, the global --cors-origins setting applies.

Index Configuration

Indexes are configured per-directory by storing a .aeordb-config/indexes.json file under the directory path. When this file changes, the engine automatically triggers a background reindex of all files in that directory.

{
  "indexes": [
    {"name": "title", "type": "string"},
    {"name": "age", "type": "u64"},
    {"name": "email", "type": ["string", "trigram"]},
    {"name": "created", "type": "timestamp"}
  ]
}

Index Types

Multi-Strategy Indexes

A single field can have multiple index types. Specify type as an array:

{"name": "title", "type": ["string", "trigram", "phonetic"]}

This creates three index files (title.string.idx, title.trigram.idx, title.phonetic.idx) from the same source field. Use the appropriate query operator to target each index type.

Source Resolution

By default, the index name is used as the JSON field name. For nested fields or parser output, use the source array:

{
  "parser": "pdf-extractor",
  "indexes": [
    {"name": "title", "source": ["metadata", "title"], "type": "string"},
    {"name": "author", "source": ["metadata", "author"], "type": ["string", "trigram"]},
    {"name": "page_count", "source": ["metadata", "pages"], "type": "u64"}
  ]
}

See Indexing & Queries for the full indexing reference.

Cron Configuration

Schedule recurring background tasks by storing /.config/cron.json:

curl -X PUT http://localhost:6830/files/.config/cron.json \
  -H "Content-Type: application/json" \
  -d '{
    "schedules": [
      {
        "id": "weekly-gc",
        "task_type": "gc",
        "schedule": "0 3 * * 0",
        "args": {},
        "enabled": true
      },
      {
        "id": "nightly-reindex",
        "task_type": "reindex",
        "schedule": "0 2 * * *",
        "args": {"path": "/data/"},
        "enabled": true
      }
    ]
  }'

Supported task types: "gc", "reindex", and "backup". The "backup" type accepts backup_dir, retention_count, and snapshot arguments (see Backup & Restore).

The schedule field uses standard 5-field cron syntax: minute hour day_of_month month day_of_week. Cron schedules can also be managed via the HTTP API at /system/cron.

Lifecycle Policy

Database lifecycle policy is stored in the database at /.aeordb-config/lifecycle.json and can be managed through the root-only /system/lifecycle API.

curl -X PUT http://localhost:6830/system/lifecycle \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "snapshot_writes_enabled": false,
    "snapshot_retention": {
      "auto_months": 1,
      "manual_months": 12
    }
  }'

snapshot_writes_enabled defaults to true. Set it to false to disallow new snapshot writes without removing or breaking existing snapshots. Existing snapshots can still be listed, read, restored, deleted, exported, and pruned. Automatic safety snapshots are skipped while this flag is disabled.

Compression

AeorDB uses zstd compression automatically when configured. To enable compression for a directory, add the compression field to the index config:

{
  "compression": "zstd",
  "indexes": [
    {"name": "title", "type": "string"}
  ]
}

Auto-Detection

When compression is enabled, the engine applies heuristics to decide whether to actually compress each file:

• Files smaller than 500 bytes are stored uncompressed (header overhead negates savings)
• Already-compressed formats (JPEG, PNG, MP4, ZIP, etc.) are stored uncompressed
• Text, JSON, XML, and other compressible types are compressed with zstd

Compression is transparent – reads automatically decompress. The content hash is always computed on the raw uncompressed data, so deduplication works regardless of compression settings.

Architecture

AeorDB is a single-file database built on an append-only write-ahead log (WAL). The database file contains all data, indexes, and metadata in one place. Understanding the architecture helps you reason about performance, recovery, and versioning behavior.

High-Level Overview

                         aeordb start
                             |
                     +-------+-------+
                     |  HTTP Server  |
                     |  (axum)       |
                     +-------+-------+
                             |
              +--------------+--------------+
              |              |              |
        +-----+----+  +-----+----+  +------+------+
        | Query    |  | Plugin   |  | Version     |
        | Engine   |  | Manager  |  | Manager     |
        +-----+----+  +-----+----+  +------+------+
              |              |              |
              |         +----+----+         |
              |         | Native  |         |
              |         | Parsers |         |
              |         +---------+         |
              +--------------+--------------+
                             |
                    +--------+--------+
                    | Storage Engine  |
                    | (StorageEngine) |
                    +--------+--------+
                             |
              +--------------+--------------+
              |              |              |
        +-----+----+  +-----+----+  +------+------+
        | Append   |  | KV Store |  | NVT         |
        | Writer   |  | (.kv)    |  | (in-memory) |
        +----------+  +----------+  +-------------+
              |              |
              +--------------+
                    |
            [  mydb.aeordb  ]    <-- single file on disk
            [ mydb.aeordb.kv ]   <-- KV index file

Native Parsers

AeorDB ships with 8 built-in format parsers (text, HTML/XML, PDF, images, audio, video, MS Office, ODF) that run as compiled Rust code during indexing. Native parsers are tried first for recognized content types; unrecognized formats fall through to the WASM plugin system. This means common file types are indexable out of the box with zero deployment overhead. See Plugin Endpoints for the full format list.

Metrics Counters

System metrics (file counts, disk sizes, throughput rates, process memory, and bounded cache diagnostics) are tracked via O(1) atomic counters or cheap in-memory snapshots that are updated inline during normal operations. On startup, live file counts and logical bytes are initialized by walking the current directory tree, while stored chunk payload bytes are initialized from KV entry metadata without reading chunk bodies. The GET /system/stats endpoint and the metrics SSE event read directly from these counters and bounded snapshots – there is no full database scan at query time. Rolling rate computation (1-minute, 5-minute, 15-minute averages) is maintained continuously, so monitoring data is always available at near-zero cost.

The Database File (.aeordb)

The .aeordb file is an append-only WAL. Every write appends a new entry to the end of the file. Entries are never modified in place (except during garbage collection).

File Layout

[File Header - 256 bytes]
  Magic: "AEOR"
  Hash algorithm, timestamps, KV/NVT pointers, HEAD hash, entry count

[Entry 1] [Entry 2] [Entry 3] ... [Entry N]
  Chunks, FileRecords, DirectoryIndexes, Snapshots, DeletionRecords, Voids

The 256-byte file header contains pointers to the KV block, NVT, and the current HEAD hash. Every entry carries its own header with magic bytes, type tag, hash algorithm, compression flag, key, and value.

Entry Types

The KV Index File (.aeordb.kv)

The KV store is a sorted array of (hash, offset) pairs stored in a separate file. It maps content hashes to byte offsets in the main .aeordb file, providing O(1) lookups when combined with the NVT.

Each entry is hash_length + 8 bytes (40 bytes for BLAKE3-256). The entries are sorted by hash, and the NVT tells you which bucket to look in, so lookups are a single seek + small scan.

KV Resize

When the KV store needs to grow, the engine enters a brief resize mode:

1. A temporary buffer KV store is created
2. New writes go to the buffer (no blocking)
3. The primary KV store is expanded
4. Buffer contents are merged into the primary
5. Buffer is discarded

Writes never block during resize.

NVT (Normalized Vector Table)

The NVT is an in-memory structure that provides fast hash-to-bucket lookups for the KV store.

How It Works

1. Normalize the hash to a scalar: first_8_bytes_as_u64 / u64::MAX produces a value in [0.0, 1.0]
2. Map the scalar to a bucket: bucket_index = floor(scalar * num_buckets)
3. The bucket points to a range in the KV store – scan that range for the exact hash

BLAKE3 hashes are uniformly distributed, so buckets stay balanced without manual tuning. The NVT starts at 1,024 buckets and doubles when the average scan length exceeds a threshold.

Scaling

Hot File WAL (Crash Recovery)

The --hot-dir flag specifies a directory for write-ahead hot files. During a write:

1. The entry is written to a hot file first (fsync’d)
2. The entry is then written to the main .aeordb file
3. On success, the hot file entry is cleared

If the process crashes between steps 1 and 2, the hot file is replayed on the next startup to recover uncommitted writes. If --hot-dir is not specified, the hot directory defaults to the same directory as the database file.

Snapshot Double-Buffering

AeorDB uses ArcSwap for lock-free concurrent reads. The in-memory directory state is wrapped in an Arc that readers clone cheaply. When a write completes:

1. The writer builds a new directory state
2. The new state is swapped in atomically via ArcSwap::store
3. Readers holding the old Arc continue using it until they finish
4. The old state is dropped when the last reader releases it

This means:

• Readers never block writers
• Writers never block readers
• Every read sees a consistent point-in-time snapshot
• No read locks, no write locks on the read path

B-Tree Directories

Small directories (under 256 entries) are stored as flat lists of child entries. When a directory exceeds 256 entries, the engine automatically converts it to a B-tree structure. This keeps directory lookups O(log n) even for directories with millions of files.

B-tree nodes are themselves stored as content-addressed entries, so they participate in versioning and structural sharing just like any other data.

Directory Propagation

When a file changes, the engine propagates the update up the directory tree:

Write /users/alice.json
  -> update /users/ directory (new child hash for alice.json)
    -> update / root directory (new child hash for users/)
      -> update HEAD (new root hash)

Each directory gets a new content hash because its contents changed. This is how the Merkle tree works – a change at any leaf creates new hashes all the way to the root. The root hash (HEAD) uniquely identifies the complete state of the database.

Unified Cache

AeorDB uses an eviction-based cache (no TTL) for frequently accessed metadata:

Cache entries live indefinitely until explicitly evicted by a write, delete, or rename that affects the underlying data. This avoids stale-read windows that TTL-based caches introduce – the cache is always consistent with the storage layer.

KV Type Index

The KV store maintains a type-based index that enables O(k) lookups by entry type (snapshots, files, directories, etc.) instead of O(n) full scans. For example, listing all snapshots requires scanning only snapshot entries rather than the entire KV store.

Next Steps

• Storage Engine – entry format, hashing, chunking, and dedup details
• Versioning – how snapshots, forks, and diff/patch work
• Indexing & Queries – how indexes are built and queried

Storage Engine

The storage engine is an append-only WAL (write-ahead log) where the log IS the database. Every write appends a new entry. The file only grows (until garbage collection reclaims unreachable entries). This design gives you crash recovery, versioning, and integrity verification as structural properties rather than bolted-on features.

Entry Format

Every entry on disk shares the same header format:

[Entry Header - 31 bytes fixed + hash_length variable]
  magic:            u32    (0x0AE012DB - marks the start of a valid entry)
  entry_version:    u8     (payload format version for this entry)
  entry_type:       u8     (Chunk, FileRecord, DirectoryIndex, etc.)
  flags:            u8     (operational flags)
  hash_algo:        u16    (BLAKE3_256 = 0x0001, SHA256 = 0x0002, etc.)
  compression_algo: u8     (None = 0x00, Zstd = 0x01)
  encryption_algo:  u8     (None = 0x00, reserved for future use)
  key_length:       u32    (length of the key field)
  value_length:     u32    (length of the value field)
  timestamp:        i64    (UTC milliseconds since epoch)
  total_length:     u32    (total bytes including header, for jump-scanning)
  hash:             [u8; N] (integrity hash, N determined by hash_algo)

[Key - key_length bytes]
[Value - value_length bytes]

Key properties:

• magic (0x0AE012DB) enables recovery scanning – find entry boundaries even in a corrupted file by scanning for magic bytes
• total_length enables jump-scanning – skip to the next entry without reading the full key/value
• hash covers entry_type + key + value – re-hash and compare to detect corruption
• entry_version enables payload evolution – the engine selects the correct parser for that entry type based on this byte. The generic WAL/header format is still v0; individual payloads can advance independently.

For BLAKE3-256 (the default), the hash is 32 bytes, making the full header 63 bytes.

Content-Addressed Hashing

Every piece of data is identified by its BLAKE3 hash. Hash inputs are prefixed by type (domain separation) to prevent collisions between different entry types:

The domain prefix ensures that a chunk’s raw data can never produce the same hash as a file path, even if the bytes are identical.

Chunking

Files are split into 256KB chunks for storage. Each chunk is content-addressed independently:

Original file (700KB):
  [Chunk 1: 256KB] -> hash_a
  [Chunk 2: 256KB] -> hash_b
  [Chunk 3: 188KB] -> hash_c

FileRecord:
  path: "/docs/report.pdf"
  content_hash: BLAKE3(file bytes)
  chunk_hashes: [hash_a, hash_b, hash_c]
  total_size: 700KB

Chunking provides:

• Deduplication: Two files sharing identical 256KB blocks store those blocks only once
• Efficient updates: Modifying 3 bytes of a 10GB file creates one new chunk, not a new copy of the entire file
• Streaming reads: Read a file by iterating its chunk hashes and fetching each chunk

Dual-Key FileRecords

FileRecords are stored at two keys to support both current reads and historical versioning:

1. Path key (file:/path) – mutable, always points to the latest version. Used for reads, metadata, indexing, and deletion. O(1) lookup.

2. Content key (filec: + serialized record) – immutable, content-addressed. The directory tree’s ChildEntry.hash points to this key.

When the version manager walks a snapshot’s directory tree, it follows ChildEntry.hash to the content key, which resolves to the FileRecord as it existed at snapshot time – not the current version. This is what makes historical reads correct.

Directories use the same pattern: dir:/path (mutable) and dirc: + data (immutable content key).

FileRecord Format

[FileRecord Value - v1]
  path_length:      u16
  path:             [u8; path_length]     (full file path)
  content_type_len: u16
  content_type:     [u8; content_type_len] (MIME type)
  total_size:       u64                    (file size in bytes)
  created_at:       i64                    (UTC milliseconds)
  updated_at:       i64                    (UTC milliseconds)
  content_hash:     [u8; hash_length]      (raw whole-file hash, BLAKE3(file bytes) by default)
  metadata_length:  u32
  metadata:         [u8; metadata_length]  (arbitrary JSON metadata)
  chunk_count:      u32
  chunk_hashes:     [u8; chunk_count * hash_length] (ordered chunk hashes)

FileRecord v0 is the same layout without content_hash. New writes emit FileRecord v1, and readers support both v0 and v1. Legacy v0 records expose an empty content_hash until they are rewritten. A forced reindex (POST /system/tasks/reindex with "force": true) rewrites older live FileRecords through the current writer and backfills this field.

Metadata fields come before the chunk list so file metadata can be read without skipping past all chunk hashes. Chunk hashes are the tail of the record for streaming reads.

Directory Propagation

When a file is stored or deleted, the change propagates up the directory tree:

Store /users/alice.json:

1. Store chunks -> [hash_a, hash_b]
2. Store FileRecord at path key + content key
3. Update /users/ DirectoryIndex (new ChildEntry for alice.json)
4. Update / root DirectoryIndex (new ChildEntry for users/)
5. Update HEAD in file header (new root hash)

Each directory gets a new content hash because one of its children changed. This chain of updates from leaf to root is what maintains the Merkle tree and makes versioning work.

Namespace Writer Serialization

The visible publish step is serialized by an engine-level namespace write guard. Individual WAL appends are already protected by the writer/KV locks, but that is not enough for a file-system namespace mutation: the mutable path key, the parent directory child entry, ancestor directory hashes, and HEAD must advance as one logical operation relative to other writers.

Buffered file writes may store chunks before taking this guard, because chunks are content-addressed and unreachable chunks are safe garbage if a later publish fails. Once a writer starts publishing FileRecords, symlinks, directory entries, or HEAD, it holds the namespace guard until the publish completes. The guard is reentrant for nested operations such as copy/restore and indexed delete, and it is scoped per StorageEngine instance.

Void Management

When garbage collection reclaims an entry, the bytes the entry occupied become a void – a region of the WAL marked as reclaimable. Voids are tracked entirely in memory by the VoidManager, which keeps two parallel indexes:

by_offset:  BTreeMap<u64, u32>            // offset → size, ordered iteration
by_size:    BTreeMap<u32, BTreeSet<u64>>  // size → set of offsets, best-fit lookup

Voids are never written into the WAL as their own records. They live in memory while the process runs and are persisted by riding along inside the hot tail on every periodic flush, so the next clean startup restores the void set without scanning the file. On a dirty startup (hot tail unreadable), voids are re-derived via a gap scan of the rebuilt KV: any byte range not covered by a live KV entry between kv_block_end and the WAL tail is registered as a void.

When a new entry needs to be written, the engine calls VoidManager::find_void(needed) before appending to the tail. If a void of sufficient size exists, the entry is written in-place at that void’s offset; otherwise the entry appends. If the chosen void is larger than the entry, the remainder is re-registered as a smaller void (when it is at least 63 bytes – the minimum useful void size for BLAKE3-256: 31-byte fixed header + 32-byte hash + 0-byte key + 0-byte value).

Two size floors govern void tracking:

Compression

Compression is a post-hash transform:

Write: raw data -> hash -> compress -> store
Read:  load -> decompress -> verify hash -> return

The hash is always computed on the raw uncompressed data. This preserves deduplication (same content = same hash regardless of compression) and integrity verification.

Each entry carries its own compression_algo byte, so compressed and uncompressed entries coexist in the same file. Currently, zstd is the only supported compression algorithm.

On-Disk Layout: Single File

A complete AeorDB lives in one .aeordb file:

┌──────────────────────────────────────────────────────────────┐
│  File Header Slot A (256 bytes) — magic + version + CRC      │
├──────────────────────────────────────────────────────────────┤
│  File Header Slot B (256 bytes) — magic + version + CRC      │
├──────────────────────────────────────────────────────────────┤
│  KV Block — bucket pages with per-page magic + CRC           │
│  (stages: 64 KB → 512 KB → 4 MB → 32 MB → 128 MB → …)        │
├──────────────────────────────────────────────────────────────┤
│  WAL — entries appended forward (chunks, file records,       │
│  directory indexes, snapshots, …)                            │
├──────────────────────────────────────────────────────────────┤
│  Hot Tail — magic + count + CRC + recent KV entries          │
└──────────────────────────────────────────────────────────────┘

No sidecar files. No journal directory. The single file is the entire database.

The file header lives in two slots: slot A at byte 0, slot B at byte 256. Each carries a u64 sequence, the header fields, a format_magic = "AEORDB\0\0" byte string, a format_version: u8, and a u32 CRC over the slot. Writers update whichever slot has the lower sequence (increment then write, then fsync). Readers parse both and pick the highest sequence with a valid CRC. A torn write on one slot leaves the other intact — the database always opens.

On open, if format_magic is wrong the engine refuses with “not an AeorDB file”. If format_version doesn’t match what this build understands, the engine refuses with a clear “DB format vN, this build expects vM” message. Format-breaking changes must bump the version.

The KV block at the head is a flat bucket array — hash → entry offset mappings indexed by a normalized vector table (NVT). Each bucket page carries [magic: u32][crc32: u32][entry_count: u16][entries…]. Lookups validate magic + CRC before scanning. If a page fails validation, the engine performs a per-bucket rebuild: using the NVT to identify which hashes belong to that bucket, it scans the WAL for those entries and reconstructs the page. The WAL is the source of truth; bucket pages are a recoverable index.

The WAL grows forward from the end of the KV block. New entries are appended.

The hot tail at EOF is a short journal of KV entries that haven’t been flushed into the bucket pages yet. On startup we read the hot tail first, populating the in-memory write buffer with the most recent unflushed entries.

Online KV Expansion

When the KV bucket pages fill, AeorDB grows the KV block to the next stage while the database is running — no restart, no downtime, no rebuild.

The expansion holds both writer and KV locks for the duration of the relocation, then releases them. Reads and writes that complete before or after the expansion proceed normally; writes during the expansion queue on the lock.

Stage progression (BLAKE3, page size 1,314 bytes):

Growth is geometric early (8× per stage) then linear (4×, 2×) to keep relocation costs bounded.

How it works (atomically)

1. Set resize_in_progress = true and resize_target_stage in the file header.
2. Scan forward from the new KV-block boundary to find the first WAL entry that starts AFTER it. This is the actual end of the growth zone — entries straddling the boundary must be included so the relocated copy is complete.
3. Bulk-copy the growth zone [old_kv_end .. actual_copy_end] to the end of the WAL (right before the hot tail). This temporarily produces two copies — the original (still readable) and the relocated copy. Crash-safe window.
4. fsync.
5. Tell the KV store to finalize: zero the new bucket pages, rehash all entries (entries that lived in the growth zone now point to their relocated offsets), update the in-memory state, write the file header with the new stage.
6. Write a Void entry over the dead tail at the old growth-zone boundary so the entry scanner stays clean on restart.

If the process crashes between steps 2 and 5, the original WAL data is intact and the next startup detects resize_in_progress = true in the header. The expansion resumes from the relocated copy.

Hot Tail

The hot tail is a small, versioned journal at the end of the file. It carries two kinds of transient state that would otherwise be lost on crash:

1. Pending KV writes — entries that exist in the in-memory write buffer but haven’t been flushed to bucket pages yet.
2. Void snapshot — the current VoidManager state, so the next clean startup restores reclaimable-space tracking without rescanning the WAL.

[Header — 21 bytes]
  magic:           [u8; 5] = AE 01 7D B1 0D
  format_version:  u8                 (top-level layout version; bumped on section changes)
  write_count:     u32                (number of write records below)
  void_count:      u32                (number of void records below)
  header_crc32:    u32                (CRC32 of the preceding 14 bytes)

[Write records — 1 + hash_length + 13 bytes each (42 for BLAKE3-256)]
  version:      u8        (per-record layout version; bumped without a full format bump)
  hash:         [u8; hash_length]
  type_flags:   u8
  offset:       u64                   (WAL position of the actual entry)
  total_length: u32                   (on-disk length of the entry)

[Void records — 13 bytes each]
  version:      u8
  offset:       u64                   (start of the reclaimable region)
  size:         u32                   (length of the region)

The hot tail is the durability boundary for both the KV and the void set. Periodic flushes (every 100 ms or whenever the write buffer hits its threshold) rewrite the hot tail in-place at header.hot_tail_offset. On the next clean startup the header is parsed, the write records reload into the write buffer, and the void records repopulate the VoidManager – all before any read serves traffic.

If the hot tail’s header CRC fails, the magic doesn’t match, or the recorded offset points past the file boundary, the engine logs a warning and triggers a dirty startup: a WAL metadata scan (via scan_entries_dirty_recovery) rebuilds the KV from scratch and recover_voids_via_gap_scan re-derives the void set from gaps in the rebuilt KV. The scan reads and verifies mutable metadata records, but skips large chunk and void payload bodies because rebuild only needs their headers and keys. Normal file reads and explicit verification tooling still validate chunk contents. No data is lost – the WAL is the source of truth; the hot tail is a fast-path index plus a void snapshot.

A magic-byte version bump is enough on its own to invalidate older hot tails: the next open with newer code sees a magic mismatch, falls into dirty startup, and rebuilds correctly. That makes the format safely evolvable without a migration tool.

fsync Strategy

Not all entries are equally important for durability:

This gives durability where it matters and performance where it doesn’t.

Serious Durability Failures

If a critical write, flush, or sync step fails, AeorDB latches the engine into a read-only safety mode. Reads and diagnostic operations remain available, but subsequent write/index/GC mutations return a durability failure instead of continuing on top of an uncertain persistence state.

On the first latch, the engine also attempts a best-effort emergency spill outside the database file. The spill contains:

• manifest.json with the failure context, DB path, offsets, counts, and spill errors.
• hot-tail.bin with the current in-memory hot-tail payload: pending KV writes plus the void snapshot.
• index-buffer.json when there are unflushed buffered index mutations.
• wal-tail.bin when the engine can copy WAL bytes from the last successfully published hot-tail boundary to the current WAL offset.

Spill destination order is:

1. AEORDB_EMERGENCY_SPILL_DIR, when set.
2. The OS user-data directory ($XDG_DATA_HOME/aeordb/emergency-spill, ~/.local/share/aeordb/emergency-spill, macOS Application Support, or Windows app data).
3. The temp fallback: /tmp/aeordb-emergency-spill on Unix-like systems.

AEORDB_EMERGENCY_WAL_SPILL_MAX_BYTES caps the WAL-tail copy. The default is 4 GiB; set it to 0 to copy the full tail. This is intentionally best-effort: the copied bytes reflect what the process can still read and write through the operating system at the time of failure.

On the next server start, AeorDB scans every spill destination for unresolved artifacts whose db_path matches the database being opened. If any are found, startup refuses to serve the normal API and prints the in-place repair command:

aeordb verify --repair --force-fix-in-place -D /path/to/database.aeordb

Repair orders matching artifacts by creation time, oldest first, and prompts before replay unless --yes is passed. Replay only copies verified WAL-tail bytes back into the database file. The external hot-tail.bin and index-buffer.json files are reported as evidence, but are not treated as primary data: after the WAL tail is restored, repair forces a WAL-to-EOF KV rebuild, re-derives reusable gaps with the void gap scanner, and publishes a new hot tail. When verify/repair finishes cleanly, each artifact directory receives an applied.json marker and future startups ignore it.

Crash Recovery

The recovery hierarchy, from least to most damage:

The magic bytes at the start of every entry enable boundary detection even in partially corrupted files. The total_length field in each header enables efficient forward scanning.

Next Steps

• Architecture – high-level system overview
• Versioning – snapshots, forks, and the Merkle tree

Versioning

AeorDB’s content-addressed Merkle tree makes versioning a structural property of the storage engine rather than an add-on feature. Every write creates new hashes up the directory tree, so every committed state is already a snapshot by definition – you just need to save a pointer to the root hash.

How Versioning Works

The database state at any point in time is fully described by its root hash (HEAD). HEAD is the hash of the root DirectoryIndex, which contains hashes of its children, which contain hashes of their children, all the way down to the chunks of individual files.

HEAD -> root DirectoryIndex
         |
         +-- /users/ (DirectoryIndex)
         |     +-- alice.json (FileRecord -> [chunk_a, chunk_b])
         |     +-- bob.json   (FileRecord -> [chunk_c])
         |
         +-- /docs/ (DirectoryIndex)
               +-- readme.md  (FileRecord -> [chunk_d])

When you write /users/alice.json, the engine creates:

1. New chunks (if the content changed)
2. A new FileRecord with new chunk hashes
3. A new /users/ DirectoryIndex with the updated child entry
4. A new root DirectoryIndex with the updated /users/ child entry
5. HEAD now points to the new root hash

The old root hash still exists and still points to the old directory tree with the old file content. Nothing was overwritten – new entries were appended.

Snapshots

A snapshot is a named reference to a root hash. Creating a snapshot saves the current HEAD so you can return to it later.

Create a Snapshot

curl -X POST http://localhost:6830/versions/snapshots \
  -H "Content-Type: application/json" \
  -d '{"name": "v1.0"}'

Snapshot deduplication: If an existing snapshot already has the same root hash as the current HEAD, create_snapshot returns the existing snapshot instead of creating a duplicate. This means taking frequent snapshots is essentially free when data hasn’t changed – no storage overhead, no redundant entries.

List Snapshots

curl http://localhost:6830/versions/snapshots

Response:

{
  "snapshots": [
    {"name": "v1.0", "root_hash": "a1b2c3...", "created_at": 1775968398000},
    {"name": "v2.0", "root_hash": "d4e5f6...", "created_at": 1775968500000}
  ]
}

Restore a Snapshot

Restoring a snapshot sets HEAD back to the snapshot’s root hash. The current state is not lost – you can snapshot it before restoring if you want to preserve it.

curl -X POST http://localhost:6830/versions/restore \
  -H "Content-Type: application/json" \
  -d '{"name": "v1.0"}'

Delete a Snapshot

curl -X DELETE http://localhost:6830/versions/snapshots/v1.0

After deleting a snapshot, entries that were only reachable through that snapshot become eligible for garbage collection.

Forks

Forks are isolated branches of the database. Writes to a fork do not affect HEAD or other forks. This is useful for testing changes, running experiments, or staging updates before promoting them.

Create a Fork

curl -X POST http://localhost:6830/versions/forks \
  -H "Content-Type: application/json" \
  -d '{"name": "experiment"}'

List Forks

curl http://localhost:6830/versions/forks

Promote a Fork

When you’re satisfied with the changes in a fork, promote it to HEAD:

curl -X POST http://localhost:6830/versions/forks/experiment/promote

Abandon a Fork

curl -X DELETE http://localhost:6830/versions/forks/experiment

Tree Walking

The content-addressed Merkle tree enables historical reads. When you walk a snapshot’s directory tree:

1. Start from the snapshot’s root hash
2. Load the root DirectoryIndex – each child entry has a hash
3. Follow child hashes to subdirectories or files
4. Each FileRecord’s ChildEntry.hash points to a content-addressed (immutable) key

Because file content keys are immutable, walking a snapshot’s tree always resolves to the data as it existed when the snapshot was taken, even if the files have been overwritten or deleted since then.

Snapshot "v1.0" root_hash: aaa111...
  -> /users/ dir_hash: bbb222...
     -> alice.json content_key: ccc333...  (resolves to Alice's v1.0 data)

Current HEAD root_hash: ddd444...
  -> /users/ dir_hash: eee555...
     -> alice.json content_key: fff666...  (resolves to Alice's current data)

Both trees can coexist because they share unchanged chunks and directories (structural sharing). Only the parts that differ consume additional storage.

Export and Import

AeorDB can export a version as a self-contained .aeordb file and import it into another database.

Export

Export creates a clean, compacted database from a single version – no history, no voids, no deleted entries:

# Export current HEAD
aeordb export --database data.aeordb --output backup.aeordb

# Export a specific snapshot
aeordb export --database data.aeordb --snapshot v1.0 --output v1.aeordb

# Export via HTTP
curl -X POST http://localhost:6830/versions/export --output backup.aeordb
curl -X POST "http://localhost:6830/versions/export?snapshot=v1.0" --output v1.aeordb

The exported file is a fully functional database that can be opened with aeordb start.

Import

Import applies an export or patch file to a target database:

# Import without promoting HEAD (inspect first)
aeordb import --database target.aeordb --file backup.aeordb

# Import and promote in one step
aeordb import --database target.aeordb --file backup.aeordb --promote

# Import via HTTP
curl -X POST http://localhost:6830/versions/import \
  -H "Content-Type: application/octet-stream" \
  --data-binary @backup.aeordb

Import does NOT automatically change HEAD. The imported version exists in the database and can be promoted explicitly when ready.

Diff and Patch

Diff extracts only the changeset between two versions. The output is a patch file – not a standalone database.

# Diff between two snapshots
aeordb diff --database data.aeordb --from v1.0 --to v2.0 --output patch.aeordb

# Diff from a snapshot to current HEAD
aeordb diff --database data.aeordb --from v1.0 --output patch.aeordb

# Diff via HTTP
curl -X POST "http://localhost:6830/versions/diff?from=v1.0&to=v2.0" --output patch.aeordb

A patch file contains only new/changed chunks, updated FileRecords, and deletion markers. Chunks shared between the two versions are not included, making patches much smaller than full exports.

Applying a Patch

# Apply patch (strict base version check)
aeordb import --database target.aeordb --file patch.aeordb

# Skip base version check
aeordb import --database target.aeordb --file patch.aeordb --force

# Apply and promote
aeordb import --database target.aeordb --file patch.aeordb --promote

If the target database’s HEAD does not match the patch’s base version, the import fails unless --force is used.

Promote

Promote sets HEAD to a specific version hash. This is separate from import so you can inspect the imported data before committing to it:

# CLI
aeordb promote --database data.aeordb --hash f6e5d4c3...

# HTTP
curl -X POST "http://localhost:6830/versions/promote" \
  -H "Content-Type: application/json" \
  -d '{"hash": "f6e5d4c3..."}'

Next Steps

• Storage Engine – how the Merkle tree and content addressing work at the byte level
• Architecture – system overview and crash recovery

Indexing & Queries

AeorDB indexes are opt-in and configured per-directory. Nothing is indexed by default – you control exactly which fields are indexed, with which strategies, and for which file types. This keeps the engine lean and predictable.

Index Configuration

Create a .aeordb-config/indexes.json file in any directory to define indexes for files in that directory:

curl -X PUT http://localhost:6830/files/users/.aeordb-config/indexes.json \
  -H "Content-Type: application/json" \
  -d '{
    "indexes": [
      {"name": "name", "type": ["string", "trigram"]},
      {"name": "age", "type": "u64"},
      {"name": "city", "type": "string"},
      {"name": "email", "type": "trigram"},
      {"name": "created_at", "type": "timestamp"}
    ]
  }'

When this file is created or updated, the engine automatically triggers a background reindex of all existing files in the directory.

Subdirectory Indexing with Glob

By default, an index config only indexes direct children of its directory. To index files across subdirectories, add a glob field:

curl -X PUT http://localhost:6830/files/sessions/.aeordb-config/indexes.json \
  -H "Content-Type: application/json" \
  -d '{
    "glob": "*/session.json",
    "indexes": [
      {"name": "patient_name", "type": ["string", "trigram"]},
      {"name": "notes", "type": "trigram", "source": ["comments", "", "text"]}
    ]
  }'

This config at /sessions/ indexes all session.json files in immediate subdirectories (e.g., /sessions/s1/session.json, /sessions/s2/session.json).

Glob patterns:

• * — matches one directory level (*/file.json matches subdir/file.json)
• ** — matches any depth (**/*.json matches a/b/c/file.json)
• ? — matches a single character

When a file is stored, the engine checks ancestor directories for glob configs. The nearest matching ancestor’s config is used. Indexes are stored at the config owner’s directory, so querying /sessions/ finds results from all matching subdirectories.

Reindexing with a glob config recursively scans all subdirectories and filters by the glob pattern.

Index Types

Multi-Strategy Indexes

A single field can be indexed with multiple strategies by passing type as an array:

{"name": "title", "type": ["string", "trigram", "phonetic"]}

This creates three separate index files for the same field:

• title.string.idx – exact match queries
• title.trigram.idx – fuzzy/substring queries
• title.phonetic.idx – phonetic queries

Use the appropriate query operator to target the desired index.

How Indexes Work

AeorDB uses a Normalized Vector Table (NVT) for index lookups. Each indexed field gets its own NVT.

The NVT Approach

1. A ScalarConverter maps each field value to a scalar in [0.0, 1.0]
2. The scalar maps to a bucket in the NVT
3. The bucket points to the matching entries

For numeric types (u64, i64, f64, timestamp), the converter tracks the observed min/max and distributes values uniformly across the [0.0, 1.0] range. This means range queries (gt, lt, between) are efficient – they resolve to a contiguous range of buckets.

For a query like WHERE age > 30:

1. converter.to_scalar(30) computes where 30 falls in the bucket range
2. All buckets after that point are candidates
3. Only those buckets are scanned

This is O(1) for the bucket lookup, with a small linear scan within the bucket.

Two-Tier Execution

Simple queries (single field, direct comparison) use direct scalar lookups – no bitmaps, no compositing. Most queries fall into this tier.

Complex queries (OR, NOT, multi-field boolean logic) build NVT bitmaps and composite them:

• Each field condition produces a bitmask over the NVT buckets
• AND = bitwise AND of masks
• OR = bitwise OR of masks
• NOT = bitwise NOT of a mask
• The final mask identifies which buckets contain results

Memory usage is bounded: a bitmask for 1M buckets is only 128KB, regardless of how many entries exist.

Source Resolution

By default, the name field in an index definition is used as the JSON key to extract the value:

{"name": "age", "type": "u64"}

This extracts the age field from {"name": "Alice", "age": 30}.

Nested Fields

For nested JSON or parser output, use the source array to specify the path:

{"name": "author", "source": ["metadata", "author"], "type": "string"}

This extracts metadata.author from a JSON structure like:

{"metadata": {"author": "Jane Smith", "title": "Report"}}

The source array supports:

• String segments for object key lookup: ["metadata", "author"]
• Integer segments for array index access: ["items", 0, "name"]

Array Fan-Out

To index every element in an array, use an empty string "" as a source segment. This “fans out” — creating one index entry per array element:

{"name": "tag", "type": "trigram", "source": ["tags", ""]}

For {"tags": ["rust", "database", "aeordb"]}, this creates three index entries for the same file. A query for tag = "rust" will find this file.

Fan-out works on objects too — "" iterates all values:

{"name": "value", "type": "string", "source": [""]}

For {"color": "red", "size": "large"}, this creates entries for both "red" and "large".

Fan-out can be chained for nested structures:

{"name": "comment_text", "type": "trigram", "source": ["comments", "", "text"]}

For {"comments": [{"text": "hello"}, {"text": "world"}]}, this creates entries for "hello" and "world".

Regex Filtering

Use a regex segment /pattern/flags to filter which keys (on objects) or elements (on arrays) are included:

{"name": "tag_value", "type": "string", "source": ["/^tag_/"]}

For {"tag_color": "red", "tag_size": "large", "name": "foo"}, this creates entries for "red" and "large" (keys matching /^tag_/), but not "foo".

Supported flags:

• i — case-insensitive matching

{"name": "name_field", "type": "string", "source": ["/^name$/i"]}

Matches keys name, Name, NAME, etc.

Plugin Mapper

For complex extraction logic, delegate to a WASM plugin:

{
  "name": "summary",
  "source": {"plugin": "my-mapper", "args": {"mode": "summary", "max_length": 500}},
  "type": "trigram"
}

The plugin receives the parsed JSON and the args object, and returns the extracted field value.

Parser Integration

For non-JSON files (PDFs, images, XML, etc.), a parser converts raw bytes into a JSON object that the indexing pipeline can work with.

Native Parsers (Built-In)

AeorDB ships with 8 native parsers that handle common formats automatically – no deployment required. During indexing, the engine tries native parsers first based on content type (with extension-based fallback for application/octet-stream). Supported formats include text, HTML/XML, PDF, images (JPEG/PNG/GIF/BMP/WebP/TIFF/SVG), audio (MP3/WAV/OGG), video (MP4/AVI/WebM/MKV/FLV), MS Office (DOCX/XLSX), and ODF (ODT/ODS). See Plugin Endpoints – Native Parsers for the full list.

If no native parser handles the content type, the engine falls through to the WASM plugin system.

WASM Parser Plugins

For custom or proprietary formats not covered by the built-in parsers, deploy a WASM parser plugin. WASM parsers receive the same input envelope and return the same JSON structure as native parsers.

Configuration

{
  "parser": "pdf-extractor",
  "parser_memory_limit": "256mb",
  "indexes": [
    {"name": "title", "source": ["metadata", "title"], "type": ["string", "trigram"]},
    {"name": "author", "source": ["metadata", "author"], "type": "phonetic"},
    {"name": "content", "source": ["text"], "type": "trigram"},
    {"name": "page_count", "source": ["metadata", "page_count"], "type": "u64"}
  ]
}

The parser receives a JSON envelope with the file data (base64-encoded) and metadata:

{
  "data": "<base64-encoded file bytes>",
  "meta": {
    "filename": "report.pdf",
    "path": "/docs/reports/report.pdf",
    "content_type": "application/pdf",
    "size": 1048576
  }
}

The parser returns a JSON object (like {"text": "...", "metadata": {"title": "...", ...}}), and the source paths in each index definition walk this JSON to extract field values.

Global Parser Registry

You can also register parsers globally by content type at /.config/parsers.json:

{
  "application/pdf": "pdf-extractor",
  "image/jpeg": "image-metadata",
  "image/png": "image-metadata"
}

When a file is stored and no parser is configured in the directory’s index config, the engine checks this registry using the file’s content type.

Failure Handling

Parser and indexing failures never prevent file storage. The file is always stored regardless of parse/index errors. If logging is enabled in the index config ("logging": true), errors are written to .logs/ under the directory.

Default Indexes

On first server start, AeorDB bootstraps a default index configuration at /.aeordb-config/indexes.json with glob: "**/*". This automatically indexes every file’s metadata across the entire database:

These indexes are stored at /.indexes/ and cover every file in the database. Because the bootstrap config uses glob: "**/*", the query and global search endpoints can use the root index for scoped virtual-field lookups and then filter candidates to the requested path. The global search endpoint (POST /files/search) works out of the box with no additional configuration.

@-Field Source Resolution

When the indexing pipeline encounters a field name starting with @, it extracts the value from the file’s metadata (FileRecord) instead of parsing the file content. This means even binary files (images, videos, PDFs) are indexed by path, filename, extension, raw whole-file hash, timestamps, size, and content type without needing a parser.

@hash is not a chunk hash and is not the content-addressed key for the serialized FileRecord. It is the hash of the reconstructed file bytes, computed and stored during file writes.

Customization

The default config at /.aeordb-config/indexes.json is only written on first boot. Existing databases keep their current config until you edit it. You can modify the config to add or remove default fields; changes trigger an automatic reindex. During reindex, AeorDB retires index files whose field/strategy is no longer present in the config. A forced reindex can also backfill FileRecord metadata migrations such as whole-file @hash.

Automatic Reindexing

When you store or update a .aeordb-config/indexes.json file, the engine automatically enqueues a background reindex task for that directory. If the config contains only virtual @ metadata fields, the task uses the metadata-only path and skips full file reads and parsers. Mixed configs that include content fields use the full indexing pipeline.

The task:

1. Reads the current index config
2. Lists all files in the directory
3. Re-runs metadata-only or full indexing for each file
4. Buffers index writes in memory and flushes them periodically
5. Reports progress via GET /system/tasks

During reindexing, queries still work but may return incomplete results. The query response includes a meta.reindexing field with the current progress:

{
  "results": [...],
  "meta": {
    "reindexing": 0.67,
    "reindexing_eta": 1775968398803,
    "reindexing_indexed": 6700,
    "reindexing_total": 10000
  }
}

Query API

Queries are submitted as POST /files/query with a JSON body:

{
  "path": "/users/",
  "where": {
    "and": [
      {"field": "age", "op": "gt", "value": 30},
      {"field": "city", "op": "eq", "value": "Portland"},
      {"not": {"field": "role", "op": "eq", "value": "banned"}}
    ]
  },
  "sort": {"field": "age", "order": "desc"},
  "limit": 50,
  "offset": 0
}

Boolean Logic

The where clause supports full boolean logic:

{
  "where": {
    "or": [
      {"field": "city", "op": "eq", "value": "Portland"},
      {
        "and": [
          {"field": "age", "op": "gt", "value": 25},
          {"field": "city", "op": "eq", "value": "Seattle"}
        ]
      }
    ]
  }
}

For backward compatibility, a flat array in where is treated as an implicit and:

{
  "where": [
    {"field": "age", "op": "gt", "value": 30},
    {"field": "city", "op": "eq", "value": "Portland"}
  ]
}

Query Operators

Next Steps

• Quick Start – hands-on tutorial with query examples
• Configuration – full index config reference
• Storage Engine – how indexed data is stored

Replication

AeorDB supports multi-node replication using content-addressed sync. Every node is a full peer — any node can accept writes. Nodes sync by comparing directory tree hashes and exchanging missing chunks. Conflicts are detected, preserved, and resolved as first-class database entities.

How It Works

AeorDB’s replication leverages its content-addressed storage architecture:

1. Every file is identified by its content hash — identical content always produces the same hash, regardless of when or where it was stored
2. Directory trees are Merkle trees — changing one file changes the tree hash all the way to the root
3. Sync is a tree comparison — two nodes compare their root hashes. If they differ, they exchange the entries that are different
4. Chunks are immutable — once stored, a chunk never changes. This makes transferring data between nodes safe and idempotent

Sync Protocol

When two nodes sync:

1. Node A asks Node B: “What changed since the last time we synced?” (tree diff)
2. Node B computes the differences and responds with a list of added, modified, and deleted files
3. Node A fetches any missing chunks from Node B
4. Node A merges the changes into its own tree, detecting conflicts
5. Node A updates its HEAD to the merged state

This process is atomic — either all changes are applied, or none are. A network failure mid-sync leaves the database unchanged.

Sync is bidirectional. After Node A pulls from Node B, Node B can pull from Node A to get any changes that originated on Node A.

Conflict Resolution

When two nodes modify the same file independently, AeorDB detects the conflict and resolves it automatically:

• Last-Write-Wins (LWW) — the version with the higher virtual timestamp becomes the “current” version
• Modify beats delete — if one node modifies a file while another deletes it, the modification wins. Work is never silently lost.
• Loser preserved — the “losing” version is stored in /.conflicts/ so it can be recovered

Conflicts are stored as regular database entries, which means they sync to all nodes automatically. A conflict resolved on any node propagates the resolution to all other nodes.

Viewing Conflicts

# List all unresolved conflicts
curl http://localhost:6830/sync/conflicts \
  -H "Authorization: Bearer $TOKEN"

# Resolve a conflict (pick the winner)
curl -X POST http://localhost:6830/sync/conflicts/assets/logo.psd \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"pick": "winner"}'

Virtual Clock

Nodes synchronize their clocks using the heartbeat mechanism. Each heartbeat carries three fields — intent_time, construct_time, and node_id — which allow nodes to compute clock offsets and network latency. This ensures that timestamps used for conflict resolution ordering are consistent across nodes to near-millisecond precision. The heartbeat is a dedicated clock-sync pulse and does not carry any stats or metrics data.

When a new node connects, it enters a honeymoon phase where only heartbeats are exchanged. The node settles its clock before any data sync begins, ensuring accurate timestamp ordering from the first sync.

Selective Sync

Nodes can sync specific path subtrees only:

{
  "sync_paths": ["/assets/**", "/docs/**"]
}

This is useful for:

• Desktop clients that only need their working directory
• Regional offices that only need their projects
• Edge nodes that serve specific content

Client = Node

The replication protocol is the same protocol that the AeorDB client uses. A desktop client syncing with a server and a server syncing with another server use the same mechanism — compare hashes, exchange chunks, merge trees.

Client Sync

Desktop clients and other non-peer applications can sync using the same protocol as replication peers, with appropriate access restrictions:

• Clients authenticate with their JWT token
• The /.system/ directory is automatically excluded from client sync results
• API key scoping rules apply — a scoped key with restricted path access only sees changes for allowed paths
• Clients can use the paths filter for selective sync (e.g., only sync /assets/**)

This means a client with a read-only key scoped to /assets/ will only see file changes under /assets/ in sync diffs, and cannot access system data, other users’ files, or paths outside its scope.

Comparison with Strong Consistency

AeorDB uses eventual consistency, not strong consistency (Raft/Paxos). This means:

For creative teams working with large assets across multiple locations, eventual consistency is the right tradeoff — availability and simplicity matter more than instant consistency.

For hands-on instructions for setting up and managing a cluster, see Cluster Operations.

Users and Workspaces

AeorDB ships with no implicit “home directory” for users. A new user exists in the system store (/.aeordb-system/users/<uuid>) but owns nothing in the file tree until an admin explicitly grants them access to one or more paths.

This is deliberate. Different deployments want different patterns:

• An interactive customer-facing app might want every user to have a personal workspace at /workspaces/<username>/.
• A team-collaboration app might want shared folders at /teams/<team-name>/ with no per-user space at all.
• A backup or sync deployment might want users that exist purely to hold API keys and never write a single file from the portal.

Hardcoding a /home/<user>/ convention into the engine would force every deployment into the first shape. Instead AeorDB exposes the primitives — create a folder, grant a permission — and leaves the policy to the operator.

The suggested convention: /workspaces/<username>/

When an admin creates a user from the portal, the “Create User” modal offers to bootstrap a personal workspace as a single extra step:

Username: [wyatt           ]
Email:    [[email protected]]

☑ Grant a personal workspace
   Path: [/workspaces/wyatt  ]   (full access for this user)

Checking the box (default) performs two operations after the user record is created:

1. POST /files/mkdir to create the folder at the chosen path.
2. POST /files/share to grant the new user crudlify permission on that folder.

The end result is identical to an admin doing both steps manually from the file browser — it’s a UX shortcut for the common case, not a new engine concept. You can:

• Override the path: type whatever makes sense for your deployment (/team-wyatt/, /u/wyatt/, /home/wyatt/, …). The default is just a suggestion.
• Skip the workspace entirely: uncheck the box for service / backup / replication users who shouldn’t have a portal-visible storage area.

Users with no grants

A user with no grants and no shares is still a valid AeorDB account. They can:

• Authenticate via POST /auth/token and receive a JWT.
• Hit any API endpoint the JWT permits (typically reads of paths shared with them later).
• Generate API keys for themselves (self-service if enabled).

What they can’t do — yet — is see or write anything in the file tree. The portal’s file browser detects this state and shows a guidance card with the user’s ID so they can request access from an admin instead of staring at a blank page with no action buttons.

Granting access after the fact

If you skipped the workspace at user-creation time (or want to grant additional folders later), the share flow is the canonical path:

1. From the file browser, navigate to (or create) the folder you want to grant.
2. Use the Share action on that folder.
3. Pick the user(s) and the permission level (crudlify for full access, or a custom subset).

This is the same engine call (POST /files/share) that the user-creation modal makes when the workspace checkbox is on — it’s just split across the two operations (mkdir + share) when an admin needs to do it incrementally.

Service / backup / replication users

For automation accounts, the recommended pattern is:

1. Create the user with the workspace checkbox unchecked.
2. Generate an API key for that user from the Keys page.
3. Grant the user the minimal permission set their job needs — e.g. read-only access to /data/exports/ for a backup job, or crudlify on /incoming/<service-name>/ for an ingestion bot.
4. Hand the API key to the automation.

The user record will never appear in the portal’s file browser because there’s nothing for them to browse — exactly the desired behavior for non-interactive accounts.

Why not auto-create /home/<user>/?

The first iteration of this design considered an engine-side rule that would auto-create a home folder for every new user. We rejected it because:

• It hardcodes a namespace. Customers wanting /team-wyatt/ would have to navigate around an auto-created /home/wyatt/ that they never asked for.
• Service users would get noise — every backup or CI account leaving an empty home folder for someone to clean up.
• Existing users (root, anyone created before the rule) would not have one — inconsistent and confusing.

Pushing the workspace creation to the portal UI, with an opt-out checkbox and an editable path, gives the same 90%-case convenience without baking a policy into the engine.

Files & Directories

AeorDB exposes a content-addressable filesystem through its file routes. Every path under /files/ represents either a file or a directory.

Endpoint Summary

Searching by metadata: Files can also be searched by their metadata – path, filename, extension, hash, size, content type, and timestamps – using virtual fields in the query API. New databases index these fields by default; legacy databases can add the default config and reindex. If a virtual-field index is absent, AeorDB falls back to a FileRecord metadata scan.

PUT /files/

Store a file at the given path. Parent directories are created automatically. If a file already exists at the path, it is overwritten (creating a new version).

Streaming uploads: Files are streamed in 256 KB chunks – the server never buffers the full file in memory. Files up to the router-level limit (10 GB) are supported. For resumable uploads of very large files, see the chunked upload protocol.

Request

• Headers:
  • Authorization: Bearer <token> (required)
  • Content-Type (optional) – auto-detected from magic bytes if omitted
• Body: raw file bytes

Response

Status: 201 Created

{
  "path": "/data/report.pdf",
  "content_type": "application/pdf",
  "size": 245678,
  "created_at": 1775968398000,
  "updated_at": 1775968398000
}

Side Effects

• If the path matches /.aeordb-config/indexes.json (or a nested variant like /data/.aeordb-config/indexes.json), a reindex task is automatically enqueued for the parent directory. Any existing pending or running reindex for that path is cancelled first. Configs containing only virtual @ metadata fields use metadata-only reindexing automatically.
• Triggers entries_created events on the event bus.
• Runs any deployed store-phase plugins.

Example

curl -X PUT http://localhost:6830/files/data/report.pdf \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/pdf" \
  --data-binary @report.pdf

Error Responses

GET /files/

Read a file or list a directory. The server determines the type automatically:

• If the path resolves to a file, the file content is streamed with appropriate headers.
• If the path resolves to a directory, a JSON object with an items array of children is returned.

Request

• Headers:
  • Authorization: Bearer <token> (required)
  • Range: bytes=<start>-<end> (optional for file reads) – single byte ranges only

Query Parameters

File Response

Status: 200 OK for full reads, 206 Partial Content for valid byte-range reads

Headers:

Body: raw file bytes (streamed)

Byte Range Reads

File reads support HTTP single byte ranges for browser media seeking and partial downloads:

curl http://localhost:6830/files/media/movie.mp4 \
  -H "Authorization: Bearer $TOKEN" \
  -H "Range: bytes=1048576-2097151" \
  -o movie.part

Supported forms:

Unsatisfiable or malformed byte ranges return 416 Range Not Satisfiable with Content-Range: bytes */<file_size>. Multi-range requests are not supported.

Directory Response

Status: 200 OK

Each entry includes path, hash, and numeric entry_type fields. Symlink entries also include a target field.

Entry types: 2 = file, 3 = directory, 8 = symlink.

Effective permissions: When a directory listing is accessed with a scoped API key or as a non-root user, each item in the response includes an effective_permissions field – an 8-character crudlify string indicating what operations the caller can perform on that item. Ancestor directory items (directories leading to the scoped path) receive read+list permissions only.

{
  "items": [
    {
      "path": "/data/report.pdf",
      "name": "report.pdf",
      "entry_type": 2,
      "hash": "a3f8c1...",
      "size": 245678,
      "created_at": 1775968398000,
      "updated_at": 1775968398000,
      "content_type": "application/pdf"
    },
    {
      "path": "/data/images",
      "name": "images",
      "entry_type": 3,
      "hash": "b2c4d5...",
      "size": 0,
      "created_at": 1775968000000,
      "updated_at": 1775968000000,
      "content_type": null
    },
    {
      "path": "/data/latest",
      "name": "latest",
      "entry_type": 8,
      "hash": "c3d5e6...",
      "target": "/data/report.pdf",
      "size": 0,
      "created_at": 1775968500000,
      "updated_at": 1775968500000,
      "content_type": null
    }
  ]
}

Paginated Directory Listing

Use limit and offset to paginate directory listings:

curl "http://localhost:6830/files/data/?limit=10&offset=20" \
  -H "Authorization: Bearer $TOKEN"

When limit or offset is provided, the response includes pagination metadata:

{
  "items": [...],
  "total": 150,
  "limit": 10,
  "offset": 20
}

Examples

Read a file:

curl http://localhost:6830/files/data/report.pdf \
  -H "Authorization: Bearer $TOKEN" \
  -o report.pdf

List a directory:

curl http://localhost:6830/files/data/ \
  -H "Authorization: Bearer $TOKEN"

Recursive Directory Listing

Use the depth and glob query parameters to list files recursively:

# List all files recursively
curl http://localhost:6830/files/data/?depth=-1 \
  -H "Authorization: Bearer $TOKEN"

# List only .psd files anywhere under /assets/
curl "http://localhost:6830/files/assets/?depth=-1&glob=*.psd" \
  -H "Authorization: Bearer $TOKEN"

# List JSON frames anywhere under /sessions/
curl "http://localhost:6830/files/sessions/?depth=-1&glob=**/frames/*.json" \
  -H "Authorization: Bearer $TOKEN"

# List one level deep
curl http://localhost:6830/files/data/?depth=1 \
  -H "Authorization: Bearer $TOKEN"

When depth > 0 or depth = -1, the response contains files only in a flat list. Directory entries are traversed but not included in the output. Recursive globs can match a basename (*.psd), a path relative to the requested directory (**/frames/*.json), or a full path.

Versioned Reads

Read a file as it was at a specific snapshot or version:

# Read file at a named snapshot
curl "http://localhost:6830/files/data/report.pdf?snapshot=v1.0" \
  -H "Authorization: Bearer $TOKEN"

# Read file at a specific version hash
curl "http://localhost:6830/files/data/report.pdf?version=a1b2c3..." \
  -H "Authorization: Bearer $TOKEN"

If both snapshot and version are provided, snapshot takes precedence. Returns 404 if the file did not exist at that version.

Error Responses

POST /files/fetch

Fetch multiple files or multiple file ranges in one request. There are two request shapes:

• paths: whole-file batch fetch, returned as a JSON object keyed by canonical path.
• items: range batch fetch, returned as an ordered items array.

Provide either paths or items, not both. Directories, missing files, system paths, and unreadable paths return 404. File bytes are encoded into JSON strings with UTF-8 lossy conversion, so binary data may contain replacement characters.

Request

• Headers:
  • Authorization: Bearer <token> (required)
  • Content-Type: application/json (required)
• Body:

{
  "paths": [
    "/data/a.txt",
    "/data/b.json"
  ],
  "max_bytes": 1048576
}

Response

Status: 200 OK

{
  "/data/a.txt": {
    "path": "/data/a.txt",
    "name": "a.txt",
    "size": 324534,
    "created_at": 1235413451345,
    "updated_at": 134513453145,
    "content_type": "text/plain",
    "content": "..."
  },
  "/data/b.json": {
    "path": "/data/b.json",
    "name": "b.json",
    "size": 2048,
    "created_at": 1235413451345,
    "updated_at": 134513453145,
    "content_type": "application/json",
    "content": "{\"ok\":true}"
  }
}

Range Fetch Mode

Use items to fetch line, character, byte, or JSON Pointer ranges. This is the preferred follow-up for search hit locators.

{
  "items": [
    {
      "id": "hit-1",
      "path": "/data/a.txt",
      "if_content_hash": "b3c1...",
      "range": { "mode": "lines", "start": 10, "end": 14 }
    },
    {
      "id": "hit-2",
      "path": "/data/b.json",
      "if_updated_at": 1775968398000,
      "range": { "mode": "json_pointer", "pointer": "/messages/0/content" },
      "max_bytes": 65536
    }
  ],
  "continue_on_error": true
}

Range-mode response:

{
  "items": [
    {
      "id": "hit-1",
      "status": "ok",
      "path": "/data/a.txt",
      "name": "a.txt",
      "size": 324534,
      "created_at": 1235413451345,
      "updated_at": 134513453145,
      "content_hash": "b3c1...",
      "content_type": "text/plain",
      "range": { "mode": "lines", "start": 10, "end": 14, "pointer": null },
      "source_size": 324534,
      "content": "...",
      "truncated": false
    }
  ],
  "has_errors": false
}

If continue_on_error is true, per-item errors have status values such as not_found, stale, invalid, too_large, or error.

Limits

Example

curl -X POST http://localhost:6830/files/fetch \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"paths":["/data/a.txt","/data/b.json"]}'

Error Responses

DELETE /files/

Delete a file or empty directory at the given path. Creates a DeletionRecord and removes the entry from its parent directory listing. The handler tries file deletion first, then directory deletion, then returns 404. Non-empty directories return 400 with “Directory is not empty (N children)”.

Request

• Headers:
  • Authorization: Bearer <token> (required)

Response

Status: 200 OK

{
  "deleted": true,
  "path": "/data/report.pdf"
}

Deleted File Access

Deleted files are invisible to users who lack the d (delete) permission on the containing directory:

• GET /files/deleted?path=/dir returns an empty list
• GET /files/{path} returns 404 for deleted files
• HEAD /files/{path} returns 404 for deleted files
• Reading a deleted file via ?version= also returns 404

Root users always have full access to deleted files.

Side Effects

• Triggers entries_deleted events on the event bus.
• Updates index entries for the deleted file.

Example

curl -X DELETE http://localhost:6830/files/data/report.pdf \
  -H "Authorization: Bearer $TOKEN"

Error Responses

PATCH /files/

PATCH is overloaded by Content-Type:

• application/json → rename the file or symlink (documented below).
• application/merge-patch+json → server-side JSON merge into the stored document. See the dedicated JSON Merge Patch reference for that mode.

The content-type alone discriminates — a merge-patch body that happens to contain a "to" key will be merged into the file, not used as a rename instruction.

Rename mode

Rename or move a file or symlink to a new path. This is a metadata-only operation – no data is copied in the content-addressed store. The file’s content hash remains the same; only the path mapping changes.

Request

• Headers:
  • Authorization: Bearer <token> (required)
  • Content-Type: application/json (required — application/merge-patch+json routes to the merge endpoint instead)
• Body:

{
  "to": "/new/path/report.pdf"
}

Response

Status: 200 OK

{
  "from": "/data/report.pdf",
  "to": "/archive/report.pdf"
}

Side Effects

• Triggers entries_created and entries_deleted events on the event bus.
• If the path is a symlink, the symlink itself is moved (not the target).
• If a file already exists at the destination path, the operation fails.

Example

# Move a file
curl -X PATCH http://localhost:6830/files/data/report.pdf \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"to": "/archive/report.pdf"}'

# Rename a symlink
curl -X PATCH http://localhost:6830/files/latest-logo \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"to": "/current-logo"}'

Error Responses

Symlinks

AeorDB supports soft symlinks — entries that point to another path. Symlinks are transparent by default: reading a symlink path returns the target’s content.

PUT /links/

Create or update a symlink.

Request Body:

{
  "target": "/assets/logo.psd"
}

Response: 201 Created

{
  "path": "/latest-logo",
  "target": "/assets/logo.psd",
  "entry_type": 8,
  "created_at": 1775968398000,
  "updated_at": 1775968398000
}

The target path does not need to exist at creation time (dangling symlinks are allowed).

Reading Symlinks

By default, GET /files/{path} follows symlinks transparently:

# Returns the content of /assets/logo.psd
curl http://localhost:6830/files/latest-logo \
  -H "Authorization: Bearer $TOKEN"

To inspect the symlink itself without following it, use ?nofollow=true:

curl "http://localhost:6830/files/latest-logo?nofollow=true" \
  -H "Authorization: Bearer $TOKEN"

Returns the symlink metadata as JSON instead of the target’s content.

Symlink Resolution

Symlinks can point to other symlinks — chains are followed recursively. AeorDB detects cycles and enforces a maximum resolution depth of 32 hops.

HEAD on Symlinks

HEAD /files/{path} returns symlink metadata as headers:

X-AeorDB-Entry-Type: symlink
X-AeorDB-Symlink-Target: /assets/logo.psd
X-AeorDB-Path: /latest-logo
X-AeorDB-Created-At: 1775968398000
X-AeorDB-Updated-At: 1775968398000

Deleting Symlinks

DELETE /files/{path} on a symlink deletes the symlink itself, not the target:

curl -X DELETE http://localhost:6830/files/latest-logo \
  -H "Authorization: Bearer $TOKEN"

{
  "deleted": true,
  "path": "latest-logo",
  "entry_type": "symlink"
}

Symlinks in Directory Listings

Symlinks appear in directory listings with entry_type: 8 and a target field:

{
  "path": "/data/latest",
  "name": "latest",
  "entry_type": 8,
  "hash": "c3d5e6...",
  "target": "/data/report.pdf",
  "size": 0,
  "created_at": 1775968500000,
  "updated_at": 1775968500000,
  "content_type": null
}

Symlink Versioning

Symlinks are versioned like files. Snapshots capture the symlink’s target path at that point in time. Restoring a snapshot restores the link, not the resolved content.

Sharing

AeorDB supports sharing files and directories with specific users and groups. Shares are stored as .permissions files in the filesystem — the sharing API is a convenience layer on top of the existing permission system.

POST /files/share

Share one or more paths with users and/or groups. For files, the permission is scoped to just that file using a path_pattern. For directories, the permission applies to everything inside.

Auth: Root only.

Request Body:

{
  "paths": ["/photos/sunset.jpg", "/photos/beach.jpg"],
  "users": ["6f94eecf-b136-47b4-9b47-c20f781f1f5b"],
  "groups": ["design-team"],
  "permissions": "crudl..."
}

At least one of users or groups must be non-empty.

Response: 200 OK

{
  "shared": 2,
  "paths": ["/photos/sunset.jpg", "/photos/beach.jpg"]
}

How it works:

• For each file path, a PermissionLink is created on the parent directory’s .permissions file with a path_pattern matching the filename — so the permission only applies to that specific file.
• For each directory path, the link is created with no path_pattern — it applies to everything in the directory.
• If a link for the same group and pattern already exists, it is updated (not duplicated).

Example:

curl -X POST http://localhost:6830/files/share \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "paths": ["/photos/sunset.jpg"],
    "users": ["6f94eecf-b136-47b4-9b47-c20f781f1f5b"],
    "permissions": "cr..l..."
  }'

GET /files/shares

List active shares for a path.

Query Parameters:

Response: 200 OK

{
  "path": "/photos/sunset.jpg",
  "shares": [
    {
      "group": "user:6f94eecf-...",
      "username": "alice",
      "allow": "cr..l...",
      "deny": "........",
      "path_pattern": "sunset.jpg"
    },
    {
      "group": "design-team",
      "allow": "crudl...",
      "deny": "........",
      "path_pattern": null
    }
  ]
}

For user:{uuid} groups, the username field is resolved from the user store. For named groups, username is null.

When querying a specific file, only shares with a matching path_pattern (or directory-wide shares with no pattern) are returned.

Example:

curl "http://localhost:6830/files/shares?path=/photos/sunset.jpg" \
  -H "Authorization: Bearer $TOKEN"

DELETE /files/shares

Revoke a specific share by removing the matching PermissionLink.

Auth: Root only.

Request Body:

{
  "path": "/photos/sunset.jpg",
  "group": "user:6f94eecf-...",
  "path_pattern": "sunset.jpg"
}

Response: 200 OK

{
  "revoked": true,
  "group": "user:6f94eecf-..."
}

Example:

curl -X DELETE http://localhost:6830/files/shares \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "path": "/photos/sunset.jpg",
    "group": "user:6f94eecf-...",
    "path_pattern": "sunset.jpg"
  }'

Error Responses (Sharing)

Link Sharing

Shareable URLs backed by scoped API keys. A share link bundles a scoped API key with a JWT token into a URL that opens the portal file browser, scoped to the shared path.

POST /files/share-link

Create a share link for one or more paths. This creates a scoped API key with user_id: null and generates a JWT token embedded in the URL.

Auth: Root only.

Request Body:

{
  "paths": ["/photos/vacation"],
  "permissions": "cr..l...",
  "expires_in_days": 30,
  "base_url": "https://files.example.com"
}

Response: 201 Created

{
  "url": "https://files.example.com/share?token=eyJ...",
  "token": "eyJhbGciOiJIUzI1NiIs...",
  "key_id": "a1b2c3d4-...",
  "permissions": "cr..l...",
  "expires_at": 1778560398000,
  "paths": ["/photos/vacation"]
}

The URL opens the portal file browser scoped to the shared path. Ancestor directories are navigable (read+list only) – only path components leading to the target are visible.

Example:

curl -X POST http://localhost:6830/files/share-link \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "paths": ["/photos/vacation"],
    "permissions": "-r--l---",
    "expires_in_days": 7,
    "base_url": "https://files.example.com"
  }'

GET /files/share-links

List active share links for a path.

Auth: Root only.

Query Parameters:

Response: 200 OK

{
  "path": "/photos/vacation",
  "links": [
    {
      "key_id": "a1b2c3d4-...",
      "label": "share-link:/photos/vacation",
      "permissions": "-r--l---",
      "expires_at": 1778560398000,
      "created_at": 1775968398000
    }
  ]
}

Example:

curl "http://localhost:6830/files/share-links?path=/photos/vacation" \
  -H "Authorization: Bearer $TOKEN"

DELETE /files/share-links/

Revoke a share link. This deletes the underlying scoped API key and invalidates the JWT.

Auth: Root only.

Response: 200 OK

{
  "revoked": true,
  "key_id": "a1b2c3d4-..."
}

Example:

curl -X DELETE http://localhost:6830/files/share-links/a1b2c3d4-... \
  -H "Authorization: Bearer $TOKEN"

Error Responses (Link Sharing)

Shared With Me

GET /files/shared-with-me

List paths that have been shared with the current user. AeorDB resolves the caller’s group memberships against the cached grants index, returning paths where those groups have access. This is used by the portal to discover entry points when the user has no root-level permissions.

Auth: Requires a user-bound token. Share tokens receive 403. Root users receive an empty list (they already have access to everything).

Response: 200 OK

{
  "paths": [
    {
      "path": "/photos/vacation/",
      "permissions": ".r..l...",
      "path_pattern": null
    }
  ]
}

Each entry includes the shared path, the permissions string granted, and an optional path_pattern (non-null when the share targets a specific file within a directory).

Example:

curl http://localhost:6830/files/shared-with-me \
  -H "Authorization: Bearer $TOKEN"

Error Responses:

HEAD /files/

Check whether a path exists and retrieve its metadata as response headers, without downloading the body. Works for both files and directories.

Request

• Headers:
  • Authorization: Bearer <token> (required)

Response

Status: 200 OK (empty body)

Headers:

Example

curl -I http://localhost:6830/files/data/report.pdf \
  -H "Authorization: Bearer $TOKEN"

HTTP/1.1 200 OK
X-AeorDB-Entry-Type: file
X-AeorDB-Path: /data/report.pdf
X-AeorDB-Size: 245678
X-AeorDB-Created-At: 1775968398000
X-AeorDB-Updated-At: 1775968398000
Content-Type: application/pdf

Error Responses

POST /files/copy

Copy files or directories to a new location. Uses content-addressed deduplication — no data is physically duplicated.

Request

• Headers:
  • Authorization: Bearer <token> (required)
  • Content-Type: application/json (required)
• Body:

{
  "paths": ["/src/file1.txt", "/src/file2.txt"],
  "destination": "/dst/"
}

Response

Status: 200 OK

{
  "copied": ["/dst/file1.txt", "/dst/file2.txt"]
}

For directories in the paths list, all contents are recursively copied. If any individual copy fails, the error is reported in an errors array but other copies continue.

Example

curl -X POST http://localhost:6830/files/copy \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "paths": ["/src/file1.txt", "/src/file2.txt"],
    "destination": "/dst/"
  }'

Global Search

POST /files/search

Search across all indexed directories in the database. Unlike POST /files/query which targets a single directory, global search fans out across every directory that has indexes.

Request body:

{
  "query": "hero-banner",
  "limit": 20,
  "offset": 0
}

The query field performs a broad fuzzy search across default-indexed text metadata fields that have fuzzy-capable indexes, such as @filename and @path.

For more targeted searches, use a structured where clause:

{
  "where": {
    "field": "@filename",
    "op": "similar",
    "value": "hero",
    "threshold": 0.3
  },
  "limit": 20
}

To find every path under a root that contains a file with a known whole-file content hash, query @hash with eq:

{
  "path": "/some/root/",
  "where": {
    "field": "@hash",
    "op": "eq",
    "value": "b3c1..."
  },
  "limit": 1000
}

@hash is the raw whole-file hash (blake3(file bytes)) stored in the FileRecord at write time. It is not the first chunk hash. New databases bootstrap an exact string index for @hash; trigram hash indexes are not created by default because exact hash lookup is the normal use case and trigram hash indexes consume substantial memory on large databases. Existing databases may need their /.aeordb-config/indexes.json updated and reindexed before @hash eq uses only the exact index; reindexing also retires old hash index strategies that are no longer present in the config. Legacy FileRecord v0 entries written before this field existed must be migrated before they can participate in exact @hash lookup; trigger a forced reindex with POST /system/tasks/reindex and "force": true to backfill them.

For agent workflows, /files/search and /files/query also support opt-in hit locators with include_matches: true. Locator responses include snippets, typed ranges, content_hash identity anchors, and fetch hints that can be used with /files/fetch range mode. See Querying for the full schema.

Response:

{
  "results": [
    {
      "path": "/assets/hero-banner.png",
      "score": 0.85,
      "directory": "/assets",
      "index_field": "@filename"
    }
  ],
  "total": 1,
  "directories_searched": 42
}

JSON Merge Patch

PATCH /files/{path} with Content-Type: application/merge-patch+json partially updates a JSON document server-side — the engine reads the stored file, merges the patch in, and writes back atomically. Clients send only the changed fields. No more read-modify-write round trips, no more lost-update races between the read and the put.

This document is the implementation guide for client SDKs and applications adopting the endpoint. For the underlying spec, see RFC 7396.

Note on the dispatcher. PATCH /files/{path} is overloaded by Content-Type: application/merge-patch+json lands here; application/json is the existing rename endpoint. The content-type alone distinguishes them — a merge-patch body that happens to contain a "to" key will be merged, not renamed.

Why this exists

Before merge-patch, updating a single field in a stored JSON document required three round trips:

1. GET /files/record.json — pull the current state.
2. Parse it, mutate locally.
3. PUT /files/record.json — push the full document back.

This is a problem because:

• It’s a lost-update race. If another writer modifies the file between your GET and PUT, your PUT silently overwrites their change.
• It’s bandwidth-expensive. A 5 MB record requires 10 MB of transfer to change a 12-byte field.
• It’s client-side complex. Every client has to implement parse/mutate/serialize correctly per schema.

Server-side merge fixes all three: only the patch (typically tiny) crosses the wire, and the engine’s write path is single-threaded so two concurrent patches to disjoint keys compose correctly.

Wire protocol

PATCH /files/{path}
Authorization: Bearer <token>
Content-Type:  application/merge-patch+json

<JSON patch body>

Query parameters:

Status codes:

Successful responses have the same shape as PUT /files/{path}:

{
  "path":         "/record.json",
  "content_type": "application/json",
  "size":         85,
  "created_at":   1779470049858,
  "updated_at":   1779470049903,
  "hash":         "bc14a77290fb594388efe43fbb4a0b31411cea40b6a725b5fcd3aa782a3cd4e4"
}

Merge semantics

The merge is recursive JSON object merge, per RFC 7396:

1. Patch is an object → each key is merged into the target:
   • null value → deletes the key from the target.
   • Object value → recursive merge into the target’s value at that key.
   • Anything else (scalar, array) → replaces the target’s value at that key.
2. Patch is anything else (top-level scalar, array, or null) → the patch replaces the entire stored document.

Arrays are replaced, not concatenated

This is a frequent surprise. RFC 7396 has no merge concept for arrays — they are always treated as opaque values.

target: {"tags": ["a", "b", "c"]}
patch:  {"tags": ["d"]}
result: {"tags": ["d"]}        // NOT ["a", "b", "c", "d"]

If you want to append to an array, you have to read it, append client-side, and PATCH the new array, or use a separate operation.

null is delete, not “set to null”

target: {"name": "Alice", "email": "a@x"}
patch:  {"email": null}
result: {"name": "Alice"}

If you genuinely need to store a JSON null for a field, you can’t do it with merge-patch. Use PUT with the full document instead.

Missing file → 201 Created

GET /files/new.json    → 404
PATCH /files/new.json with {"a": 1}    → 201 Created
GET /files/new.json    → {"a": 1}

The missing file is treated as {} for merge purposes.

Depth bound

By default the merge recurses to arbitrary depth. The ?depth=N query parameter bounds that recursion. The sign is meaningful:

The signed distinction only fires for object values at the boundary. Scalars and null always behave the same regardless of sign — null deletes at the current merge level, scalars insert/replace.

Why positive vs negative?

The use cases are genuinely different.

Positive depth — “I want to update top-level fields and atomically swap a known subtree”:

target: {"user": {"name": "Alice", "prefs": {"theme": "dark"}}, "session": "abc"}
patch:  {"user": {"prefs": {"theme": "light"}}}
?depth=1
result: {"user": {"prefs": {"theme": "light"}}, "session": "abc"}
                  ▲
              user is REPLACED — name is lost.
              "session" is preserved (not in the patch).

Negative depth — “I want to update top-level fields but leave nested state alone, even if the caller accidentally includes deeper data”:

target: {"user": {"name": "Alice", "prefs": {"theme": "dark"}}, "scalar": "old"}
patch:  {"user": {"prefs": {"theme": "light"}}, "scalar": "new"}
?depth=-1
result: {"user": {"name": "Alice", "prefs": {"theme": "dark"}}, "scalar": "new"}
                  ▲
              user is PRESERVED — patch's user object is ignored entirely.
              "scalar" is updated (scalars always merge regardless of sign).

Negative depth is a defensive primitive: it lets a service that knows it should only update shallow fields enforce that on the server, so a buggy or malicious client can’t accidentally rewrite a nested subtree.

Counting levels

depth=N means N levels of merge actually happen. The outer merge of the patch into the target counts as 1.

• depth=1: only top-level keys merge. Their values replace (positive) or are preserved (negative).
• depth=2: top-level merges, plus one recursion into object values. Level-3 objects replace/preserve.
• depth=3: three levels of merging happen. Level-4 objects replace/preserve.
• depth=0: no levels of merging — the patch is written as the new document.

Examples

All examples assume BASE=http://localhost:6830 and JWT=$(...) set to a valid bearer token.

Update a single field

# Stored: {"name": "Alice", "age": 30}
curl -X PATCH "$BASE/files/user.json" \
  -H "Authorization: Bearer $JWT" \
  -H "Content-Type: application/merge-patch+json" \
  -d '{"age": 31}'
# Result: {"name": "Alice", "age": 31}

Delete a field

# Stored: {"name": "Alice", "email": "a@x", "phone": "555-1212"}
curl -X PATCH "$BASE/files/user.json" \
  -H "Authorization: Bearer $JWT" \
  -H "Content-Type: application/merge-patch+json" \
  -d '{"phone": null}'
# Result: {"name": "Alice", "email": "a@x"}

Update a nested field without disturbing siblings

# Stored: {"user": {"name": "Alice", "prefs": {"theme": "dark", "lang": "en"}}}
curl -X PATCH "$BASE/files/user.json" \
  -H "Authorization: Bearer $JWT" \
  -H "Content-Type: application/merge-patch+json" \
  -d '{"user": {"prefs": {"theme": "light"}}}'
# Result: {"user": {"name": "Alice", "prefs": {"theme": "light", "lang": "en"}}}

Swap a subtree atomically (?depth=+1)

When you want to replace prefs wholesale rather than merging:

# Stored: {"user": {"name": "Alice", "prefs": {"theme": "dark", "lang": "en"}}}
curl -X PATCH "$BASE/files/user.json?depth=1" \
  -H "Authorization: Bearer $JWT" \
  -H "Content-Type: application/merge-patch+json" \
  -d '{"user": {"prefs": {"theme": "light"}}}'
# Result: {"user": {"prefs": {"theme": "light"}}}
#                 ▲ user.name is gone (user object was replaced wholesale)

Protect nested state from accidental writes (?depth=-1)

Useful in a service that exposes shallow user-profile updates but never wants its API to touch nested session/credential blobs even if a caller includes them by mistake:

# Stored: {"profile": {"name": "Alice"}, "credentials": {"token": "secret"}}
curl -X PATCH "$BASE/files/user.json?depth=-1" \
  -H "Authorization: Bearer $JWT" \
  -H "Content-Type: application/merge-patch+json" \
  -d '{"profile": {"name": "Bob"}, "credentials": {"token": "compromised"}}'
# Result: {"profile": {"name": "Alice"}, "credentials": {"token": "secret"}}
#         ▲ Both nested objects are preserved — the patch's depths are ignored.

Note that scalars at the top level still update — ?depth=-1 is “shallow merge that doesn’t touch the depths,” not “noop.”

Create a new document via PATCH

# /new.json does not exist
curl -X PATCH "$BASE/files/new.json" \
  -H "Authorization: Bearer $JWT" \
  -H "Content-Type: application/merge-patch+json" \
  -d '{"hello": "world"}'
# Status: 201 Created
# Stored: {"hello": "world"}

Wholesale replace (?depth=0 — equivalent to PUT)

curl -X PATCH "$BASE/files/doc.json?depth=0" \
  -H "Authorization: Bearer $JWT" \
  -H "Content-Type: application/merge-patch+json" \
  -d '{"replaced": true}'
# Stored: {"replaced": true}  (whatever was there before is gone)

In practice you should just use PUT for this. ?depth=0 exists for callers that compute the depth dynamically and want the boundary to degrade cleanly.

Concurrency

Safe by default

The engine’s write path is single-threaded — writes are serialized through the write buffer and WAL. Two concurrent merge-patches to the same file are applied one at a time by the server. Each merge reads the latest persisted state (including the other writer’s prior merge), applies its own patch on top, and writes.

This means:

• Disjoint key updates compose correctly. Writer A’s {"a": 1} and Writer B’s {"b": 2} arriving simultaneously yield {"a": 1, "b": 2} regardless of which runs first.
• Overlapping key updates are last-writer-wins. This is inherent to merge semantics — there’s no per-key clock or vector to detect concurrency. If two writers both update "theme", the second to be applied wins.

Compare-and-swap is not currently supported

The endpoint does not yet honor If-Match: <content-hash>. If you have a strict ordering requirement (e.g., “fail if anyone modified this file since I last read it”), do a GET first, capture the hash, then PUT with the new document and compare. We may add an opt-in CAS header in a future revision; track [the followup issue] if you need it.

When you should still GET first

• You need to make a decision based on the current state (“only set published to true if draft_count >= 1”). The server can’t conditionally merge, only blindly merge.
• You need to read the result of the merge — the response body is metadata only, not the merged content. Do a GET after.

Limits

These caps apply because the engine has to hold both the existing document and the patched result in memory simultaneously. If you have records larger than 10 MB, split them into multiple files and merge each independently.

Migration from read-modify-write

If you have existing client code like this:

// OLD: 3 round trips, race-prone
const resp = await fetch(`${base}/files/user.json`, { headers });
const doc = await resp.json();
doc.prefs.theme = 'light';
await fetch(`${base}/files/user.json`, {
  method: 'PUT',
  headers: { ...headers, 'Content-Type': 'application/json' },
  body: JSON.stringify(doc),
});

The migration is mechanical:

// NEW: 1 round trip, race-safe under server-side merge
await fetch(`${base}/files/user.json`, {
  method: 'PATCH',
  headers: { ...headers, 'Content-Type': 'application/merge-patch+json' },
  body: JSON.stringify({ prefs: { theme: 'light' } }),
});

Pick a depth mode based on what your code path expects:

Implementation checklist for SDKs

A minimal client wrapper should:

• Expose mergePatch(path, patch, { depth }) returning the response metadata.
• Set Content-Type: application/merge-patch+json (no other variant — the dispatcher uses this exact string to discriminate from rename).
• Encode depth as a signed query-string integer when provided. depth=0 is legal; depth=-1 is legal; omit the query param entirely for unbounded.
• Handle 201 and 200 as success. Both have the same body shape.
• Surface 413 and 415 distinctly from generic 4xx; they typically indicate a configuration bug (wrong content-type on the stored file, oversize doc) rather than a transient failure.
• Do not retry on 4xx. Merge patches are not idempotent in general — retrying a delete-via-null after a successful first call would re-apply against the now-changed state.
• Document explicitly that arrays replace wholesale and null deletes, so users don’t get surprised.

Query API

The query engine supports indexed field queries with boolean combinators, pagination, sorting, aggregations, projections, and an explain mode.

Endpoint Summary

POST /files/query

Execute a query against indexed fields within a directory path.

Request Body

{
  "path": "/users",
  "where": {
    "field": "age",
    "op": "gt",
    "value": 21
  },
  "limit": 20,
  "offset": 0,
  "order_by": [{"field": "name", "direction": "asc"}],
  "after": null,
  "before": null,
  "include_total": true,
  "select": ["@path", "@score", "name"],
  "include_matches": false,
  "max_matches_per_result": 5,
  "snippet_chars": 160,
  "match_context_lines": 2,
  "max_locator_scan_bytes": 268435456,
  "explain": false
}

Hit locators are opt-in. They are generated only for the current page after authorization filtering, not during index lookup.

Query Operators

Each field query is an object with field, op, and value:

{"field": "age", "op": "gt", "value": 21}

Comparison Operators

Text Search Operators

These operators require the appropriate index type to be configured.

Fuzzy Operator Options

The fuzzy operator supports additional parameters:

{
  "field": "name",
  "op": "fuzzy",
  "value": "alice",
  "fuzziness": "auto",
  "algorithm": "damerau_levenshtein"
}

Similar Operator Options

{
  "field": "name",
  "op": "similar",
  "value": "alice",
  "threshold": 0.3
}

Boolean Combinators

Combine multiple conditions using and, or, and not:

AND

All conditions must match:

{
  "where": {
    "and": [
      {"field": "age", "op": "gt", "value": 21},
      {"field": "status", "op": "eq", "value": "active"}
    ]
  }
}

OR

At least one condition must match:

{
  "where": {
    "or": [
      {"field": "status", "op": "eq", "value": "active"},
      {"field": "status", "op": "eq", "value": "pending"}
    ]
  }
}

NOT

Invert a condition:

{
  "where": {
    "not": {"field": "status", "op": "eq", "value": "deleted"}
  }
}

Nested Boolean Logic

Combinators can be nested up to 32 levels deep. Queries exceeding this depth are rejected with a 400 error. Additionally, a single query can return at most 100,000 results – requests that would exceed this ceiling are truncated.

{
  "where": {
    "and": [
      {"field": "age", "op": "gt", "value": 21},
      {
        "or": [
          {"field": "role", "op": "eq", "value": "admin"},
          {"field": "role", "op": "eq", "value": "moderator"}
        ]
      }
    ]
  }
}

Legacy Array Format

An array at the top level is sugar for AND:

{
  "where": [
    {"field": "age", "op": "gt", "value": 21},
    {"field": "status", "op": "eq", "value": "active"}
  ]
}

Response Format

Standard Query Response

{
  "items": [
    {
      "path": "/users/alice.json",
      "size": 256,
      "content_type": "application/json",
      "created_at": 1775968398000,
      "updated_at": 1775968398000,
      "score": 1.0,
      "matched_by": ["name"]
    }
  ],
  "has_more": true,
  "total_count": 150,
  "next_cursor": "eyJwYXRoIjoiL3VzZXJzL2JvYi5qc29uIn0=",
  "prev_cursor": "eyJwYXRoIjoiL3VzZXJzL2Fhcm9uLmpzb24ifQ==",
  "meta": {
    "reindexing": 0.67,
    "reindexing_eta": 1775968398803,
    "reindexing_indexed": 670,
    "reindexing_total": 1000,
    "reindexing_stale_since": 1775968300000
  }
}

Result Fields

Each result object contains:

Hit Locators

When include_matches is true, /files/query and /files/search add bounded hit locators to each returned result. This is designed for agents that need to search first and then fetch only relevant ranges.

For JSON files, matching indexed fields are reported as field-value locators with JSON Pointer fetch hints:

{
  "path": "/users/alice.json",
  "content_hash": "b3c1...",
  "matches": [
    {
      "id": "m_0001",
      "query": "Alice",
      "matched_text": "Alice",
      "field": "name",
      "operator": "eq",
      "source": {
        "type": "field-value",
        "field": "name",
        "json_pointer": "/name",
        "value_type": "string"
      },
      "range": {
        "char": { "start": 0, "end": 5, "unit": "unicode-scalar", "basis": "field-value" }
      },
      "fetch": {
        "json_pointer": "/name",
        "preferred": "json_pointer"
      },
      "snippet": {
        "text": "Alice",
        "highlight": [{ "start": 0, "end": 5, "unit": "unicode-scalar" }],
        "truncated_before": false,
        "truncated_after": false
      },
      "confidence": "exact",
      "scan_status": "complete"
    }
  ],
  "matches_truncated": false,
  "locator_status": "complete"
}

Metadata matches such as @filename use source.type = "metadata". Plain-text or unstructured UTF-8 file matches use source.type = "stored-file" and include byte, character, line, and column ranges plus byte_range and line_range fetch hints.

Use POST /files/fetch range mode to fetch the returned ranges. Pass if_content_hash or if_updated_at from the search result to avoid fetching a stale range after the file changes.

Sorting

Sort results by one or more fields:

{
  "order_by": [
    {"field": "name", "direction": "asc"},
    {"field": "created_at", "direction": "desc"}
  ]
}

Pagination

Offset-Based

{
  "limit": 20,
  "offset": 40
}

Cursor-Based

Use after or before with cursor values from a previous response:

{
  "limit": 20,
  "after": "eyJwYXRoIjoiL3VzZXJzL2JvYi5qc29uIn0="
}

Projection (select)

Return only specific fields in each result. Use @-prefixed names for built-in metadata fields:

{
  "select": ["@path", "@score", "name", "email"]
}

Envelope fields (has_more, next_cursor, total_count, meta) are never stripped by projection.

Aggregations

Run aggregate computations instead of returning individual results.

Request

{
  "path": "/orders",
  "where": {"field": "status", "op": "eq", "value": "complete"},
  "aggregate": {
    "count": true,
    "sum": ["total", "tax"],
    "avg": ["total"],
    "min": ["total"],
    "max": ["total"],
    "group_by": ["status"]
  }
}

Response

The response shape depends on whether group_by is used. Aggregation results are returned as a JSON object.

Explain Mode

Inspect the query execution plan without running the full query. Useful for debugging index usage and performance.

{
  "path": "/users",
  "where": {"field": "age", "op": "gt", "value": 21},
  "explain": "plan"
}

Virtual Fields

Virtual fields let you query file metadata directly. Prefix a field name with @ to query against built-in file attributes instead of indexed document fields.

AeorDB uses virtual-field indexes when they are configured. New databases bootstrap default virtual-field indexes at /.aeordb-config/indexes.json; older databases can add the same config and run a forced reindex to backfill them. If no index exists for a supported virtual field, AeorDB falls back to scanning FileRecord metadata for compatibility.

Available Virtual Fields

Supported Operators

String virtual fields (@path, @filename, @file_name, @extension, @content_type, @hash) support:

eq, contains, in, gt, lt, similar (trigram), fuzzy (edit distance), phonetic (soundex/metaphone), match (fused multi-strategy)

Numeric virtual fields (@size, @created_at, @updated_at) support:

eq, gt, lt, between, in

How Virtual Fields Work

Virtual fields are derived from the FileRecord header and payload. Query execution is index-first:

• Default indexes. New databases index every built-in virtual field for exact lookup by default. @filename also gets trigram and phonetic indexes for contains, similar, fuzzy, phonetic, and match.
• Ancestor glob indexes. If a query is scoped below a glob-indexed directory, AeorDB can use the ancestor index and filter candidates back down to the requested path.
• Scan fallback. If a database does not have a matching virtual-field index, AeorDB scans FileRecords under the query path so legacy queries continue to work. This fallback is O(n) over files under the path.
• Combinable with indexed fields. You can mix virtual fields and indexed fields in the same where clause using boolean combinators (and, or, not).
• Canonical aliases. @file_name is accepted as an alias but is indexed and queried as @filename.

Virtual Field Examples

Find all PDFs by extension:

curl -X POST http://localhost:6830/files/query \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "path": "/",
    "where": {"field": "@extension", "op": "eq", "value": "pdf"}
  }'

Fuzzy filename search (handles typos):

curl -X POST http://localhost:6830/files/query \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "path": "/docs",
    "where": {"field": "@filename", "op": "similar", "value": "report", "threshold": 0.3}
  }'

Find large files over 10 MB:

curl -X POST http://localhost:6830/files/query \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "path": "/",
    "where": {"field": "@size", "op": "gt", "value": 10485760}
  }'

Find images by content type:

curl -X POST http://localhost:6830/files/query \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "path": "/",
    "where": {"field": "@content_type", "op": "contains", "value": "image/"}
  }'

Combine multiple virtual fields – large PDFs:

curl -X POST http://localhost:6830/files/query \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "path": "/",
    "where": {
      "and": [
        {"field": "@extension", "op": "eq", "value": "pdf"},
        {"field": "@size", "op": "gt", "value": 1048576}
      ]
    }
  }'

Global Search

Search across all indexed directories in the database.

POST /files/search

At least one of query or where is required.

Broad Search

Discovers all directories with fuzzy-capable indexes (trigram, phonetic, soundex, dmetaphone) and searches the term against every matching field:

curl -X POST http://localhost:6830/files/search \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query": "alice", "limit": 20}'

Structured Search

Same where clause syntax as /files/query, but searches across all directories that have the requested field indexed:

curl -X POST http://localhost:6830/files/search \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"where": {"field": "@size", "op": "gt", "value": 1048576}}'

Combined

Broad search filtered by structured conditions:

curl -X POST http://localhost:6830/files/search \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query": "report", "where": {"field": "@extension", "op": "eq", "value": "pdf"}}'

Response

Same format as /files/query, plus a source field per result indicating which directory’s index matched. The locator fields below are present only when include_matches is true:

{
  "results": [
    {
      "path": "/users/alice.json",
      "score": 0.95,
      "matched_by": ["@filename"],
      "source": "/",
      "size": 256,
      "content_type": "application/json",
      "created_at": 1775968398000,
      "updated_at": 1775968398000,
      "content_hash": "b3c1...",
      "matches": [],
      "matches_truncated": false,
      "locator_status": "complete"
    }
  ],
  "has_more": false,
  "total_count": 1
}

Examples

Simple equality query

curl -X POST http://localhost:6830/files/query \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "path": "/users",
    "where": {"field": "status", "op": "eq", "value": "active"},
    "limit": 10
  }'

Fuzzy name search with pagination

curl -X POST http://localhost:6830/files/query \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "path": "/users",
    "where": {"field": "name", "op": "similar", "value": "alice", "threshold": 0.4},
    "limit": 20,
    "order_by": [{"field": "name", "direction": "asc"}],
    "include_total": true
  }'

Complex boolean query

curl -X POST http://localhost:6830/files/query \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "path": "/products",
    "where": {
      "and": [
        {"field": "price", "op": "between", "value": 10, "value2": 100},
        {
          "or": [
            {"field": "category", "op": "eq", "value": "electronics"},
            {"field": "category", "op": "eq", "value": "books"}
          ]
        },
        {"not": {"field": "status", "op": "eq", "value": "discontinued"}}
      ]
    },
    "order_by": [{"field": "price", "direction": "asc"}],
    "limit": 50
  }'

Aggregation with grouping

curl -X POST http://localhost:6830/files/query \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "path": "/orders",
    "where": {"field": "year", "op": "eq", "value": 2026},
    "aggregate": {
      "count": true,
      "sum": ["total"],
      "avg": ["total"],
      "group_by": ["status"]
    }
  }'

Error Responses

Version API

AeorDB provides Git-like version control through snapshots (named points in time) and forks (divergent branches of the data).

Endpoint Summary

Snapshots

POST /versions/snapshots

Create a named snapshot of the current HEAD.

If lifecycle configuration has "snapshot_writes_enabled": false, this endpoint returns 403 Forbidden. Existing snapshots remain available for listing, reading, restoring, deleting, exporting, and pruning.

Request Body:

{
  "name": "v1.0",
  "metadata": {
    "description": "First stable release",
    "author": "alice"
  }
}

Response: 201 Created

{
  "name": "v1.0",
  "root_hash": "a1b2c3d4e5f6...",
  "created_at": 1775968398000,
  "metadata": {
    "description": "First stable release",
    "author": "alice"
  }
}

Example:

curl -X POST http://localhost:6830/versions/snapshots \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "v1.0", "metadata": {"description": "First stable release"}}'

Error Responses:

GET /versions/snapshots

List all snapshots.

Response: 200 OK

{
  "items": [
    {
      "name": "v1.0",
      "root_hash": "a1b2c3d4e5f6...",
      "created_at": 1775968398000,
      "metadata": {"description": "First stable release"}
    },
    {
      "name": "v2.0",
      "root_hash": "f6e5d4c3b2a1...",
      "created_at": 1775969000000,
      "metadata": {}
    }
  ]
}

Example:

curl http://localhost:6830/versions/snapshots \
  -H "Authorization: Bearer $TOKEN"

POST /versions/restore

Restore a named snapshot, making it the current HEAD. Requires root.

Request Body:

{
  "name": "v1.0"
}

Response: 200 OK

{
  "restored": true,
  "name": "v1.0"
}

Example:

curl -X POST http://localhost:6830/versions/restore \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "v1.0"}'

Error Responses:

DELETE /versions/snapshots/

Delete a named snapshot. Requires root.

Response: 200 OK

{
  "deleted": true,
  "name": "v1.0"
}

Example:

curl -X DELETE http://localhost:6830/versions/snapshots/v1.0 \
  -H "Authorization: Bearer $TOKEN"

Error Responses:

Forks

Forks create a divergent branch of the data, optionally based on a named snapshot.

POST /versions/forks

Create a new fork.

Request Body:

{
  "name": "experiment",
  "base": "v1.0"
}

Response: 201 Created

{
  "name": "experiment",
  "root_hash": "a1b2c3d4e5f6...",
  "created_at": 1775968398000
}

Example:

curl -X POST http://localhost:6830/versions/forks \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "experiment", "base": "v1.0"}'

Error Responses:

GET /versions/forks

List all active forks.

Response: 200 OK

{
  "items": [
    {
      "name": "experiment",
      "root_hash": "a1b2c3d4e5f6...",
      "created_at": 1775968398000
    }
  ]
}

Example:

curl http://localhost:6830/versions/forks \
  -H "Authorization: Bearer $TOKEN"

POST /versions/forks/{name}/promote

Promote a fork’s state to HEAD, making it the active version. Requires root.

Response: 200 OK

{
  "promoted": true,
  "name": "experiment"
}

Example:

curl -X POST http://localhost:6830/versions/forks/experiment/promote \
  -H "Authorization: Bearer $TOKEN"

Error Responses:

DELETE /versions/forks/

Abandon a fork (soft delete). Requires root.

Response: 200 OK

{
  "abandoned": true,
  "name": "experiment"
}

Example:

curl -X DELETE http://localhost:6830/versions/forks/experiment \
  -H "Authorization: Bearer $TOKEN"

Error Responses:

File-Level Version Access

Read, restore, and view history for individual files at specific historical versions.

Reading Files at a Version

Use query parameters on the standard file read endpoint:

# Read a file as it was at a named snapshot
curl "http://localhost:6830/files/assets/logo.psd?snapshot=v1.0" \
  -H "Authorization: Bearer $TOKEN"

# Read a file at a specific version hash
curl "http://localhost:6830/files/assets/logo.psd?version=a1b2c3d4..." \
  -H "Authorization: Bearer $TOKEN"

Returns the file content exactly as it was at that version, with the same headers as a normal file read. If both snapshot and version are provided, snapshot takes precedence.

Error Responses:

GET /versions/history/

View the change history of a single file across all snapshots. Returns entries ordered newest-first, each with a change_type indicating what happened to the file at that snapshot.

Response: 200 OK

{
  "path": "assets/logo.psd",
  "history": [
    {
      "snapshot": "v2.0",
      "timestamp": 1775969000000,
      "change_type": "modified",
      "size": 512000,
      "content_type": "image/vnd.adobe.photoshop",
      "content_hash": "f6e5d4c3..."
    },
    {
      "snapshot": "v1.0",
      "timestamp": 1775968398000,
      "change_type": "added",
      "size": 256000,
      "content_type": "image/vnd.adobe.photoshop",
      "content_hash": "a1b2c3d4..."
    }
  ]
}

Change types:

If the file has never existed in any snapshot, returns 200 with an empty history array.

Example:

curl http://localhost:6830/versions/history/assets/logo.psd \
  -H "Authorization: Bearer $TOKEN"

POST /versions/restore/

Restore a single file from a historical version to the current HEAD. Requires root.

Before restoring, an automatic safety snapshot is created to preserve the current state. If lifecycle configuration has "snapshot_writes_enabled": false, this safety snapshot is skipped and the restore still proceeds. If snapshot writes are enabled but the safety snapshot cannot be created, the restore is rejected.

Request Body:

{
  "snapshot": "v1.0"
}

Or using a version hash:

{
  "version": "a1b2c3d4..."
}

If both are provided, snapshot takes precedence.

Response: 200 OK

{
  "restored": true,
  "path": "assets/logo.psd",
  "from_snapshot": "v1.0",
  "auto_snapshot": "pre-restore-2026-04-14T05-01-01Z",
  "size": 256000
}

The auto_snapshot field contains the name of the safety snapshot created before the restore. You can use this snapshot to recover the pre-restore state if needed.

When snapshot writes are disabled, auto_snapshot is null:

{
  "restored": true,
  "path": "assets/logo.psd",
  "from_snapshot": "v1.0",
  "auto_snapshot": null,
  "size": 256000
}

Example:

curl -X POST http://localhost:6830/versions/restore/assets/logo.psd \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"snapshot": "v1.0"}'

Error Responses:

Pre-Hashed Upload Protocol

AeorDB provides a 4-phase upload protocol for efficient, deduplicated file transfers. Clients split files into chunks, hash them locally, and only upload chunks the server does not already have.

When to use this protocol: Inline uploads via PUT /files/{path} are capped at 100 MB. Files larger than 100 MB must use this chunked upload protocol. It is also beneficial for large batches of files because the dedup check (phase 2) skips chunks already on the server.

Protocol Overview

1. Negotiate – GET /blobs/config to learn the hash algorithm and chunk size.
2. Dedup check – POST /blobs/check with a list of chunk hashes to find which are already stored.
3. Upload – PUT /blobs/chunks/{hash} for each needed chunk.
4. Commit – POST /blobs/commit to atomically assemble chunks into files.

Endpoint Summary

Phase 1: GET /blobs/config

Retrieve the server’s hash algorithm, chunk size, and hash prefix. This endpoint is public (no authentication required).

Response

Status: 200 OK

{
  "hash_algorithm": "blake3",
  "chunk_size": 262144,
  "chunk_hash_prefix": "chunk:"
}

How to Compute Chunk Hashes

The server computes chunk hashes as:

hash = blake3("chunk:" + chunk_bytes)

Clients must use the same formula. The prefix ("chunk:") is prepended to the raw bytes before hashing, not to the hex-encoded hash.

Example

curl http://localhost:6830/blobs/config

Phase 2: POST /blobs/check

Send a list of chunk hashes to determine which ones the server already has (deduplication). Only upload the ones in the needed list.

The request body is a JSON manifest and is capped at 32 MiB. Clients syncing very large files or large batches should split /blobs/check calls before that limit.

Request Body

{
  "hashes": [
    "a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2",
    "f6e5d4c3b2a1f6e5d4c3b2a1f6e5d4c3b2a1f6e5d4c3b2a1f6e5d4c3b2a1f6e5"
  ]
}

Response

Status: 200 OK

{
  "have": [
    "a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2"
  ],
  "needed": [
    "f6e5d4c3b2a1f6e5d4c3b2a1f6e5d4c3b2a1f6e5d4c3b2a1f6e5d4c3b2a1f6e5"
  ]
}

Example

curl -X POST http://localhost:6830/blobs/check \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"hashes": ["a1b2c3...", "f6e5d4..."]}'

Error Responses

Phase 3: PUT /blobs/chunks/

Upload a single chunk. The server verifies the hash matches the content before storing.

Request

• URL parameter: {hash} – hex-encoded blake3 hash of "chunk:" + chunk_bytes
• Headers:
  • Authorization: Bearer <token> (required)
• Body: raw chunk bytes

Hash Verification

The server recomputes the hash from the uploaded bytes:

computed = blake3("chunk:" + body_bytes)

If the computed hash does not match the URL parameter, the upload is rejected.

Response

Status: 201 Created (new chunk stored)

{
  "status": "created",
  "hash": "f6e5d4c3b2a1..."
}

Status: 200 OK (chunk already exists – dedup)

{
  "status": "exists",
  "hash": "f6e5d4c3b2a1..."
}

Compression

Blob staging stores chunk bytes exactly as uploaded. AeorDB does not blindly compress /blobs/chunks/ payloads because large media files are often already compressed, and commit can use raw chunk headers for fast metadata validation.

Example

curl -X PUT http://localhost:6830/blobs/chunks/f6e5d4c3b2a1... \
  -H "Authorization: Bearer $TOKEN" \
  --data-binary @chunk_001.bin

Error Responses

Phase 4: POST /blobs/commit

Atomically commit multiple files from previously uploaded chunks. Each file specifies its path, content type, and the ordered list of chunk hashes that compose it.

The request body is a JSON manifest and is capped at 32 MiB. This is separate from the 10 GB raw chunk upload limit because /blobs/commit carries paths and hash references, not file bytes.

During commit, AeorDB records the raw whole-file content hash (blake3(file bytes)) in the file metadata. That stored value backs @hash searches; it is not derived from the first chunk.

By default, AeorDB streams the ordered stored chunks and computes that hash on the server. Trusted sync clients that already computed the raw file hash can send content_hash and size with each file. For raw stored chunks, AeorDB then validates chunk existence and byte length from KV metadata and can publish the FileRecord without rereading every chunk body.

Request Body

{
  "files": [
    {
      "path": "/data/report.pdf",
      "content_type": "application/pdf",
      "content_hash": "9b01f3e3d06f...",
      "size": 15234212,
      "chunk_hashes": [
        "a1b2c3d4e5f6...",
        "f6e5d4c3b2a1..."
      ]
    },
    {
      "path": "/data/image.png",
      "content_type": "image/png",
      "chunk_hashes": [
        "1234abcd5678..."
      ]
    }
  ]
}

Response

Status: 200 OK

The response contains a summary of the commit operation.

Example

curl -X POST http://localhost:6830/blobs/commit \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "files": [
      {
        "path": "/data/report.pdf",
        "content_type": "application/pdf",
        "content_hash": "9b01f3e3d06f...",
        "size": 15234212,
        "chunk_hashes": ["a1b2c3d4...", "f6e5d4c3..."]
      }
    ]
  }'

Error Responses

Full Upload Workflow

Here is a complete workflow for uploading a file:

# 1. Get server configuration
CONFIG=$(curl -s http://localhost:6830/blobs/config)
CHUNK_SIZE=$(echo $CONFIG | jq -r '.chunk_size')

# 2. Split file into chunks and hash them
# (pseudo-code: split report.pdf into 256KB chunks, hash each with blake3)
# chunk_hashes=["hash1", "hash2", ...]
# content_hash=blake3(report.pdf raw bytes)
# size=report.pdf raw byte length

# 3. Check which chunks are needed
DEDUP=$(curl -s -X POST http://localhost:6830/blobs/check \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"hashes": ["hash1", "hash2"]}')

# 4. Upload only the needed chunks
for hash in $(echo $DEDUP | jq -r '.needed[]'); do
  curl -X PUT "http://localhost:6830/blobs/chunks/$hash" \
    -H "Authorization: Bearer $TOKEN" \
    --data-binary @"chunk_$hash.bin"
done

# 5. Commit the file
curl -X POST http://localhost:6830/blobs/commit \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "files": [{
      "path": "/data/report.pdf",
      "content_type": "application/pdf",
      "content_hash": "whole-file-hash",
      "size": 15234212,
      "chunk_hashes": ["hash1", "hash2"]
    }]
  }'

System Operations

Administrative endpoints for garbage collection, background tasks, cron scheduling, metrics, health checks, backup/restore, and user/group management. Most system endpoints require root access.

Endpoint Summary

Garbage Collection

Background Tasks

Cron Scheduling

Lifecycle Configuration

Backup & Restore

Monitoring

API Key Management

User Management

Group Management

Email Configuration

Garbage Collection

POST /system/gc

Run garbage collection synchronously. Identifies and removes orphaned entries not reachable from the current HEAD.

Query Parameters:

Response: 200 OK

The response contains GC statistics (entries scanned, reclaimed bytes, etc.).

Example:

# Dry run
curl -X POST "http://localhost:6830/system/gc?dry_run=true" \
  -H "Authorization: Bearer $TOKEN"

# Actual GC
curl -X POST http://localhost:6830/system/gc \
  -H "Authorization: Bearer $TOKEN"

Error Responses:

Background Tasks

POST /system/tasks/reindex

Enqueue a reindex task for a directory path. Re-scans all files and rebuilds index entries.

Request Body:

{
  "path": "/data/",
  "force": false,
  "metadata_only": false,
  "index_flush_writes": 262144,
  "index_flush_ms": 30000
}

Response: 200 OK

{
  "id": "task-uuid-here",
  "task_type": "reindex",
  "status": "pending"
}

Example:

curl -X POST http://localhost:6830/system/tasks/reindex \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"path": "/data/", "metadata_only": true}'

POST /system/tasks/gc

Enqueue a background GC task (non-blocking).

Request Body:

{
  "dry_run": false
}

Response: 200 OK

{
  "id": "task-uuid-here",
  "task_type": "gc",
  "status": "pending"
}

Example:

curl -X POST http://localhost:6830/system/tasks/gc \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"dry_run": false}'

GET /system/tasks

List all tasks with their current progress.

Response: 200 OK

{
  "items": [
    {
      "id": "task-uuid-here",
      "task_type": "reindex",
      "status": "running",
      "args": {"path": "/data/"},
      "progress": 0.45,
      "eta_ms": 1775968500000
    }
  ]
}

Each task includes progress (0.0-1.0) and eta_ms (estimated completion timestamp) if available.

Example:

curl http://localhost:6830/system/tasks \
  -H "Authorization: Bearer $TOKEN"

GET /system/tasks/

Get a single task by ID.

Response: 200 OK

{
  "id": "task-uuid-here",
  "task_type": "reindex",
  "status": "running",
  "args": {"path": "/data/"},
  "progress": 0.45,
  "eta_ms": 1775968500000
}

Error Responses:

DELETE /system/tasks/

Cancel a task.

Response: 200 OK

{
  "id": "task-uuid-here",
  "status": "cancelled"
}

Example:

curl -X DELETE http://localhost:6830/system/tasks/task-uuid-here \
  -H "Authorization: Bearer $TOKEN"

Cron Scheduling

Tip: The portal Settings page provides an intuitive UI for scheduling garbage collection. Navigate to Settings → Garbage Collector to configure.

GET /system/cron

List all cron schedules.

Response: 200 OK

{
  "items": [
    {
      "id": "nightly-gc",
      "schedule": "0 2 * * *",
      "task_type": "gc",
      "args": {"dry_run": false},
      "enabled": true
    }
  ]
}

POST /system/cron

Create a new cron schedule.

Request Body:

{
  "id": "nightly-gc",
  "schedule": "0 2 * * *",
  "task_type": "gc",
  "args": {"dry_run": false},
  "enabled": true
}

Response: 201 Created

Example:

curl -X POST http://localhost:6830/system/cron \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "nightly-gc",
    "schedule": "0 2 * * *",
    "task_type": "gc",
    "args": {"dry_run": false},
    "enabled": true
  }'

Error Responses:

PATCH /system/cron/

Update a cron schedule. All fields are optional – only provided fields are changed.

Request Body:

{
  "enabled": false,
  "schedule": "0 3 * * *"
}

Response: 200 OK

Returns the updated schedule.

Error Responses:

DELETE /system/cron/

Delete a cron schedule.

Response: 200 OK

{
  "id": "nightly-gc",
  "deleted": true
}

Error Responses:

Lifecycle Configuration

Lifecycle configuration is stored inside the database at /.aeordb-config/lifecycle.json. Missing fields use safe defaults, so older databases that only have snapshot_retention continue to allow snapshot writes.

GET /system/lifecycle

Return the current lifecycle policy.

Response: 200 OK

{
  "snapshot_writes_enabled": true,
  "snapshot_retention": {
    "auto_months": 0,
    "manual_months": 0
  }
}

PUT /system/lifecycle

Replace the lifecycle policy.

Request Body:

{
  "snapshot_writes_enabled": false,
  "snapshot_retention": {
    "auto_months": 1,
    "manual_months": 12
  }
}

When snapshot_writes_enabled is false, manual POST /versions/snapshots and snapshot rename requests return 403. Automatic safety snapshots, such as file-restore and pre-GC snapshots, are skipped; the underlying operation continues when it can safely proceed without writing a new snapshot.

Backup & Restore

POST /versions/export

Export the database (or a specific version) as an .aeordb archive file.

Query Parameters:

Response: 200 OK

• Content-Type: application/octet-stream
• Content-Disposition: attachment; filename="export-{hash_prefix}.aeordb"
• Body: binary archive data

Example:

# Export HEAD
curl -X POST http://localhost:6830/versions/export \
  -H "Authorization: Bearer $TOKEN" \
  -o backup.aeordb

# Export a specific snapshot
curl -X POST "http://localhost:6830/versions/export?snapshot=v1.0" \
  -H "Authorization: Bearer $TOKEN" \
  -o backup-v1.aeordb

# Export by hash
curl -X POST "http://localhost:6830/versions/export?hash=a1b2c3d4..." \
  -H "Authorization: Bearer $TOKEN" \
  -o backup.aeordb

POST /versions/diff

Create a patch file representing the difference between two versions.

Query Parameters:

Response: 200 OK

• Content-Type: application/octet-stream
• Content-Disposition: attachment; filename="patch-{hash_prefix}.aeordb"
• Body: binary patch data

Example:

curl -X POST "http://localhost:6830/versions/diff?from=v1.0&to=v2.0" \
  -H "Authorization: Bearer $TOKEN" \
  -o patch-v1-v2.aeordb

POST /versions/import

Import a backup or patch file. Body limit: 10 MB.

Query Parameters:

Request:

• Headers:
  • Authorization: Bearer <token> (required)
• Body: raw .aeordb file bytes

Response: 200 OK

{
  "status": "success",
  "backup_type": "export",
  "entries_imported": 1500,
  "chunks_imported": 3200,
  "files_imported": 450,
  "directories_imported": 30,
  "deletions_applied": 5,
  "version_hash": "a1b2c3d4e5f6...",
  "head_promoted": true
}

Example:

curl -X POST "http://localhost:6830/versions/import?promote=true" \
  -H "Authorization: Bearer $TOKEN" \
  --data-binary @backup.aeordb

Error Responses:

POST /versions/promote

Promote an arbitrary version hash to HEAD.

Query Parameters:

Response: 200 OK

{
  "status": "success",
  "head": "a1b2c3d4e5f6..."
}

Example:

curl -X POST "http://localhost:6830/versions/promote?hash=a1b2c3d4e5f6..." \
  -H "Authorization: Bearer $TOKEN"

Error Responses:

Monitoring

GET /system/stats

System statistics endpoint. Returns a structured JSON snapshot of all engine metrics. All values are read from O(1) atomic counters — this endpoint is safe to poll frequently with no performance impact.

Response: 200 OK

{
  "identity": {
    "version": "0.9.0",
    "database_path": "/data/mydb.aeordb",
    "hash_algorithm": "Blake3_256",
    "chunk_size": 262144,
    "node_id": 1,
    "uptime_seconds": 86400
  },
  "counts": {
    "files": 150000,
    "directories": 23000,
    "symlinks": 500,
    "chunks": 420000,
    "snapshots": 12,
    "forks": 2
  },
  "sizes": {
    "disk_total": 2147483648,
    "kv_file": 86114304,
    "logical_data": 1800000000,
    "chunk_data": 1200000000,
    "void_space": 5242880,
    "dedup_savings": 600000000
  },
  "throughput": {
    "writes_per_sec": { "1m": 42.3, "5m": 38.1, "15m": 35.7, "peak_1m": 120.0 },
    "reads_per_sec": { "1m": 156.2, "5m": 140.5, "15m": 138.0, "peak_1m": 450.0 },
    "bytes_written_per_sec": { "1m": 435200, "5m": 392000, "15m": 367000 },
    "bytes_read_per_sec": { "1m": 16065536, "5m": 14450000, "15m": 14200000 }
  },
  "latency": {
    "write": { "p50": 5.6, "p95": 15.4, "p99": 20.5 },
    "read": { "p50": 2.1, "p95": 8.3, "p99": 12.0 },
    "query": { "p50": 4.2, "p95": 22.0, "p99": 45.0 },
    "flush": { "p50": 1.2, "p95": 5.0, "p99": 12.0 }
  },
  "health": {
    "disk_usage_percent": 48.5,
    "kv_fill_ratio": 0.72,
    "dedup_hit_rate": 0.33,
    "gc_last_reclaimed_bytes": 1048576,
    "write_buffer_depth": 42
  },
  "memory": {
    "process": {
      "rss_bytes": 2147483648,
      "peak_rss_bytes": 3221225472,
      "virtual_bytes": 8589934592,
      "data_bytes": 1610612736,
      "swap_bytes": 0,
      "thread_count": 32,
      "fd_count": 128
    },
    "index_cache": {
      "cached_indexes": 16,
      "dirty_indexes": 4,
      "deleted_indexes": 0,
      "pending_mutations": 512,
      "total_mutations": 102400,
      "flushes": 7,
      "flushed_indexes": 92,
      "entries": 2500000,
      "values": 350000,
      "estimated_bytes": 734003200,
      "top_cached_indexes": []
    },
    "directory_cache": {
      "entries": 12000,
      "estimated_bytes": 16777216
    },
    "caches": {
      "permissions_entries": 128,
      "index_config_entries": 16,
      "grants_index_entries": 4
    },
    "estimated_engine_owned_bytes": 750780416
  },
  "sync": {
    "active_peers": 2,
    "failing_peers": 0,
    "last_sync_ms": 1776563922032,
    "sync_lag_entries": { "peer_2": 0, "peer_3": 15 }
  }
}

Response sections:

Note: The previous GET /system/stats returned a flat object computed via O(n) iteration. The new response is structured into nested sections and is O(1) — no performance concerns polling at high frequency.

sizes.logical_data is the sum of live file sizes reachable from the current HEAD tree. sizes.chunk_data is the stored payload size of unique chunk entries in the KV index, initialized from entry metadata without reading chunk bodies. sizes.void_space is tracked reusable space inside the append log; it is not filesystem free space.

memory.process is sampled from the operating system. On Linux this uses /proc/self/status plus /proc/self/fd; on macOS it uses Mach task information plus /dev/fd. Platform-specific fields that are unavailable are reported as 0. memory.index_cache.estimated_bytes, memory.directory_cache.estimated_bytes, and memory.estimated_engine_owned_bytes are best-effort estimates intended for diagnosis and trend monitoring; they are not allocator-exact byte accounting.

Example:

curl http://localhost:6830/system/stats \
  -H "Authorization: Bearer $TOKEN"

GET /system/health

Public health check endpoint. No authentication required. Once the database is ready, this returns only a minimal status object – no detailed internal checks are exposed.

Response: 200 OK

{
  "status": "healthy",
  "version": "0.9.5"
}

During startup, clean opens, dirty startup, or WAL/KV recovery, AeorDB binds HTTP before the storage engine is ready. In that state, /system/health still returns 200 OK with startup progress:

{
  "status": "starting",
  "phase": "rebuild_kv_scan",
  "message": "Scanning WAL entries for dirty startup recovery",
  "version": "0.9.5",
  "progress": 0.42,
  "eta": {
    "seconds": 480,
    "at": "2026-06-12T03:05:00Z"
  }
}

progress is an overall startup fraction from 0.0 to 1.0. eta is null when unknown, or an object with estimated seconds remaining and an RFC 3339 timestamp. Non-health routes return 503 Service Unavailable until the full application router is ready.

For detailed system diagnostics after startup, use GET /system/stats instead (requires authentication).

Example:

curl http://localhost:6830/system/health

GET /system/metrics

Prometheus-format metrics endpoint. Requires authentication.

Response: 200 OK

• Content-Type: text/plain; version=0.0.4; charset=utf-8
• Body: Prometheus text exposition format

Memory gauges are updated when /system/metrics is rendered and by the periodic metrics pulse:

Example:

curl http://localhost:6830/system/metrics \
  -H "Authorization: Bearer $TOKEN"

API Key Management

POST /auth/keys/admin

Create a new API key. The plaintext key is returned only once – store it securely. Requires root.

Request Body:

{
  "user_id": "550e8400-e29b-41d4-a716-446655440000"
}

Response: 201 Created

{
  "key_id": "660e8400-e29b-41d4-a716-446655440001",
  "api_key": "aeor_660e8400_a1b2c3d4e5f6...",
  "user_id": "550e8400-e29b-41d4-a716-446655440000",
  "created_at": "2026-04-13T10:00:00Z"
}

Example:

curl -X POST http://localhost:6830/auth/keys/admin \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"user_id": "550e8400-e29b-41d4-a716-446655440000"}'

GET /auth/keys/admin

List all API keys (metadata only – no secrets). Requires root.

Response: 200 OK

{
  "items": [
    {
      "key_id": "660e8400-e29b-41d4-a716-446655440001",
      "user_id": "550e8400-e29b-41d4-a716-446655440000",
      "created_at": "2026-04-13T10:00:00Z",
      "is_revoked": false
    }
  ]
}

DELETE /auth/keys/admin/

Revoke an API key. Revoked keys cannot be used to obtain tokens. Requires root.

Response: 200 OK

{
  "revoked": true,
  "key_id": "660e8400-e29b-41d4-a716-446655440001"
}

Error Responses:

User Management

POST /system/users

Create a new user. Requires root.

Request Body:

{
  "username": "alice",
  "email": "[email protected]",
  "tags": ["editor", "us-west"]
}

Response: 201 Created

{
  "user_id": "550e8400-e29b-41d4-a716-446655440000",
  "username": "alice",
  "email": "[email protected]",
  "is_active": true,
  "tags": ["editor", "us-west"],
  "created_at": 1775968398000,
  "updated_at": 1775968398000
}

GET /system/users

List all users. Requires root.

Response: 200 OK

{
  "items": [
    {
      "user_id": "550e8400-e29b-41d4-a716-446655440000",
      "username": "alice",
      "email": "[email protected]",
      "is_active": true,
      "created_at": 1775968398000,
      "updated_at": 1775968398000
    }
  ]
}

GET /system/users/

Get a single user. Requires root.

Response: 200 OK (same shape as the user object above)

Error Responses:

PATCH /system/users/

Update a user. All fields are optional. Requires root.

Request Body:

{
  "username": "alice_updated",
  "email": "[email protected]",
  "is_active": true,
  "tags": ["editor", "us-west", "senior"]
}

Tags are admin-only – users cannot modify their own tags, preventing privilege escalation through self-assigned group membership.

Response: 200 OK (returns the updated user)

DELETE /system/users/

Deactivate a user (soft delete – sets is_active to false). Requires root.

Response: 200 OK

{
  "deactivated": true,
  "user_id": "550e8400-e29b-41d4-a716-446655440000"
}

Group Management

Groups define path-level access control rules using query-based membership. Users are matched into groups by querying safe fields on their user record, including tags.

Tag-Based Group Membership

When query_field is set to "tags", three special operators are available:

Standard operators (eq, gt, etc.) also work with tags – they match against the comma-joined tag string.

POST /system/groups

Create a new group. Requires root.

Request Body:

{
  "name": "editors",
  "default_allow": "/content/*",
  "default_deny": "/system/*",
  "query_field": "tags",
  "query_operator": "has",
  "query_value": "editor"
}

Response: 201 Created

{
  "name": "editors",
  "default_allow": "/content/*",
  "default_deny": "/system/*",
  "query_field": "role",
  "query_operator": "eq",
  "query_value": "editor",
  "created_at": 1775968398000,
  "updated_at": 1775968398000
}

GET /system/groups

List all groups. Requires root.

Response: 200 OK (object with items array of group objects)

GET /system/groups/

Get a single group. Requires root.

Error Responses:

PATCH /system/groups/

Update a group. All fields are optional. Requires root.

Request Body:

{
  "default_allow": "/content/*",
  "query_value": "senior-editor"
}

The query_field value is validated against a whitelist of safe fields. Attempting to use an unsafe field returns a 400 error.

Error Responses:

DELETE /system/groups/

Delete a group. Requires root.

Response: 200 OK

{
  "deleted": true,
  "name": "editors"
}

Error Responses:

Email Configuration

AeorDB supports sending email notifications (e.g., when files are shared via POST /files/share). Email can be configured using either SMTP or OAuth providers.

GET /system/email-config

Retrieve the current email configuration. Sensitive fields (passwords, client secrets, refresh tokens) are masked as "--------" in the response, and a "configured": true field is added.

Auth: Root only.

Response: 200 OK

{
  "provider": "smtp",
  "host": "smtp.example.com",
  "port": 587,
  "username": "[email protected]",
  "password": "--------",
  "from_address": "[email protected]",
  "from_name": "AeorDB",
  "tls": "starttls",
  "configured": true
}

PUT /system/email-config

Save email configuration. Supports two provider types: SMTP and OAuth.

Auth: Root only.

SMTP Configuration:

{
  "provider": "smtp",
  "host": "smtp.example.com",
  "port": 587,
  "username": "[email protected]",
  "password": "secret",
  "from_address": "[email protected]",
  "from_name": "AeorDB",
  "tls": "starttls"
}

OAuth Configuration:

{
  "provider": "oauth",
  "oauth_provider": "gmail",
  "client_id": "...",
  "client_secret": "...",
  "refresh_token": "...",
  "from_address": "[email protected]",
  "from_name": "AeorDB"
}

Response: 200 OK

POST /system/email-test

Send a test email to verify the current configuration.

Auth: Root only.

Request Body:

{
  "to": "[email protected]"
}

Response: 200 OK

{
  "sent": true,
  "message": "Test email sent successfully"
}

On failure:

{
  "sent": false,
  "error": "Connection refused: smtp.example.com:587"
}

Share Notifications

When files are shared via POST /files/share, email notifications are automatically sent to recipients if email is configured. If email is not configured, sharing still works – notifications are silently skipped.

Plugin Endpoints

AeorDB supports deploying WebAssembly (WASM) plugins that extend the database with custom logic. HTTP plugins are invoked by their plugin path under the /plugins namespace.

Native Parsers

AeorDB includes 8 built-in format parsers that run automatically during indexing – no WASM deployment needed. Native parsers are tried first; if the content type is not recognized, the engine falls through to any deployed WASM parser plugin.

Supported Formats

How Native Parsers Work

1. When a file is stored and indexing runs, the engine checks the file’s content type against the native parser registry.
2. If the content type is application/octet-stream or empty, the engine falls back to extension-based matching.
3. If a native parser handles the format, it returns structured JSON that feeds into the indexing pipeline.
4. If no native parser matches, the engine falls through to the WASM plugin system (checking /.config/parsers.json or the directory’s parser field).

Native parsers have zero overhead compared to WASM – they run as compiled Rust code in the same process. WASM parser plugins remain available for custom or proprietary formats not covered by the built-in parsers.

Endpoint Summary

PUT /plugins/

Deploy a WASM plugin at the given plugin path. If a plugin already exists at this path, it is replaced.

{name} in the URL is the plugin path/key used for storage and invocation. The optional name query parameter is display metadata only.

Request

• URL parameters:
  • {name} – plugin path/key used by /plugins/{name}/invoke
• Query parameters:
  • name (optional) – display name (defaults to the {name} URL segment)
  • plugin_type (optional) – plugin type string (defaults to "wasm")
  • version (optional) – plugin package version
  • author (optional) – plugin author
• Headers:
  • Authorization: Bearer <token> (required)
• Body: raw WASM binary bytes

Response

Status: 200 OK

Returns the plugin metadata:

{
  "plugin_id": "2f1f4a7a-30f4-4a75-bde0-8e3c6eaf51e4",
  "name": "my-plugin",
  "path": "my-plugin",
  "plugin_type": "wasm",
  "created_at": "2026-04-13T10:00:00Z",
  "version": "1.2.3",
  "author": "Plugin Author",
  "checksum": "blake3:4f3c...",
  "updated_at": "2026-04-13T10:15:00Z"
}

Example

curl -X PUT "http://localhost:6830/plugins/my-plugin?version=1.2.3&author=Plugin%20Author" \
  -H "Authorization: Bearer $TOKEN" \
  --data-binary @plugin.wasm

Error Responses

POST /plugins/{name}/invoke

Invoke a deployed plugin. The request body is wrapped in a PluginRequest envelope with metadata, then passed to the WASM runtime.

Request

• URL parameters:
  • {name} – plugin path/key
• Headers:
  • Authorization: Bearer <token> (required)
  • Content-Type – depends on what the plugin expects
• Body: raw request payload (passed to the plugin as arguments)

Plugin Request Envelope

The server wraps the raw body into:

{
  "arguments": "<raw body bytes>",
  "metadata": {
    "name": "my-plugin",
    "path": "/plugins/my-plugin",
    "plugin_path": "my-plugin"
  }
}

Plugin Response

Plugins return a PluginResponse envelope:

{
  "status_code": 200,
  "content_type": "application/json",
  "headers": {
    "x-custom-header": "value"
  },
  "body": "<response bytes>"
}

The server maps these fields to the HTTP response. For security, only safe headers are forwarded:

• Headers starting with x-
• cache-control, etag, last-modified, content-disposition, content-language, content-encoding, vary

If the plugin returns data that is not a valid PluginResponse, it is sent as raw application/octet-stream bytes (backward compatibility).

WASM Host Functions

Plugins have access to the following host functions for interacting with the database:

• CRUD: read, write, and delete files
• Extraction: read UTF-8 line or character ranges without buffering the full file through the plugin boundary
• Query: execute queries and aggregations against the engine
• Context: access request metadata

Example

curl -X POST http://localhost:6830/plugins/my-plugin/invoke \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"input": "hello"}'

Error Responses

GET /plugins

List all deployed plugins.

Request

• Headers:
  • Authorization: Bearer <token> (required)

Response

Status: 200 OK

{
  "items": [
    {
      "plugin_id": "2f1f4a7a-30f4-4a75-bde0-8e3c6eaf51e4",
      "name": "my-plugin",
      "path": "my-plugin",
      "plugin_type": "wasm",
      "created_at": "2026-04-13T10:00:00Z",
      "version": "1.2.3",
      "author": "Plugin Author",
      "checksum": "blake3:4f3c...",
      "updated_at": "2026-04-13T10:15:00Z"
    }
  ]
}

Example

curl http://localhost:6830/plugins \
  -H "Authorization: Bearer $TOKEN"

DELETE /plugins/

Remove a deployed plugin.

Request

• URL parameters:
  • {name} – plugin path/key
• Headers:
  • Authorization: Bearer <token> (required)

Response

Status: 200 OK

{
  "removed": true,
  "path": "my-plugin"
}

Example

curl -X DELETE http://localhost:6830/plugins/my-plugin \
  -H "Authorization: Bearer $TOKEN"

Error Responses

Events & Webhooks

AeorDB publishes real-time events via Server-Sent Events (SSE). Clients can subscribe to a filtered stream of engine events for live updates.

Endpoint Summary

GET /system/events

Open a persistent Server-Sent Events connection. The server pushes events as they occur and sends periodic keepalive pings.

When the stream is available, the server is ready to serve the full API. For unfiltered streams, and for streams that explicitly include server_ready, the first event is a synthetic server_ready event with the current ready state. This event is generated per connection; it is not a one-time broadcast that clients can miss.

Query Parameters

Request

curl -N http://localhost:6830/system/events \
  -H "Authorization: Bearer $TOKEN"

Filtered Stream

Subscribe to only specific event types:

curl -N "http://localhost:6830/system/events?events=entries_created,entries_deleted" \
  -H "Authorization: Bearer $TOKEN"

Filter by path prefix:

curl -N "http://localhost:6830/system/events?path_prefix=/data/users" \
  -H "Authorization: Bearer $TOKEN"

Combine both:

curl -N "http://localhost:6830/system/events?events=entries_created&path_prefix=/data/" \
  -H "Authorization: Bearer $TOKEN"

Response Format

The response is an SSE stream. Each event has the standard SSE fields:

id: evt-uuid-here
event: entries_created
data: {"event_id":"evt-uuid-here","event_type":"entries_created","timestamp":1775968398000,"payload":{"entries":[{"path":"/data/report.pdf"}]}}

Event Envelope

Each event is a JSON object with:

Event Types

Server Ready Event

The server_ready event is emitted as the first SSE event for each eligible /system/events connection. Because /system/events is only served by the full application router, receiving this event means the server has finished startup and is ready for normal API traffic.

It is sent when:

• no events filter is supplied
• server_ready is included in the events filter

It is not sent when an events filter excludes it. This preserves strict event filtering for clients that only want a specific event type.

{
  "status": "ready",
  "version": "0.9.5",
  "startup_time": 1781233139578,
  "uptime_ms": 6500
}

Heartbeat Event

The heartbeat event is used exclusively for clock synchronization between nodes. It fires every 15 seconds and carries only three fields:

{
  "intent_time": 1776563925000,
  "construct_time": 1776563925003,
  "node_id": 1
}

The delta between intent_time and construct_time is used by peers to measure clock offset and network latency. The heartbeat does not contain any stats or metrics data — it is a lightweight clock-sync mechanism only.

Breaking change: Prior versions included stats fields (file counts, disk usage, etc.) in the heartbeat payload. These fields have been removed. Use the metrics event for monitoring data.

Metrics Event

The metrics event delivers system metrics to connected clients every 15 seconds (configurable, independent of the heartbeat interval). All values are computed in O(1) from atomic counters — subscribing to this event has no performance impact on the server.

Subscribe to metrics:

curl -N "http://localhost:6830/system/events?events=metrics" \
  -H "Authorization: Bearer $TOKEN"

Payload:

{
  "counts": {
    "files": 150000,
    "directories": 23000,
    "symlinks": 500,
    "chunks": 420000,
    "snapshots": 12,
    "forks": 2
  },
  "sizes": {
    "disk_total": 2147483648,
    "kv_file": 86114304,
    "logical_data": 1800000000,
    "chunk_data": 1200000000,
    "void_space": 5242880,
    "dedup_savings": 600000000
  },
  "throughput": {
    "writes_per_sec": { "1m": 42.3, "5m": 38.1, "15m": 35.7, "peak_1m": 120.0 },
    "reads_per_sec": { "1m": 156.2, "5m": 140.5, "15m": 138.0, "peak_1m": 450.0 },
    "bytes_written_per_sec": { "1m": 435200, "5m": 392000, "15m": 367000 },
    "bytes_read_per_sec": { "1m": 16065536, "5m": 14450000, "15m": 14200000 }
  },
  "health": {
    "disk_usage_percent": 48.5,
    "kv_fill_ratio": 0.72,
    "dedup_hit_rate": 0.33,
    "write_buffer_depth": 42
  },
  "memory": {
    "process": {
      "rss_bytes": 2147483648,
      "peak_rss_bytes": 3221225472,
      "virtual_bytes": 8589934592,
      "data_bytes": 1610612736,
      "swap_bytes": 0,
      "thread_count": 32,
      "fd_count": 128
    },
    "index_cache": {
      "cached_indexes": 16,
      "dirty_indexes": 4,
      "deleted_indexes": 0,
      "pending_mutations": 512,
      "total_mutations": 102400,
      "flushes": 7,
      "flushed_indexes": 92,
      "entries": 2500000,
      "values": 350000,
      "estimated_bytes": 734003200,
      "top_cached_indexes": []
    },
    "directory_cache": {
      "entries": 12000,
      "estimated_bytes": 16777216
    },
    "caches": {
      "permissions_entries": 128,
      "index_config_entries": 16,
      "grants_index_entries": 4
    },
    "estimated_engine_owned_bytes": 750780416
  }
}

memory.process is sampled from the operating system. memory.index_cache.estimated_bytes, memory.directory_cache.estimated_bytes, and memory.estimated_engine_owned_bytes are best-effort diagnostic estimates, not allocator-exact accounting.

Payload sections:

Migration note: If your dashboard previously subscribed to ?events=heartbeat for monitoring data, switch to ?events=metrics. If you need both clock data and metrics (uncommon), subscribe to ?events=heartbeat,metrics.

Keepalive

The server sends a keepalive ping every 30 seconds to prevent connection timeouts:

: ping

Path Prefix Matching

The path prefix filter checks two locations in the event payload:

1. Batch events: payload.entries[].path – matches if any entry’s path starts with the prefix
2. Single-path events: payload.path – matches if the path starts with the prefix

Connection Behavior

• The connection stays open indefinitely until the client disconnects.
• If the client falls behind (lagged), missed events are silently dropped.
• Reconnecting clients should use the last received id for gap detection (standard SSE Last-Event-ID header).

JavaScript Example

const evtSource = new EventSource(
  'http://localhost:6830/system/events?events=entries_created',
  { headers: { 'Authorization': 'Bearer ' + token } }
);

evtSource.addEventListener('entries_created', (event) => {
  const data = JSON.parse(event.data);
  console.log('Files created:', data.payload.entries);
});

evtSource.onerror = (err) => {
  console.error('SSE error:', err);
};

GET /events/me

A per-user SSE channel that delivers ONLY events addressed to the authenticated user. The server filters the event bus and forwards an event only when its recipient_user_id matches the JWT’s sub claim. Generic events with no recipient (heartbeats, system metrics, file uploads, etc.) are NOT delivered here — those go through /system/events.

This channel is the security boundary for personal notifications: each user can only see events sent specifically to them, even if multiple users are subscribed simultaneously.

Request

curl -N http://localhost:6830/events/me \
  -H "Authorization: Bearer $TOKEN"

EventSource (browsers can’t set Authorization headers on SSE):

const evt = new EventSource('/events/me?token=' + encodeURIComponent(token));
evt.addEventListener('files_shared', (e) => {
  const payload = JSON.parse(e.data).payload;
  alert(`${payload.from} shared ${payload.path} with you`);
});

Event Types Currently Routed Here