docs: BLUEPRINT_FORMAT.md + README updates for the v2 container (terrain-water task 02)

Byte-accurate v2 contract (magic/version, tagged sections, registered
FourCC table, validation rules, re-encode sentinels, extension path)
plus the v1 legacy summary — v1 stays live per the safety-net plan.
READMEs updated additively: parser dispatch, dual-write outputs,
harness usage, server params cross-check, wire-format hazard rewording
(u8 on v2 path, range-checked; append-only rule unchanged).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Stewart Howe 2026-08-06 07:31:03 -04:00
parent 8532d88771
commit a7e6c17422
7 changed files with 185 additions and 22 deletions

View file

@ -8,16 +8,24 @@ define what things are and how to calculate them; they never remember what is cu
### `MapDataParser.cs` — the data bridge ### `MapDataParser.cs` — the data bridge
Deserializes the binary `.dat` blueprint written by `/Tools` into a `WorldBlueprint` held in RAM: Deserializes the binary `.dat` blueprint written by `/Tools` into a `WorldBlueprint` held in RAM:
map size, the float heightmap, the biome map, town locations, and **all four tiers** of A\* road map size, the float heightmap, the biome map, town locations, **all four tiers** of A\* road
vectors (Highways, Branch Roads, Rugged Roads, Trails). vectors (Highways, Branch Roads, Rugged Roads, Trails), and — from v2 files — the embedded
generation params (seed, sizes, impact centre, provenance) and each town's highway-node flag.
Read order is fixed and must match the writer exactly: magic string `"ISLA_V1"` → map size → **Two formats are live** (byte-accurate contract: `Core/Scripts/BLUEPRINT_FORMAT.md`). The parser
per-pixel `{height float, biome int}` in x-major order → towns → the four road tiers in order. dispatches on the file's first byte:
- **v2 (primary)** — raw `ISLA` magic, u32 version gate, then tagged sections
`[u32 tag][u64 length][payload]`. Unknown tags are skipped by length, so future sections are
invisible to older readers. Validated on read: version, MapSize bounds, section lengths against
the file, biome/tier ordinal ranges.
- **v1 (legacy)** — the positional `"ISLA_V1"` format, still written beside v2 as `_v1.dat` and
still loadable (with a deprecation warning) until a future removal task.
**Wire-format hazard.** Biome and town-tier enums are serialized as their **ordinal values**, with **Wire-format hazard.** Biome and town-tier enums are serialized as their **ordinal values**
no version gate and no range validation on read. **Never reorder or insert members** in `Enums.cs` (u8 in v2, i32 in v1). **Never reorder or insert members** in `Enums.cs` — append only, at the end.
append only, at the end. Reordering silently reinterprets every pixel of every existing `.dat`. v2's range checks make drift fail loudly at parse time; legacy v1 has no validation and silently
(`RoadTier` is exempt: road tiers are stored in separate file sections, so that enum never hits disk.) reinterprets every pixel. (`RoadTier` is exempt: road tiers are stored in separate file sections —
tagged in v2, positional in v1 — so that enum never hits disk.)
### `ChunkData.cs` + `Constants.cs` — voxel containers and tuning ### `ChunkData.cs` + `Constants.cs` — voxel containers and tuning
- **Chunk dimensions:** `24 × 24` horizontal, `256` vertical (`Constants.cs`). - **Chunk dimensions:** `24 × 24` horizontal, `256` vertical (`Constants.cs`).

View file

@ -0,0 +1,116 @@
# BLUEPRINT_FORMAT — the `.dat` container, v2 (and the v1 legacy summary)
The `.dat` blueprint is the sole handoff from the offline generator (`/Tools`) to the server
(`/Server`). This document is the byte-accurate contract. Code authority: constants in
`BlueprintFormat.cs`, writer in `BlueprintWriter.cs`, reader in `MapDataParser.cs`. On any
disagreement between this file and the code, the code wins and this file must be corrected.
- **Byte order: little-endian throughout** (BinaryWriter/BinaryReader platform default; stated here
because the format itself records no endianness marker).
- **File naming:** `user://MapData_Seed_<seed>.dat`, where `<seed>` is the generator's **resolved**
noise seed. The server looks the file up by config `WorldSeed` — a config seed of 0 ("randomize")
therefore never finds the file it just generated (historical hazard H4; config-side, unchanged).
- The generator currently **dual-writes**: v2 under the primary name, legacy v1 beside it as
`MapData_Seed_<seed>_v1.dat`. The v1 fallback is scheduled for removal in a future task.
## Format detection
The reader dispatches on the **first byte** of the file:
| First byte | Format | Why it's unambiguous |
|---|---|---|
| `0x49` (`'I'`) | **v2** — opens with the raw 4 bytes `ISLA` | v2's magic is raw bytes, deliberately not a .NET string |
| `0x07` | **v1 (legacy)** — opens with the length-prefixed .NET string `"ISLA_V1"` (prefix byte 7) | v1 loads intact, with a deprecation warning |
| anything else | rejected loudly, `null` return | |
## v2 layout
```
[4 B] magic: raw bytes 'I','S','L','A' (== little-endian u32 0x414C5349)
[4 B] format version: u32 = 2 (any other value -> loud reject, null)
[...] sections, sequentially, until EOF
```
### Section framing
Every section: `[u32 tag][u64 payload-length in bytes][payload]`.
- **Reader rule: known tag → parse; unknown tag → skip payload-length bytes and continue.** This is
the forward-compatibility property the v2 redesign exists to buy: a reader that predates a section
(a future water layer, for instance) loads the file and never sees it.
- The u64 length is deliberate headroom (a u32 caps a section at 4 GiB; a 16K-map height section
would already be 1 GiB, and future per-pixel sections should never have to shard on length).
- Tag values are FourCC codes stored as little-endian u32, so the on-disk bytes read as ASCII in a
hex dump. **All tags are registered in `BlueprintFormat.cs` and in the table below — one place
each. Never reuse a retired tag value.**
- **Rules enforced by the reader:** the params section must be **first**; duplicate tags are an
error; every declared length is checked against the remaining file before the payload is read; a
parsed section must consume exactly its declared length.
### Registered sections (v2, current content)
| Tag (ASCII / u32) | Payload | Notes |
|---|---|---|
| `PRMS` / `0x534D5250` | `i32 WorldSeed` · `i32 MapSize` · `f32 CraterRadius` · `f32 DensityMultiplier` · `f32 ImpactCenterX` · `f32 ImpactCenterY` · `str GeneratedUtc` · `str GeneratorGitHash` | Mandatory, first. The **resolved** generation inputs — the file is self-describing; the server cross-checks seed and MapSize against config and warns loudly on desync. `str` = .NET length-prefixed UTF-8 (fine *inside* a length-framed section; only the file header must avoid it). `CraterRadius` is f32 because `ConfigManager.CraterRadius` is a float in code. `GeneratorGitHash` is the repo short hash or `""`. |
| `HGTS` / `0x53544748` | `MapSize²` × `f32` height, **X outer / Y inner** | Mandatory. The second index is the map's north/south axis; the server consumes it as world **Z**. (This axis convention was never written down for v1 — it is now normative.) Length must equal `4·MapSize²`. |
| `BIOM` / `0x4D4F4942` | `MapSize²` × `u8` biome ordinal, same pixel order | Mandatory. Ordinals from `Enums.cs::Biome`**append-only, never reorder** (the ordinal IS the wire value). Writer refuses ordinals > 255; reader rejects ordinals ≥ the known biome count (parse-time validation; the palette's runtime Dirt fallback for in-memory values is unchanged). Length must equal `MapSize²`. |
| `TOWN` / `0x4E574F54` | `i32 count`, then per town: `f32 X` · `f32 Y` · `u8 tier` · `u8 isHighwayNode` (0/1) | Tier ordinals from `Enums.cs::TownTier`, append-only, range-checked on read. The highway-node flag is what the generator's road topology was built from (v1 dropped it); carried and exposed on the parsed blueprint, consumed by nothing server-side yet. |
| `RDHW` / `0x57484452` | `i32 pathCount`, then per path: `i32 pointCount` + `pointCount` × (`f32 X` · `f32 Y`) | Highway tier. **Tags, not file position, identify the tier** — the v1 order-fragility is gone. |
| `RDBR` / `0x52424452` | same layout | Branch tier. |
| `RDRG` / `0x47524452` | same layout | Rugged tier. |
| `RDTL` / `0x4C544452` | same layout | Trail tier. |
Missing `TOWN`/road sections load as empty lists with a warning; missing `PRMS`/`HGTS`/`BIOM` is an
error. Section order after `PRMS` is not significant (the reference writer emits the table order).
### Reader validation (v2 path)
1. Magic and version gate — loud, specific errors; `null` return (the caller's null-check aborts
world load cleanly).
2. `MapSize` sanity bound **before any allocation**: `256 ≤ MapSize ≤ 32768`.
3. Every section length checked against the remaining file length; truncation is a loud error, not
a read-past-end crash.
4. Section parsers must consume exactly their declared length.
5. Biome and town-tier ordinals range-checked against the known enum counts.
6. After load, `ServerChunkManager` compares embedded `WorldSeed`/`MapSize` against
`ServerConfig.json` and logs a prominent desync warning on mismatch (warning, not abort).
### Sentinel params (re-encoded files)
A v2 file produced by re-encoding a v1 source (e.g. the round-trip harness) cannot know the
original generation inputs. It carries: `WorldSeed = 0`, `CraterRadius = -1`, `DensityMultiplier =
-1`, `ImpactCenter = (-1, -1)` (`MapSize` is always real). The seed cross-check skips sentinel
seed 0. Provenance (`GeneratedUtc`, `GeneratorGitHash`) is stamped at **write** time and describes
the file, not the original generation.
### Size
`total = 8 (header) + Σ per section (12 + payload)`. The pixel grid dominates: `5·MapSize²` bytes
(4 height + 1 biome) ≈ **320 MiB at 8K**, vs v1's `8·MapSize²` ≈ 512 MiB — the u8 biome section
saves ~192 MiB at 8K.
### Adding a new section (the intended extension path)
1. Register a fresh FourCC in `BlueprintFormat.cs` and in the table above.
2. Emit it from `BlueprintWriter.WriteV2` (any position after `PRMS`).
3. Parse it in `MapDataParser.LoadV2`'s tag dispatch.
4. Old readers skip it automatically; **no version bump is needed for additive sections.** Bump
`VERSION` only for changes that alter the meaning of *existing* bytes.
## v1 legacy summary (still readable, still dual-written, removal pending)
Positional and untagged — every field's meaning derives from its offset; no lengths, no checksums,
no skip capability. Layout: length-prefixed string `"ISLA_V1"``i32 MapSize``MapSize²` ×
(`f32 height` · **`i32` biome ordinal**) X-outer/Y-inner → `i32 townCount` + per town (`f32 X` ·
`f32 Y` · `i32 tier`) → four road blocks **identified by position** (Highway → Branch → Rugged →
Trail), each `i32 pathCount` + per path `i32 pointCount` + points. The v1 reader stops after the
Trail block and ignores trailing bytes. No validation beyond the header string. It does not carry
generation params, the impact centre, or the highway-node flag.
## Shared wire-format rule (both formats)
**`Biome` and `TownTier` ordinals are the serialized values** (u8 in v2, i32 in v1). The enums in
`Enums.cs` are declared without explicit numeric values, so their ordinals are positional:
**append new members only at the end; never reorder or insert.** v2 adds parse-time range checks,
which turn enum drift from silent reinterpretation into a loud load failure — but the append-only
rule is still what keeps old files *meaning* the same thing.

View file

@ -24,15 +24,25 @@ rendering.
### Shared identifiers ### Shared identifiers
- **`Enums.cs`** — `Biome`, `TownTier`, `MapHalf`, `RoadTier`. - **`Enums.cs`** — `Biome`, `TownTier`, `MapHalf`, `RoadTier`.
**`Biome` and `TownTier` are the `.dat` wire format.** They are declared without explicit values, **`Biome` and `TownTier` are the `.dat` wire format** (u8 in the v2 container, i32 in legacy
so each member's ordinal *is* the number written to disk, with no version gate and no validation on v1). They are declared without explicit values, so each member's ordinal *is* the number written
read. **Append only, at the end — never reorder or insert.** Doing so silently reinterprets every to disk. **Append only, at the end — never reorder or insert.** The v2 reader range-checks
pixel and every settlement in every existing blueprint. ordinals so drift fails loudly at parse time, but only the append-only rule keeps old files
*meaning* the same thing. Legacy v1 has no gate and no validation at all.
`RoadTier` is exempt: road tiers live in separate sections of the file, so it is never serialized. `RoadTier` is exempt: road tiers live in separate sections of the file, so it is never serialized.
### World data ### World data
- **`MapDataParser.cs`** — decodes the `.dat` blueprint into a `WorldBlueprint` (heightmap, biome map, - **`BLUEPRINT_FORMAT.md`** — the byte-accurate `.dat` container contract (v2 tagged sections +
towns, four road tiers). the v1 legacy summary). Read this before touching the writer or parser.
- **`BlueprintFormat.cs`** — the single registry of v2 constants: magic, version gate, section
tags (FourCC), validation bounds, re-encode sentinels.
- **`BlueprintWriter.cs`** — writes a `WorldBlueprint` as a v2 file. Blueprint-typed on purpose:
the map generator and the round-trip harness are both just callers.
- **`MapDataParser.cs`** — decodes a `.dat` blueprint into a `WorldBlueprint` (heightmap, biome map,
towns, four road tiers, and — v2 only — the embedded generation params and per-town highway-node
flag). Dispatches on the first byte: v2 tagged-section files get validation (version gate, size
bounds, section-length and ordinal range checks); legacy v1 files still load, intact, with a
deprecation warning.
- **`ChunkData.cs`** — one chunk's density and block-ID fields, `+1` padded on every axis so the - **`ChunkData.cs`** — one chunk's density and block-ID fields, `+1` padded on every axis so the
mesher can reach into the neighbouring chunk, plus the per-column data the renderer needs. mesher can reach into the neighbouring chunk, plus the per-column data the renderer needs.
- **`Constants.cs`** — chunk dimensions, `ISO_LEVEL`, `VOXEL_SCALE`, and the visual/road tunables - **`Constants.cs`** — chunk dimensions, `ISO_LEVEL`, `VOXEL_SCALE`, and the visual/road tunables

View file

@ -8,11 +8,15 @@ future multiplayer setup this is the side that dictates terrain and ships chunk
Attached to the `World` root node of `Scenes/Main.tscn`. Runs the whole 3D world at boot. Attached to the `World` root node of `Scenes/Main.tscn`. Runs the whole 3D world at boot.
### Startup ### Startup
1. Loads `ServerConfig.json` and the seed's `.dat` blueprint. 1. Loads `ServerConfig.json` and the seed's `.dat` blueprint (v2 or legacy v1 — the parser
2. **Finds the Capitol** in the parsed town list and uses it as the world origin point. dispatches automatically; see `Core/Scripts/BLUEPRINT_FORMAT.md`).
3. Converts its pixel position to chunk coordinates (`pixel / CHUNK_SIZE`). 2. **Cross-checks the blueprint's embedded params** (v2 only) against the config and logs a
4. Builds a `(2 × ChunkRadius)²` grid of chunks around it — **synchronously, all at boot**. prominent `BLUEPRINT/CONFIG DESYNC` warning if the seed or MapSize disagree — a config edited
5. Teleports the `Camera3D` to 120 m above the Capitol, looking down. after generation is loud now, not silent.
3. **Finds the Capitol** in the parsed town list and uses it as the world origin point.
4. Converts its pixel position to chunk coordinates (`pixel / CHUNK_SIZE`).
5. Builds a `(2 × ChunkRadius)²` grid of chunks around it — **synchronously, all at boot**.
6. Teleports the `Camera3D` to 120 m above the Capitol, looking down.
**Two things to know about startup.** The chunk grid is built in one blocking pass with no **Two things to know about startup.** The chunk grid is built in one blocking pass with no
streaming or unloading, so `ChunkRadius` directly controls boot cost — 32 means 4,096 chunks and streaming or unloading, so `ChunkRadius` directly controls boot cost — 32 means 4,096 chunks and

View file

@ -33,7 +33,8 @@ so they only help between paths. Expect a regeneration to take minutes, or to ne
| File | What it is | | File | What it is |
|---|---| |---|---|
| `MapData_Seed_<seed>.dat` | The binary blueprint the server reads (~512 MB at 8K) | | `MapData_Seed_<seed>.dat` | The binary blueprint the server reads — **v2 tagged container** (~336 MB at 8K; contract in `Core/Scripts/BLUEPRINT_FORMAT.md`) |
| `MapData_Seed_<seed>_v1.dat` | The same content in the legacy v1 format (~512 MB), dual-written as a safety net until a future removal task |
| `Map_Seed_<seed>.png` | Visual snapshot of the map, for reviewing and picking seeds | | `Map_Seed_<seed>.png` | Visual snapshot of the map, for reviewing and picking seeds |
On Linux: `~/.local/share/godot/app_userdata/islaApocolypse/`. On Linux: `~/.local/share/godot/app_userdata/islaApocolypse/`.

View file

@ -23,6 +23,19 @@ tiny or black PNG.
Note the generator writes the `.dat` on its own — the capture wrapper only matters for the Note the generator writes the `.dat` on its own — the capture wrapper only matters for the
full-resolution image. full-resolution image.
## `RoundTripHarness.tscn`
The blueprint format regression test (script: `Tools/Scripts/RoundTripHarness.cs`). Run it
headless from a terminal — it never generates, it round-trips an existing blueprint through the
real parser and v2 writer and asserts semantic equality:
```
Godot --headless --path <repo> res://Tools/Scenes/RoundTripHarness.tscn
```
Defaults to the preserved reference blueprint (`user://reference_1409879727/`); override with the
`HARNESS_V1_PATH` / `HARNESS_V2_PATH` environment variables. Exit code 0 = pass.
## Generating a new world ## Generating a new world
1. Open `MapPreview.tscn` (or `Scenes/MapCaptureTool.tscn` for the full-resolution PNG). 1. Open `MapPreview.tscn` (or `Scenes/MapCaptureTool.tscn` for the full-resolution PNG).

View file

@ -18,7 +18,10 @@ Generates the entire 2D blueprint. Roughly in order:
5. **Biomes and towns** — biome zoning by height and temperature; tiered town placement (Capitol, 5. **Biomes and towns** — biome zoning by height and temperature; tiered town placement (Capitol,
Hubs, Villages, Outposts, POIs) filtered by slope, water proximity and spacing. Hubs, Villages, Outposts, POIs) filtered by slope, water proximity and spacing.
6. **Roads** — see below. 6. **Roads** — see below.
7. **Export** — writes the `.dat` blueprint, then renders the PNG snapshot. 7. **Export** — writes the `.dat` blueprint (dual-write: the v2 tagged container under the primary
seed name, plus the legacy v1 format beside it as `_v1.dat` — see
`Core/Scripts/BLUEPRINT_FORMAT.md`), then renders the PNG snapshot. The v2 file embeds the
resolved generation params (seed, MapSize, crater radius, density, impact centre, provenance).
### The road network ### The road network
@ -52,7 +55,15 @@ Road colours on the snapshot, useful for identifying a road: **red** = Highway,
**dark brown** = Rugged, **light brown** = Trail. **dark brown** = Rugged, **light brown** = Trail.
### Outputs ### Outputs
`MapData_Seed_<seed>.dat` and `Map_Seed_<seed>.png`, both to `user://`. `MapData_Seed_<seed>.dat` (v2), `MapData_Seed_<seed>_v1.dat` (legacy dual-write), and
`Map_Seed_<seed>.png`, all to `user://`.
## `RoundTripHarness.cs`
The blueprint format regression test (scene: `Tools/Scenes/RoundTripHarness.tscn`). Loads a
known-good blueprint through the real parser, re-writes it as v2 through the real writer, re-loads
it, and asserts semantic equality (heights bitwise, biomes, towns, every road point). Headless,
seconds per cycle, no generation. Exit code 0 = pass.
## Rules ## Rules
1. **No magic numbers.** Distances, radii and thresholds derive from `MapSize` or `scaleFactor`, so 1. **No magic numbers.** Distances, radii and thresholds derive from `MapSize` or `scaleFactor`, so