Define the persistent observation store and legacy migration #2

New Issue

2026-08-31T08:03:03Z

xavierk commented

2026-08-31 08:03:03 +00:00

Parent map: Chart Fenris’s persistent TUI monitoring redesign

Question

What persistent observation model should Fenris use to represent monitoring periods, hourly active/idle/powered-off/unknown observations, compact long-term aggregates, recent raw samples, projection inputs, and schema versioning? Decide retention boundaries and an idempotent migration from history.jsonl and hourly.jsonl that preserves existing observation history and recovers safely from interruption.

Parent map: [Chart Fenris’s persistent TUI monitoring redesign](https://git.bongbetic.com/xavierk/Fenris/issues/1) ## Question What persistent observation model should Fenris use to represent monitoring periods, hourly active/idle/powered-off/unknown observations, compact long-term aggregates, recent raw samples, projection inputs, and schema versioning? Decide retention boundaries and an idempotent migration from history.jsonl and hourly.jsonl that preserves existing observation history and recovers safely from interruption.

xavierk added this to the Wayfinder: Fenris persistent TUI monitoring redesign milestone 2026-08-31 08:03:03 +00:00

xavierk added the wayfinder:grilling label 2026-08-31 08:03:03 +00:00

xavierk added a new dependency 2026-08-31 08:03:19 +00:00

#4 Define the lifespan projection and confidence model

xavierk added a new dependency 2026-08-31 08:03:19 +00:00

#5 Verify NVMe endurance signals and projection constraints

xavierk self-assigned this 2026-08-31 08:51:13 +00:00

xavierk commented

2026-08-31 09:18:51 +00:00

Resolution — decided by grilling, all questions approved with recommendations.

Substrate & access: one SQLite database in WAL mode at /var/lib/fenris/observations.db; root-owned, group-readable via the packaging-created fenris read group; the TUI opens it read-only. No /run snapshot layer.

Entities:

samples — recent raw SMART samples (ts, controller identity, raw DUW/DUR ints, percentage_used, available_spare, media_errors, power_on_hours, power_cycles, unsafe_shutdowns, temperature, critical_warning)
hour_observations — one row per UTC hour: usage-habit split (seconds_active/seconds_idle/seconds_powered_off/seconds_unknown), DUW/DUR deltas, temp min/avg/max, sample count, coverage flag. Classification thresholds belong to the projection model (#4), not the store.
day_aggregates — one row per UTC day (habit-evidence grain), keyed on UTC so derivation from hour rows is monotonic and DST-ambiguous days never exist.
monitoring_periods — started_at, ended_at (NULL = open), end_cause enum (user_disabled, migrated, …). Powered-off time stays inside a period; deliberately disabled time does not.
controller_segments — boundaries where controller identity changes or DUW decreases; write deltas never cross a segment.
endurance_baseline — verified rated-TBW override in bytes plus provenance (source URL, doc revision, entry date), edited via the CLI; /etc/fenris/ keeps operational config only.
Projections are not stored (recomputed on read); no latest-status table.

Retention: raw samples 14 days, pruned opportunistically by the collector; hour observations and day aggregates indefinite.

Migration (runs on first new-version collection): no-op if the legacy-import marker exists; history.jsonl is the sole authority (import raw, derive hours and days, ignore hourly.jsonl — diff and log mismatches only); one implicit monitoring period opens at the first legacy sample and closes with end_cause = migrated; single transaction (interruption leaves the DB fully pre- or post-migration); legacy files renamed to *.migrated only after commit, never deleted; malformed lines quarantined with a logged count, never silently dropped.

Versioning: PRAGMA user_version plus ordered transactional migration steps; the collector refuses to run against an unknown newer version.

Collector health: not stored — failures go to the journal per #7; the freshest sample timestamp is the store's staleness signal.

Assets:

**Resolution** — decided by grilling, all questions approved with recommendations. **Substrate & access**: one SQLite database in WAL mode at `/var/lib/fenris/observations.db`; root-owned, group-readable via the packaging-created `fenris` read group; the TUI opens it read-only. No `/run` snapshot layer. **Entities**: - `samples` — recent raw SMART samples (ts, controller identity, raw DUW/DUR ints, percentage_used, available_spare, media_errors, power_on_hours, power_cycles, unsafe_shutdowns, temperature, critical_warning) - `hour_observations` — one row per UTC hour: usage-habit split (`seconds_active`/`seconds_idle`/`seconds_powered_off`/`seconds_unknown`), DUW/DUR deltas, temp min/avg/max, sample count, coverage flag. Classification thresholds belong to the projection model ([#4](https://git.bongbetic.com/xavierk/Fenris/issues/4)), not the store. - `day_aggregates` — one row per UTC day (habit-evidence grain), keyed on UTC so derivation from hour rows is monotonic and DST-ambiguous days never exist. - `monitoring_periods` — `started_at`, `ended_at` (NULL = open), `end_cause` enum (`user_disabled`, `migrated`, …). Powered-off time stays inside a period; deliberately disabled time does not. - `controller_segments` — boundaries where controller identity changes or DUW decreases; write deltas never cross a segment. - `endurance_baseline` — verified rated-TBW override in bytes plus provenance (source URL, doc revision, entry date), edited via the CLI; `/etc/fenris/` keeps operational config only. - Projections are **not** stored (recomputed on read); no latest-status table. **Retention**: raw samples 14 days, pruned opportunistically by the collector; hour observations and day aggregates indefinite. **Migration** (runs on first new-version collection): no-op if the legacy-import marker exists; `history.jsonl` is the sole authority (import raw, derive hours and days, ignore `hourly.jsonl` — diff and log mismatches only); one implicit monitoring period opens at the first legacy sample and closes with `end_cause = migrated`; single transaction (interruption leaves the DB fully pre- or post-migration); legacy files renamed to `*.migrated` only after commit, never deleted; malformed lines quarantined with a logged count, never silently dropped. **Versioning**: `PRAGMA user_version` plus ordered transactional migration steps; the collector refuses to run against an unknown newer version. **Collector health**: not stored — failures go to the journal per [#7](https://git.bongbetic.com/xavierk/Fenris/issues/7); the freshest sample timestamp is the store's staleness signal. **Assets**: - [ADR 0001 — Observation store: a single SQLite database](https://git.bongbetic.com/xavierk/Fenris/src/branch/main/docs/adr/0001-observation-store-sqlite.md) - [CONTEXT.md — new glossary terms: observation store, hour observation, day aggregate, controller segment, endurance baseline](https://git.bongbetic.com/xavierk/Fenris/src/branch/main/CONTEXT.md)

xavierk closed this issue

2026-08-31 09:18:52 +00:00

xavierk referenced this issue

2026-08-31 09:18:54 +00:00

Chart Fenris’s persistent TUI monitoring redesign #1

Sign in to join this conversation.