Define trustworthy hourly history and first-data availability #66

New Issue

2026-09-11T04:08:37Z

xavierk commented

2026-09-11 04:08:37 +00:00

Parent map: Fenris TUI polish and hourly history

Question

What end-to-end observation-history contract will let real collection populate hourly usage promptly, without fabricating missing evidence or altering projection-confidence rules?

HITL grilling + domain-modeling. Investigate existing local code before asking the human for decisions; this is not an implementation task.

Starting evidence (source inspection, deployed store unverified): src/fenris/collect.py:103–126 invokes run_collection; src/fenris/collector.py:278–325 writes samples but does not invoke hour/day derivation. src/fenris/tui.py:119–161 consumes day aggregates only. Existing hour classification/day derivation and store tables should be evaluated for reuse. History has no projection warm-up gate; zero-valued sparklines can also appear blank.

Resolve: responsibility and timing for sample → hour observation → day aggregate; first honest visible point and partial-hour handling; UTC storage versus display-day boundaries; sample/counter support, true zero versus unknown, paused intervals, missing samples, resets and controller changes; safe handling of pre-existing raw history without invented hourly allocation; historical scope/retention requirements. Specify why counters/history are unavailable independently of lifespan readiness. Fixing the missing aggregation path belongs in the future implementation contract, not in this session.

Answer must state freshness/data-availability guarantees, preservation constraints, and concrete examples that the graph and warm-up decisions can consume. Do not claim waiting alone repairs missing aggregation.

Parent map: [Fenris TUI polish and hourly history](https://git.bongbetic.com/xavierk/Fenris/issues/65) ## Question What end-to-end observation-history contract will let real collection populate hourly usage promptly, without fabricating missing evidence or altering projection-confidence rules? HITL grilling + domain-modeling. Investigate existing local code before asking the human for decisions; this is not an implementation task. Starting evidence (source inspection, deployed store unverified): `src/fenris/collect.py:103–126` invokes `run_collection`; `src/fenris/collector.py:278–325` writes samples but does not invoke hour/day derivation. `src/fenris/tui.py:119–161` consumes day aggregates only. Existing hour classification/day derivation and store tables should be evaluated for reuse. History has no projection warm-up gate; zero-valued sparklines can also appear blank. Resolve: responsibility and timing for sample → hour observation → day aggregate; first honest visible point and partial-hour handling; UTC storage versus display-day boundaries; sample/counter support, true zero versus unknown, paused intervals, missing samples, resets and controller changes; safe handling of pre-existing raw history without invented hourly allocation; historical scope/retention requirements. Specify why counters/history are unavailable independently of lifespan readiness. Fixing the missing aggregation path belongs in the future implementation contract, not in this session. Answer must state freshness/data-availability guarantees, preservation constraints, and concrete examples that the graph and warm-up decisions can consume. Do not claim waiting alone repairs missing aggregation.

xavierk added this to the Wayfinder: Fenris TUI polish and hourly history milestone 2026-09-11 04:08:37 +00:00

xavierk added the wayfinder:grilling label 2026-09-11 04:08:37 +00:00

xavierk added a new dependency 2026-09-11 04:08:51 +00:00

#68 Choose daily graph encoding and hourly drill-down

xavierk added a new dependency 2026-09-11 04:08:51 +00:00

#70 Approve the Fenris TUI polish specification and handoff

xavierk self-assigned this 2026-09-11 04:54:36 +00:00

xavierk commented

2026-09-11 04:58:19 +00:00

Decision checkpoint — not a resolution

Human answers in live grilling:

Selected day means the user's local calendar day, not a UTC display day.
Keep raw samples for 14 days, hour observations/day aggregates indefinitely, with all retained history browsable. Display ranges limit the viewport, not retention. Missing detail is unavailable, not reconstructed from daily totals.

Still open: timezone selection/history behavior; sufficient retained evidence for exact local boundaries (including fractional UTC offsets and DST); first visible point and partial intervals; safe cross-boundary/gap attribution; aggregation publication/recovery guarantees. UTC storage/projection-day contracts have not been changed by the local-calendar display choice. Ticket remains open and claimed pending further live decisions.

Read-only source scout confirms normal collection writes raw samples but does not derive hours/days (src/fenris/collector.py:268–325); history UI reads persisted days (src/fenris/tui.py:119–161). Existing derivation helpers require correction before reuse. This is source evidence only; deployed version/store not inspected. Waiting alone is not a repair.

## Decision checkpoint — not a resolution Human answers in live grilling: - Selected day means the user's **local calendar day**, not a UTC display day. - Keep raw samples for **14 days**, hour observations/day aggregates **indefinitely**, with all retained history browsable. Display ranges limit the viewport, not retention. Missing detail is unavailable, not reconstructed from daily totals. Still open: timezone selection/history behavior; sufficient retained evidence for exact local boundaries (including fractional UTC offsets and DST); first visible point and partial intervals; safe cross-boundary/gap attribution; aggregation publication/recovery guarantees. UTC storage/projection-day contracts have not been changed by the local-calendar display choice. Ticket remains open and claimed pending further live decisions. Read-only source scout confirms normal collection writes raw samples but does not derive hours/days (`src/fenris/collector.py:268–325`); history UI reads persisted days (`src/fenris/tui.py:119–161`). Existing derivation helpers require correction before reuse. This is source evidence only; deployed version/store not inspected. Waiting alone is not a repair.

xavierk commented

2026-09-11 05:02:40 +00:00

Decision checkpoint — live recommendations accepted, not final resolution

Human accepted Q3–Q6:

Display uses current system timezone, visibly labelled; history regroups when that timezone changes. UTC storage and UTC projection-confidence days remain unchanged. DST calendar days can contain 23/25 hours; repeated local hours have distinct offsets.
Retain timestamped usage intervals indefinitely alongside hour/day summaries; full raw samples retain the 14-day policy. Boundary-crossing bytes whose split is unknowable remain explicitly unallocated rather than proportionally divided. Existing history lacking sufficient detail remains incomplete. This extends retained derived evidence; it does not promise exact local allocations from sparse counters.
First valid sample pair permits a measured partial-hour value labelled "so far", without extrapolation, only where attribution to that hour is supported. First sample alone reads "Awaiting another sample". Measured zero is 0 B; missing evidence is not zero.
Collector owns sample → interval → hour/day derivation and publishes consistently before reporting collection success. TUI is read-only; next successful refresh sees published updates. No hourly batch delay or projection-readiness gate.

Remaining decisions: ambiguous intervals/gaps; partial elapsed summaries and deliberate-disable exclusions; controller-history browsing; safe repair/publication/pruning constraints for existing stores. No production changes made.

## Decision checkpoint — live recommendations accepted, not final resolution Human accepted Q3–Q6: - Display uses current system timezone, visibly labelled; history regroups when that timezone changes. UTC storage and UTC projection-confidence days remain unchanged. DST calendar days can contain 23/25 hours; repeated local hours have distinct offsets. - Retain timestamped usage intervals indefinitely alongside hour/day summaries; full raw samples retain the 14-day policy. Boundary-crossing bytes whose split is unknowable remain explicitly unallocated rather than proportionally divided. Existing history lacking sufficient detail remains incomplete. This extends retained derived evidence; it does not promise exact local allocations from sparse counters. - First valid sample pair permits a measured partial-hour value labelled "so far", without extrapolation, only where attribution to that hour is supported. First sample alone reads "Awaiting another sample". Measured zero is `0 B`; missing evidence is not zero. - Collector owns sample → interval → hour/day derivation and publishes consistently before reporting collection success. TUI is read-only; next successful refresh sees published updates. No hourly batch delay or projection-readiness gate. Remaining decisions: ambiguous intervals/gaps; partial elapsed summaries and deliberate-disable exclusions; controller-history browsing; safe repair/publication/pruning constraints for existing stores. No production changes made.

xavierk referenced this issue

2026-09-11 05:10:57 +00:00

Reconcile unallocated usage with UTC projection evidence #71

xavierk commented

2026-09-11 05:12:02 +00:00

Resolution — approved hourly-history and first-data contract

Human chose local-calendar display and existing retention in Q1–Q2, accepted Q3–Q6 recommendations, then accepted all Q7–Q10 edge-case recommendations. This comment consolidates those live decisions and supersedes the interim checkpoints. Planning only: no production implementation or deployed-store diagnosis.

Ownership, publication and first data

Collector owns sample → usage interval → hour observation/day aggregate derivation. Publish a consistent validated result before reporting collection success; readers must not see new samples advertised as fully derived while dependent history is missing. Failure remains explicit with prior valid history preserved. TUI remains read-only: its next successful refresh sees published updates, independent of whether TUI was running during collection. No hourly batch wait and no projection-confidence gate on usage history.

One successful sample establishes counter/health/freshness evidence, not a usage delta: display "Awaiting another sample". First usable sample pair may display measured partial-hour usage "so far" where interval attribution supports it. A usable pair has valid ordered timestamps and supported, nonnegative monotonic counters within the same controller segment; monitored totals additionally require an interval fully inside one monitoring period. A cross-hour pair yields a real interval total, not two invented hour values. No promise that merely waiting five minutes creates a point: default cadence remains unchanged, and collection success plus compatible evidence are prerequisites.

Measured zero is 0 B, visibly distinct from missing evidence. Missing/unsupported/invalid counters are not converted to zero; acquisition failure retains the existing whole-run failure contract. A store fault, insufficient samples, unallocated usage, paused monitoring, derivation/repair failure and unavailable projection evidence are different facts. Final status wording/precedence belongs to Define monitoring signals and honest warm-up estimates. Optional reads remain uncommitted; writes are the graph default. No animation or readiness estimate substitutes for successful published evidence.

Calendar and retained evidence

Storage timestamps and projection evidence days stay UTC. Browsing uses the current system timezone, visibly labelled, and historical grouping changes when that timezone changes. Use actual timezone/calendar boundaries rather than assuming every day lasts 24 hours. Repeated local hours have distinct offsets; DST can produce 23/25-hour days and fractional UTC offsets must work without synthetic splitting.

Retain timestamped usage intervals indefinitely alongside hour observations/day aggregates. Full raw samples retain the 14-day policy, subject to the necessary boundary-anchor exception below. All retained history is browsable; selected display ranges do not delete history. Existing summaries without sufficient detail cannot acquire invented interval/hour precision.

Attribution, elapsed time and boundaries

Preserve measured interval totals. Never divide them proportionally across hours/calendar days or assign them to an endpoint as if that were observed timing. An interval entirely inside a local day and monitoring period can contribute its total to that local-day total even when individual hour shares are unknown. Otherwise show it separately as unallocated; do not present an incomplete allocated subtotal as a complete total. Never double count interval totals and their supported summaries.

Missing samples reduce usage-habit coverage where classification is unknown, but do not erase a compatible measured gap total. Byte-allocation completeness and usage-habit classification coverage are distinct; existing permitted power-on-hours evidence remains the only established gap-classification inference. Do not infer zero writes from missing samples.

Partial summaries represent elapsed time only. Future time is neither zero nor unknown; deliberately disabled time is excluded, not an hour activity state. Intervals crossing a deliberate pause cannot distinguish monitored from paused writes: exclude their ambiguous bytes from monitored totals and explain why, preserving original evidence. Resume needs compatible new evidence; no bridge across a pause. External service stops remain unexplained in-period gaps, not deliberate disables.

Default history view is the current controller segment. Older segments remain browsable with explicit reset/replacement boundaries. Never form deltas across segment boundaries or silently combine different drives. Current history viewport does not redefine the existing projection-history eligibility policy.

Existing-history repair and pruning

Automatically derive only what surviving raw evidence supports, transactionally and idempotently. Preserve original evidence and valid historical summaries; an empty or incomplete reconstruction must not overwrite valid older history. Unsupported historical precision stays explicitly unavailable, not repaired by interpolation or waiting. Existing derivation helpers require correction and boundary validation before reuse.

Prune raw samples only after corresponding durable derivation. Retain the necessary boundary anchors until successor evidence is durably represented; this is an explicit exception to deleting every sample exactly at 14 days. A failed repair must remain visible and preserve retryable evidence. Store-fault/newer-schema refusal and no-destructive-recreation rules remain binding. Legacy import, normal collection, recovery, pruning and read-only consumption must obey one history contract; no TUI-side repair writer.

Concrete acceptance examples for future implementation

10:10 and 10:15 compatible samples, 20 MB increase, one monitoring period: show 20 MB observed so far, not a full-hour extrapolation. An unchanged counter yields visible 0 B for the evidenced interval, not a fabricated zero for the rest of the hour.
10:55 → 11:05, 100 MB increase: preserve 100 MB interval total; hourly split unknown, not 50/50. Count once in a local-day total if wholly within that day/period.
Interval straddles local midnight: preserve total separately; neither local-day share is invented. For UTC+05:30, do not split an existing UTC-hour total at local midnight without supporting interval evidence.
Only first sample, acquisition failures or absent aggregates: distinct insufficient-evidence/failure facts, not blank zero sparklines or a projection-warm-up explanation.
Pause at 10:12, resume 11:00: paused seconds excluded; an interval spanning that boundary does not supply monitored write totals. At 11:10 future minutes are not counted as unknown.
Counter reset or controller replacement at 10:30: prior segment remains browsable; no cross-boundary delta even though both samples fall in one hour/day.
DST fallback: repeated hours distinguished by offsets; changing system timezone regroups the same retained evidence without rewriting measured history.
Repair rerun produces no duplicates; failure preserves usable evidence; retention cannot delete the only evidence needed for unfinished derivation. Older day-only summaries remain day-only when finer evidence no longer exists.

Explicit acceptance amendments and remaining projection decision

This extends ADR 0001 retention with permanent usage intervals and necessary raw boundary anchors; UTC storage/day evidence and long-term hour/day retention remain intact. It amends ADR 0005's fixed 3,600-second validation invariant for partial summaries: validate against represented elapsed time, excluding deliberate-disable time, with no future-time fabrication. This is a companion-spec acceptance amendment, not permission to edit the frozen redesign spec or change projection thresholds.

A newly precise interaction remains: ADR 0002 retains aggregate in-period gap writes but also consumes UTC daily evidence. Exact allocation across UTC midnight, projection horizons or partial days cannot be silently invented. Reconcile unallocated usage with UTC projection evidence owns that decision and blocks warm-up/spec approval. No daily allocation algorithm or confidence-rule amendment is implied by this resolution. Graph exploration can proceed using the measured/unallocated display contract above.

Source evidence and glossary asset

Read-only inspection: src/fenris/collector.py:268–325 writes normal raw samples without hour/day derivation; src/fenris/tui.py:119–161 reads persisted days. Existing helper defects include within-hour-only legacy deltas (src/fenris/legacy.py:128–158,277–294), whole-hour period-overlap accounting/current-day future denominator (src/fenris/day_aggregate.py:29–164), and unwired pruning (src/fenris/pruning.py:10–31). These are source-level implementation obligations, not claims about the deployed store/version. Waiting alone cannot repair missing aggregation.

Glossary asset: CONTEXT.md now defines Usage interval, Unallocated usage and Local display day, and sharpens Hour observation/Day aggregate. No production code, release gates or frozen specification changed.

## Resolution — approved hourly-history and first-data contract Human chose local-calendar display and existing retention in Q1–Q2, accepted Q3–Q6 recommendations, then accepted all Q7–Q10 edge-case recommendations. This comment consolidates those live decisions and supersedes the interim checkpoints. Planning only: no production implementation or deployed-store diagnosis. ### Ownership, publication and first data Collector owns sample → usage interval → hour observation/day aggregate derivation. Publish a consistent validated result before reporting collection success; readers must not see new samples advertised as fully derived while dependent history is missing. Failure remains explicit with prior valid history preserved. TUI remains read-only: its next successful refresh sees published updates, independent of whether TUI was running during collection. No hourly batch wait and no projection-confidence gate on usage history. One successful sample establishes counter/health/freshness evidence, not a usage delta: display "Awaiting another sample". First usable sample pair may display measured partial-hour usage "so far" where interval attribution supports it. A usable pair has valid ordered timestamps and supported, nonnegative monotonic counters within the same controller segment; monitored totals additionally require an interval fully inside one monitoring period. A cross-hour pair yields a real interval total, not two invented hour values. No promise that merely waiting five minutes creates a point: default cadence remains unchanged, and collection success plus compatible evidence are prerequisites. Measured zero is `0 B`, visibly distinct from missing evidence. Missing/unsupported/invalid counters are not converted to zero; acquisition failure retains the existing whole-run failure contract. A store fault, insufficient samples, unallocated usage, paused monitoring, derivation/repair failure and unavailable projection evidence are different facts. Final status wording/precedence belongs to [Define monitoring signals and honest warm-up estimates](https://git.bongbetic.com/xavierk/Fenris/issues/67). Optional reads remain uncommitted; writes are the graph default. No animation or readiness estimate substitutes for successful published evidence. ### Calendar and retained evidence Storage timestamps and projection evidence days stay UTC. Browsing uses the current system timezone, visibly labelled, and historical grouping changes when that timezone changes. Use actual timezone/calendar boundaries rather than assuming every day lasts 24 hours. Repeated local hours have distinct offsets; DST can produce 23/25-hour days and fractional UTC offsets must work without synthetic splitting. Retain timestamped usage intervals indefinitely alongside hour observations/day aggregates. Full raw samples retain the 14-day policy, subject to the necessary boundary-anchor exception below. All retained history is browsable; selected display ranges do not delete history. Existing summaries without sufficient detail cannot acquire invented interval/hour precision. ### Attribution, elapsed time and boundaries Preserve measured interval totals. Never divide them proportionally across hours/calendar days or assign them to an endpoint as if that were observed timing. An interval entirely inside a local day and monitoring period can contribute its total to that local-day total even when individual hour shares are unknown. Otherwise show it separately as unallocated; do not present an incomplete allocated subtotal as a complete total. Never double count interval totals and their supported summaries. Missing samples reduce usage-habit coverage where classification is unknown, but do not erase a compatible measured gap total. Byte-allocation completeness and usage-habit classification coverage are distinct; existing permitted power-on-hours evidence remains the only established gap-classification inference. Do not infer zero writes from missing samples. Partial summaries represent elapsed time only. Future time is neither zero nor unknown; deliberately disabled time is excluded, not an hour activity state. Intervals crossing a deliberate pause cannot distinguish monitored from paused writes: exclude their ambiguous bytes from monitored totals and explain why, preserving original evidence. Resume needs compatible new evidence; no bridge across a pause. External service stops remain unexplained in-period gaps, not deliberate disables. Default history view is the current controller segment. Older segments remain browsable with explicit reset/replacement boundaries. Never form deltas across segment boundaries or silently combine different drives. Current history viewport does not redefine the existing projection-history eligibility policy. ### Existing-history repair and pruning Automatically derive only what surviving raw evidence supports, transactionally and idempotently. Preserve original evidence and valid historical summaries; an empty or incomplete reconstruction must not overwrite valid older history. Unsupported historical precision stays explicitly unavailable, not repaired by interpolation or waiting. Existing derivation helpers require correction and boundary validation before reuse. Prune raw samples only after corresponding durable derivation. Retain the necessary boundary anchors until successor evidence is durably represented; this is an explicit exception to deleting every sample exactly at 14 days. A failed repair must remain visible and preserve retryable evidence. Store-fault/newer-schema refusal and no-destructive-recreation rules remain binding. Legacy import, normal collection, recovery, pruning and read-only consumption must obey one history contract; no TUI-side repair writer. ### Concrete acceptance examples for future implementation - 10:10 and 10:15 compatible samples, 20 MB increase, one monitoring period: show 20 MB observed so far, not a full-hour extrapolation. An unchanged counter yields visible `0 B` for the evidenced interval, not a fabricated zero for the rest of the hour. - 10:55 → 11:05, 100 MB increase: preserve 100 MB interval total; hourly split unknown, not 50/50. Count once in a local-day total if wholly within that day/period. - Interval straddles local midnight: preserve total separately; neither local-day share is invented. For UTC+05:30, do not split an existing UTC-hour total at local midnight without supporting interval evidence. - Only first sample, acquisition failures or absent aggregates: distinct insufficient-evidence/failure facts, not blank zero sparklines or a projection-warm-up explanation. - Pause at 10:12, resume 11:00: paused seconds excluded; an interval spanning that boundary does not supply monitored write totals. At 11:10 future minutes are not counted as unknown. - Counter reset or controller replacement at 10:30: prior segment remains browsable; no cross-boundary delta even though both samples fall in one hour/day. - DST fallback: repeated hours distinguished by offsets; changing system timezone regroups the same retained evidence without rewriting measured history. - Repair rerun produces no duplicates; failure preserves usable evidence; retention cannot delete the only evidence needed for unfinished derivation. Older day-only summaries remain day-only when finer evidence no longer exists. ### Explicit acceptance amendments and remaining projection decision This extends ADR 0001 retention with permanent usage intervals and necessary raw boundary anchors; UTC storage/day evidence and long-term hour/day retention remain intact. It amends ADR 0005's fixed 3,600-second validation invariant for partial summaries: validate against represented elapsed time, excluding deliberate-disable time, with no future-time fabrication. This is a companion-spec acceptance amendment, not permission to edit the frozen redesign spec or change projection thresholds. A newly precise interaction remains: ADR 0002 retains aggregate in-period gap writes but also consumes UTC daily evidence. Exact allocation across UTC midnight, projection horizons or partial days cannot be silently invented. [Reconcile unallocated usage with UTC projection evidence](https://git.bongbetic.com/xavierk/Fenris/issues/71) owns that decision and blocks warm-up/spec approval. No daily allocation algorithm or confidence-rule amendment is implied by this resolution. Graph exploration can proceed using the measured/unallocated display contract above. ### Source evidence and glossary asset Read-only inspection: `src/fenris/collector.py:268–325` writes normal raw samples without hour/day derivation; `src/fenris/tui.py:119–161` reads persisted days. Existing helper defects include within-hour-only legacy deltas (`src/fenris/legacy.py:128–158,277–294`), whole-hour period-overlap accounting/current-day future denominator (`src/fenris/day_aggregate.py:29–164`), and unwired pruning (`src/fenris/pruning.py:10–31`). These are source-level implementation obligations, not claims about the deployed store/version. Waiting alone cannot repair missing aggregation. Glossary asset: `CONTEXT.md` now defines Usage interval, Unallocated usage and Local display day, and sharpens Hour observation/Day aggregate. No production code, release gates or frozen specification changed.

xavierk closed this issue

2026-09-11 05:12:03 +00:00

xavierk added a new dependency 2026-09-11 05:12:04 +00:00

#71 Reconcile unallocated usage with UTC projection evidence

xavierk referenced this issue

2026-09-11 05:12:26 +00:00

Fenris TUI polish and hourly history #65

xavierk referenced this issue

2026-09-13 20:11:30 +00:00

Define monitoring signals and honest warm-up estimates #67

xavierk referenced this issue

2026-09-13 20:31:27 +00:00