Reconcile unallocated usage with UTC projection evidence #71
Notifications
Due Date
No due date set.
Blocks
Depends on
#67 Define monitoring signals and honest warm-up estimates
xavierk/Fenris
#70 Approve the Fenris TUI polish specification and handoff
xavierk/Fenris
Reference: xavierk/Fenris#71
Reference in New Issue
Block a user
Parent map: Fenris TUI polish and hourly history
Question
How do retained usage intervals whose exact UTC-hour/day allocation is unknown contribute to UTC day aggregates, projection rates, coverage and qualifying-day evidence without fabricating allocations or changing the accepted lifespan mathematics/confidence thresholds?
HITL grilling + domain-modeling. Planning only. Read the resolution of Define trustworthy hourly history and first-data availability, CONTEXT.md, ADR 0001, ADR 0002 and ADR 0005. Claim before work; live human answers required.
The accepted history contract retains measured interval totals, does not split bytes proportionally across hours/days, excludes ambiguous pause-crossing bytes from monitored totals, displays current-system local calendar days, and leaves UTC projection evidence/thresholds intact. ADR 0002 also preserves aggregate counter deltas through unexplained in-period gaps and uses UTC daily write rates for habit change and qualification. Their interaction needs an explicit decision, not an implementation guess.
Resolve with examples:
Answer must state exact eligibility/accounting and unavailability facts, plus acceptance examples. Distinguish missing byte allocation from unknown activity classification. If the existing contracts cannot coexist, explicitly identify the minimal proposed amendment and obtain human approval; do not silently alter thresholds, lifespan math, or gap preservation.
This blocks the qualifying-evidence/ETA decision and final companion-spec approval. It does not block exploring graph encoding under the already accepted measured/unallocated display contract. No production code, release gates, or runtime diagnosis.
Decision checkpoint — Q1/Q2 accepted, not final resolution
Human accepted both recommendations in live grilling:
Remaining: practical availability consequences of boundary ambiguity; exact window/evidence endpoints; partial-day qualification and elapsed denominators; same-day reset/replacement and legacy day-only eligibility; concrete acceptance reconciliation. No final resolution or approval of unasked details implied.
Read-only source audit found no explicit unknown-write-total representation in projection inputs (
src/fenris/projection.py:131–146,228–250), UTC-midnight-rounded horizon starts with read-clock denominators (:228–250), and date-only segment filtering plus unfiltered scenario history (:481–516). Current source is not authoritative where it conflicts with ADR 0002. No production edits or runtime diagnosis.Decision checkpoint — Q3–Q6 accepted, not final resolution
Human accepted all recommendations in the next live round:
Still reconciling final edge cases and exact acceptance examples; ticket remains open and claimed. No production changes.
Resolution — approved UTC projection accounting contract
Human accepted Q1–Q2, Q3–Q6, then Q7–Q9 in live grilling. This comment consolidates those decisions and supersedes both interim checkpoints. Planning only: no production implementation, release gates, or deployed-store diagnosis.
1. Measured totals, allocation and eligibility
Retained usage intervals are measured evidence, not estimated daily allocations. For any requested span, count a compatible interval's total exactly once when its complete span is inside that requested span and one monitoring period, with eligible controller provenance. It may supply a complete window total even when its individual UTC-hour/day shares are unknown. Do not add the same bytes again through hour/day summaries; valid summaries and their underlying evidence are alternative representations, not additive sources.
A positive interval crossing a requested boundary cannot supply that window's unknown share. Preserve its measured total separately as unallocated usage. Never split proportionally, assign to the ending day, or silently drop its possible contribution and label the subtotal a complete total. A day aggregate must distinguish a known allocated subtotal from a complete daily total, with explicit missing/unallocated evidence.
Withhold a rate whenever its monitored numerator cannot be established: unresolved boundary shares, missing initial/resume/reset counter support, or legacy eligibility ambiguity are not zero. Do not shorten the requested window or remove unknown monitored seconds merely to obtain a number. A compatible recovery interval wholly inside the requested span can establish aggregate gap writes without establishing daily distribution or activity classification. Missing activity classification alone does not invalidate a measured aggregate total.
A compatible, monotonic interval with zero counter delta proves zero writes throughout its represented span, including a requested subspan. This is direct counter evidence, not proportional interpolation. It does not classify activity or prove anything outside that span.
2. Projection endpoints and denominator
Let T be the latest published usage-evidence endpoint. The 7/28/90-day scenario windows end at T and start exactly 7/28/90 × 86,400 seconds earlier. Do not round the start to UTC midnight or advance the rate denominator merely because a reader refreshes. Evidence age remains a separate fact.
The default sustained regime is eligible observation history capped at 90 days, ending at T. A detected habit-change regime starts at the first divergence day's UTC midnight. Any unresolved share at those boundaries invokes the same unavailable-rate rule; there is no silent fallback to a more convenient start.
For the chosen span, the rate denominator is wall-clock seconds inside monitoring periods, including powered-off and unknown time, excluding deliberate-disable time. Numerator and denominator describe the same requested span. An unexplained in-period gap stays in the denominator, with measured aggregate recovery writes retained when supported. A pause-crossing interval cannot establish which writes were monitored: preserve original evidence, exclude its ambiguous bytes from monitored totals, and withhold any rate that consequently lacks a complete monitored numerator. No bridge across controller-segment boundaries.
Elapsed unknown time and staleness must remain visible; a frozen rate endpoint is not proof that collection remains healthy. Future time is never included in partial summaries.
3. UTC evidence, partial dates and exact warm-up gate
Projection evidence remains UTC; local graph regrouping never changes eligibility. Byte-allocation completeness and classification coverage are independent.
Coverage uses known-classified seconds divided by represented elapsed monitored seconds, excluding deliberately disabled and future time. Use elapsed seconds, not an assumed full day or unweighted full-day percentages. Existing classification rules remain binding; no new inference from missing samples is authorized.
A qualifying day is a distinct UTC date whose represented monitored span has coverage at least 50%. A current partial date counts provisionally and can lose qualification as unknown time accumulates. Count a date once, not once per hour, interval, or controller fragment. Entirely paused dates have no represented monitored span and do not count. Do not add a separate, undocumented requirement for a sample inside each qualifying date when permitted classification evidence establishes coverage.
Warm-up clears when the current controller segment has at least 14 distinct represented UTC dates, at least 12 of which qualify. Old poor dates do not permanently veto this gate. Supported still requires at least 14 qualifying dates in the current segment plus every other existing condition. Thus 12 qualifying + 2 poor dates clears warm-up but remains Limited. Same-day segment breaks use only the current segment's eligible portion for its new-segment warm-up; a shared UTC date does not import old-segment qualification.
Habit-change comparisons require completed, consecutive UTC days with evaluable daily write totals. Do not compress missing calendar dates into adjacent-row windows or treat missing/disabled time as zero daily usage. The existing 7-versus-preceding-28-day means, factor-of-two trigger and three-consecutive-day requirement remain unchanged. Regime age is elapsed age, not stored-row count. The current unfinished date cannot establish a completed-day habit change.
4. Confidence and unavailability
An affected scenario rate is withheld while other independently computable rates remain visible. If the headline regime lacks a complete monitored numerator, no lifespan number renders; supply its specific evidence-unavailable reason rather than a zero-rate or generic warming-up explanation.
A complete positive headline rate may render while daily shares remain unknown, but dependent checks cannot silently pass: unknown daily totals cannot establish habit change or pass the burst/concentration guard. Supported is blocked whenever a required confidence check cannot be established.
Only horizons old enough to be covered by eligible history participate. There is no new 28/90-day minimum-history gate, and no added minimum count of available horizons. Distinguish a not-yet-reached horizon from one whose calendar extent exists but whose bytes are unresolved: omitting the latter's number must not silently remove a failed prerequisite for Supported. Apply the existing burst guard to the observed eligible portion of the trailing 28 days; short history does not skip the guard, and unknown daily totals block it.
Keep existing baseline, positive-rate, 80% coverage, freshness, factor-of-two horizon agreement, burst fraction, young-regime, degraded-identity and segment gates. In particular, zero measured rate keeps its existing no-finite-projection meaning; an unknown rate is a different fact. Keep the existing endurance formula and cumulative W_t operand; do not substitute regime writes for cumulative endurance consumption.
Accepted availability trade-off: ordinary samples may cross every midnight, and exact daily shares can remain unknowable indefinitely. Likewise a requested horizon may cut a positive interval. No promise that waiting 14 days repairs these facts or produces Supported confidence. This effort adds no interpolation, collector-cadence change, or allocation-bounds engine. First graph data remains independent of lifespan readiness.
5. Controller segments and older summaries
Never compute a delta across a reset/replacement boundary, even within one UTC hour/day. Same-identity reset retains prior eligible habit evidence, subject to the existing current-segment re-warm gate before a lifespan number can render. Re-warming does not itself reconstruct missing counter support; the numerator-completeness rule still applies.
Controller-identity change, including to/from a blank key, quarantines previous identity from every projection horizon, not merely the headline. Returning to a previously seen key does not retroactively undo an intervening identity-change quarantine. History browsing may still show older segments explicitly.
Preserve trusted legacy day-only summaries at their actual represented precision. A summary can supply a total only when the requested window contains its entire represented span and its monitoring-period/controller eligibility is known. Never invent interval/hour precision or cut a day-only total at a rolling boundary. Mixed legacy summaries that cannot separate eligible from ineligible writes remain browsable but cannot supply affected projection totals. Missing metadata is not guessed. Valid historical evidence must not be erased by a less complete reconstruction; existing repair, transactional publication, retention and store-fault guarantees remain binding.
6. Acceptance examples for the companion specification
7. Explicit amendments and handoff
This is an explicit companion-spec amendment to ADR 0002: allocation/missing-numerator unavailability; evaluability requirements for existing confidence tests; exact evidence-anchored windows; provisional UTC-day qualification; and the clarified 14-distinct/12-qualifying warm-up gate separate from 14-qualifying Supported. Numeric thresholds and lifespan mathematics are not changed. Completed consecutive-day habit comparisons and correct controller quarantine enforce existing contracts rather than preserving source defects. The prior history decision's ADR 0001 retention and ADR 0005 partial-summary amendments remain binding.
Read-only source evidence:
src/fenris/projection.py:131–146,228–250lacks unknown-byte inputs and rounds date windows;:345–349,518–524implements inconsistent warming/qualifying behavior;:481–516uses date-only segment filtering and unfiltered scenario history;:548–561substitutes regime bytes for cumulative W_t.src/fenris/day_aggregate.py:34–60,98–165needs elapsed-time and completeness-aware accounting. These are future implementation obligations, not deployed-runtime claims or permission to expand this planning effort into a bug-fix campaign.Glossary asset:
CONTEXT.mdadds Byte-allocation completeness and Qualifying day; pre-existing glossary edits are preserved. Frozendocs/spec/fenris-redesign.mdand production files are unchanged by this session. Decision detail lives here, not duplicated on the parent map.Define monitoring signals and honest warm-up estimates now has the accounting facts needed for its status/precedence table and conditional ETA decisions. No new decision ticket is needed: remaining UI wording, combined states, layout and final specification approval already have owners.