Record approved browser modes in research scope

Research browser shell and language workload validity
2026-09-25 23:47:06 +05:30 · 2026-09-25 23:44:03 +05:30
2 changed files with 135 additions and 139 deletions
@@ -0,0 +1,135 @@
 # Establish browser, shell, and language workload validity
 Research for [Establish browser, shell, and language workload validity](https://git.bongbetic.com/xavierk/odin/issues/5), part of [Find the way to Odin’s build-ready specification](https://git.bongbetic.com/xavierk/odin/issues/1).
 Access date: **2026-09-25**. This is a candidate analysis, not the final suite. No browsers, toolchains, or dependencies were installed; no benchmarks or builds were run.
 ## Recommendation for the selection decision
 Preserve five distinct measurement purposes: browser application responsiveness, shell interaction, process startup, compilation, and warmed application execution. A single fast loop cannot represent all five. Retain upstream benchmark names, versions, metrics, and correctness rules; an Odin aggregate is a separate decision.
 A practical initial option is **Speedometer 3.1**, a small Bash latency/throughput set, and optional language packs with pinned workload inputs. JetStream, MotionMark, representative compilation projects, and larger JVM applications fit extended investigation. Installed software answers “how usable is this machine as configured”; pinned reference software improves comparison across machines. Store these as distinct modes rather than treating them as interchangeable.
 ## 1. Browser candidates and official measurement rules
 BrowserBench’s live index currently points to **Speedometer 3.1**, **JetStream 3.0**, and **MotionMark 1.3.2**. [1]
 | Candidate | What it measures | Conditional role and interpretation |
 | --- | --- | --- |
 | Speedometer 3.1 | End-to-end simulated web-app interactions: TodoMVC frameworks, editors, charts, and news applications. | Strong default browser-responsiveness candidate. Preserve the full default suite and iteration protocol for an upstream-style result. It does not measure browser process startup or network loading. |
 | JetStream 3.0 | JavaScript/WebAssembly workloads, combining startup, average, and worst-case execution behavior. | Useful extended engine-compute view. “Startup” is workload first-iteration behavior, not opening the browser. Its 77 default workloads and upstream geometric aggregation should remain intact. |
 | MotionMark 1.3.2 | Browser graphics scenes adjusted toward a target frame rate, with confidence intervals. | Useful optional browser-rendering view where a suitable graphics/display path exists. It is not a standalone GPU benchmark or evidence that the intended hardware driver is active. |
 Speedometer’s release/3.1 source currently resolves to commit `1386415be8fef2f6b6bbdbe1828872471c5d802a`. Its `package.json` still says `3.0.0-alpha`, demonstrating why package metadata alone is not a benchmark identity. The release source defaults to **10 iterations** and an **800×600 workload viewport**; URL parameters can change iterations, suites, warmup, and measurement behavior. The result includes individual measurements and JSON exports. Preserve these settings and raw results; do not silently shorten the suite and report an ordinary “Speedometer 3.1” score. [2]
 Official Speedometer instructions recommend a latest stable browser, a separate clean profile, closed background applications/tabs, a focused benchmark page, AC power, no interaction, and cooling between runs where needed. A pinned historical browser is useful for longitudinal comparison but must be identified as a reference-browser experiment. Speedometer aggregates inverse geometric-mean durations, then averages scores across iterations and reports a 95% confidence interval around that mean. Retain that native statistic and uncertainty. Any Odin median across complete benchmark runs is a separate statistic; do not relabel either or average heterogeneous subtest times. [2][3]
 JetStream explicitly says scores are not comparable across its versions. Its startup, worst-case, and average components are deliberately part of the metric; generic harness warmups must not erase those startup observations. MotionMark depends on drawing-area class, viewport, refresh behavior, and browser rendering. Its instructions recommend maximizing the window at the default resolution and 60 Hz for comparison across browsers; a changed display policy is a recorded experimental condition, not something Odin should silently impose. [4][5]
 ### Offline hosting and licensing
 Speedometer documents a local static HTTP server, so a prepared source/assets snapshot can be served on loopback without fetching the public site during measurement. Its root license permits source/binary redistribution with the copyright, conditions, and disclaimer retained. That does not replace the license notices of bundled frameworks, fonts, datasets, or other third-party assets. JetStream and MotionMark’s deployed HTML also carries permissive two-condition notices, while their workload provenance points to multiple projects. Before distributing a pack, inventory the exact selected files and retain their individual notices. [2][4][5][6]
 Recommended preparation contract: content-addressed assets, release/commit identity, dependency/license manifest, complete local resource closure, and a future offline-network audit. Bind the server to loopback and prepare everything before timing. A copied public page without its subresources is not a verified offline benchmark. Keep upstream workload code unchanged where feasible; record every adapter patch. The existence of localhost support does not prove the entire pack is self-contained until that artifact is checked.
 ### Automation and the Linux boundary
 Two viable routes serve different purposes:
 1. **Native installed browser plus matching WebDriver.** Speedometer itself documents Selenium testing with ChromeDriver, GeckoDriver, and other browser drivers. This route can use distro-supported native browsers on Void/musl or other systems outside Playwright’s matrix, provided the actual browser/driver pair exists and is qualified. Use explicit browser and driver paths and versions. Selenium Manager can download software by default; its documented offline mode disables network requests/downloads. Its documentation still describes limited Linux architecture support, so automatic management cannot be presumed to cover aarch64 or musl. Native automation remains a capability to prove, not a promise. [7]
 2. **Pinned browser supplied with Playwright on its supported platforms.** Current documentation lists Debian 12/13 and Ubuntu 22.04/24.04/26.04 on x86-64/arm64. Its bundled Firefox/WebKit builds require glibc; musl distributions are unsupported. Stock Firefox cannot simply be substituted because Playwright requires its patched Firefox. Branded Chrome/Edge support and custom executable paths also do not establish universal compatibility. [8]
 **Odin can launch on a system even when its browser module is unavailable.** On musl, use a qualified native browser/driver pair or report an explicit capability skip. A glibc container/chroot or software-rendered browser is a separate environment and cannot transparently stand in for native desktop usability.
 The user has approved **both headed and headless browser modes**, with terminal controls/results and distinct measurement labels. Chrome’s unified headless mode shares browser code with headed Chrome, but Playwright also provides a distinct headless-shell binary and documents behavioral differences. Sharing code does not establish equal scores, display scheduling, GPU access, or compositor behavior. Label a headless cohort explicitly and avoid claiming compliance with focused-window desktop run conditions. A headed cohort needs a graphical session and the official run conditions. Do not merge their reference distributions or invent a conversion factor. The default mode for each run profile remains a selection decision. [8][9]
 Automation should start the pinned page, observe completion, and collect its full result without repeated polling/instrumentation inside timed work. Correctness failures, missing workloads, or timeouts invalidate a full-suite result. Shortened or filtered runs can be Odin browser probes with their own identities; they are not substitutes for official full-suite statistics.
 Record browser binary/version/channel, driver and automation versions, full flags, profile policy, headed/headless/shell distinction, viewport/device scale, display resolution/refresh, X11/Wayland/virtual display, renderer and acceleration evidence when exposed, kernel/libc/architecture, power state, suite digest, and all query parameters. An unavailable GPU detail is unknown, not proof of hardware acceleration. Browser startup, if desired, should be a separate process-to-ready-page test with its cache/profile policy declared.
 ## 2. Meaningful Bash and shell measurements
 Bash startup depends on invocation. Login shells read `/etc/profile` and the first readable user login profile; interactive non-login shells read `.bashrc`; noninteractive shells may execute the file named by `BASH_ENV`. `--noprofile` and `--norc` control different paths. Therefore “time bash” is underspecified. [10]
 | Candidate | Measurement definition | Boundary to preserve |
 | --- | --- | --- |
 | Clean process startup | Directly launch a known Bash binary with controlled environment/startup-file policy and a trivial command; measure process launch through exit. | This includes loader/process/exit costs. It is not prompt readiness. Explicitly control `BASH_ENV`, inherited functions, locale, and working directory. |
 | Interactive readiness | Launch a clean interactive shell under a PTY; measure until a declared prompt-ready marker, then measure prompt return after a builtin command. | Account for PTY setup and marker instrumentation. `bash -i -c exit` does not observe a real prompt. Test login and non-login separately if both are offered. |
 | Builtin workload throughput | Fixed arithmetic, parameter expansion, arrays, and parsing over seeded in-memory inputs, returning a verified checksum. | Bounded output; no per-iteration `date`, `cat`, `grep`, or other subprocesses. Input size and Bash version are part of the workload identity. |
 | External-command/pipeline workload | Fixed commands, input corpus, and pipe structure, with exact executable versions. | Measures the combined shell, process-launch, tool, and I/O stack. Report it as such. |
 | User configuration readiness | Opt-in observation using the user’s real startup files and prompt configuration. | These files execute arbitrary user-configured commands, may contact networks or change state, and may not be repeatable. Report local usability separately from the clean reference result. |
 **Harness trap:** Hyperfine defaults to an intermediate `/bin/sh` on Unix and calibrates/subtracts shell-spawn time. For Bash startup, use its direct-command mode or another direct process timer so the Bash process is the measured payload. Hyperfine also supports warmups and repetitions; warmed filesystem caches are a declared condition, not “cold boot.” Do not globally drop caches or alter system tuning just to manufacture a shell metric. [11]
 Treat time-to-first-prompt and prompt-to-next-prompt as distributions, not a single best sample. Bound captured PTY output, support cancellation, and keep configured-shell timeout/error details. Store configuration fingerprints with care; benchmark records should not copy potentially secret startup-file contents.
 ## 3. Language workload candidates
 Each result describes a **workload + implementation + toolchain + environment**, not an intrinsic language ranking. Numeric kernels are useful but should not be the only evidence. Libraries, parsing, allocation, and application behavior matter; Python’s C-backed JSON module and a Rust regex library do not represent the same implementation simply because both are called language tests.
 | Language / candidate | Useful workload selection | Evidence, tradeoffs, and role |
 | --- | --- | --- |
 | Python: pyperformance + pyperf | `python_startup`, `python_startup_no_site`, `json_loads`/`json_dumps`, one regex workload, and optionally a pure-Python workload such as `richards`. | Maintained Python project favors real applications. Current PyPI metadata: pyperformance **1.14.0**, Python ≥3.10; pyperf **2.10.0**, Python ≥3.9. Suite docs contain older requirements, so prefer release metadata. MIT project license; inspect selected workload notices. Small subsets fit standard profiles. [12] |
 | Rust: rustc-perf subsets | Runtime groups include parsing, text search, compression, hash maps, and numeric/graphics kernels; compile candidates include pinned real crates such as `syn`. | Official compiler-performance project clearly separates compilation from generated-program execution. Runtime suite is explicitly **experimental**, with changing workloads; pin a commit and validate chosen outputs. MIT infrastructure, separate compile-benchmark licenses. Avoid adopting the full collector by default: it has profiling/environment dependencies. [13] |
 | C++: LLVM test-suite subset | A small C++ application/proxy workload such as the serial miniFE variant, plus a smaller verified workload; optionally time compiling the same pinned sources. | Suite compiles/runs whole programs, checks reference output, and records compile and execution times separately. CMake/lit and dependencies increase setup cost. Apache-2.0 with LLVM exceptions for project code; third-party directories have separate terms. Good standard/extended candidate after footprint qualification. [14] |
 | Java: selected workloads under JMH | A packaged, reviewed data-processing/allocation workload with validated results and configured warmup/forks. | JMH **1.37** is the current published Maven release inspected. It is a harness, not a representative suite by itself; samples explain pitfalls rather than defining an Odin score. GPLv2 with the Classpath exception on designated files. Suitable where a small, controlled JVM workload is desired. [15] |
 | Java: DaCapo application subset | `lusearch` for search, `h2` for database-style execution, or another justified application. | Latest inspected release **23.11-MR2-chopin**; application suite with nontrivial memory use and output validation. Release notes establish Java 11–21 compatibility, not arbitrary newer JDKs. Harness Apache-2.0; component programs retain their licenses. Better extended candidate than a compulsory quick test. [16] |
 pyperformance explicitly says it is not tuned for PyPy. CPython, PyPy, free-threaded builds, optional JIT builds, and distro build choices require distinct metadata and qualification. Interpreter startup with and without `site` measures different initialization; preserve both names if offered. Some pyperformance dependencies have native extensions, so a Python package being source-available does not prove wheel availability on every musl/architecture combination. [12]
 ## 4. Fair measurement protocol
 **Startup, compilation, and warmed execution need separate records.** For Rust/C++, compile once before execution measurements unless compilation is the workload. For Java, bytecode compilation with `javac` differs from runtime JIT work. For Python, process startup/imports differ from repeated execution in an initialized interpreter. If the workload intentionally includes setup, say so and apply the same rule on every system.
 pyperf uses calibration, multiple processes, warmups, values, and metadata. Its fast mode explicitly trades accuracy for speed. JMH provides forks, warmup/measurement iterations, state setup, and result consumption to avoid dead-code elimination; its maintainers warn that a harness does not eliminate benchmarking mistakes. Retain cold/startup samples and warmed samples separately. Use predetermined warmup/measurement rules and report instability; do not keep warming until a favorable result appears. [15][17]
 Proposed compilation policy for selection:
 - Pin sources, datasets, dependency lockfiles, language standard/edition, compiler identity, target triple, optimization flags, linker, libc, and library versions.
 - Use a documented release optimization configuration. Cargo’s release defaults include `opt-level=3`, incremental off, and 16 codegen units; its bench profile inherits release. Record overrides and dependency profiles. C++ should use an explicitly chosen optimization level and standard; `-Ofast` changes standards-compliance assumptions and cannot silently replace a strict floating-point contract. [18]
 - Keep baseline-ISA builds and host-tuned builds distinct. `target-cpu=native`/`-march=native`, SIMD dispatch, LTO, PGO, allocator choice, and thread counts can materially change results. A single binary is not portable across x86_64 and aarch64; equal flags do not mean equal generated instructions.
 - Compilation tests distinguish clean builds, no-op rebuilds, and controlled incremental edits. Pre-fetch dependencies and hold build parallelism/cache policy fixed. Network resolution is not compilation speed; compiler caches must be controlled or explicitly measured.
 - Validate outputs/checksums and numerical tolerances. Prevent constant folding/dead work, but avoid timing validation if the workload definition excludes it. Do not compare implementations with different precision, algorithms, data sizes, or hidden thread counts under a shared metric name.
 An **installed-toolchain mode** best reveals current developer experience. A **reference-toolchain mode** uses pinned, prepared artifacts for stronger cross-machine comparison, with acquisition size, libc/ISA support, licenses, and security updates owned explicitly. It must not install into or replace the user’s default toolchain during a run. Report compiler/runtime version changes as changes in execution conditions, not unexplained hardware improvements.
 ## 5. Profile options, budgets, and unavailable tools
 The following are provisional **application-domain budgets**, not measured runtime promises or the whole Odin run duration:
 | Profile | Candidate scope | Budget policy to evaluate |
 | --- | --- | --- |
 | Quick | Browser capability probe; clean Bash startup and one builtin workload; startup/small verified work on selected installed languages. | Approximately 30–60 seconds for this domain. No full browser score if the official suite does not fit. Short observations carry lower-confidence status. |
 | Standard | Full Speedometer 3.1 on one qualified browser; Bash set; selected language pack(s) with normal warmup/repetitions. | Reserve minutes rather than seconds; a pilot target is 2–5 minutes for browser and 1–3 minutes per selected language pack. Slow hosts may exceed it. |
 | Extended | JetStream, optional MotionMark, more process forks, compilation workloads, selected DaCapo applications, additional browsers/toolchains. | User-visible per-module time/memory/disk ceilings; an initial 15–30-minute application allocation needs calibration on low-end physical hardware. |
 Preparation/download/compilation costs are shown separately unless compilation is the named workload. Check tools and resource requirements before beginning. Do not shorten an upstream test behind the user’s back to meet a deadline: mark it incomplete and preserve diagnostics.
 Missing interpreter/compiler/browser/driver, unsupported ABI, unavailable display, insufficient resources, timeout, and failed output validation are different outcomes. None is a zero performance measurement. Skips reduce **capability coverage**; the score-design ticket must determine eligibility for an aggregate. Optional-tool absence should leave completed results usable. No hardware/browser/type of VM was executed here to substantiate duration estimates.
 ## Remaining decisions and evidence gaps
 Both browser modes are already approved. Select the default mode for each profile; the native-browser/driver and reference-browser support matrix; exact upstream full-suite versus Odin-probe identities; language subsets; installed/reference toolchain policy; fixed compilation/warmup rules; and measured profile budgets. Browser automation overhead, all-assets-offline closure, per-asset redistribution inventory, selected-suite behavior on musl/aarch64 and newer JDKs, and correctness/variance qualification remain future proof work. An application score must not conceal these unresolved cohort boundaries.
 ## Sources and method
 Primary sources inspected **2026-09-25**. Context7 resolution preceded queries for Speedometer/WebKit, Bash, Hyperfine, Playwright, pyperformance/pyperf, Rust, LLVM, JMH, GCC, and Selenium. Exact Speedometer and pyperformance lookups returned unrelated entries; those were rejected and their own repositories inspected. Queries stayed within the three-command limit per lookup; no quota error occurred.
 1. [BrowserBench current index](https://browserbench.org/).
 2. Speedometer release source: [commit](https://github.com/WebKit/Speedometer/commit/1386415be8fef2f6b6bbdbe1828872471c5d802a), [parameters](https://github.com/WebKit/Speedometer/blob/1386415be8fef2f6b6bbdbe1828872471c5d802a/resources/params.mjs), [result client](https://github.com/WebKit/Speedometer/blob/1386415be8fef2f6b6bbdbe1828872471c5d802a/resources/main.mjs), [package metadata](https://github.com/WebKit/Speedometer/blob/1386415be8fef2f6b6bbdbe1828872471c5d802a/package.json), [local hosting](https://github.com/WebKit/Speedometer/blob/1386415be8fef2f6b6bbdbe1828872471c5d802a/Development.md).
 3. [Speedometer 3.1 instructions](https://browserbench.org/Speedometer3.1/instructions.html), [workload descriptions](https://browserbench.org/Speedometer3.1/about.html), [measurement/scoring explanation](https://github.com/WebKit/Speedometer/blob/main/README.md).
 4. [JetStream 3.0 implementation page](https://browserbench.org/JetStream3.0/), [in-depth methodology and provenance](https://browserbench.org/JetStream3.0/in-depth.html).
 5. [MotionMark 1.3.2 implementation page](https://browserbench.org/MotionMark1.3.2/), [methodology, display requirements, and version history](https://browserbench.org/MotionMark1.3.2/about.html).
 6. [Speedometer release license](https://github.com/WebKit/Speedometer/blob/1386415be8fef2f6b6bbdbe1828872471c5d802a/LICENSE).
 7. [Speedometer browser-driver testing documentation](https://github.com/WebKit/Speedometer/blob/1386415be8fef2f6b6bbdbe1828872471c5d802a/Testing.md), [Selenium Manager: explicit drivers, offline mode, and architecture limitations](https://www.selenium.dev/documentation/selenium_manager/).
 8. [Playwright system requirements](https://playwright.dev/docs/intro), [browser distributions and headless modes](https://github.com/microsoft/playwright/blob/main/docs/src/browsers.md), [musl limitations](https://github.com/microsoft/playwright/blob/main/docs/src/docker.md).
 9. [Chrome unified headless and headless-shell distinction](https://developer.chrome.com/docs/chromium/headless).
 10. [GNU Bash startup files](https://www.gnu.org/software/bash/manual/html_node/Bash-Startup-Files.html), [invocation options](https://www.gnu.org/software/bash/manual/html_node/Invoking-Bash.html).
 11. [Hyperfine README: intermediate-shell correction, direct mode, warmups, repetitions](https://github.com/sharkdp/hyperfine/blob/master/README.md).
 12. [pyperformance purpose and license](https://github.com/python/pyperformance/blob/main/README.rst), [PyPI release metadata](https://pypi.org/pypi/pyperformance/json), [pyperf release metadata](https://pypi.org/pypi/pyperf/json), [benchmark manifest](https://github.com/python/pyperformance/blob/main/pyperformance/data-files/benchmarks/MANIFEST), [workload descriptions](https://github.com/python/pyperformance/blob/main/doc/benchmarks.rst), [dependency/runtime guidance](https://github.com/python/pyperformance/blob/main/doc/usage.rst), [startup implementation](https://github.com/python/pyperformance/blob/main/pyperformance/data-files/benchmarks/bm_python_startup/run_benchmark.py).
 13. [rustc-perf purpose/licenses](https://github.com/rust-lang/rustc-perf/blob/master/README.md), [compile suite](https://github.com/rust-lang/rustc-perf/blob/master/collector/compile-benchmarks/README.md), [experimental runtime suite](https://github.com/rust-lang/rustc-perf/blob/master/collector/runtime-benchmarks/README.md), [runtime groups](https://github.com/rust-lang/rustc-perf/tree/master/collector/runtime-benchmarks), [collector requirements](https://github.com/rust-lang/rustc-perf/blob/master/collector/README.md).
 14. [LLVM test-suite guide](https://llvm.org/docs/TestSuiteGuide.html), [testing/output-validation model](https://llvm.org/docs/TestingGuide.html), [serial miniFE description](https://github.com/llvm/llvm-test-suite/blob/main/MultiSource/Benchmarks/DOE-ProxyApps-C++/miniFE/README), [license and third-party exceptions](https://github.com/llvm/llvm-test-suite/blob/main/LICENSE.TXT).
 15. [JMH purpose and cautions](https://github.com/openjdk/jmh/blob/master/README.md), [published version metadata](https://repo.maven.apache.org/maven2/org/openjdk/jmh/jmh-core/maven-metadata.xml), [parameter/warmup/fork example](https://github.com/openjdk/jmh/blob/master/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_27_Params.java), [result consumption and profilers](https://github.com/openjdk/jmh/blob/master/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_35_Profilers.java), [license](https://github.com/openjdk/jmh/blob/master/LICENSE).
 16. [DaCapo latest release](https://github.com/dacapobench/dacapobench/releases/tag/v23.11-MR2-chopin), [release compatibility and workload notes](https://github.com/dacapobench/dacapobench/blob/0db32562cf169730c163d88df2eeb28217ca7d03/benchmarks/RELEASE_NOTES.md), [purpose/reporting/license guidance](https://github.com/dacapobench/dacapobench/blob/master/README.md).
 17. [pyperf runner configuration](https://github.com/psf/pyperf/blob/main/doc/runner.rst), [process/calibration architecture](https://github.com/psf/pyperf/blob/main/doc/run_benchmark.rst).
 18. [Cargo profile defaults](https://doc.rust-lang.org/cargo/reference/profiles.html), [rustc code-generation options](https://doc.rust-lang.org/rustc/codegen-options/index.html), [GCC optimization semantics](https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html).
@@ -1,139 +0,0 @@
 # Safe CPU, memory, and kernel measurements for Odin
 Research for [Establish safe CPU, memory, and kernel measurements](https://git.bongbetic.com/xavierk/odin/issues/2), part of [Find the way to Odin’s build-ready specification](https://git.bongbetic.com/xavierk/odin/issues/1).
 **Accessed:** 25 September 2026. **Status:** recommendations for later decisions, not a selected workload suite. No benchmarks, stress tests, privilege changes, or host configuration changes were performed.
 ## Findings that shape the decision
 Odin can reuse established workloads and Linux interfaces, but no single tool measures system usability, peak performance, and hardware health. Keep four outcomes distinct:
 - **Performance measurement:** completed work per second or elapsed time under a specified workload.
 - **Responsiveness:** foreground request or wakeup latency, including its tail, under a specified competing load.
 - **Pressure observation:** time lost to CPU, memory, or I/O contention during that interval.
 - **Health finding:** a detected verification failure or reported hardware error, with the observation's coverage.
 This separation follows the tools' actual contracts: sysbench benchmarks operations; schbench measures artificial requests and scheduling delays; PSI measures stalls; memtester checks memory contents. Stress-ng explicitly says it was **not intended as a precise benchmark suite**. Successful stress completion establishes only that the chosen work completed without detected failures under those conditions. [sysbench][sysbench] [schbench][schbench] [PSI][psi] [memtester][memtester-man] [stress-ng][stress-readme]
 ## Candidate comparison
 Maintenance observations describe inspected releases or repository activity, not a guarantee of future support. GPL notices and third-party dependencies need checking again for the exact distributable artifacts. Perf is distributed in the Linux source tree, whose COPYING specifies GPL-2.0-only with the syscall exception and notes that other licenses may also apply; preserve the selected tool and dependency notices. [Linux COPYING][kernel-license]
 | Candidate | Useful measurement and limitations | Controls, output, portability, maintenance |
 |---|---|---|
 | **sysbench CPU / memory** | CPU events are a small prime-search workload, not a general application-performance model. The memory test defaults to a **1 KiB block**; its `100G` default is cumulative transfer size, not RAM allocation. A default run therefore cannot stand for DRAM bandwidth. | Thread, event, duration, warmup and percentile controls; human-readable built-in reports require a pinned parser. Upstream advertises x86_64 and aarch64 packages. GPL-2.0-or-later. Latest published release inspected: 1.0.20, April 2020; repository push activity March 2025. Void still packages 1.0.20. [README][sysbench] [CPU source][sysbench-cpu] [memory source][sysbench-memory] [release][sysbench-release] [metadata][sysbench-meta] [Void][void-sysbench] |
 | **schbench** | Reports synthetic request latency, wakeup latency, and requests/second. Closer to responsiveness than peak throughput. Its server-inspired matrix workload deliberately penalizes preemption using per-CPU locks; this is not a desktop interaction model. | Runtime, worker/message counts, work size, request rate, affinity and JSON percentiles. Source has x86 and aarch64 paths; Linux pthread/futex interfaces, no normal root requirement found. Musl behavior remains unverified. GPLv2. Inspected upstream commit `6300b8f`, June 2025. [methodology][schbench] [source][schbench-source] |
 | **stress-ng** | Appropriate candidate for controlled background load and selected verification diagnostics. Bogo operations are unsuitable as Odin's cross-test score foundation. | Explicit byte/worker/time caps, verification, YAML and differentiated exit codes. Upstream documents musl builds and testing on ARM64/x86-64. GPL-2.0-or-later; release 0.22.01, September 2026. Void's recipe has explicit musl handling. [README][stress-readme] [manual][stress-man] [release][stress-release] [Void][void-stress] |
 | **STREAM** | Established sustainable memory-bandwidth kernels: Copy, Scale, Add, Triad. No memory-latency or comprehensive error-detection result. Each array must exceed cache requirements; a small cache-resident run is not a compliant STREAM result. | Portable C; multicore uses OpenMP and its runtime. Array size is a build parameter in reference 5.10. Text output and numerical validation. Reference source dates to 2013: stable method, not evidence of a modern portability test matrix. Custom license permits use/redistribution but imposes result-naming/run-rule conditions. [source and license][stream-source] [run rules][stream-rules] |
 | **lmbench `lat_mem_rd`** | Pointer-chain latency over sizes/strides exposes cache, memory and TLB behavior. Its manual acknowledges vulnerability to stride-sensitive prefetchers; do not present this as an architecture-independent “true RAM latency.” | Warmup, repetitions, bounded size, text pairs. GPLv2 COPYING inspected; Intel's repository is active but contains old documentation. Portability and selected-file licensing need validation before adoption; not a recommended mandatory dependency yet. [manual][lmbench-memory] [README][lmbench-readme] [COPYING][lmbench-license] [metadata][lmbench-meta] |
 | **cyclictest / perf** | Cyclictest measures timer wakeup latency; perf supplies diagnostic counters. Neither is a substitute for foreground application response. | Cyclictest offers duration, histogram and JSON; source is GPL-2.0-only, current rt-tests release 2.11. Its startup tests permission to enter SCHED_FIFO even when ordinary policy is selected. Perf depends on kernel/PMU access and can emit JSON. Treat both as optional diagnostic coverage. [cyclictest][cyclictest] [privilege check][rt-utils] [rt-tests release][rt-release] [perf][perf-stat] |
 | **memtester** | Online checking of allocated memory, not all installed RAM. Failures can involve memory, CPU, temperature or power; the result does not identify a replaceable DIMM by itself. | Byte size and finite iteration count; default iterations are infinite. Text plus exit-bit mask. Record actual allocation and locking, not only exit status. GPL-2.0-only. Version 4.7.1's December 2024 fix addresses stricter C23/GCC 15 compilation; Void recipe inspected still selects 4.6.0. [manual][memtester-man] [source][memtester-source] [changelog][memtester-changelog] [Void][void-memtester] |
 | **Memtest86+** | Offline, bootable diagnostics reach almost all memory without the resident OS. They cannot run as an ordinary in-terminal stage. | GPLv2. Stable v8.10, May 2026, lists x86, x86-64 and LoongArch64. Current main additionally lists AArch64 with UEFI boot. This is a **stable/development difference**, not certified aarch64 coverage. [stable README][memtest-stable] [development README][memtest-main] [release][memtest-release] |
 | **EDAC / rasdaemon** | Hardware-error telemetry, complementary to active tests. Availability depends on hardware, firmware, drivers and exposed events. | Read EDAC counters where accessible; optionally consume an existing rasdaemon history. Rasdaemon monitors kernel trace events and has database backends; its repository shows September 2026 activity and GPLv2 metadata. Starting it is a separate privileged monitoring action, not necessary to read available counters. [EDAC ABI][edac-abi] [rasdaemon][rasdaemon] [metadata][ras-meta] |
 ## Measuring usability under contention
 **Recommendation:** evaluate paired idle and loaded foreground-request measurements as a first-class candidate. Run the same bounded foreground work alone, with a fixed CPU background load, and with separately controlled memory pressure. Preserve normal scheduling policy, work size, thread count, placement, requested arrival rate, actual throughput, sample count, and latency distribution. Report absolute latency and degradation relative to idle; a fast idle result can coexist with poor responsiveness under contention.
 Existing tools can supply the observations. Schbench records both request and wakeup latency, supports a fixed request rate, and can coexist with an independently bounded stress-ng background worker. Sysbench's rate-limited engine is another candidate: its source adds measured queue time to event duration and reports queue length. These remain synthetic proxies for foreground work; neither measures keyboard-to-pixel delay, terminal rendering, browser interaction, or application launch by itself. Those need their own workload definitions. [schbench methodology][schbench] [schbench source][schbench-source] [sysbench queue][sysbench-core] [sysbench timer][sysbench-timer]
 Schbench needs qualification before selection. Its README says warmup defaults to five seconds, while the inspected source defaults to zero and bypasses its warmup reset in request-rate mode. It uses `gettimeofday`, so clock adjustments can contaminate timing. Set options explicitly, retain the revision, validate timing conditions, and decide whether its deliberate preemption penalty fits Odin's goal. Do not adapt work size independently on every system and then compare the resulting latency as equal work. [source][schbench-source]
 Cyclictest is a useful separate scheduler diagnostic. Its default behavior can hold `/dev/cpu_dma_latency` at zero and suppress deep idle states; `--default-system` avoids that tuning. The source's unconditional real-time privilege check prevents assuming that an ordinary-policy configuration is universally unprivileged. Its timer latency, especially under SCHED_FIFO, is a different measurement from a normal foreground application's response. [manual][cyclictest] [source][cyclictest-source] [privileges][rt-utils]
 ## Kernel telemetry and compatibility
 **PSI:** Read system and, where available, workload-cgroup `cpu`, `memory`, and `io` pressure. `some` is time when at least some tasks are stalled; `full` is time when all non-idle tasks are stalled together. The cumulative `total` counter permits interval deltas; rolling 10/60/300-second averages can smear a short benchmark across adjacent phases. **System-wide CPU `full` is undefined and exposed as zero for compatibility**—zero there cannot mean perfect responsiveness. PSI exists in the inspected Linux 4.20 source, but requires `CONFIG_PSI` and may be disabled by default pending `psi=1`. Probe actual files and readability, not only kernel version. [PSI][psi] [4.20 documentation][psi-420] [Kconfig][kconfig]
 **Capacity and pressure:** Record usable RAM, `MemAvailable`, swap capacity/usage, process or cgroup memory, major faults, and swap/reclaim activity where exposed. `MemAvailable` is an estimate of memory available without swapping, not an allocation guarantee. Swap occupancy alone does not establish current pressure. `/proc/stat` supplies CPU time and steal time, but kernel documentation explicitly warns that `iowait` is unreliable. Correlate these observations with latency and PSI; do not derive a definitive bottleneck from CPU utilization or a single counter. [proc documentation][proc]
 **Cgroups:** Observe effective CPU affinity/cpuset, CPU quota, memory/swap limits and relevant ancestors. Host RAM and online CPU counts may exceed what the benchmark is allowed to use. Cgroup v2 `cpu.stat` records throttling; `memory.events` separates high-limit reclaim, OOM conditions and kills. Parse by key: the kernel explicitly allows new `memory.stat` entries in the middle. These interfaces are independent of a particular init system, but writable delegation and enabled controllers are not guaranteed on Void, deb, rpm, containers, or user sessions. [cgroup v2][cgroup]
 **Perf:** Treat hardware counters as enrichment. `CONFIG_PERF_EVENTS`, CPU PMU support, virtualization, `perf_event_paranoid`, capabilities and distribution policy can limit access. Kernel documentation recommends `CAP_PERFMON` over broad `CAP_SYS_ADMIN`; Odin should describe missing access rather than lowering system security settings. Record event identity and time-running percentage when multiplexing occurs. PMU-specific cache and pipeline events are not universal normalized scores. Perf's manual also warns of overhead at short sampling intervals, particularly below 100 ms. [security][perf-security] [Kconfig][kconfig] [perf stat][perf-stat]
 **Architecture/libc:** Proc/sysfs/cgroup interfaces offer the strongest common layer across x86_64/aarch64 and glibc/musl. Workload binaries still need distinct, verified artifacts and recorded toolchain flags. Upstream stress-ng explicitly documents musl; sysbench's advertised architectures and Void recipes are useful evidence, but none of the inspected material certifies Odin's whole four-way architecture/libc matrix. STREAM additionally needs a compatible OpenMP runtime; rt-tests has library dependencies. A package recipe proves availability intent, not successful operation. Missing checks should carry reasons such as unsupported, permission denied, unavailable dependency, or insufficient safe resources. [stress-ng][stress-readme] [sysbench][sysbench] [Void recipes][void-sysbench] [STREAM][stream-source] [rt-tests Makefile][rt-makefile]
 ## Resource safety and cancellation
 The following is a proposed execution contract, with exact caps left to the safety decision:
 1. Calculate a conservative working budget from current `MemAvailable`, effective cgroup/ancestor headroom, expected tool/runtime overhead, and a retained reserve. Recheck while running. There is no sourced universal percentage that guarantees safety when other programs allocate concurrently.
 2. Where delegated cgroup v2 control exists, put disposable workers in their own subtree and keep the supervisor outside that subtree. `memory.high` induces reclaim/throttling and **is not a hard cap**; `memory.max` bounds charged memory and can invoke OOM inside the cgroup. `memory.swap.max` separately controls swap. Use deliberate limits and record them because they change results. Caps reduce risk; they cannot guarantee that an unrelated system-wide shortage never kills a process. [cgroup v2][cgroup]
 3. Reserve intentional pressure for explicitly selected, isolated work. Without reliable containment or enough reserve, recommend pressure **observation** and small bounded workloads, and report unavailable active-pressure coverage. Allocation success alone is insufficient: memtester's own manual warns about overcommit, swapping and OOM affecting other programs. [memtester][memtester-man]
 4. Never expose memtester's physical-address/device modes in the normal benchmark path. They overwrite the mapped region and can crash the system when it belongs to another process or the kernel. For ordinary allocations, verify locked bytes and completed patterns/iterations. Linux permits unprivileged locking up to `RLIMIT_MEMLOCK`; larger locking requires suitable privilege, commonly `CAP_IPC_LOCK`. The tool's “run as root” advice should not force the entire TUI to run as root. [manual][memtester-man] [Linux mlock][mlock]
 5. Use finite work/time limits and a supervisor deadline. For stress-ng, enable only reviewed stressors, verification where supported, and no OOM respawn (`--oomable`). Its `--oom-avoid` is a heuristic with measurement overhead, not containment. In 0.22.01, `--vm-bytes` describes a total across VM workers; other stressors have different allocation semantics, so retain the exact version and options. [stress-ng manual][stress-man]
 6. Cancel the worker process group gracefully, then terminate remaining descendants after a defined grace period; use `cgroup.kill` when accessible. Preserve a cancelled/partial result. Stress-ng documents SIGINT cleanup, but its timeout can overrun during uninterruptible calls or cleanup. No userspace deadline guarantees immediate cancellation of an uninterruptible kernel task. [manual][stress-man] [cgroup kill][cgroup]
 ## Repetition, kernel settings, and TUI overhead
 **Recommendation:** record warmup separately, repeat bounded measurements, retain all repetitions and dispersion, and report a median only at a clearly defined level. STREAM's official report takes the **best** iteration after discarding the first; a median of repeated STREAM run results is a different statistic. Do not silently relabel its internal minimum as a median, combine raw milliseconds with MB/s, or replace an unavailable result with zero. Overall score normalization belongs to the scoring decision. [STREAM source][stream-source]
 Record kernel/build identity, visible preemption/scheduler settings, CPU topology and allowed CPUs, NUMA placement, THP policy, libc, workload/compiler version and flags, governor/driver, boost, power source and temperature observations. NUMA placement and THP policy affect what memory workload is actually measured. Kernel CPUFreq documentation explains that `scaling_cur_freq` can be a requested state rather than measured frequency, and boost depends on thermal/power conditions and package load. A governor name or falling frequency alone does not prove thermal throttling. Correlate sustained performance with available temperatures, thermal trip/cooling states, and power/frequency evidence; unavailable sensors remain unavailable. [CPUFreq][cpufreq] [NUMA][numa] [THP][thp] [thermal interfaces][thermal]
 For “single core,” specify whether the worker is pinned and how the core is chosen on heterogeneous CPUs. For “multicore,” specify workers relative to allowed CPUs, SMT and quota; do not silently change those rules between systems. Preserve the machine's existing configuration for the baseline. Potential governor, scheduler, THP, affinity or kernel changes should be advice or separately labelled experiments, not automatic optimization before measuring.
 The TUI competes for CPU time, memory bandwidth, cache and terminal I/O. Recommend throttled graph updates, buffered logs, no expensive animation during timed sections, and a quiet measurement mode that preserves cancellation. Avoid hiding this by reserving a core without recording it: that reduces tested capacity. Later validation should compare quiet versus normal rendering on the slowest supported machines and establish an overhead budget. This is a proposed qualification experiment, not evidence that a particular redraw rate is already safe. Lmbench explicitly warns about competing cache/CPU work; PSI's own Kconfig notes overhead can show up in synthetic scheduler stress tests. [lmbench][lmbench-readme] [Kconfig][kconfig]
 ## Memory errors, VM coverage, and remaining decisions
 Online memtester cannot touch RAM occupied by the kernel or other processes. It may allocate less than requested and may continue unlocked; inspected 4.7.1 source can then still exit zero if its pattern checks succeed. Parse allocation/locking evidence alongside exit bits and report “no errors detected in the tested allocation,” with size, iterations and duration. A mismatch warrants investigation, not an automatic RAM-replacement diagnosis. [manual][memtester-man] [source][memtester-source]
 EDAC counters reset at driver initialization or explicit reset; preserve counter baselines and `seconds_since_reset` without resetting them. Corrected errors merit attention, but uncorrected errors may cause a panic before a counter increments. DIMM labels can depend on board-specific userspace mapping. Therefore missing EDAC nodes, zero observed deltas, and an empty rasdaemon history cannot certify error-free RAM. Offer offline follow-up where supported; decide how development-only AArch64 Memtest86+ support should be presented. [EDAC ABI][edac-abi] [EDAC model][edac] [Memtest86+ stable][memtest-stable] [development][memtest-main]
 VMs can validate packaging, libc/architecture execution, permissions, telemetry fallbacks, cgroup containment and result handling. Guest CPU/memory scores describe the guest allocation and host scheduling conditions; steal time is useful context. They do not certify the host's DIMMs, ECC pipeline, cooling, physical memory-channel bandwidth, or representative bare-metal scheduler tails. Rasdaemon's upstream QEMU tests intentionally inject virtual nonfatal events: useful for exercising decoding, not proving physical hardware health. [proc][proc] [rasdaemon CI description][rasdaemon]
 **Recommended next decision:** shortlist sysbench for a narrow CPU baseline, STREAM for bandwidth, schbench for responsiveness qualification, selected stress-ng workers for bounded load, and memtester plus available EDAC/RAS for diagnostics. Keep perf/cyclictest optional; defer mandatory memory-latency scoring until a candidate is validated. Final selection remains open.
 The human-facing decisions still needed are the foreground workload's meaning; fixed versus relative background load; inclusion of responsiveness in the median score; safe resource reserves and privileges; repetition/time allocation within the 10–20 minute standard run; treatment of heterogeneous cores and missing coverage; and whether offline/development-tool guidance belongs in the first release.
 **Evidence limits:** no candidate was built or executed across the target matrix. Context7 resolved Linux kernel, sysbench, memtester and rt-tests documentation. Stress-ng, STREAM and schbench searches returned unrelated libraries, so no false library match was used; their owning sources were inspected directly. Memtester's upstream HTTPS site failed certificate validation; the report uses the original source/manpage/changelog preserved by Debian, cross-checked against the Void 4.6.0 source archive checksum, and identifies the version difference. Exact artifact compatibility, parser contracts, resource budgets, and score repeatability require later qualification.
 [sysbench]: https://github.com/akopytov/sysbench/blob/master/README.md
 [sysbench-cpu]: https://github.com/akopytov/sysbench/blob/master/src/tests/cpu/sb_cpu.c
 [sysbench-memory]: https://github.com/akopytov/sysbench/blob/master/src/tests/memory/sb_memory.c
 [sysbench-core]: https://github.com/akopytov/sysbench/blob/master/src/sysbench.c
 [sysbench-timer]: https://github.com/akopytov/sysbench/blob/master/src/sb_timer.h
 [sysbench-release]: https://github.com/akopytov/sysbench/releases/tag/1.0.20
 [sysbench-meta]: https://api.github.com/repos/akopytov/sysbench
 [void-sysbench]: https://github.com/void-linux/void-packages/blob/master/srcpkgs/sysbench/template
 [schbench]: https://kernel.googlesource.com/pub/scm/linux/kernel/git/mason/schbench/+/6300b8f3a8922c61ea6bb2cdfa1901a42c0cc6fc/README.md
 [schbench-source]: https://kernel.googlesource.com/pub/scm/linux/kernel/git/mason/schbench/+/6300b8f3a8922c61ea6bb2cdfa1901a42c0cc6fc/schbench.c
 [stress-readme]: https://github.com/ColinIanKing/stress-ng/blob/V0.22.01/README.md
 [stress-man]: https://github.com/ColinIanKing/stress-ng/blob/V0.22.01/stress-ng.1
 [stress-release]: https://github.com/ColinIanKing/stress-ng/releases/tag/V0.22.01
 [void-stress]: https://github.com/void-linux/void-packages/blob/master/srcpkgs/stress-ng/template
 [stream-source]: https://www.cs.virginia.edu/stream/FTP/Code/stream.c
 [stream-rules]: https://www.cs.virginia.edu/stream/ref.html
 [lmbench-memory]: https://github.com/intel/lmbench/blob/master/doc/lat_mem_rd.8
 [lmbench-readme]: https://github.com/intel/lmbench/blob/master/README
 [lmbench-license]: https://github.com/intel/lmbench/blob/master/COPYING
 [lmbench-meta]: https://api.github.com/repos/intel/lmbench
 [cyclictest]: https://kernel.googlesource.com/pub/scm/utils/rt-tests/rt-tests/+/62da2befac98f811af8e56f2b7992fb09faa33d6/src/cyclictest/cyclictest.8
 [cyclictest-source]: https://kernel.googlesource.com/pub/scm/utils/rt-tests/rt-tests/+/62da2befac98f811af8e56f2b7992fb09faa33d6/src/cyclictest/cyclictest.c
 [rt-utils]: https://kernel.googlesource.com/pub/scm/utils/rt-tests/rt-tests/+/62da2befac98f811af8e56f2b7992fb09faa33d6/src/lib/rt-utils.c
 [rt-release]: https://kernel.googlesource.com/pub/scm/utils/rt-tests/rt-tests/+/62da2befac98f811af8e56f2b7992fb09faa33d6
 [rt-makefile]: https://kernel.googlesource.com/pub/scm/utils/rt-tests/rt-tests/+/62da2befac98f811af8e56f2b7992fb09faa33d6/Makefile
 [perf-stat]: https://github.com/torvalds/linux/blob/master/tools/perf/Documentation/perf-stat.txt
 [kernel-license]: https://github.com/torvalds/linux/blob/master/COPYING
 [perf-security]: https://docs.kernel.org/admin-guide/perf-security.html
 [memtester-man]: https://sources.debian.org/data/main/m/memtester/4.7.1-1/memtester.8
 [memtester-source]: https://sources.debian.org/data/main/m/memtester/4.7.1-1/memtester.c
 [memtester-changelog]: https://sources.debian.org/data/main/m/memtester/4.7.1-1/CHANGELOG
 [void-memtester]: https://github.com/void-linux/void-packages/blob/master/srcpkgs/memtester/template
 [mlock]: https://man7.org/linux/man-pages/man2/mlock.2.html
 [memtest-stable]: https://github.com/memtest86plus/memtest86plus/blob/v8.10/README.md
 [memtest-main]: https://github.com/memtest86plus/memtest86plus/blob/main/README.md
 [memtest-release]: https://github.com/memtest86plus/memtest86plus/releases/tag/v8.10
 [edac-abi]: https://github.com/torvalds/linux/blob/master/Documentation/ABI/testing/sysfs-devices-edac
 [edac]: https://docs.kernel.org/driver-api/edac.html
 [rasdaemon]: https://github.com/mchehab/rasdaemon/blob/master/README.rst
 [ras-meta]: https://api.github.com/repos/mchehab/rasdaemon
 [psi]: https://docs.kernel.org/accounting/psi.html
 [psi-420]: https://github.com/torvalds/linux/blob/v4.20/Documentation/accounting/psi.txt
 [kconfig]: https://github.com/torvalds/linux/blob/master/init/Kconfig
 [proc]: https://docs.kernel.org/filesystems/proc.html
 [cgroup]: https://docs.kernel.org/admin-guide/cgroup-v2.html
 [cpufreq]: https://docs.kernel.org/admin-guide/pm/cpufreq.html
 [numa]: https://docs.kernel.org/admin-guide/mm/numa_memory_policy.html
 [thp]: https://docs.kernel.org/admin-guide/mm/transhuge.html
 [thermal]: https://docs.kernel.org/driver-api/thermal/sysfs-api.html
Author	SHA1	Message	Date
Codex	80be5a7ba9	Record approved browser modes in research scope	2026-09-25 23:47:06 +05:30
Codex	c5fc4e7ecb	Research browser shell and language workload validity	2026-09-25 23:44:03 +05:30