xavierk/voculinux
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

feat(void): publish built releases with publish-release.sh

Browse Source 
The release is built on the maintainer's x86_64 Void host, not in CI. The
bongbetic-ci runner is ARM64 and cannot build the glibc x86_64 package. A tag
push no longer starts the xbps workflow: it is manual dispatch only, and its
header says why.

- build-release.sh release refuses an uncommitted or untracked checkout before
  any container starts. It writes BUILT_FROM (the tag and the commit it was
  built from), which nothing recorded before.
- publish-release.sh TAG --out-dir DIR publishes a built out-dir to the Gitea
  release API in the workflow's order. It refuses unless origin's tag peels to
  the BUILT_FROM commit, the sdist digest matches, and the client key is the
  committed one. --dry-run runs every check and prints the planned calls
  without sending one. The token comes from GITEA_TOKEN or the tea login and is
  never printed.
- Tests run both scripts offline against a local bare origin and a stateful
  Gitea stand-in.
- docs/RELEASE_PROCESS.md and packaging/void/README.md describe the local
  sequence: merge, annotated tag by ref, clean build, dry-run, publish, checksum.
This commit is contained in:
xavierk
2026-11-07 01:46:38 +05:30
parent 64020a132d
commit dc7e964981
10 changed files with 1186 additions and 116 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitea/workflows/xbps-release.yml
+27 -19
View File
@@ -1,22 +1,32 @@
name: xbps-release
# Signed Void xbps repository release (tickets #28 and #29). A human pushes a
# version tag; this job does everything else through packaging/void/
# build-release.sh, the same script a local dry run and the smoke gate call:
# Signed Void xbps repository release (tickets #28 and #29). Manual dispatch only.
#
# 1. the tag must EXACTLY equal "v" + __version__ (src/vocalinux/version.py),
# and the committed Void template must already name that version;
# 2. the XBPS_SIGNING_KEY secret must be non-empty and RSA 4096, and its public
# half must be packaging/void/repository-key.plist;
# 3. the canonical sdist is built from the tag (Python 3.11,
# uv 0.12/0.13, scripts/build_reproducible_wheel.py);
# 4. every template builds in the pinned Void glibc image with podman (no
# signing key in that phase), then the repository is signed in a second
# container with no network, then voculinux is installed and smoked in a
# clean image with no key;
# 5. the sdist, the signed repository and the client key are published to the
# tag's release and to the rolling `latest` release that the client
# registration URL serves (README.md).
# A tag push does NOT start this workflow. Releases are currently built and
# published from the maintainer's own x86_64 Void host, not from CI: the only
# runner with the bongbetic-ci label is ARM64 Docker on another server, and it
# cannot build the glibc x86_64 release. The local sequence is in
# docs/RELEASE_PROCESS.md (Step 7 and the Void section of Step 8) and
# packaging/void/README.md ("Releasing"):
# 1. create and push the annotated tag on the verified origin/main commit;
# 2. packaging/void/build-release.sh release TAG --key-file FILE --out-dir DIR,
# in a clean checkout of the tag;
# 3. packaging/void/publish-release.sh TAG --out-dir DIR, after --dry-run.
#
# This job is kept for a future x86_64 host-mode runner with rootless podman. It
# runs the same release path as the local run, and its publish step makes the
# same API calls as publish-release.sh. A dispatch's ref is a branch, so the tag
# gate below refuses it before anything is built or signed.
#
# The job's steps: the tag must EXACTLY equal "v" + __version__
# (src/vocalinux/version.py), and the committed Void template must already name
# that version; the XBPS_SIGNING_KEY secret must be non-empty and RSA 4096, and
# its public half must be packaging/void/repository-key.plist; the canonical sdist
# is built from the tag; every template builds in the pinned Void glibc image with
# podman, then the repository is signed in a second container with no network,
# then voculinux is installed and smoked in a clean image with no key; and the
# staged set is published to the tag's release and to the rolling `latest`
# release that the client registration URL serves (README.md).
#
# Nothing here is built or uploaded by hand, nothing commits or pushes to any
# branch, and no template is bumped: the post-release template bump is a normal
@@ -25,9 +35,7 @@ name: xbps-release
# only, never Docker. Runner requirements: packaging/void/README.md.
on:
push:
tags:
- "v*"
workflow_dispatch:
permissions:
contents: write
docs/RELEASE_PROCESS.md
+52 -45
View File
@@ -39,23 +39,25 @@ publish succeeds.
### Void xbps repository (required for a Void release)
The signed Void repository is released by `.gitea/workflows/xbps-release.yml`
on the fork's Gitea host, not by `release.yml`. The whole release is one script,
`packaging/void/build-release.sh`, which the workflow and a local dry run both
call. Set it up once:
The signed Void repository is built and published from the maintainer's own
x86_64 Void host, not from CI. The whole build is one script,
`packaging/void/build-release.sh`, and the publish is one more,
`packaging/void/publish-release.sh`. The Gitea workflow
`.gitea/workflows/xbps-release.yml` is manual dispatch only: a tag push does not
start it, because its only runner (label `bongbetic-ci`) is ARM64 and cannot build
the glibc x86_64 release. Set this up once on the release host:
| what | value |
|---|---|
| Secret | `XBPS_SIGNING_KEY`: the PEM-encoded RSA 4096 repository private key (Gitea repository settings, Actions, Secrets) |
| Signing key | the PEM-encoded RSA 4096 repository private key, kept in the maintainer's own backup. It is passed to `build-release.sh` as a file on tmpfs (`/dev/shm`) and removed after the run; it is never committed |
| Public key | `packaging/void/repository-key.plist`, committed, fingerprint `fa:d1:00:32:ce:b1:b9:fd:73:05:05:76:75:54:37:25` |
| Runner | `bongbetic-ci`, a host job with podman (rootless, with user namespaces), python3.11, uv 0.12/0.13 and tmpfs `/dev/shm`. Requirements: `packaging/void/README.md`, "Runner requirements". Never Docker |
| Token | the job token must be allowed to create releases and upload release assets |
| Host | x86_64 Void with rootless podman (user namespaces), python3.11, uv 0.12/0.13 and tmpfs `/dev/shm`. Requirements: `packaging/void/README.md`, "Runner requirements". Never Docker |
| Gitea login | a `tea` login for `https://git.bongbetic.com` whose token may create releases and upload assets, or `GITEA_TOKEN` in the environment for `publish-release.sh` |
Without the secret the workflow fails and publishes nothing; it never signs
with a different key either: the release path refuses any key whose public half
is not `repository-key.plist`. The private key is never committed; only the
public `.plist` is. Key generation and the check that a private key matches the
committed public key are in `packaging/void/README.md`.
Without a valid key the build stops before any container starts. It never signs
with a different key: the release path refuses any key whose public half is not
`repository-key.plist`. Only the public `.plist` is committed. Key generation and the
check that a private key matches the committed public key are in `packaging/void/README.md`.
The carried Void templates in `packaging/void/srcpkgs/` track the project's own
pins. `python3-PyAudio` and `python3-pysocks` replace stock Void packages below
@@ -109,7 +111,7 @@ Install `just` from https://just.systems or the distro package `just`.
- [ ] Commit all version changes
- [ ] Push branch and create PR
- [ ] After PR merge, create tag: `git tag -a vX.Y.Z-PHASE -m "Release X.Y.Z-PHASE"`
- [ ] Push tag: `git push origin vX.Y.Z-PHASE` (this also starts the Gitea xbps workflow, see Step 8)
- [ ] Create and push the annotated tag on the verified `origin/main` commit: `git push origin refs/tags/vX.Y.Z-PHASE`, never `--tags`. Pushing a tag starts no workflow; the Void release is built and published by hand (Step 8)
### After the release run finishes
- [ ] Run `just verify-release` - the published release verifies as published
@@ -351,8 +353,8 @@ It has two parts, handled at two times, and both go through a normal pull reques
release, so a stale template stops the tag, before any build.
- **Checksum: after the tag, in a pull request.** The release build replaces
`checksum=` in its build tree only, with the sha256 of the sdist it builds, and
prints `checksum=<sha256>` in the workflow log and in the release notes
(`sdist sha256:`). Copy that value into the template in a pull request through
prints `checksum=<sha256>`, and the published release notes carry it as
`sdist sha256:`. Copy that value into the template in a pull request through
the normal test, lint and type gates. Never commit it directly to `main`, and
never hand-edit a release asset.
@@ -515,10 +517,15 @@ Key changes since 0.4.1-alpha:
Full changelog: https://github.com/VocaHQ/vocalinux/releases"
# Push tag
git push origin v0.5.0-beta
# Push the one tag by its ref, never with --tags
git push origin refs/tags/v0.5.0-beta
```
**Void release (this fork):** tag the verified `origin/main` commit by its full SHA, then push
that one ref: `git tag -a v0.19.0 <sha> -m "Voculinux 0.19.0"` and
`git push origin refs/tags/v0.19.0`. Never `git push --tags`: this clone holds upstream tags
that must not reach the fork's Gitea host. Then follow the Void section of Step 8.
### Step 8: Monitor Release
After pushing the tag, the GitHub Actions workflow will automatically:
@@ -551,36 +558,35 @@ After pushing the tag, the GitHub Actions workflow will automatically:
Monitor at: https://github.com/VocaHQ/vocalinux/actions
#### Void xbps repository (Gitea tag workflow)
#### Void xbps repository (local release)
Pushing the tag to `origin` (the fork's Gitea repository) also starts
`.gitea/workflows/xbps-release.yml`. Pushing a tag is a human act and is
approved each time: the workflow has no schedule and no manual trigger.
Pushing the tag starts nothing. The Void release is run by hand on the release host,
from a clean checkout of the tag, in this order. Replace `vX.Y.Z` with the release.
1. **Tag gate.** `build-release.sh check-tag`: the tag must be exactly `v` plus
`__version__`, and the committed template must name that release. Any other
tag exits 1 and publishes nothing.
2. **Key gate.** The `XBPS_SIGNING_KEY` secret must be non-empty. It is written
with noclobber to a `0600` file on tmpfs and removed by a final step that
always runs.
3. **Runner check.** podman, python3.11, uv 0.12/0.13, openssl, git and curl
must be on the runner, or the job names what is missing and stops.
4. **Release.** `build-release.sh release` builds the canonical sdist from the
tag, builds every template in the pinned Void glibc container
without the key, signs the repository in a second container with no network,
and installs `voculinux` from that repository in a clean image without the key.
A local dry run is `just void-release vX.Y.Z` with `XBPS_SIGNING_KEY_FILE`
set: the same script, the same containers, nothing published.
5. **Publish.** The staged set (sdist, packages, signatures, `x86_64-repodata`,
`FINGERPRINT`, the client `.plist`) attaches to the tag's release on Gitea and
replaces the assets of the rolling `latest` release, which the client
registration URL
(`https://git.bongbetic.com/xavierk/voculinux/releases/download/latest`)
serves. Push tags in version order: pushing an older tag refreshes `latest`
with the older repository.
1. **Merge the release pull request** to `main`, then confirm which commit `origin/main` is:
`git fetch origin && git rev-parse origin/main`.
2. **Create and push the annotated tag** on that commit (Step 7).
3. **Check out the tag cleanly:** `git worktree add /tmp/voculinux-vX.Y.Z vX.Y.Z`. A worktree
at a tag creates no tag. Work only in that worktree.
4. **Build:** from that checkout, run
`packaging/void/build-release.sh release vX.Y.Z --key-file /dev/shm/xbps-sign-key.pem --out-dir /path/to/out`.
The out-dir must be absent or empty. The run refuses an uncommitted checkout and a key
that is not the committed one, both before any container starts. It writes `sdist/`,
`xbps/`, `SDIST_SHA256` and `BUILT_FROM` (the tag and the commit it was built from), and
prints `checksum=<sha256>`. Remove the key file afterwards.
5. **Dry-run the publish:** `packaging/void/publish-release.sh vX.Y.Z --out-dir /path/to/out --dry-run`.
It runs every check (the tag is on `origin` at the built commit, the digest, the key)
and prints the API calls it would make. It sends no request and reads no token.
6. **Publish:** `packaging/void/publish-release.sh vX.Y.Z --out-dir /path/to/out`. It attaches the
sdist, packages, signatures, `FINGERPRINT` and the client `.plist` to the tag's release, then
replaces the assets of the rolling `latest` release that the client registration URL serves.
The repodata is uploaded last. The token comes from `GITEA_TOKEN`, or from the `tea` login
for `https://git.bongbetic.com`.
7. **Record the checksum** in a post-release pull request (section 3.5).
Nothing in this run is built or uploaded by hand, and the workflow never commits
or pushes to any branch. The post-release checksum pull request is section 3.5.
Neither script commits or pushes to any branch or tag, and nothing is built or uploaded by
hand. Publish tags in version order: publishing an older tag refreshes `latest` with the
older repository.
### Step 9: Post-Release Tasks
@@ -601,7 +607,7 @@ or pushes to any branch. The post-release checksum pull request is section 3.5.
- [ ] Verify PyPI package was published (if applicable), and that its wheel sha256
matches the line for that wheel in `SHA256SUMS`
- [ ] Verify website was deployed (check voculinux.com)
- [ ] Verify the Gitea xbps run succeeded, and that both the tag release and
- [ ] Verify the Void publish succeeded, and that both the tag release and
`latest` carry the full signed set (`sdist/voculinux-*.tar.gz`,
`*.x86_64.xbps`, `*.sig2`, `x86_64-repodata`, `FINGERPRINT`, the `.plist`),
that the fingerprint matches the one in `README.md`, and that the sdist
@@ -636,6 +642,7 @@ git push origin v0.5.1-beta
| Version | Date | Phase | Notes |
|---------|------|-------|-------|
| 0.19.0 | 2026-11-07 | Stable | First Voculinux release: a hard fork for Void Linux (glibc, x86_64) with a signed xbps repository, niri and Dank Material Shell support, a per-backend memory-aware model default, and PipeWire system audio that never falls back to the microphone |
| 0.2.0-alpha | 2024 | Alpha | Initial alpha |
| 0.3.0-alpha | 2024 | Alpha | - |
| 0.4.0-alpha | 2024 | Alpha | Multi-language support |
justfile
+6 -6
View File
@@ -134,12 +134,12 @@ native-smoke distro="debian:12":
void-smoke:
bash packaging/void/smoke.sh
# Dry-run or perform the signed Void release for TAG (vX.Y.Z[-phase]) exactly
# as the Gitea tag workflow does: packaging/void/build-release.sh release.
# Stages the complete set under publish/ and publishes nothing. Needs
# XBPS_SIGNING_KEY_FILE (an RSA 4096 private key whose public half is
# packaging/void/repository-key.plist) and the release tools in
# packaging/void/README.md. Podman only, never Docker.
# Build the signed Void release for TAG (vX.Y.Z[-phase]) exactly as the release
# host does: packaging/void/build-release.sh release, from a clean checkout of
# the tag. Stages the complete set under publish/ and publishes nothing; publish
# it with packaging/void/publish-release.sh. Needs XBPS_SIGNING_KEY_FILE (an RSA
# 4096 private key whose public half is packaging/void/repository-key.plist) and
# the release tools in packaging/void/README.md. Podman only, never Docker.
void-release TAG:
bash packaging/void/build-release.sh release "{{TAG}}" \
--key-file "${XBPS_SIGNING_KEY_FILE:?set XBPS_SIGNING_KEY_FILE to the RSA 4096 key}" \
packaging/void/README.md
+55 -35
View File
@@ -10,7 +10,8 @@ repository first.
```
packaging/void/
├── README.md this file
├── build-release.sh the release path: gate, sdist, signed repo, install smoke
├── build-release.sh the release path: gate, clean checkout, sdist, signed repo, install smoke
├── publish-release.sh publishes a built out-dir to the Gitea release API (dry run first)
├── container-release.sh the container half: build, sign, install (one per phase)
├── render-voculinux.sh pins version, _tag and checksum in the build tree only
├── smoke.sh ticket #28 gate: the release path with a throwaway key
@@ -161,7 +162,7 @@ xbps versions cannot contain `-`. The xbps spelling of a release is
| `__version__` (tag `v` + it) | xbps `version` | distfile and release asset |
|---|---|---|
| `0.18.5` | `0.18.5` | `voculinux-0.18.5.tar.gz` |
| `0.19.0` | `0.19.0` | `voculinux-0.19.0.tar.gz` |
| `0.19.0-alpha` | `0.19.0alpha` | `voculinux-0.19.0alpha.tar.gz` |
| `0.19.0-beta` | `0.19.0beta` | `voculinux-0.19.0beta.tar.gz` |
| `0.19.0-rc1` | `0.19.0rc1` | `voculinux-0.19.0rc1.tar.gz` |
@@ -186,7 +187,7 @@ refuses to continue unless exactly one `checksum=` line exists and reads back as
the computed digest, and it refuses if the committed template changed.
After a release, the digest of the published sdist is recorded in the template
by a **pull request**: the workflow log and the release notes both print
by a **pull request**: the build output and the published release notes both carry
`checksum=<sha256>`, and the pull request copies it into
`packaging/void/srcpkgs/voculinux/template`. Nothing commits to `main`.
@@ -194,9 +195,10 @@ by a **pull request**: the workflow log and the release notes both print
```sh
just void-smoke # the gate: build, sign with a throwaway key, install, smoke
just void-release v0.18.5 # a dry run of the release, with XBPS_SIGNING_KEY_FILE set
just void-release v0.19.0 # build the release set, with XBPS_SIGNING_KEY_FILE set
packaging/void/build-release.sh names v0.19.0-beta # offline: the names a tag maps to
packaging/void/build-release.sh check-tag v0.19.0-beta # offline: the tag gate
packaging/void/publish-release.sh v0.19.0 --out-dir DIR --dry-run # every check, the planned calls, no request
```
Podman only, never Docker. The base image digest and the void-packages commit
@@ -230,13 +232,17 @@ every `srcpkgs` template.
## Signed repository
`packaging/void/build-release.sh release TAG --key-file FILE --out-dir DIR`
is the whole release, and the only path that signs anything. It is the path the
tag workflow runs, and `just void-release` runs it locally.
is the whole release, and the only path that signs anything. The maintainer runs
it on the release host (see "Releasing" below); `just void-release` runs the same
command with `publish/` as the out-dir.
Fail-closed, in this order, before any container starts:
- the tag must be exactly `v` plus `__version__`, and the committed template must
name that version and tag;
- the checkout must be clean: no uncommitted or untracked change. The sdist is built
from the working tree, so anything else is not the tag. The commit it is built from
is recorded in `BUILT_FROM`;
- the key must be non-empty, valid, RSA 4096, and its public half must equal
`repository-key.plist` byte for byte (a different key never publishes);
- the canonical sdist must build with Python 3.11 and uv 0.12/0.13. Only its
@@ -256,8 +262,9 @@ The committed templates must be unchanged after the build; the run fails if they
are not. `DIR` must be absent or empty, and the run writes `DIR/sdist/`,
`DIR/xbps/repo/` (packages, signatures, `x86_64-repodata`), `DIR/xbps/keys/`
(the client `.plist` named by its fingerprint, and `FINGERPRINT`) and
`DIR/SDIST_SHA256`. A final check accepts only that set: nothing from musl, no
other architecture, every package signed, the key the pinned one.
`DIR/SDIST_SHA256` and `DIR/BUILT_FROM` (the tag and the commit it was built
from). A final check accepts only that set: nothing from musl, no other
architecture, every package signed, the key the pinned one.
### The repository key
@@ -273,8 +280,10 @@ fa:d1:00:32:ce:b1:b9:fd:73:05:05:76:75:54:37:25
`tests/test_void_signed_repo.py` derives the fingerprint from this file
and fails if either README shows a different one.
- **Private half**: never committed, never in the repository, never on a
build host's disk beyond the run. It exists as the `XBPS_SIGNING_KEY`
CI secret (PEM text, newlines intact) and in the maintainer's own backup.
build host's disk beyond the run. It is kept in the maintainer's own backup,
and the release reads it as a file on tmpfs (`/dev/shm`) that is removed
afterwards. The `XBPS_SIGNING_KEY` CI secret holds the same PEM text for the
workflow, which is manual dispatch only.
- **Generating a new key** (only if the key is ever replaced; clients then
have to re-pin, so this is a release-notes event):
@@ -292,35 +301,47 @@ openssl rsa -in xbps-sign-key.pem -pubout 2>/dev/null | openssl md5
python3 -c "import plistlib,sys; sys.stdout.buffer.write(plistlib.load(open('packaging/void/repository-key.plist','rb'))['public-key'])" | openssl md5
```
### Releasing: the Gitea workflow
### Releasing: the local sequence
A human pushes a version tag; `.gitea/workflows/xbps-release.yml` does the
rest. It never runs on a schedule or a manual dispatch.
Pushing a tag starts nothing. `.gitea/workflows/xbps-release.yml` is manual dispatch
only, kept for a future x86_64 host-mode runner; its one runner today is ARM64 and
cannot build this release. The sequence is:
1. **Tag gate.** `build-release.sh check-tag` (see above). Nothing else runs first.
2. **Key gate.** The CI secret `XBPS_SIGNING_KEY` must be non-empty. The key is
written with noclobber to a `0600` file on tmpfs (`/dev/shm`), never to disk.
3. **Runner check.** The tools in [Runner requirements](#runner-requirements)
must be present, or the job stops and names what is missing.
4. **Release.** `build-release.sh release`, with the key file (above).
5. **Publish.** The staged set goes to the tag's release and to the rolling
`latest` release, which the client registration URL serves. Assets:
`sdist/voculinux-<version>.tar.gz`, the packages, their `.sig2` signatures,
`x86_64-repodata` (last), `FINGERPRINT` and the `.plist`. Nothing is built or
uploaded by hand.
6. **Cleanup.** The key file is removed by a step that always runs.
1. **Merge the release pull request** to `main`. `origin/main` is then the commit to release.
2. **Create and push the annotated tag** on that commit:
`git tag -a vX.Y.Z <sha> -m "Voculinux X.Y.Z"`, then `git push origin refs/tags/vX.Y.Z`.
Never `git push --tags`: the clone holds upstream tags.
3. **Build in a clean checkout of the tag:** `git worktree add /tmp/voculinux-vX.Y.Z vX.Y.Z`,
then, in that worktree,
`packaging/void/build-release.sh release vX.Y.Z --key-file /dev/shm/xbps-sign-key.pem --out-dir /path/to/out`.
Remove the key file when the run ends.
4. **Dry-run the publish:** `packaging/void/publish-release.sh vX.Y.Z --out-dir /path/to/out --dry-run`.
It checks that origin's `vX.Y.Z` peels to the commit in `BUILT_FROM`, that the sdist
matches `SDIST_SHA256`, and that the client key is the committed one, then prints the
calls it would make. It sends no request and reads no token.
5. **Publish:** `packaging/void/publish-release.sh vX.Y.Z --out-dir /path/to/out`. The tag's
release gets the sdist, packages, signatures, `FINGERPRINT` and the client `.plist`, and
the rolling `latest` release that the client registration URL serves is replaced, with the
repodata last. The token comes from `GITEA_TOKEN`, or from the `tea` login for
`https://git.bongbetic.com`. It goes into a 0600 header file that is removed on exit and is
never printed.
6. **Post-release pull request:** record the `checksum=` the build printed in the template
(section 3.5 of `docs/RELEASE_PROCESS.md`).
Then the template's checksum is recorded by a pull request (above), and the
release notes show the sdist digest to copy from.
`publish-release.sh` and the workflow's publish step make the same API calls: the sdist,
packages, signatures and client key, then the repodata, to the tag's release and then to
`latest`. Any non-2xx response stops the publish. Publishing an older tag refreshes `latest`
with the older repository, so publish in version order.
Secret setup, once: Gitea repository settings, Actions, Secrets, add
`XBPS_SIGNING_KEY` with the PEM private key. The job token must be allowed to
create releases and upload release assets.
Secret setup for the workflow, once: Gitea repository settings, Actions, Secrets, add
`XBPS_SIGNING_KEY` with the PEM private key. The job token must be allowed to create
releases and upload release assets.
### Runner requirements
### Runner requirements (the release host)
The job runs on the `bongbetic-ci` label as a host job. It is not a `container:`
job, and it never uses Docker. It needs:
The release runs on the maintainer's x86_64 Void host. The workflow's `runs-on` label
(`bongbetic-ci`) is ARM64 today and cannot run it. Requirements, which a future x86_64
host-mode runner would also need. It never uses Docker:
| requirement | why | check |
|---|---|---|
@@ -333,8 +354,7 @@ job, and it never uses Docker. It needs:
| about 25 GiB free disk | the void-packages bootstrap and the whisper.cpp build | `df -h` |
| egress to `github.com`, `repo-default.voidlinux.org`, `ghcr.io`, `pypi.org`, `files.pythonhosted.org`, `git.bongbetic.com` | clone, bootstrap, base image, locked tools, distfiles, Gitea API | the build phase |
The job's `timeout-minutes` is 240. A cold build takes most of that: the
whisper.cpp Vulkan build is the long step.
A cold build is long: the whisper.cpp Vulkan build is the long step.
## Client setup
packaging/void/build-release.sh
+26 -3
View File
@@ -12,9 +12,11 @@
# src/vocalinux/version.py, and the committed Void template must already
# name that version and tag. Exits 1 otherwise.
# build-release.sh release TAG --key-file FILE --out-dir DIR
# the whole release: gate, canonical sdist, signed repository,
# completeness check. Writes DIR/sdist and DIR/xbps. Publishes nothing;
# the workflow publishes DIR. DIR must be absent or empty.
# the whole release: gate, clean-checkout gate, canonical sdist, signed
# repository, completeness check. Writes DIR/sdist, DIR/xbps,
# DIR/SDIST_SHA256 and DIR/BUILT_FROM (the tag and the commit built from).
# Publishes nothing; packaging/void/publish-release.sh publishes DIR.
# DIR must be absent or empty, and the checkout must be clean.
# build-release.sh smoke
# the same pipeline for the current version, signed with a throwaway
# RSA4096 key in a temporary directory. Proves the path works; nothing is
@@ -127,6 +129,19 @@ check_template() {
fail "committed template _tag is '$have', not '$TAG'; update packaging/void/srcpkgs/voculinux/template"
}
# release_checkout: a release is built from a clean git checkout, and the commit
# it is built from is recorded in DIR/BUILT_FROM. The sdist is built from the
# working tree (tracked edits and non-ignored new files included), so anything
# but a clean tree is not the tag. Sets SOURCE_COMMIT.
release_checkout() {
git -C "$ROOT" rev-parse --git-dir >/dev/null 2>&1 ||
fail "a release is built from a git checkout of $TAG, and records its commit; $ROOT is not one"
[ -z "$(git -C "$ROOT" status --porcelain --untracked-files=normal)" ] ||
fail "the working tree has uncommitted or untracked changes; build $TAG from a clean checkout"
SOURCE_COMMIT=$(git -C "$ROOT" rev-parse --verify HEAD) ||
fail "could not read the commit HEAD"
}
# check_key KEY_FILE: RSA 4096, and for a release the public half must be the
# committed repository-key.plist key. Sets FINGERPRINT and writes the client
# key to $WORK/client.plist.
@@ -281,6 +296,9 @@ verify_set() {
run_pipeline() {
derive "$1"
check_template
if [ "$MODE" = release ]; then
release_checkout
fi
WORK=$(mktemp -d "${TMPDIR:-/tmp}/voculinux-release.XXXXXX")
cleanup() {
status=$?
@@ -317,6 +335,11 @@ run_pipeline() {
phase_install
verify_set
printf '%s\n' "$SDIST_SHA256" >"$OUT/SDIST_SHA256"
if [ "$MODE" = release ]; then
# The commit this set was built from: publish-release.sh refuses to publish
# unless origin's tag peels to it.
printf 'tag=%s\ncommit=%s\n' "$TAG" "$SOURCE_COMMIT" >"$OUT/BUILT_FROM"
fi
echo "template bump for the post-release pull request: checksum=$SDIST_SHA256"
}
packaging/void/publish-release.sh
Executable
+362
View File
@@ -0,0 +1,362 @@
#!/usr/bin/env bash
# Publish an already-built Void release to the fork's Gitea release API (spec #16).
#
# A release is built and verified by packaging/void/build-release.sh, locally on
# the maintainer's x86_64 Void host (packaging/void/README.md, "Releasing"). This
# script publishes that out-dir and nothing else:
#
# publish-release.sh TAG --out-dir DIR [--dry-run]
#
# Before any API request it checks that:
# 1. TAG passes the tag gate (build-release.sh check-tag): exactly v plus
# __version__, and the committed template names this release;
# 2. DIR/BUILT_FROM names TAG and the commit the set was built from;
# 3. origin's refs/tags/TAG resolves to that commit. An annotated tag is compared
# by the commit it peels to; a lightweight tag is compared directly;
# 4. the sdist matches DIR/SDIST_SHA256, the client key is the committed
# packaging/void/repository-key.plist, and the repository has its packages,
# signatures and index.
# It then publishes the staged set the way the Gitea workflow's publish step does
# (.gitea/workflows/xbps-release.yml): the tag's release first, then the rolling
# `latest` release, with the repodata uploaded last.
#
# --dry-run runs every check above, reads no token, makes no API call, and prints
# the calls it would make, in order.
#
# Token: GITEA_TOKEN when it is set. Otherwise the token of the tea login for
# GITEA_HOST in $TEA_CONFIG (default ${XDG_CONFIG_HOME:-$HOME/.config}/tea/config.yml).
# The token is written to a 0600 header file in a private temporary directory,
# which is removed on exit. It is never printed.
#
# Environment: GITEA_HOST, GITEA_API (the repository's API root), REMOTE (default
# origin), TEA_CONFIG. Tests point them at a stub; a release uses the defaults.
set -euo pipefail
CDPATH=""
HERE=$(cd -- "$(dirname -- "$0")" && pwd)
ROOT=$(cd -- "$HERE/../.." && pwd)
PLIST="$HERE/repository-key.plist"
GITEA_HOST=${GITEA_HOST:-https://git.bongbetic.com}
GITEA_API=${GITEA_API:-https://git.bongbetic.com/api/v1/repos/xavierk/voculinux}
REMOTE=${REMOTE:-origin}
TEA_CONFIG=${TEA_CONFIG:-${XDG_CONFIG_HOME:-$HOME/.config}/tea/config.yml}
usage() {
echo "usage: publish-release.sh TAG --out-dir DIR [--dry-run]" >&2
exit 2
}
fail() {
echo "publish-release.sh: $*" >&2
exit 1
}
plan() {
printf ' %s\n' "$*"
}
urlencode() {
local s=$1 out="" c i
for ((i = 0; i < ${#s}; i++)); do
c=${s:i:1}
case $c in
[a-zA-Z0-9._~-]) out+=$c ;;
*) out+=$(printf '%%%02X' "'$c") ;;
esac
done
printf '%s' "$out"
}
# Reads the token of the tea login for GITEA_HOST and writes the header file
# named by argv[3]. Prints nothing else; a failure names the config, never a token.
TEA_READER=$(cat <<'PY'
import re
import sys
config, host, out = sys.argv[1:4]
try:
with open(config, encoding="utf-8") as handle:
text = handle.read()
except OSError:
sys.exit(f"no tea config at {config}")
# tea keeps a YAML list of logins; each block opens with "- name:".
for block in re.split(r"^\s*-\s+name:", text, flags=re.M)[1:]:
fields = {}
for line in block.splitlines():
match = re.match(r"\s*(url|token):\s*(.*?)\s*$", line)
if match:
fields[match.group(1)] = match.group(2).strip("'\"")
if fields.get("url", "").rstrip("/") == host and fields.get("token"):
with open(out, "w", encoding="utf-8") as handle:
handle.write("Authorization: token " + fields["token"] + "\n")
sys.exit(0)
sys.exit(f"no tea login for {host} with a token in {config}")
PY
)
TAG=""
OUT=""
DRY_RUN=0
[ "$#" -ge 1 ] || usage
TAG=$1
shift
while [ "$#" -gt 0 ]; do
case $1 in
--out-dir)
[ "$#" -ge 2 ] || usage
OUT=$2
shift 2
;;
--dry-run)
DRY_RUN=1
shift
;;
*) usage ;;
esac
done
[ -n "$OUT" ] || usage
# Gate 1: the tag gate, offline. Exactly v plus __version__, and the template names it.
"$HERE/build-release.sh" check-tag "$TAG" >/dev/null ||
fail "tag $TAG fails the tag gate (build-release.sh check-tag); refusing to publish"
names=$("$HERE/build-release.sh" names "$TAG") || fail "could not derive the names of $TAG"
SDIST_NAME=$(printf '%s\n' "$names" | sed -n 's/^SDIST_NAME=//p')
# Gate 2: the out-dir is the set build-release.sh built for this tag.
[ -d "$OUT" ] || fail "$OUT is not a directory"
OUT=$(cd -- "$OUT" && pwd)
[ -f "$OUT/BUILT_FROM" ] ||
fail "$OUT has no BUILT_FROM record: it was not written by build-release.sh release"
BUILT_TAG=$(sed -n 's/^tag=//p' "$OUT/BUILT_FROM")
BUILT_COMMIT=$(sed -n 's/^commit=//p' "$OUT/BUILT_FROM")
[ "$BUILT_TAG" = "$TAG" ] || fail "$OUT was built for '$BUILT_TAG', not '$TAG'"
printf '%s\n' "$BUILT_COMMIT" | grep -Eq '^[0-9a-f]{40}$' ||
fail "BUILT_FROM in $OUT names no commit"
shopt -s nullglob
[ -f "$OUT/sdist/$SDIST_NAME" ] && [ "$(find "$OUT/sdist" -mindepth 1 -maxdepth 1 | wc -l)" -eq 1 ] ||
fail "$OUT/sdist must hold exactly $SDIST_NAME"
[ -f "$OUT/SDIST_SHA256" ] || fail "$OUT has no SDIST_SHA256"
want_digest=$(cat "$OUT/SDIST_SHA256")
got_digest=$(sha256sum "$OUT/sdist/$SDIST_NAME" | cut -d' ' -f1)
[ "$got_digest" = "$want_digest" ] || fail "sdist/$SDIST_NAME does not match SDIST_SHA256"
[ -f "$OUT/xbps/keys/FINGERPRINT" ] || fail "$OUT has no FINGERPRINT record"
FINGERPRINT=$(sed -n 's/^fingerprint: //p' "$OUT/xbps/keys/FINGERPRINT")
[ -n "$FINGERPRINT" ] || fail "FINGERPRINT in $OUT names no fingerprint"
cmp -s "$PLIST" "$OUT/xbps/keys/$FINGERPRINT.plist" ||
fail "the client key in $OUT is not the committed packaging/void/repository-key.plist"
packages=("$OUT"/xbps/repo/*.x86_64.xbps)
signatures=("$OUT"/xbps/repo/*.sig2)
index=("$OUT"/xbps/repo/*-repodata*)
[ "${#packages[@]}" -gt 0 ] || fail "$OUT holds no packages"
[ "${#signatures[@]}" -eq "${#packages[@]}" ] || fail "every package needs exactly one signature"
[ "${#index[@]}" -gt 0 ] || fail "$OUT has no repository index (x86_64-repodata)"
# Gate 3: origin's tag is the commit the set was built from. Both refs are asked
# for in one call: the tag object, and the commit it peels to (`^{}`).
remote_refs=$(git -C "$ROOT" ls-remote --tags "$REMOTE" "refs/tags/$TAG" "refs/tags/$TAG^{}") ||
fail "could not read the tags of $REMOTE"
direct=$(printf '%s\n' "$remote_refs" | awk -v ref="refs/tags/$TAG" '$2 == ref { print $1 }')
peeled=$(printf '%s\n' "$remote_refs" | awk -v ref="refs/tags/$TAG^{}" '$2 == ref { print $1 }')
[ -n "$direct" ] ||
fail "tag $TAG is not on $REMOTE; push the annotated tag first (packaging/void/README.md)"
[ "$(printf '%s\n' "$direct" | wc -l)" -eq 1 ] || fail "$REMOTE has more than one $TAG"
if [ -n "$peeled" ]; then
REMOTE_COMMIT=$peeled
else
REMOTE_COMMIT=$direct
fi
[ "$REMOTE_COMMIT" = "$BUILT_COMMIT" ] ||
fail "$REMOTE's $TAG points at $REMOTE_COMMIT, but $OUT was built from $BUILT_COMMIT; refusing to publish"
echo "gates passed: $TAG is on $REMOTE at $BUILT_COMMIT, the commit $OUT was built from"
# The staged set, in upload order: the sdist, packages, signatures and the client
# key first; the repodata last, so no client sees an index for packages not yet up.
cd "$OUT"
FILES=(
sdist/*.tar.gz
xbps/repo/*.x86_64.xbps
xbps/repo/*.sig2
xbps/keys/FINGERPRINT
xbps/keys/*.plist
xbps/repo/*-repodata*
)
NAMES=$(for f in "${FILES[@]}"; do basename "$f"; done)
SDIST_DIGEST=$want_digest
NOTES=$(printf '%s\n\n%s\n%s\n' \
"Signed xbps repository, Void Linux glibc x86_64, built from ${TAG} by packaging/void/build-release.sh and published by packaging/void/publish-release.sh. This is a third-party repository published by this project, not by upstream Void Linux. Registration: README.md." \
"$(cat xbps/keys/FINGERPRINT)" \
"sdist sha256: ${SDIST_DIGEST} (voculinux-*.tar.gz, the source the packages are built from)")
# Every request goes through api(), which is the only place curl runs. In a dry
# run each helper prints its planned call instead and no request is made.
api() {
local method=$1 url=$2 code
shift 2
code=$(curl -sS -o "$BODY" -w '%{http_code}' -X "$method" -H "@$AUTH" "$@" "$url") ||
fail "request failed: $method $url"
case $code in
2[0-9][0-9]) ;;
*)
cat "$BODY" >&2
echo >&2
fail "$method $url returned HTTP $code"
;;
esac
}
json_field() {
python3 -I -c 'import json,sys; print(json.load(open(sys.argv[1]))[sys.argv[2]])' "$BODY" "$1"
}
release_payload() {
python3 -I -c 'import json,sys; print(json.dumps({"tag_name": sys.argv[1], "target_commitish": sys.argv[2], "name": sys.argv[3], "body": sys.argv[4], "draft": False, "prerelease": False}))' "$@"
}
# release_id TAG: sets REL_ID to the release's id, or to "" when the tag has no
# release. Sets a variable rather than printing: a failure inside $(...) in an
# argument position would otherwise be swallowed.
release_id() {
local code
REL_ID=""
code=$(curl -sS -o "$BODY" -w '%{http_code}' -H "@$AUTH" \
"$GITEA_API/releases/tags/$(urlencode "$1")") || fail "request failed: GET releases/tags/$1"
case $code in
200) REL_ID=$(json_field id) ;;
404) ;;
*)
cat "$BODY" >&2
echo >&2
fail "GET releases/tags/$1 returned HTTP $code"
;;
esac
}
# ensure_release TAG NAME NOTES: sets REL_ID, creating the release at the built
# commit when the tag has none.
ensure_release() {
local payload
if [ "$DRY_RUN" -eq 1 ]; then
plan "GET $GITEA_API/releases/tags/$1"
plan "POST $GITEA_API/releases only when $1 has no release: tag_name=$1 target_commitish=$BUILT_COMMIT"
REL_ID="<$1 release id>"
return 0
fi
release_id "$1"
if [ -z "$REL_ID" ]; then
payload=$(release_payload "$1" "$BUILT_COMMIT" "$2" "$3")
api POST "$GITEA_API/releases" -H 'Content-Type: application/json' -d "$payload"
REL_ID=$(json_field id)
fi
}
# delete_assets ID MODE: MODE is "all", or "same" for only the assets whose names
# this run uploads again (a rerun on a tag release leaves nothing stale behind).
delete_assets() {
local listing aid aname
if [ "$DRY_RUN" -eq 1 ]; then
plan "GET $GITEA_API/releases/$1/assets"
if [ "$2" = all ]; then
plan "DELETE $GITEA_API/releases/$1/assets/<asset id> for every asset"
else
plan "DELETE $GITEA_API/releases/$1/assets/<asset id> for each asset this set uploads again"
fi
return 0
fi
api GET "$GITEA_API/releases/$1/assets"
listing=$(python3 -I -c 'import json,sys; [print(a["id"], a["name"]) for a in json.load(open(sys.argv[1]))]' "$BODY")
[ -n "$listing" ] || return 0
while read -r aid aname; do
if [ "$2" = all ] || printf '%s\n' "$NAMES" | grep -qxF -- "$aname"; then
api DELETE "$GITEA_API/releases/$1/assets/$aid"
echo "deleted: $aname"
fi
done <<<"$listing"
}
upload_assets() {
local f name
for f in "${FILES[@]}"; do
name=$(basename "$f")
if [ "$DRY_RUN" -eq 1 ]; then
plan "POST $GITEA_API/releases/$1/assets?name=$(urlencode "$name") attachment=$f"
continue
fi
api POST "$GITEA_API/releases/$1/assets?name=$(urlencode "$name")" \
-F "attachment=@${f};filename=${name}"
echo "uploaded: $name"
done
}
# verify_assets ID: every file this run uploaded is on the release.
verify_assets() {
local have missing
if [ "$DRY_RUN" -eq 1 ]; then
plan "GET $GITEA_API/releases/$1/assets (every uploaded name must be listed)"
return 0
fi
api GET "$GITEA_API/releases/$1/assets"
have=$(python3 -I -c 'import json,sys; [print(a["name"]) for a in json.load(open(sys.argv[1]))]' "$BODY" | sort)
missing=$(printf '%s\n' "$NAMES" | sort | comm -23 - <(printf '%s\n' "$have"))
[ -z "$missing" ] || fail "release $1 is missing uploaded assets: $missing"
}
if [ "$DRY_RUN" -eq 1 ]; then
echo "dry run: no request is sent and no token is read. Planned calls, in order:"
echo " Authorization: token <redacted>, API $GITEA_API"
echo "tag release $TAG (commit $BUILT_COMMIT):"
REL_ID=""
BODY=""
ensure_release "$TAG" "$TAG" "$NOTES"
TAG_ID=$REL_ID
delete_assets "$TAG_ID" same
upload_assets "$TAG_ID"
verify_assets "$TAG_ID"
echo "latest release (rolling, the registration URL serves it):"
ensure_release latest latest "$NOTES"
LATEST_ID=$REL_ID
plan "PATCH $GITEA_API/releases/$LATEST_ID name=latest, body=the same notes, with ${TAG} named"
delete_assets "$LATEST_ID" all
upload_assets "$LATEST_ID"
verify_assets "$LATEST_ID"
echo "dry run complete: nothing was published"
exit 0
fi
# Token: GITEA_TOKEN first (the workflow's job token), then the tea login.
AUTH_DIR=$(mktemp -d "${TMPDIR:-/tmp}/publish-release.XXXXXX")
trap 'rm -rf "$AUTH_DIR"' EXIT
AUTH="$AUTH_DIR/authorization.h"
BODY="$AUTH_DIR/body.json"
umask 077
if [ -n "${GITEA_TOKEN:-}" ]; then
printf 'Authorization: token %s\n' "$GITEA_TOKEN" >"$AUTH"
else
python3 -I -c "$TEA_READER" "$TEA_CONFIG" "$GITEA_HOST" "$AUTH" ||
fail "no tea login for $GITEA_HOST with a token (set GITEA_TOKEN, or log in with tea)"
fi
echo "publishing ${TAG} (built from ${BUILT_COMMIT}) to ${GITEA_API}"
echo "== tag release ${TAG}"
ensure_release "$TAG" "$TAG" "$NOTES"
TAG_ID=$REL_ID
delete_assets "$TAG_ID" same
upload_assets "$TAG_ID"
verify_assets "$TAG_ID"
# The rolling release the registration URL points at. The tag it was created
# with stays where it was first made; its assets are what clients download.
echo "== latest release"
LATEST_NOTES=$(printf '%s\n\n%s\n' \
"Rolling signed xbps repository, currently built from ${TAG} (${BUILT_COMMIT})." \
"$NOTES")
ensure_release latest latest "$LATEST_NOTES"
LATEST_ID=$REL_ID
PATCH_PAYLOAD=$(python3 -I -c 'import json,sys; print(json.dumps({"name": sys.argv[1], "body": sys.argv[2]}))' latest "$LATEST_NOTES")
api PATCH "$GITEA_API/releases/$LATEST_ID" -H 'Content-Type: application/json' -d "$PATCH_PAYLOAD"
delete_assets "$LATEST_ID" all
upload_assets "$LATEST_ID"
verify_assets "$LATEST_ID"
echo "OK: signed repository published to ${TAG} and latest"
tests/test_void_publish_release.py
+576
View File
@@ -0,0 +1,576 @@
"""Offline guards for publishing a built Void release (spec #16; packaging/void/publish-release.sh).
``publish-release.sh`` publishes an out-dir that ``build-release.sh release``
already built and verified. These tests run the real script in a scratch git
checkout whose ``origin`` is a local bare repository, so the tag gate and the
``ls-remote`` comparison are exercised for real. The Gitea API is the stateful
``curl`` stand-in from ``test_xbps_release_workflow.py`` (``FakeGitea``): every
request, its order and its authentication are recorded, and nothing leaves the
machine. The release itself is never run.
The expected values come from the documented rules (the tag must be on origin
at the built commit, the repodata goes last, the token never leaves the header
file), not from the script.
"""
import hashlib
import os
import re
import shutil
import subprocess
from pathlib import Path
from typing import List, Optional
import pytest
from test_xbps_release_workflow import FakeGitea
REPO_ROOT = Path(__file__).resolve().parents[1]
VOID = REPO_ROOT / "packaging" / "void"
BUILD_RELEASE = VOID / "build-release.sh"
PUBLISH_RELEASE = VOID / "publish-release.sh"
SMOKE_SH = VOID / "smoke.sh"
PINS = VOID / "tool_checksums.txt"
PLIST = VOID / "repository-key.plist"
TEMPLATE = VOID / "srcpkgs" / "voculinux" / "template"
TAG = "v0.19.0"
XBPS = "0.19.0"
SDIST = "voculinux-0.19.0.tar.gz"
HOST = "https://git.bongbetic.com"
API = f"{HOST}/api/v1/repos/xavierk/voculinux"
TOKEN = "gitea-token-from-the-environment"
TEA_TOKEN = "tea-token-from-the-config-file"
FINGERPRINT = "fa:d1:00:32:ce:b1:b9:fd:73:05:05:76:75:54:37:25"
GIT_ENV = {
"GIT_CONFIG_GLOBAL": os.devnull,
"GIT_CONFIG_NOSYSTEM": "1",
"GIT_AUTHOR_NAME": "fixture",
"GIT_AUTHOR_EMAIL": "fixture@example.invalid",
"GIT_COMMITTER_NAME": "fixture",
"GIT_COMMITTER_EMAIL": "fixture@example.invalid",
"GIT_TERMINAL_PROMPT": "0",
}
def _git(cwd: Path, *args: str) -> str:
result = subprocess.run(
["git", *args],
cwd=cwd,
env={**os.environ, **GIT_ENV},
capture_output=True,
text=True,
check=True,
timeout=60,
)
return result.stdout.strip()
def _smoke_packages() -> List[str]:
match = re.search(r'^PKGS="([^"]+)"', SMOKE_SH.read_text(encoding="utf-8"), re.M)
assert match, "smoke.sh must declare its ordered PKGS list"
return match.group(1).split()
class Release:
"""A scratch checkout with an origin, a built out-dir, and a fake Gitea on PATH."""
def __init__(self, root: Path) -> None:
self.root = root
self.tree = root / "checkout"
self.origin = root / "origin.git"
self.out = root / "publish"
self.home = root / "home"
self.tmp = root / "tmp"
for directory in (self.home, self.tmp):
directory.mkdir()
self._write_tree()
_git(root, "init", "-q", "--bare", str(self.origin))
_git(self.tree, "remote", "add", "origin", str(self.origin))
self.gitea = FakeGitea(root)
def _write_tree(self) -> None:
pkg = self.tree / "packaging" / "void"
pkg.mkdir(parents=True)
for source in (BUILD_RELEASE, PUBLISH_RELEASE, SMOKE_SH, PINS, PLIST):
shutil.copy2(source, pkg / source.name)
(pkg / "srcpkgs" / "voculinux").mkdir(parents=True)
text = TEMPLATE.read_text(encoding="utf-8")
text = re.sub(r"^version=.*$", f"version={XBPS}", text, flags=re.M)
text = re.sub(r"^_tag=.*$", f"_tag={TAG}", text, flags=re.M)
(pkg / "srcpkgs" / "voculinux" / "template").write_text(text, encoding="utf-8")
src = self.tree / "src" / "vocalinux"
src.mkdir(parents=True)
(src / "version.py").write_text('__version__ = "0.19.0"\n', encoding="utf-8")
_git(self.tree, "init", "-q")
_git(self.tree, "add", "-A")
_git(self.tree, "commit", "-q", "-m", "release fixture")
def commit(self, message: str) -> str:
(self.tree / "CHANGELOG.fixture").write_text(message + "\n", encoding="utf-8")
_git(self.tree, "add", "-A")
_git(self.tree, "commit", "-q", "-m", message)
return _git(self.tree, "rev-parse", "HEAD")
def head(self) -> str:
return _git(self.tree, "rev-parse", "HEAD")
def push_tag(self, tag: str = TAG, annotated: bool = True, at: Optional[str] = None) -> None:
target = at or "HEAD"
if annotated:
_git(self.tree, "tag", "-a", tag, target, "-m", f"Voculinux {tag}")
else:
_git(self.tree, "tag", tag, target)
_git(self.tree, "push", "-q", "origin", f"refs/tags/{tag}")
def stage(self, built_commit: str, tag: str = TAG) -> None:
"""Lay out publish/ the way build-release.sh writes it, for ``built_commit``."""
sdist = self.out / "sdist"
repo = self.out / "xbps" / "repo"
keys = self.out / "xbps" / "keys"
for directory in (sdist, repo, keys):
directory.mkdir(parents=True)
payload = b"canonical sdist bytes"
(sdist / SDIST).write_bytes(payload)
(self.out / "SDIST_SHA256").write_text(hashlib.sha256(payload).hexdigest() + "\n")
for name in _smoke_packages():
pkg = f"{name}-{XBPS}_1"
(repo / f"{pkg}.x86_64.xbps").write_bytes(b"package " + pkg.encode())
(repo / f"{pkg}.x86_64.xbps.sig2").write_bytes(b"signature " + pkg.encode())
(repo / "x86_64-repodata").write_bytes(b"signed index")
(keys / "FINGERPRINT").write_text(
f"signed-by: Voculinux <xbps@voculinux.com>\nfingerprint: {FINGERPRINT}\n",
encoding="utf-8",
)
shutil.copy2(PLIST, keys / f"{FINGERPRINT}.plist")
(self.out / "BUILT_FROM").write_text(f"tag={tag}\ncommit={built_commit}\n")
def staged_names(self) -> List[str]:
return sorted(
p.name
for sub in ("sdist", "xbps/repo", "xbps/keys")
for p in (self.out / sub).iterdir()
)
def run(
self,
*args: str,
tag: str = TAG,
token: Optional[str] = TOKEN,
tea_config: Optional[Path] = None,
fail: str = "",
) -> subprocess.CompletedProcess:
env = {
"PATH": f"{self.gitea.bin}{os.pathsep}{os.environ['PATH']}",
"HOME": str(self.home),
"TMPDIR": str(self.tmp),
"GITEA_HOST": HOST,
"GITEA_API": API,
"TEA_CONFIG": str(tea_config or self.home / "no-tea-config.yml"),
"FAKE_GITEA_STATE": str(self.gitea.state_file),
"FAKE_GITEA_BASE": API,
"FAKE_GITEA_FAIL": fail,
**GIT_ENV,
}
if token is not None:
env["GITEA_TOKEN"] = token
return subprocess.run(
["bash", str(self.tree / "packaging" / "void" / "publish-release.sh"), tag, *args],
cwd=self.tree,
env=env,
capture_output=True,
text=True,
timeout=60,
check=False,
)
def publish(self, *extra: str, **kwargs) -> subprocess.CompletedProcess:
return self.run("--out-dir", str(self.out), *extra, **kwargs)
@pytest.fixture()
def release(tmp_path: Path) -> Release:
if shutil.which("git") is None or shutil.which("bash") is None:
pytest.skip("git and bash are required")
return Release(tmp_path)
def _published(release: Release) -> Release:
"""A tree whose origin holds v0.19.0 at HEAD, and a staged out-dir for HEAD."""
release.push_tag()
release.stage(release.head())
return release
def _uploads(gitea: FakeGitea, tag: str) -> List[str]:
return gitea.uploads(tag)
# ---------------------------------------------------------------------------
# The gates: nothing is published unless origin's tag is the built commit
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("annotated", [True, False], ids=["annotated", "lightweight"])
def test_a_tag_at_the_built_commit_publishes_both_releases(
release: Release, annotated: bool
) -> None:
release.push_tag(annotated=annotated)
release.stage(release.head())
result = release.publish()
assert result.returncode == 0, result.stdout + result.stderr
assert "gates passed" in result.stdout
expected = release.staged_names()
assert SDIST in expected
for tag in (TAG, "latest"):
assert sorted(release.gitea.asset_names(tag)) == expected, tag
uploaded = _uploads(release.gitea, tag)
assert sorted(uploaded) == expected, f"{tag}: {uploaded}"
assert uploaded[-1] == "x86_64-repodata", "the index goes last, after its packages"
def test_a_tag_that_is_not_on_origin_is_refused(release: Release) -> None:
release.stage(release.head())
result = release.publish()
assert result.returncode != 0
assert f"tag {TAG} is not on origin" in result.stderr
assert not release.gitea.state["log"], "no request may be made for an unpublished tag"
def test_a_tag_that_only_shares_a_prefix_with_the_release_is_refused(release: Release) -> None:
"""origin holds v0.19.0x; the release is v0.19.0. A prefix match must not pass."""
_git(release.tree, "tag", "-a", "v0.19.0x", "-m", "other", "HEAD")
_git(release.tree, "push", "-q", "origin", "refs/tags/v0.19.0x")
release.stage(release.head())
result = release.publish()
assert result.returncode != 0
assert f"tag {TAG} is not on origin" in result.stderr
assert not release.gitea.state["log"]
def test_a_tag_on_another_commit_is_refused(release: Release) -> None:
built = release.head()
release.stage(built)
later = release.commit("a commit after the build")
release.push_tag(at=later)
result = release.publish()
assert result.returncode != 0
assert f"points at {later}, but" in result.stderr
assert f"built from {built}" in result.stderr
assert not release.gitea.state["log"]
def test_an_out_dir_without_a_build_record_is_refused(release: Release) -> None:
_published(release)
(release.out / "BUILT_FROM").unlink()
result = release.publish()
assert result.returncode != 0
assert "no BUILT_FROM record" in result.stderr
assert not release.gitea.state["log"]
def test_an_out_dir_built_for_another_tag_is_refused(release: Release) -> None:
release.push_tag()
release.stage(release.head(), tag="v0.18.5")
result = release.publish()
assert result.returncode != 0
assert "was built for 'v0.18.5'" in result.stderr
assert not release.gitea.state["log"]
def test_a_sdist_that_does_not_match_its_digest_is_refused(release: Release) -> None:
_published(release)
(release.out / "sdist" / SDIST).write_bytes(b"tampered after the build")
result = release.publish()
assert result.returncode != 0
assert "does not match SDIST_SHA256" in result.stderr
assert not release.gitea.state["log"]
def test_a_client_key_other_than_the_committed_plist_is_refused(release: Release) -> None:
_published(release)
(release.out / "xbps" / "keys" / f"{FINGERPRINT}.plist").write_text("<plist/>\n")
result = release.publish()
assert result.returncode != 0
assert "not the committed packaging/void/repository-key.plist" in result.stderr
assert not release.gitea.state["log"]
def test_a_repository_without_its_index_is_refused(release: Release) -> None:
_published(release)
(release.out / "xbps" / "repo" / "x86_64-repodata").unlink()
result = release.publish()
assert result.returncode != 0
assert "has no repository index" in result.stderr
assert not release.gitea.state["log"]
def test_a_tag_that_fails_the_tag_gate_is_refused(release: Release) -> None:
"""v0.19.0-beta is not the release: the tag gate refuses it before origin is read."""
_published(release)
result = release.publish(tag="v0.19.0-beta")
assert result.returncode != 0
assert "fails the tag gate" in result.stderr
assert not release.gitea.state["log"]
def test_usage_errors_exit_2(release: Release) -> None:
"""No --out-dir is a usage error, and so is a call with no tag at all."""
no_out_dir = release.run()
assert no_out_dir.returncode == 2, no_out_dir.stderr
assert "usage: publish-release.sh" in no_out_dir.stderr
bare = subprocess.run(
["bash", str(release.tree / "packaging" / "void" / "publish-release.sh")],
capture_output=True,
text=True,
check=False,
)
assert bare.returncode == 2, bare.stderr
# ---------------------------------------------------------------------------
# Dry run: every gate, the planned calls, and no request at all
# ---------------------------------------------------------------------------
def test_a_dry_run_plans_every_call_and_makes_none(release: Release) -> None:
_published(release)
result = release.publish("--dry-run")
assert result.returncode == 0, result.stdout + result.stderr
assert "dry run" in result.stdout
assert release.gitea.state["log"] == [], "a dry run must not reach the API"
lines = result.stdout.splitlines()
tag_section = lines.index(f"tag release {TAG} (commit {release.head()}):")
latest_section = lines.index("latest release (rolling, the registration URL serves it):")
assert tag_section < latest_section
for start, end in ((tag_section, latest_section), (latest_section, len(lines))):
uploads = [line for line in lines[start:end] if "/assets?name=" in line]
assert uploads, "a section plans its uploads"
assert uploads[-1].endswith("attachment=xbps/repo/x86_64-repodata"), uploads[-1]
assert f"POST {API}/releases only when {TAG} has no release" in result.stdout
def test_a_dry_run_never_prints_or_reads_a_token(release: Release) -> None:
_published(release)
result = release.publish("--dry-run", token=TOKEN)
assert result.returncode == 0, result.stdout + result.stderr
assert TOKEN not in result.stdout + result.stderr
assert "Authorization: token <redacted>" in result.stdout
def test_a_dry_run_needs_no_token_at_all(release: Release) -> None:
_published(release)
result = release.publish("--dry-run", token=None)
assert result.returncode == 0, result.stdout + result.stderr
def test_a_dry_run_still_refuses_a_tag_that_is_not_on_origin(release: Release) -> None:
release.stage(release.head())
result = release.publish("--dry-run")
assert result.returncode != 0
assert "is not on origin" in result.stderr
assert not release.gitea.state["log"]
# ---------------------------------------------------------------------------
# The publish itself, against the stateful stand-in
# ---------------------------------------------------------------------------
def test_the_latest_release_is_created_at_the_built_commit(release: Release) -> None:
_published(release)
assert release.publish().returncode == 0
built = release.head()
latest = release.gitea.release("latest")
assert latest["target_commitish"] == built
tag_release = release.gitea.release(TAG)
assert tag_release["target_commitish"] == built
assert not tag_release["draft"] and not latest["draft"]
assert TAG in latest["body"], "latest says which tag it was built from"
assert built in latest["body"]
def test_every_request_is_to_the_fork_api_and_carries_the_token(release: Release) -> None:
_published(release)
assert release.publish().returncode == 0
log = release.gitea.state["log"]
assert log
for entry in log:
assert entry["url"].startswith(API + "/"), entry["url"]
assert entry["auth"] == [f"Authorization: token {TOKEN}"], entry
posts = [e["data"]["tag_name"] for e in log if e["url"] == f"{API}/releases"]
assert posts == [TAG, "latest"], "the tag release is filled before latest is touched"
def test_the_notes_state_the_sdist_digest_exactly(release: Release) -> None:
_published(release)
assert release.publish().returncode == 0
digest = (release.out / "SDIST_SHA256").read_text(encoding="utf-8").strip()
assert f"sdist sha256: {digest}" in release.gitea.release("latest")["body"]
assert FINGERPRINT in release.gitea.release(TAG)["body"]
def test_a_rerun_replaces_same_named_assets_on_the_tag_release(release: Release) -> None:
_published(release)
release.gitea.seed_release(TAG, [SDIST, "x86_64-repodata"])
before = {a["name"]: a["id"] for a in release.gitea.release(TAG)["assets"]}
assert release.publish().returncode == 0
after = {a["name"]: a["id"] for a in release.gitea.release(TAG)["assets"]}
assert after[SDIST] != before[SDIST]
assert after["x86_64-repodata"] != before["x86_64-repodata"]
assert sorted(after) == release.staged_names()
def test_latest_replaces_every_asset_it_already_held(release: Release) -> None:
_published(release)
release.gitea.seed_release("latest", ["voculinux-0.18.5_1.x86_64.xbps", "stale.txt"])
assert release.publish().returncode == 0
assert sorted(release.gitea.asset_names("latest")) == release.staged_names()
assert (
len([r for r in release.gitea.state["releases"].values() if r["tag_name"] == "latest"]) == 1
)
def test_uploads_come_only_from_the_staged_set(release: Release) -> None:
_published(release)
assert release.publish().returncode == 0
staged = set(release.staged_names())
for tag in (TAG, "latest"):
assert set(_uploads(release.gitea, tag)) == staged
for entry in release.gitea.state["log"]:
for form in entry["forms"]:
path = re.match(r"attachment=@(.*?);", form).group(1)
assert path.startswith(("sdist/", "xbps/")), f"not from the staged set: {path}"
def test_the_colon_in_the_fingerprint_name_is_percent_encoded(release: Release) -> None:
_published(release)
assert release.publish().returncode == 0
plist_urls = [e["url"] for e in release.gitea.state["log"] if ".plist" in e["url"]]
assert plist_urls and all(":" not in u.split("name=")[1] for u in plist_urls), plist_urls
assert f"{FINGERPRINT}.plist" in release.gitea.asset_names("latest")
@pytest.mark.parametrize(
"fail",
[
"POST|/releases|403", # the token may not create releases
"POST|/assets?|500", # an upload fails midway
"DELETE|/assets/|500",
"GET|/releases/tags/|500",
"PATCH|/releases/|500",
],
)
def test_any_non_2xx_response_fails_the_publish(release: Release, fail: str) -> None:
_published(release)
release.gitea.seed_release("latest", ["stale"])
release.gitea.seed_release(TAG, ["x86_64-repodata"])
result = release.publish(fail=fail)
assert result.returncode != 0, f"{fail} must fail the publish"
assert "HTTP" in result.stderr or "failed" in result.stderr
def test_a_failed_tag_release_never_touches_latest(release: Release) -> None:
_published(release)
result = release.publish(fail="POST|/assets?|500")
assert result.returncode != 0
assert not any(r["tag_name"] == "latest" for r in release.gitea.state["releases"].values())
# ---------------------------------------------------------------------------
# The token: from GITEA_TOKEN or the tea login, never printed, never left behind
# ---------------------------------------------------------------------------
def _tea_config(path: Path, token: str) -> Path:
path.write_text(
"logins:\n"
" - name: other\n"
" url: https://example.invalid\n"
" token: token-for-another-host\n"
" default: true\n"
" - name: xavierk\n"
f" url: {HOST}\n"
f" token: {token}\n"
" default: false\n"
" user: xavierk\n"
"preferences:\n"
" editor: false\n",
encoding="utf-8",
)
return path
def test_the_token_comes_from_the_tea_login_for_this_host_and_is_never_printed(
release: Release,
) -> None:
_published(release)
config = _tea_config(release.root / "tea.yml", TEA_TOKEN)
result = release.publish(token=None, tea_config=config)
assert result.returncode == 0, result.stdout + result.stderr
assert TEA_TOKEN not in result.stdout + result.stderr
for entry in release.gitea.state["log"]:
assert entry["auth"] == [f"Authorization: token {TEA_TOKEN}"], entry
def test_no_token_and_no_tea_login_refuses_before_any_request(release: Release) -> None:
_published(release)
config = _tea_config(release.root / "tea.yml", TEA_TOKEN)
other = release.root / "other-host.yml"
other.write_text(config.read_text(encoding="utf-8").replace(HOST, "https://elsewhere.invalid"))
result = release.publish(token=None, tea_config=other)
assert result.returncode != 0
assert f"no tea login for {HOST}" in result.stderr
assert not release.gitea.state["log"]
def test_the_header_file_is_removed_when_the_publish_ends(release: Release) -> None:
_published(release)
assert release.publish().returncode == 0
assert list(release.tmp.glob("publish-release.*")) == []
def test_a_failed_publish_also_removes_the_header_file(release: Release) -> None:
_published(release)
assert release.publish(fail="POST|/releases|403").returncode != 0
assert list(release.tmp.glob("publish-release.*")) == []
def test_the_token_never_appears_in_the_output_of_a_failed_request(release: Release) -> None:
_published(release)
result = release.publish(fail="POST|/releases|403")
assert TOKEN not in result.stdout + result.stderr
# ---------------------------------------------------------------------------
# The script itself
# ---------------------------------------------------------------------------
def test_the_script_is_executable_bash_that_fails_closed() -> None:
text = PUBLISH_RELEASE.read_text(encoding="utf-8")
assert text.startswith("#!/usr/bin/env bash\n")
assert "set -euo pipefail" in text
assert os.access(PUBLISH_RELEASE, os.X_OK)
def test_the_script_never_enables_tracing_or_echoes_the_header() -> None:
code = "\n".join(
line
for line in PUBLISH_RELEASE.read_text(encoding="utf-8").splitlines()
if not line.lstrip().startswith("#")
)
assert "set -x" not in code
assert 'cat "$AUTH"' not in code
assert 'echo "$GITEA_TOKEN"' not in code and "printf '%s' \"$GITEA_TOKEN\"" not in code
def test_the_workflow_publish_step_and_this_script_name_the_same_api() -> None:
"""Two copies of the publish logic: they must publish to the same place."""
workflow = (REPO_ROOT / ".gitea" / "workflows" / "xbps-release.yml").read_text(encoding="utf-8")
assert "GITEA_API: https://git.bongbetic.com/api/v1/repos/xavierk/voculinux" in workflow
assert "https://git.bongbetic.com/api/v1/repos/xavierk/voculinux" in PUBLISH_RELEASE.read_text(
encoding="utf-8"
)
script = PUBLISH_RELEASE.read_text(encoding="utf-8")
assert "xbps/repo/*-repodata*" in workflow and "xbps/repo/*-repodata*" in script
tests/test_void_release_build.py
+62
View File
@@ -90,6 +90,17 @@ def make_tree(
version_py if version_py is not None else f'"""Version."""\n\n__version__ = "{version}"\n',
encoding="utf-8",
)
# A release is built from a clean git checkout, so the fixture is one: committed,
# with nothing left uncommitted. Tests that want a dirty tree change it after this.
git = ["git", "-c", "user.name=fixture", "-c", "user.email=fixture@example.invalid"]
subprocess.run(["git", "init", "-q"], cwd=root, check=True, capture_output=True)
subprocess.run(["git", "add", "-A"], cwd=root, check=True, capture_output=True)
subprocess.run(
[*git, "-c", "commit.gpgsign=false", "commit", "-q", "-m", "fixture"],
cwd=root,
check=True,
capture_output=True,
)
return root
@@ -466,6 +477,7 @@ def test_the_checks_run_before_the_sdist_and_before_any_container() -> None:
pipeline = _body(text, "run_pipeline")
order = [
pipeline.index("check_template"),
pipeline.index("release_checkout"),
pipeline.index("check_key"),
pipeline.index("build_sdist"),
pipeline.index("phase_build"),
@@ -473,6 +485,56 @@ def test_the_checks_run_before_the_sdist_and_before_any_container() -> None:
assert order == sorted(order), order
def test_a_release_records_the_commit_it_is_built_from() -> None:
"""publish-release.sh refuses unless origin's tag peels to this commit."""
text = _code(BUILD_RELEASE)
pipeline = _body(text, "run_pipeline")
assert 'printf \'tag=%s\\ncommit=%s\\n\' "$TAG" "$SOURCE_COMMIT" >"$OUT/BUILT_FROM"' in pipeline
assert "rev-parse --verify HEAD" in _body(text, "release_checkout")
@pytest.fixture()
def no_container(tmp_path: Path) -> tuple:
"""A podman that must never run, with a log that proves it did not."""
bindir = tmp_path / "nocontainer-bin"
bindir.mkdir()
log = tmp_path / "nocontainer-calls.log"
script = bindir / "podman"
script.write_text(f'#!/bin/sh\necho "$@" >> "{log}"\nexit 97\n', encoding="utf-8")
script.chmod(0o755)
return f"{bindir}{os.pathsep}{os.environ['PATH']}", log
def test_a_release_refuses_an_uncommitted_checkout_before_anything_runs(
tmp_path: Path, no_container: tuple
) -> None:
path, log = no_container
tree = make_tree(tmp_path / "tree", "0.18.5", template_version="0.18.5", template_tag="v0.18.5")
(tree / "scratch.txt").write_text("not committed\n", encoding="utf-8")
out = tmp_path / "out"
tmp = tmp_path / "work"
tmp.mkdir()
result = run_build(tree, "release", "v0.18.5", "--out-dir", str(out), path=path, tmp=tmp)
assert result.returncode == 1, result.stdout + result.stderr
assert "uncommitted or untracked changes" in result.stderr
assert not log.exists() and not out.exists()
def test_a_release_refuses_a_tree_that_is_not_a_git_checkout(
tmp_path: Path, no_container: tuple
) -> None:
path, log = no_container
tree = make_tree(tmp_path / "tree", "0.18.5", template_version="0.18.5", template_tag="v0.18.5")
shutil.rmtree(tree / ".git")
out = tmp_path / "out"
tmp = tmp_path / "work"
tmp.mkdir()
result = run_build(tree, "release", "v0.18.5", "--out-dir", str(out), path=path, tmp=tmp)
assert result.returncode == 1, result.stdout + result.stderr
assert "is not one" in result.stderr
assert not log.exists() and not out.exists()
def test_the_key_is_never_echoed_or_copied() -> None:
text = _code(BUILD_RELEASE)
assert "set -x" not in text
tests/test_void_signed_repo.py
+2 -2
View File
@@ -92,7 +92,7 @@ def _gitea_workflow_text() -> str:
def test_the_xbps_job_lives_in_the_gitea_workflow_not_the_github_release() -> None:
"""One signing path: the tag-triggered Gitea workflow runs build-release.sh."""
"""One signing path: the manual Gitea workflow runs build-release.sh, and no tag starts it."""
github = _release_workflow_text()
for marker in (
"build-xbps",
@@ -108,7 +108,7 @@ def test_the_xbps_job_lives_in_the_gitea_workflow_not_the_github_release() -> No
assert SIGNING_SECRET in gitea, "the Gitea workflow must read the XBPS_SIGNING_KEY secret"
assert "podman" in gitea, "the Gitea workflow must run the pipeline with podman"
workflow = yaml.safe_load(gitea)
assert workflow[True] == {"push": {"tags": ["v*"]}}, "tag pushes are the only trigger"
assert set(workflow[True]) == {"workflow_dispatch"}, "manual dispatch is the only trigger"
#: Registration URL every client snippet names: the rolling ``latest``
tests/test_xbps_release_workflow.py
+18 -6
View File
@@ -1,7 +1,8 @@
"""Offline guards for the signed xbps release workflow (tickets #28 and #29).
Releasing the Void repository is a tag push on the fork's Gitea host. The
workflow ``.gitea/workflows/xbps-release.yml`` gates the tag, stages the
Releasing the Void repository is a local maintainer run on an x86_64 Void host;
the workflow ``.gitea/workflows/xbps-release.yml`` is manual dispatch only and is
kept for a future x86_64 host-mode runner. It gates the tag, stages the
signing key on tmpfs, runs ``packaging/void/build-release.sh`` (the one release
path), and publishes exactly what that script staged to the tag's release and
to the rolling ``latest`` release the client registration URL serves.
@@ -135,10 +136,16 @@ def _run_step(
# ---------------------------------------------------------------------------
def test_triggers_only_on_version_tag_pushes() -> None:
"""A tag push is a human act: no schedule, no dispatch, no PR trigger."""
def test_is_manual_dispatch_only_and_never_runs_on_a_tag_push() -> None:
"""A tag push must not start the job: releases are built and published locally.
The only runner with the bongbetic-ci label is ARM64 and cannot build the
glibc x86_64 release, so a tag-triggered run would fail (or worse, publish a
partial set). No schedule, no push, no pull request trigger.
"""
triggers = _workflow()[True] # YAML 1.1 reads the bare key `on` as True
assert triggers == {"push": {"tags": ["v*"]}}, triggers
assert set(triggers) == {"workflow_dispatch"}, triggers
assert "push" not in triggers and "schedule" not in triggers
def test_grants_only_contents_write() -> None:
@@ -462,7 +469,12 @@ def main(argv):
elif arg == "-X":
method, i = argv[i + 1], i + 2
elif arg == "-H":
headers.append(argv[i + 1])
value = argv[i + 1]
if value.startswith("@"):
# curl reads a header from a file, which keeps the token off argv.
with open(value[1:], encoding="utf-8") as handle:
value = handle.read().strip()
headers.append(value)
i += 2
elif arg == "-d":
data, i = argv[i + 1], i + 2