Roadmap
Single source of truth for scope narrowed during implementation of the v0.1 line plus the larger pre-/post-v1.0 work pulled forward from FEATURES.md and the old PROJECT-DETAILS.md §6.2 table. Distinct from FEATURES.md (up-front product scope), PROJECT-DETAILS.md §6.2 (high-level release-line summary), and docs/project/VERSIONING.md (canonical milestone gates) — those capture what was planned or gated. This file captures what is queued, with a per-entry priority.
Priority taxonomy
Every entry below carries one of five priorities. See VERSIONING.md
for the milestone scheme this hangs off.
gate-v0.5— Blocks the v0.5 external-tester milestone. Must land before v0.5 ships. ~13 entries today.gate-v1.0— Blocks the v1.0 SemVer-stability commitment. Must land in the v0.x line between v0.5 and v1.0. Production polish + epic completion. ~35 entries today.v0.x— Desirable pre-v1.0; lands opportunistically in v0.x releases as effort permits. No specific gate. ~20 entries today.v1.x— Post-v1.0 feature additions on the stable v1.0 line. Windows agent, K8s operator, TUI, saga checkpoint-resume, telemetry gateway, etc.v2.x+— Architectural post-v1.0: federation, supercluster, cloud KMS, marketplace, Web UI.
Entries may carry a partial priority — e.g. gate-v0.5 (rich-rule canonicalisation); v0.x (the rest) — when only some sub-items block a gate.
How to use this file
- When deferring scope mid-task: add an entry with a
Priority:field. Cite the epic + task that produced the deferral and the source-of-truth file/line. - When planning the next release: filter on
Priority: gate-v0.5(orgate-v1.0), prioritise within each gate. - When the deferred work lands: move to a
### Doneblock with a one-line “landed in commit/PR” note, or delete if cleanup is obvious.
Format: each entry is a #### heading; body opens with **Priority**: then What / Why deferred / Acceptance / References bullets.
Tracker integration
tools/trackerctl/config/release-order.yaml groups entries into tracker buckets that match the priority taxonomy (gate-v0.5, gate-v1.0, v0.x, v1.x, v2.x+). Reorder within a bucket there to change tracker-issue ordering; re-tag a priority here to move an entry between buckets.
gate-v0.5 — blocks v0.5 external-tester milestone
service stdlib module — OpenRC / sysvinit / launchd backends
- Priority: gate-v0.5 work complete (OpenRC backend landed — verified on Alpine in the cross-distro matrix; sysvinit backend also landed); launchd (macOS) is post-v1.0 (v1.x).
- What: Epic 08 task 11f ships the
servicemodule with systemd-only. Hosts with a different init system (Alpine’s default OpenRC, Gentoo OpenRC/sysvinit, older RHEL/CentOS sysvinit, macOS launchd) getservice.ErrNoBackendfrom mutating ops. Add provider implementations:openrc(Alpine / Gentoo) — task 11f2 — landed:internal/statemgmt/stdlib/service/openrc.go(rc-service for exists/status/start/stop + rc-update for the default-runlevel enable/disable & show; Lookup runs the three queries since OpenRC has nosystemctl showequivalent), wired indefaultProviderafter systemd via a/run/openrcmarker + rc-service/rc-update presencesysvinit— landed:internal/statemgmt/stdlib/service/sysvinit.go. Runtime viaservice <name> start|stop|status(universal); boot-enable per host —chkconfig(RHEL/CentOS) orupdate-rc.d+ the/etc/rc[2-5].dstart-symlink scan (Debian/Devuan), whichever is detected. Existence is filesystem-based (/etc/init.d/<name>) since there is no universal “exists” command. Wired indefaultProviderafter OpenRC; unit-tested (no sysvinit distro in the cross-distro matrix yet, so a live matrix join is a follow-up).launchd(macOS) — post-v1.0
- Why deferred: systemd covers the whole Epic 08 cross-distro Docker matrix (Debian 12, Ubuntu 22.04/24.04, RHEL 9, Rocky 9 — Alpine 3.19 defaults to OpenRC but a systemd variant exists). One backend at a time keeps PRs reviewable; the Provider interface + detection skeleton is in place.
- Acceptance: openrc backend added (rc-service / rc-update wrappers); auto-detect picks the right init system per host; the
servicemodule’s idempotency tests pass on Alpine 3.19 (OpenRC) in addition to the systemd distros. openrc landed (the gate-v0.5 backend) + sysvinit landed (post-v1.0, pulled forward: chkconfig + update-rc.d); onlylaunchd(macOS) remains post-v1.0. Remaining: the cross-distro matrix harness exercising the Alpine (OpenRC) idempotency end-to-end, and a sysvinit-distro matrix join. - References: Epic 08 task 11f;
internal/statemgmt/stdlib/service/{detect_linux.go,sysvinit.go};internal/statemgmt/stdlib/service/openrc.goas the template.
package stdlib module — dnf, apk, zypper, pacman backends
- Priority: gate-v0.5 work complete (dnf + apk backends landed — verified on Rocky + Alpine in the cross-distro matrix; zypper + pacman backends also landed and unit-tested); live SUSE/Arch matrix coverage is gate-v1.0.
- What: Epic 08 task 11e ships the
packagemodule with apt-only (Debian / Ubuntu). On hosts where no supported package manager is detected the module returnspkg.ErrNoBackend(“no supported package manager detected on this host”) rather than silently doing nothing. Add provider implementations for the other Linux package managers:dnf(RHEL 8+ / Rocky / Fedora) — task 11e2 — landed:internal/statemgmt/stdlib/pkg/dnf.go(dnf install/remove + rpm query; name-version pinning; rpm exit-1 → not-installed), wired indefaultProvider(apt probed first, then dnf+rpm)apk(Alpine) — task 11e3 — landed:internal/statemgmt/stdlib/pkg/apk.go(single apk binary: add/del +apk list --installedquery; name=version pinning; digit-led-token disambiguation against name-glob over-match), wired indefaultProviderafter dnfzypper(openSUSE / SLES) — landed:internal/statemgmt/stdlib/pkg/zypper.go(openSUSE is rpm-based, so it reuses the dnf query path —rpm -q+parseRpmQuery;zypper --non-interactive install|remove, pin via zypper’s exact-editionname=version), wired indefaultProviderafter dnfpacman(Arch) — landed:internal/statemgmt/stdlib/pkg/pacman.go(single binary:-S/-R --noconfirm+pacman -Qquery; version pinning errors — Arch is rolling, so the repos carry only the current version and aversion:request returnsErrVersionPinUnsupportedrather than silently installing latest), wired indefaultProviderafter apk
- Why deferred: One backend at a time keeps PRs reviewable; the Provider interface + detection skeleton is in place, so adding a backend is a new file + a branch in
detect_linux.go+ per-backend tests. - Acceptance: dnf + apk backends added (with appropriate command-line parsers); auto-detect picks the right backend per host; the Epic 08 cross-distro Docker matrix (Debian 12, Ubuntu 22.04/24.04, RHEL 9, Rocky 9, Alpine 3.19) passes the
packagemodule’s idempotency tests. All three v0.5 backends (apt/dnf/apk) landed, plus zypper + pacman pulled forward from post-v1.0, so all five Linux package managers are implemented. The only remaining gate is the cross-distro matrix harness (separate gate-v0.5 entry) actually exercising RHEL/Rocky/Alpine idempotency end-to-end; zypper/pacman are unit-tested (no openSUSE/Arch image in the matrix yet — a matrix join is a follow-up). - References: Epic 08 task 11e;
internal/statemgmt/stdlib/pkg/detect_linux.godefaultProvider;internal/statemgmt/stdlib/pkg/{apt.go,dnf.go}as templates (zypper.go,pacman.go).
firewalld stdlib module — whole-zone management, masquerade/forward-port, direct rules
- Priority: gate-v0.5 work complete (rich-rule canonicalisation landed); v0.x (the rest).
- What: Epic 08 task 11 ships the
firewalldmodule (manage one item — aservice, aport, or arich_rule— in an existing firewalld zone viafirewall-cmd --permanent --zone=Z --query-/add-/remove-…; statespresent/absent; runs--reloadafter a change unlessreload: false). Reserved for v0.x:- Whole-zone management (declare the complete set of services / ports / rich rules / sources / interfaces on a zone and prune the rest) and zone creation; binding interfaces or source addresses to a zone; default-zone management; per-zone target (
ACCEPT/REJECT/DROP). - Toggles for masquerade, ICMP-block (and ICMP-block-inversion), forward ports, ICMP types, and protocol items;
--directrules; ipset management; lockdown / panic mode. - Runtime-only (non-permanent) changes (v1.0 always operates on
--permanentso changes survive a reboot). Canonical-form rich-rule comparisonlanded: rich rules are compared by canonical form (canon.go) — Check lists the zone’s stored rich rules (--list-rich-rules) and runs both the declared rule and each stored rule through the same syntactic canonicaliser (whitespace, attribute quoting, and intra-element attribute order), so a re-formatted rule still matches. The canonicaliser is syntactic only — value-semantic rewrites (e.g. firewalld lowercasing a MAC, normalising a CIDR) aren’t captured, so operators write such values as firewalld stores them.firewallis its own (planned) abstraction module that dispatches across iptables / nftables / firewalld; firewalld is its own backend here.
- Whole-zone management (declare the complete set of services / ports / rich rules / sources / interfaces on a zone and prune the rest) and zone creation; binding interfaces or source addresses to a zone; default-zone management; per-zone target (
- Why deferred: “enable / disable this one service / port / rich rule on this zone” is the v0.1 scope; whole-zone management needs a diffing pass over multiple
--list-…outputs, masquerade / forward-ports / direct rules each have their own flag families and corner cases, and the rich-rule canonicaliser is a real grammar parse. TheProvider(Has/Add/Remove/Reload) and theItem(Kind+Value) types extend cleanly. - Acceptance: a re-formatted rich rule (different whitespace or attribute order) still matches the stored one (met — rich-rule canonicalisation landed). v0.x: a
manage_zone: truedeclaration replaces the zone’s full service/port/rich-rule/source/interface set;masquerade: truetoggles masquerade;forward_port: { port: 80, proto: tcp, to_port: 8080 }round-trips; adirect_rule:declaration round-trips; a stopped firewalld → permanent change persists, reload reports clearly. - References: Epic 08 task 11
_(landed)_;internal/statemgmt/stdlib/firewalld/firewalld.gopackage comment;internal/statemgmt/stdlib/firewalld/params.go.
firewall abstraction — deny action, IPv6 on iptables, chain/table overrides, service catalog expansion
- Priority: gate-v0.5 work complete (IPv6 on iptables + service-catalog expansion both landed); v0.x (the rest).
- What: Epic 08 task 11 ships the
firewallcross-backend abstraction (one declaration → “allow this service / port inbound”, auto-detected backend, hard-coded standard inbound chain /publiczone / IPv4 — seeinternal/statemgmt/stdlib/firewall/firewall.gofor the full translation table). Reserved for v0.x:action: deny(v1.0 is allow-only — translate to-j DROP/drop/ a firewalld deny rich-rule).landed: the iptables backend now runs two sub-applies (v4 + v6) and aggregates the results, sofamily: both(or implicit dual-stack) on the iptables backendfirewallopens both families by default — like nftables (inet) and firewalld. When ip6tables is absent the IPv6 half is skipped gracefully (the IPv4 rule still applies; the StateResult Comment + Diff flag the IPv6 skip loudly). Afamily:override (force a single family) remains v0.x under the chain/table/family-overrides item below.chain/table/familyoverrides on the iptables / nftables backends (v1.0 hard-codes filter+INPUT+ipv4 / inet+filter+input). Operators who need a different chain currently bypass the abstraction and use the backend module directly.Named-service catalog expansionlanded: the catalog is now multi-port (each name maps to a list of port/proto pairs, expanded to one rule per port per family —samba→ 137/udp + 138/udp + 139/tcp + 445/tcp), a curated set of firewalld-native names was added (dhcpv6-client,cockpit,mountd,nfs,kerberos,dns, …), and an/etc/serviceslookup behindstrict_catalog: falseresolves off-catalog names against the host (canonical + alias, tcp/udp/sctp/dccp). Mirroring firewalld’s full ~150 built-ins, and firewalld’s rich (non-port) service semantics, stay v0.x.- Per-source filtering (
source: 10.0.0.0/8) and richer rich-rule-style matches — v1.0 abstraction is allow-port-from-anywhere only. - nftables backend chain creation — v1.0 requires
inet filter inputto already exist (see the nftables module’s V1X entry). - Backend attribution in
StateResult.Comment(e.g. “applied via firewalld”) — v1.0 passes the backend’s Comment through verbatim.
- Why deferred: “allow one service / port on the standard inbound” is the v0.1 scope; the rest each open a meaningful design surface (
action: denycascades into iptables vs firewalld rich-rule asymmetry; dual-stack iptables needs two backend invocations and result-merging; catalog expansion needs a firewalld-services parser or a hand-curated multi-port table). TheModule+BackendDetector+buildSubDeclshape extends cleanly. - Acceptance:
firewall.service: sshopens both v4 and v6 on every backend by default (met — dual-stack iptables landed); the catalog acceptsdhcpv6-clientandsamba(every backend, samba expanding to 4 ports × the backend’s families) (met — catalog expansion landed). v0.x:action: denyround-trips on every backend;chain: custom/family: ipv6override on the iptables backend round-trips;source: 10.0.0.0/8filters the allow on every backend; the StateResult Comment carriesvia <backend>. - References: Epic 08 task 11
_(landed)_;internal/statemgmt/stdlib/firewall/firewall.gopackage comment;internal/statemgmt/stdlib/firewall/services.go. All three backends are now exercised live across the cross-distro matrix viatest/e2e/state/smoke.firewall.sh: iptables (auto-detected, incl. dual-stack + catalog) and nftables (pinned, over a pre-createdinet filter input) on every distro, and firewalld (auto-detected once its daemon is up) on the systemd distros where its D-Bus interface responds in-container.
security stdlib module — AppArmor, SELinux file contexts / ports / modules / logins, absent semantics
- Priority: gate-v0.5 work complete (AppArmor per-profile modes landed); v0.x (the rest).
- What: Epic 08 task 11 shipped the
securitymodule’s SELinux ops (mode: enforcing|permissive|disabledfor the global SELinux mode — persistent via/etc/selinux/config+ runtime viasetenforce;boolean: NAME+value: on|offfor SELinux booleans viasetsebool -P). Reserved for v0.x:AppArmor per-profile modeslanded:apparmor.profile: <name>+apparmor.profile_mode: enforce|complain|disablesets a profile’s mode viaaa-enforce/aa-complain/aa-disable, idempotent againstaa-status --json(disable↔ the profile being unloaded). A secondAppArmorProviderwas added alongside the SELinuxProvider, dispatched by which params are set. Still v0.x: AppArmor framework on/off (whole-subsystem enable/disable) and profile load/reload viaapparmor_parser.- SELinux file contexts (
semanage fcontext -a -t TYPE PATTERN+restorecon -R PATH) — by far the deepest SELinux daily-ops surface beyond booleans. - SELinux port labels (
semanage port -a -t TYPE -p PROTO PORT); SELinux policy module install (semodule -i / -r); SELinux login / user mappings (semanage login). state: absentfor booleans (v1.0 toggle-by-value —value: off— only) and for mode (semantics are ambiguous: “ensure NOT enforcing”?). Avalue:-less boolean op that just queries / reports could pair naturally with absent.- A
persist: falseopt-out that only flips the runtime viasetenforcewithout touching/etc/selinux/config(uncommon but a real use case for short-lived debugging). - Managing SELinux on hosts where
getenforceis missing but/etc/selinux/configis present (split the Provider so persistent-only ops don’t require the user-space tools).
- Why deferred: “ensure SELinux is in this mode” + “ensure this boolean is in this state” covers the day-2 SELinux operations operators most commonly automate; AppArmor’s per-profile mode flips are a meaningfully different shape (parsing
aa-status --jsonetc.), andfcontext/semanage port/etc. each carry their own grammars and patterns. TheProviderinterface + the op-dispatch inModule.Check/Applyextend cleanly. - Acceptance:
apparmor.profile: /usr/bin/foo(the profile name asaa-statuskeys it) +apparmor.profile_mode: enforce|complain|disableround-trips and is idempotent againstaa-status --json(met — AppArmor per-profile modes landed). v0.x:selinux.fcontext: "/srv/www(/.*)?"+selinux.context: httpd_sys_content_tround-trips, withrestoreconinvoked when the rule changes;selinux.port: 8443/tcp+selinux.context: http_port_tround-trips; aselinux.module:declaration installs / removes a policy module viasemodule;state: absentfor a boolean removes the persistent override; apersist: falseruntime flip leaves/etc/selinux/configuntouched. - References: Epic 08 task 11
_(landed)_;internal/statemgmt/stdlib/security/security.gopackage comment;internal/statemgmt/stdlib/security/params.go.
system stdlib module — reboot disconnect-tolerance, cross-distro reboot detection, locale dual-file, absent semantics for reboot/locale
- Priority: v0.x (the rest). Cross-distro reboot detection + reboot disconnect-tolerance (
system.rebooted) both landed — gate-v0.5 complete. - What: Epic 08 task 11 ships the
systemmodule with three operations (exactly one per declaration):banner: motd|issue|issue_net+content;reboot: true+when_file(default/var/run/reboot-required) +delay(0–60 min, default 1);locale: <LANG>. Reserved for v0.x:Reboot result delivery across the disconnectdisconnect-tolerance landed (gate-v0.5): the agent stamps its Linux boot-ID (/proc/sys/kernel/random/boot_id, read once, cached) on every heartbeat; the server’sConnectionManagertracks it per agent and, on a change (the host rebooted), emits asystem.rebootedevent (Source= agent-ID;Data= old/new boot-ID +was_stale) via the injectedEventPublisher. This distinguishes a reboot from a transient flap and gives the deferred “the reboot happened” signal the synchronous RPC reply can’t (thedelay: 0race). Still v0.x: correlating the event with the originating reboot command in the dispatcher (so adelay: 0reboot command is marked completed-via-reboot rather than timed-out), and persisting the per-agent boot-ID across server restarts (in-memory today → a reboot landing exactly during server downtime is not detected).- Arch reboot-needed detection — the marker file +
needs-restarting -rnow cover Debian/Ubuntu and RHEL/Rocky/Fedora (landed:IsRebootNeededchecks the marker, then a binary-detectedneeds-restarting -r/dnf needs-restarting -rprobe). Arch’scheckservices/needrestartis a separate pattern still deferred; Alpine has no reboot-required convention (marker-only there). - Unconditional reboot (
force: true) — v1.0 requires a marker so every Apply is gated. A forced-reboot escape hatch is sometimes needed (e.g. after a manual config change with no marker to set). - Debian’s
/etc/default/localedual-file — v1.0 writes/etc/locale.confonly (the systemd canonical path). Debian (and downstream Ubuntu) historically uses/etc/default/locale; pre-systemd hosts and some images still read that. A dual-write or platform-detected target path is V1X. - Per-
LC_*overrides (LC_ALL,LC_MESSAGES, etc.) and console keymap / X11 keyboard layout management (vconsole.conf,00-keyboard.conf). absentsemantics forreboot(cancel a scheduled reboot viashutdown -c) and forlocale(revert to the compile-time default — ambiguous, may simply mean removing the LANG= line).
- Why deferred: “manage these three settings idempotently” is the v0.1 scope; reboot-across-the-disconnect needs a follow-up-event mechanism the engine doesn’t yet have, RHEL-flavoured reboot detection involves a different tool and exit-code dance, and the locale dual-file is a small but real distro divergence. The op-dispatch in
Module.Check/Applyand theProviderinterface extend cleanly — each V1X item is roughly one more method or one more branch. - Acceptance: a
reboot: truedecl on RHEL 9 detects need viadnf needs-restarting -rexit code without an/var/run/reboot-requiredmarker (met — cross-distro detection landed); the agent delivers asystem.rebootedevent after a reboot Apply, with the boot-ID change as proof (met — boot-ID rides the heartbeat; the server’s ConnectionManager emitssystem.rebootedon the change). v0.x: areboot: truewithforce: truereboots unconditionally; alocale: en_US.UTF-8decl on Debian writes both/etc/locale.confand/etc/default/locale; anlc_all:param round-trips intoLC_ALL=in the locale conf;state: absentforrebootcancels a pendingshutdown -r. - References: Epic 08 task 11
_(landed)_;internal/statemgmt/stdlib/system/{system.go,params.go}; reboot disconnect-tolerance —internal/agent/metadata.go(boot-ID on the heartbeat),internal/controlplane/connection_manager.go(emitRebooted),internal/events/taxonomy.go(system.rebooted),pkg/api/server/server.go(publisher wiring).
lvm stdlib module — existing-VG PV-set mgmt, LV resize, metadata, thin/cache/snapshot
- Priority: v0.x (the rest). LV resize + VG PV-set mgmt landed (gate-v0.5 complete).
- What: Epic 08 task 11 ships the
lvmmodule with three ops (exactly one per decl):pv: <device>(pvcreate/pvremove);vg: <name>+pvs: [<device>, …](vgcreate/vgremove -y);lv: <name>+vg: <vgname>+size: <human>xorextents: <N>%{FREE|VG|PVS|ORIGIN}(lvcreate -y -n <lv> {-L size|-l extents} <vg>/lvremove -y <vg>/<lv>). Reserved for v0.x:Existing-LV resizelanded (grow-only, size-based): a size-based LV below its declared size is grown vialvextend -L <size> [--resizefs] <vg>/<lv>(setresize_fs: trueto grow the contained fs). “At least” semantics — a declared size ≤ live is satisfied, so shrink (lvreduce, filesystem-dangerous) is never performed and the check is idempotent across extent rounding.extents:-based resize stays v0.x (target depends on live free space).Existing-VG PV-set managementlanded (gate-v0.5): an existing VG’s PV set is reconciled against the declaredpvs:— adds viavgextend, removes viavgreduce. Paths are matched against LVM’spv_nameafter resolving symlinks (so/dev/disk/by-id/…works). No-f:vgreducerefuses to drop a PV still holding LV extents (operatorspvmoveoff first). Still v0.x:vgchangeallocation policy, missing-PV cleanup, and automaticpvmove.- Existing-LV resize —
lvextendandlvresize --resizefs;lvreduce(fundamentally dangerous — filesystem must support shrink). v1.0 errors out on a mismatched-size existing LV by way of doing nothing about it (we report exists=true and don’t reconcile size). - LV metadata: tags, allocation policy, stripes / mirror / RAID levels, thin pools + thin volumes, cache origin / pool, snapshots (origin + snapshot LV pair).
- PV metadata:
--metadatasize, allocation tags, restore from a backup. - Filesystem creation on an LV — v1.0 punts to the
diskmodule on the resulting device, orcmdwithmkfs.X; anlvmmkfs:shortcut would land cleanly.
- Why deferred: “create/remove an LVM object” is the v0.1 scope; PV-set mgmt on existing VGs needs a diffing pass over
vgs -o pv_name, LV resize needs filesystem coordination, and thin / cache / RAID each open their own option family. The Provider (HasPV/CreatePV/…/RemoveLV) and the Op-dispatch extend cleanly — each V1X item is roughly one more Provider method or one more branch inapplyVG/applyLV. - Acceptance: an
lv:decl with a new size larger than the live LV runslvextend(with--resizefswhenresize_fs: true) (met — LV resize landed); avg:decl with a differentpvs:set than the live VG reconciles (extends or reduces) idempotently (met — VG PV-set mgmt landed). v0.x:tags: [app=web]on an LV round-trips; a thin-poollv:decl creates the pool + the thin volume; a snapshotlv:declares an origin and creates the COW LV. - References: Epic 08 task 11
_(landed)_;internal/statemgmt/stdlib/lvm/{lvm.go,params.go,provider.go,provider_linux.go}(vgPVDrift/reconcileExisting/GetVGPVs/ExtendVG/ReduceVG/Canonicalize). The create lifecycle (pvcreate/vgcreate/lvcreate), the LV grow (lvextend), and the VG PV-set reconcile (vgextend) are now exercised live across the cross-distro matrix viatest/e2e/state/smoke.lvm.sh;extents:-resize,vgreduce, and theabsentpath remain follow-ups (see the matrix-harness entry below).
disk stdlib module — partition mgmt, resize, label/UUID, encryption, fstype catalog expansion
- Priority: gate-v0.5 work complete (filesystem resize landed for ext2/3/4, xfs, btrfs, f2fs); v0.x (the rest).
- What: Epic 08 task 11 ships the
diskmodule (single op: ensuredevice:has filesystemfstype:—presentorabsent— gated by an explicitforce: truefor any apply that would destroy existing data; mkfs binary resolved per fstype from a curated 9-entry catalog: ext2/3/4, xfs, btrfs, f2fs, vfat, exfat, swap). Reserved for v0.x:- Partition management via parted / sgdisk: create / remove / resize partitions, partition flags (boot, lvm, raid, esp), label types (GPT vs MBR), partition labels. Partitioning is destructive enough to deserve its own module (
partition?) — co-existing withdisk(filesystem layer) once both ship. - Filesystem resize without destroying data via
resize_fs: true—ext2/3/4,xfs,btrfs,f2fslanded; each grows only when the fs doesn’t already fill the block device (blockdev --getsize64). ext is device-based (dumpe2fsblock-count×size →resize2fs <device>); xfs and btrfs are mounted/by-mountpoint (xfs_info→xfs_growfs <mnt>;btrfs fi show --raw→btrfs filesystem resize max <mnt>), with the mountpoint resolved viafindmntand an unmounted device a clear error. f2fs is the inverse — offline/device-based (resize.f2fs <device>, a mounted device is the error), with a section-aware fill check that reads the on-disk superblock (block count + section geometry) since f2fs has no version-stable size tool. All four are exercised live (mkfs + resize) against loop-backed devices by the cross-distro matrix (test/e2e/state/smoke.disk.sh), each gated on tool availability (btrfs/f2fs skip on Rocky 9). - Filesystem label and UUID management (no re-format):
tune2fs -L <label> -U <uuid>(ext),xfs_admin -L -U,btrfs filesystem label,swaplabel -L -U(swap),dosfslabel(vfat). - Encryption (LUKS via cryptsetup) —
luksFormat,luksOpen/Close, key slot management, passphrase rotation; integration withmountfor opening at boot. - fstype catalog expansion: ntfs (
mkfs.ntfsvia ntfs-3g), zfs (zpool create), bcachefs, reiserfs (legacy), jfs (legacy), tmpfs (it’s a mount-time fs, but adiskshortcut could be useful). - Re-format on mismatch without explicit
force: true— a per-decl policy likeon_mismatch: reformat|error|warnwould let operators opt into more permissive defaults; v1.0 always errors on mismatch unlessforce: true.
- Partition management via parted / sgdisk: create / remove / resize partitions, partition flags (boot, lvm, raid, esp), label types (GPT vs MBR), partition labels. Partitioning is destructive enough to deserve its own module (
- Why deferred: “ensure this device has this filesystem” is the v0.1 scope; partitioning is a different (more dangerous) operation surface, fs-resize is per-fstype and per-fs-state, encryption introduces a key-management dimension, and the catalog expansion is mostly more
mkfs.<fstype>mappings + signature handling. The Provider (GetFilesystem/MakeFilesystem/WipeFilesystem) extends cleanly withResizeFilesystem,RelabelFilesystem, etc. - Acceptance: a
diskdecl withresize_fs: trueextends the fs to fill the device, idempotently (met — ext/xfs/btrfs/f2fs fs-resize landed; all four wired into the cross-distro matrix via loop devices). v0.x: apartition:decl (in a newpartitionmodule, sibling todisk) creates/dev/sdb1at the declared start/size/type, idempotent againstparted -m print; alabel: mylabeldecl on an existing ext4 device updates the label without re-format; aluks: { passphrase: …, key_slot: 0 }decl initialises LUKS on the device;fstype: ntfsround-trips via ntfs-3g;on_mismatch: warnreports the drift in the StateResult Comment but doesn’t error. - References: Epic 08 task 11
_(landed)_;internal/statemgmt/stdlib/disk/{disk.go,params.go,provider.go,provider_linux.go}(per-fstype resize dispatch +parseXfsInfoBytes/parseBtrfsShowBytes/parseF2fsSuperblock/mountpointOf).
network stdlib module — boot-survive configuration, per-family addresses, DNS / NTP / domain mgmt
- Priority: gate-v0.5 work complete (boot-survive via netplan + networkd landed); v0.x (the rest).
- What: Epic 08 task 11 ships the
networkmodule (single op: reconcile one interface’s runtime state via iproute2 —addresses: [...],mtu:,up: <bool>— viaip -j addr showfor Check andip addr add/del/ip link set mtu/up/downfor Apply). Reserved for v0.x:Boot-survive / persistent configurationlanded for networkd + netplan:persist: networkd|netplan|autorenders a systemd-networkd*.networkunit (/etc/systemd/network/10-kscore-<iface>.network) or a netplan YAML doc (/etc/netplan/90-kscore-<iface>.yaml) mirroring the declared addresses + mtu, additive to the runtime reconcile (the file is for the next boot; runtime is already live viaip, so nothing is auto-activated).autopicks netplan when/etc/netplanexists, else networkd. Still v0.x: NetworkManagersystem-connections/, Debian/etc/network/interfaces, RHELifcfg-*renderers;upis runtime-only (not rendered).- Per-family address management — declare IPv4 and IPv6 sets independently (
ipv4_addresses: [...]+ipv6_addresses: [...]); v1.0 merges them into oneaddresses:list. - Address scope, valid_lft, preferred_lft, broadcast, peer per
ip addr addoptions. - DNS resolvers, search domains, NTP servers — these live in /etc/resolv.conf, systemd-resolved, NetworkManager, /etc/systemd/timesyncd.conf, etc. Distinct V1X module(s) (
resolver/ntp). - Wireless (
wpa_supplicant, NM Wi-Fi), 802.1X, WireGuard / OpenVPN / IPsec — vendor / protocol modules. - Interface creation / removal for physical NICs (impossible) and SR-IOV VFs (possible); v1.0 errors out with
ErrInterfaceNotFoundif the interface doesn’t exist. Virtual-interface creation is thebond/bridge/vlanmodules’ job. - Address-removal safety — v1.0 will happily strip an in-use IP that the operator forgot to declare; a
dry_run_safety: truemode would surface the in-use-IP risk before applying.
- Why deferred: “ensure this interface has these addresses, this MTU, this admin state right now” is the v0.1 scope — the runtime layer is what the operator most often needs in day-2 ops (container hosts, transient overlays, troubleshooting). Persistent config is a distro-render problem that adds ~5× the surface; per-family + scope + lft attributes are V1X polish on top. The Provider (
GetInterface/AddAddress/DelAddress/SetMTU/SetLinkUp) extends cleanly along all of those axes. - Acceptance: a
persist: networkddecl writes a syntactically-valid*.networkfile matching the runtime config and survives a reboot (met — networkd + netplan persist landed). v0.x:ipv4_addresses:+ipv6_addresses:round-trip independently;valid_lft: 3600on an address is preserved by Check;dns_resolvers: [1.1.1.1, …]writes the host’s resolver config (systemd-resolved or /etc/resolv.conf); adry_run_safety: truemode logs the planned removals without applying them. - References: Epic 08 task 11
_(landed)_;internal/statemgmt/stdlib/network/network.gopackage comment;internal/statemgmt/stdlib/network/params.go.
route stdlib module — persistent configuration, route attributes, source-routing rules, multipath
- Priority: gate-v0.5 (persistent configuration); v0.x (the rest)
- What: Epic 08 task 11 ships the
routemodule (one routing-table entry per declaration viaip route replace/ip route del; identity keyed on(destination, metric, table);gateway:+interface:are reconciled). Boot-survive persistence landed (gate-v0.5):persist: networkd|netplan|autorenders the route via the sharednetpersisthelper — a networkd[Route]drop-in (<iface>.network.d/<slug>.conf, merged + create-if-absent base) or a per-route netplanroutes:document;persistrequiresinterface:. Reserved for v0.x:- Remaining persist backends: NetworkManager static-routes, /etc/sysconfig/network-scripts/route-*, /etc/network/interfaces
post-up ip route add …. (netplan’stable:is numeric, and multiple routes on one interface via separate netplan files conflict — networkd drop-ins are the multi-route backend.) - Route attributes:
proto(boot / dhcp / static / kernel),scope(link / host / global),src(source IP for the route),mtu,advmss,pref(high / medium / low),onlink,realms,congctl. - Multipath nexthops —
nexthop via X weight 1 nexthop via Y weight 2style ECMP routes. - Source-routing policy rules via
ip rule add— a separaterulemodule that pairs withrouteand thetable:knob this module already exposes. - VRF awareness beyond the
table:knob (thevrfinterface type + L3 master). - IPv6 specific:
expires(lifetime),prefmed/high/low (RA preference). - Default-table inference from metric (some operator workflows imply a table from a metric range — v1.0 keeps them orthogonal).
- Remaining persist backends: NetworkManager static-routes, /etc/sysconfig/network-scripts/route-*, /etc/network/interfaces
- Why deferred: “ensure this one route exists / doesn’t” via the modern
ip route replaceidempotency is the v0.1 scope; persistence is the same distro-renderer problem thenetworkmodule faces; attributes are an option-by-option extension; multipath nexthops + policy rules are meaningful new surfaces. The Provider (GetRoute/ReplaceRoute/DelRoute) extends cleanly with extraRouteSpec/RouteEntryfields. - Acceptance: a
persist: networkd|netplan|autodecl renders a syntactically-valid route entry in the host’s config_(met: networkd drop-in + netplan)_;proto: static+scope: link+src: 10.0.0.5round-trip; anexthop:list creates a multipath route; a separaterulemodule declares afrom 10.0.0.0/24 lookup vpnpolicy rule and theroutemodule’stable: vpndeclarations populate that table. - References: Epic 08 task 11
_(landed)_;internal/statemgmt/stdlib/route/{route.go,params.go,render.go,persist.go};internal/statemgmt/stdlib/netpersist/.
bond stdlib module — in-place attribute / member reconciliation, persistent configuration, slave attributes
- Priority: gate-v0.5 (persistent configuration); v0.x (the rest)
- What: Epic 08 task 11 ships the
bondmodule (create / delete a Linux bonding interface at runtime viaip link add … type bond mode … [miimon N]+ip link set <member> master <bond>). Reserved for v0.x:- In-place attribute reconciliation on an existing bond:
mode,miimon,xmit_hash_policy,lacp_rate,ad_select,primary,primary_reselect,fail_over_mac,num_grat_arp,all_slaves_active, etc. v1.0 considers an existing bond converged regardless of attrs (operators delete + recreate to change). - Member-set reconciliation on an existing bond — adding / removing slaves without destroying the bond. v1.0 enslaves declared members only at create time.
Persistent / boot-survive configurationlanded (gate-v0.5):persist: networkd|netplan|autorenders the bond via the sharednetpersisthelper — a<bond>.netdev+ a[Network] Bond=enslave drop-in per member (networkd; absent cleans up by glob) or a singlebonds:netplan document. Remaining v0.x: NetworkManager, ifupdown.- Slave-level attributes: per-slave queue id, priority.
- In-place attribute reconciliation on an existing bond:
- Why deferred: “create or delete this bond” is the v0.1 scope; changing bond mode on a live aggregation has subtle implications (LACP renegotiation, traffic interruption, slave release/reattach) that operators typically want behind an explicit step. The Provider (
GetLink/CreateBond/DeleteLink/SetMaster) extends cleanly withSetBondAttr/ClearMastermethods for the V1X path. - Acceptance: a
mode:change on a live bond reconciles viaecho <mode> > /sys/class/net/<bond>/bonding/mode(or down-up-cycle if required) and reports the before→after in the Diff;members: [eth0, eth1, eth2]against a live bond witheth0, eth1adds eth2;members: [eth0]againsteth0, eth1removes eth1 (ip link set eth1 nomaster); apersist: networkddecl renders a*.netdev+*.networkpair_(met: networkd .netdev + enslave drop-ins, and netplan)_. - References: Epic 08 task 11
_(landed)_;internal/statemgmt/stdlib/bond/{bond.go,params.go,render.go,persist.go};internal/statemgmt/stdlib/netpersist/(NetdevPersist). Runtime create + networkd persist are exercised live across the cross-distro matrix viatest/e2e/state/smoke.netdev.sh(over a dummy member).
bridge stdlib module — in-place attribute / port reconciliation, per-port attributes, persistent configuration
- Priority: gate-v0.5 (persistent configuration); v0.x (the rest)
- What: Epic 08 task 11 ships the
bridgemodule (create / delete a Linux bridge interface at runtime viaip link add … type bridge [stp_state 1]+ip link set <member> master <bridge>). Reserved for v0.x:- In-place attribute reconciliation on an existing bridge: stp_state, forward_delay, hello_time, max_age, ageing_time, vlan_filtering, vlan_default_pvid, mcast_snooping, mcast_querier, group_fwd_mask.
- Port-set reconciliation on an existing bridge — adding / removing ports without destroying it. v1.0 attaches declared ports only at create time.
- Per-port bridge attributes: state (disabled / listening / learning / forwarding), priority, path-cost, pvid, learning, unicast_flood, mcast_flood, mcast_router, neigh_suppress.
- VLAN-aware bridge filtering (
bridge vlan add vid 10 dev port) — a separatebridge_vlanop or sub-module would land cleanly. Persistent / boot-survive configurationlanded (gate-v0.5):persist: networkd|netplan|autorenders a<bridge>.netdev([Bridge] STP=) + a[Network] Bridge=enslave drop-in per port (networkd; absent cleans up by glob) or a singlebridges:netplan document. Remaining v0.x: NetworkManager, ifupdown.
- Why deferred: “create or delete this bridge” is the v0.1 scope; live-bridge attribute changes (especially stp_state) interrupt connected traffic and operators typically want behind an explicit step. The Provider (
GetLink/CreateBridge/DeleteLink/SetMaster) extends cleanly along these axes. - Acceptance: a
members:change on a live bridge attaches/detaches ports without re-creating;stp: trueon a live STP-disabled bridge enables STP and reports the change;port_pvid: { eth0: 10, eth1: 20 }round-trips; avlan_filtering: truebridge takes a list ofbridge_vlan:declarations that populate the VLAN table. (Persist_(met: networkd .netdev + enslave drop-ins, and netplan)_.) - References: Epic 08 task 11
_(landed)_;internal/statemgmt/stdlib/bridge/{bridge.go,params.go,render.go,persist.go};internal/statemgmt/stdlib/netpersist/(NetdevPersist). Runtime create + networkd persist are exercised live across the cross-distro matrix viatest/e2e/state/smoke.netdev.sh(over a dummy port).
vlan stdlib module — in-place attribute reconciliation, QinQ, VLAN ranges, persistent configuration
- Priority: gate-v0.5 (persistent configuration); v0.x (the rest)
- What: Epic 08 task 11 ships the
vlanmodule (create / delete an 802.1Q VLAN interface at runtime viaip link add link <parent> name <name> type vlan id <id>). Reserved for v0.x:- In-place attribute reconciliation on an existing VLAN:
id,parent, ingress-qos-map, egress-qos-map, reorder_hdr, gvrp, mvrp, loose_binding. - QinQ / 802.1ad stacked-VLAN tagging (
proto 802.1ad). - VLAN ranges — declare 100-200 in one decl that creates that many subinterfaces.
- Bridge VLAN filtering (
bridge vlan add vid 10 dev port) — see thebridgemodule’s V1X scope. Persistent / boot-survive configurationlanded (gate-v0.5):persist: networkd|netplan|autorenders a<vlan>.netdev([VLAN] Id=) + a[Network] VLAN=enslave drop-in on the parent (networkd; absent cleans up by glob) or a singlevlans:netplan document. Remaining v0.x: NetworkManager, ifupdown.
- In-place attribute reconciliation on an existing VLAN:
- Why deferred: “create or delete this VLAN” is the v0.1 scope; the in-place attribute change has subtle implications (VLAN id change effectively reroutes the L2 segment), and QinQ + VLAN ranges open new param-shape questions (a range op would have a different
name:semantic). The Provider (GetLink/CreateVLAN/DeleteLink) extends cleanly. - Acceptance: an
id:change on a live VLAN reports drift and reconciles via delete-and-recreate (orip link set <vlan> type vlan id <new>if the kernel supports it);proto: 802.1adround-trips;id_range: 100-200creates 101 VLAN interfaces in one Apply; apersist: networkddecl renders a*.networkfor the VLAN_(met: networkd .netdev + parent enslave drop-in, and netplan)_. - References: Epic 08 task 11
_(landed)_;internal/statemgmt/stdlib/vlan/{vlan.go,params.go,render.go,persist.go};internal/statemgmt/stdlib/netpersist/(NetdevPersist). Runtime create + networkd persist are exercised live across the cross-distro matrix viatest/e2e/state/smoke.netdev.sh(over a dummy parent).
Cross-distro state stdlib docker matrix harness
- Priority: gate-v0.5 — harness landed; grows per-module as backends land
- What: Epic 08 task 13 Layer C. Landed: all five distros (Debian 12, Ubuntu 22.04/24.04, Rocky 9, Alpine 3.19) run green via
make test-cross-distro. Each boots in a privileged container with its real init system (systemd / OpenRC); a static CGO-free apply-harness (test/e2e/state/harness/, runs on glibc + musl) applies the smoke fixture twice and asserts the second pass is a zero-change no-op — sopackage(apt/dnf/apk),service(systemd/OpenRC),user/group(shadow-utils / BusyBox), andhostname(hostnamectl //etc/hostname+hostname(1)) are exercised against the live system, not mocked. The olddocker-compose.ymlscaffold was replaced by adocker run -d+docker execorchestrator (run.sh) because the service module needs a booted init as PID 1, which compose’s run-to-exit model can’t provide. Docker auto-skip preserved. - Remaining: the fixtures exercise
file+package+service+user/group+hostname+disk+firewall+lvm+bond/bridge/vlan+mount+cron/at/systemd_timer+timezone/sysctltoday — every stdlib module with distro-varying, container-safe live behaviour is now covered. The modules still outside the matrix are excluded for a concrete reason, documented intest/e2e/state/README.md§ “Modules deliberately not in the matrix”: the hermetic ones (file/link/cmd/config/ssh/archive) ride thestate_integration_test.gosuite instead;network/route/swap/kernel_module/security/systemare matrix-hostile (they would disturb the host or can’t run isolated in a container);git/langpkgneed outbound network. So this entry is complete as the integration point — all three firewall backends (iptables, nftables, firewalld) are now exercised live, leaving no live matrix follow-ups. (mountjoined viasmoke.mount.sh(loop-backed ext4, container mount namespace);cron/at/systemd_timerviasmoke.sched.sh(crontab entry +atqueue entry + a.timerunit on systemd distros, each self-gated on its tooling);timezone/sysctlviasmoke.sysconf.sh(timedatectl-or-symlink + a per-netnsnet.*key with its persist drop-in).) (bond/bridge/vlanjoined viasmoke.netdev.sh, which installs a JSON-capableiproute2, buildsdummyscaffold interfaces, and applies a fixture that creates one bond, one bridge, and one VLAN at runtime (ip link add type …) — each withpersist: networkd, so the sharednetpersistrenderer is exercised alongside the runtime path. The interfaces live in the container’s own network namespace (so nothing on the host is touched), and each type is gated on its kernel module. netplan-pinned persist, in-place attribute/member reconciliation, and theabsentpath remain follow-ups.) (firewalljoined viasmoke.firewall.sh, which exercises all three backends as separate harness invocations — iptables (auto-detected, dual-stack + catalog) and nftables (pinned, over a pre-createdinet filter input) on every distro, then firewalld (auto-detected once its daemon is up) on the systemd distros wherefirewall-cmd --stateresponds; the rules apply in the container’s own network namespace, so nothing on the host is touched. Live nftables coverage surfaced a real backend idempotency bug —parseChainRulesfailed on thenft --handlechain-handle line — fixed separately in #194. firewalld self-gates where its in-container D-Bus interface returns NoReply, e.g. Debian 12 / Ubuntu 24.04.) (lvmjoined viasmoke.lvm.sh, which installslvm2and drives the module over loop-backed PVs through its create lifecycle (pv→vg→lv), an LV grow (lvextend), and a VG PV-set reconcile (vgextend, gated on a second free loop) — each a separate harness invocation so the grow/extend paths re-apply idempotently; teardown is guaranteed by a rawlvm+dmsetup+losetuptrap.extents:-resize,vgreduce, and theabsentpath remain follow-ups.) Three distros were added to exercise the backends that landed after the original five — openSUSE (zypper) + Arch (pacman) forpackage, and Devuan (sysvinit) forservice— and all three were certified green viamake test-cross-distro. sysvinit boots a keep-alive container (its ops are script-based, no PID-1 init needed);wait_readywaits for the boot’s apt setup to finish (PID 1 ==sleep) so the smoke doesn’t race it.diskruns in its own loop-device-backed phase (smoke.disk.sh): each fstype gets a blank-loopmkfsscenario and a small-fs-on-grown-deviceresizescenario (xfs/btrfs mounted, ext/f2fs offline), self-gating on the hostloopmodule, per-fstype tool availability (btrfs/f2fs skip on Rocky 9), and per-loop-device availability (a scenario that can’t allocate a loop — e.g. when snapd holds most of the host’s loops — is skipped, not failed). Caveats: on Alpine the BusyBox backend has nousermod/groupmod, so modifying an existing account’s scalar fields (or a group’s GID) returnsErrModUnsupported(the create/delete/lookup + supplementary-group paths the harness exercises are fully supported); andhostnamerequires detaching Docker’s/etc/hostnamebind-mount (smoke.shrunsumount /etc/hostname) since the bind-mount makes the inode unreplaceable (EBUSY) — a Docker-only step, no-op on a real host. - Not in CI: privileged + systemd-as-PID-1 needs cgroup write access; the entry is a manual / Docker-host gate (the shared CI runner pool is unprivileged). Certified green on a Docker host across all five distros.
- References:
test/e2e/state/README.md;test/e2e/state/{run.sh,smoke.sh,smoke.disk.sh,smoke.systemd.yaml,smoke.openrc.yaml};test/e2e/state/harness/;internal/statemgmt/stdlib/{user,group}/{detect,shadow,busybox}_linux.go(the shadow / BusyBox backend split); the per-module gate-v0.5 entries above forfirewall,firewalld,security,system,lvm,disk,network,route,bond,bridge,vlanname the per-distro acceptance each module must clear inside this harness as it lands.
Encrypt CA material at rest
- Priority: gate-v0.5 — complete (landed).
- What: Epic 09 task 5 shipped
internal/identity.FileCAStorageas plaintext PEM (cert0644, key0600) under a0700directory. PROJECT-DETAILS §4.10’s “optional encryption key” on persisted CA material is now met byinternal/identity.EncryptedFileCAStorage: a drop-inCAStorageover the same directory layout that seals the private-key files as AES-256-GCM envelopes (cert files stay public PEM). The master key is sourced via the neutralinternal/masterkeyresolver shared with Epic 10’s secrets backend (env:/file:/inline:). The provider selects it via the newidentity.encryption_keyserver-config field; empty keeps the plaintext surface (backward-compatible). Existing plaintext deployments migrate in place withkscore-identity ca encrypt --storage-path <dir> --key <source>(refuses to double-encrypt; all-or-nothing on detection). - Acceptance: met —
EncryptedFileCAStorageround-trips withFileCAStorage(key-only encryption; wrong-key + tamper rejected via the envelope’s fingerprint guard + GCM auth); theca encryptmigration command lands; and the gate testTestEmbeddedProvider_EncryptedStorageEndToEndboots a clean embedded provider with encryption enabled, persists the sealed CA, reloads it across a restart with the right key, and refuses to boot with the wrong key. The shared resolver was extracted tointernal/masterkeyfirst (a behaviour-preserving refactor, separate PR). - References:
internal/identity/{ca_storage_encrypted,ca_envelope,ca_migrate}.go;internal/masterkey/;internal/config/identity.go(encryption_key);cmd/kscore-server/identity.go(newCAStorage);internal/cli/identity/ca_encrypt.go; PROJECT-DETAILS §4.10.
Hugo docs site
- Priority: gate-v0.5 — complete (built, navigable, link-checked, deploy-ready; only standing up the live
docs.keystone-core.iohosting remains, and that domain is still aspirational). - What: The v0.1.x doc surface is rendered Markdown under
docs/(README.md,docs/project/*.md,docs/runbooks/*.md,docs/adr/*.md) — discoverable through Forgejo’s web UI, navigable via the subtreeREADME.mdindex pages added during the v0.1.x first-impression doc pass. v0.5 graduates this into a Hugo site (using the Hextra theme rather than the originally-named Docsy — lighter, no npm/PostCSS pipeline, built-in offline full-text search; decision recorded here) with per-page navigation, full-text search, and the structure the epic-19 §Documentation block originally called for (reference/, operations/, …). - Progress: PR 1 landed — a building site.
make docs-siterenders todocs/public/via Hugo Extended (make install-hugo); the canonical Markdown trees are mounted in place (docs/hugo.toml[[module.mounts]]:docs/project→/docs/reference,docs/runbooks→/docs/operations,docs/adr→/docs/adr), so nothing is moved or duplicated anddocs/project/stays the source of truth (incl. the gendocs CLI/config/API refs). Section landing pages + the home page live underdocs/content/. CI gates the build (ci-fastlint job). A localalertshortcode shim renders the Docsy-style callout a few docs use. Seedocs/SITE.md. PR 2 landed — navigation + titles. Each section_index.mdcarries a Hugocascademap that sets per-pagetitle+ sidebarweight(and hides the mountedREADMEs) so the canonical files underdocs/project/stay pristine — no front matter added to them. The reference sidebar mirrorsdocs/project/README.md’s intent grouping (operator entry points → reference → design → security → governance → testing → development → lifecycle), the operations runbooks order by operational lifecycle, and acronym titles (Design/DCO/RFC/AI/E2E) render correctly; the home page gained section cards. PR 3 landed — rendered-site link-check + publish path. A link render hook (docs/layouts/_default/_markup/render-link.html) rewrites the canonical docs’ relative links to in-site URLs (for mounted docs) or absolute Codeberg source URLs (for repo-root files / source code), so the rendered site is self-consistent without editing any canonical file;make docs-links-siteruns lychee over the rendereddocs/public/and is gated in CI.deploy/docs/README.mddocuments the publish path (servedocs/public/atdocs.keystone-core.io). The only thing left is standing up the actual hosting for that (still-aspirational) domain — infrastructure work, not docs work. - Why now (v0.5): pulled forward from a prior v1.x position. v0.5 is the “external-tester ready” milestone, and a wider tester audience benefits from a polished, searchable doc experience over plain Markdown — particularly for the auto-generated CLI / config / API references, which become genuinely browsable when rendered with navigation. Pre-v0.5 (the v0.1.x soft-launch audience), the Markdown surface is sufficient because the audience is invited and willing to navigate the repo directly.
- Acceptance:
docs/builds a Hugo site underdocs/public/viamake docs-site; auto-generated CLI / config / API references regenerate viamake docs-syncinto the Hugo content tree (one source of truth — edits go to the auto-gen source, not the Hugo content copy); the published site mirrors the structure ofdocs/project/with per-page navigation; site hosted underkeystone-core.io/docs(or a documented alternative if the domain isn’t yet provisioned at v0.5 cut); link-check CI gate (lycheealready in pipeline) covers the rendered site in addition to the Markdown source. - References: epic 19 task 9
_(landed)_(auto-gen CLI/config/API references);docs/project/{CLI,CONFIGURATION,API}-REFERENCE.md+GETTING-STARTED.md(Hugo-content-tree sources);docs/README.md+docs/project/README.md+docs/runbooks/README.md(v0.1.x subtree indexes that the Hugo navigation will replace); AGENTS.md §5 (canonical-doc-surfaces pointer); FEATURES.md §1 v0.5 (pulled forward from v1.x).
Expanded getting-started guides (per-domain tutorials)
- Priority: gate-v0.5 — complete (all six guides landed across PRs 1–3).
- What: Epic 19 §Documentation listed five getting-started guides (install, first cluster, first command, first state apply, first blueprint, first module). v0.1.x ships one consolidated ~30-minute walkthrough at
docs/project/GETTING-STARTED.mdthat covers install + agent online + command + state + audit. The per-domain tutorials (blueprint authoring, module authoring + publishing, secrets, GitOps integration, audit/policy queries, HA cluster topology) round out the v0.5 external-tester doc experience. - Why deferred: One thorough walkthrough satisfied the epic-19 acceptance line (“Quick-start guide can be followed end-to-end on a fresh Ubuntu VM in <30 minutes”). The longer-form per-domain content benefits from the Hugo site (also gate-v0.5) being live so the new guides land into the proper site IA rather than as ad-hoc Markdown files — so it pairs with Hugo in the v0.5 gate.
- Acceptance: Six guides under
docs/content/docs/getting-started/in the Hugo content tree (the live tree is single-language, so noen/segment); each is runnable end-to-end on a fresh host; each is link-checked. - Progress: PR 1 landed — the
getting-startedsection + the two “extend the system” guides: Authoring a Blueprint (kscore-blueprint init/validate/lint/info/apply/rollback/bundle, grounded in themodules/examples/blueprints/examples) and Authoring & Publishing a Module (the Starlarkinit→validate→test→run→build→sign→publish→installflow +kscore-registry serve). Every command was verified against the real CLIs (built fromcmd/); the guides are Hugo-native content (site-relative links, link-checked bymake docs-links-site, sodocs/contentis excluded from the source.mdlink gate). PR 2 landed — the two fully-runnable operational guides: Managing Secrets (theencrypted_filebackend config + master-key resolver, thenkscore-secrets put/get/list/delete) and Querying the Audit Log & Policies (kscore-audit log/stats/report/export, plus authoring + locallyvalidate/eval-ing a Rego policy — verified to compile +DENYas shown). PR 3 landed — the two partial-feature guides, each scoped to what actually runs with honest v1.0/v1.1 callouts: GitOps Integration (the webhook receiver config +kscore-gitops verifyover an http-step workflow — verified to run — +rollback; explicit that v0.1 is the inbound-hooks surface, not full repo-sync) and HA Cluster Topology (theclusterconfig surface, single-node bring-up, and thekscore-cluster status/leader/members/transfer-leader/rebalance/backupCLI; explicit that automatic failover reassignment + the multi-process form are still in progress). All six guides are now live underdocs/content/docs/getting-started/, link-checked and rendered in the Hugo site’s Getting Started section. - References: epic 19 task 9
_(landed)_;docs/project/GETTING-STARTED.md(the consolidated quick start);docs/content/docs/getting-started/; Hugo docs site entry under gate-v0.5.
gate-v1.0 — blocks v1.0 SemVer-stability commitment
Module system boot wiring (loader PolicyChecker/Hosts/trust-policy + runtime registration)
- Priority: gate-v1.0
- What: Epic 14 builds the plugin/module system as runtime-agnostic, all-seams (the recurring dep-light pattern): task 3’s capability backends take injected
capability.Hosts(realinternal/secrets/os-exec/net-http); task 4’s verifier takes an injected*verify.TrustPolicy; task 10’sloader.ModuleLoadertakes an injectedPolicyChecker(the production impl = an adapter over the Epic-12internal/policy.Engine), a*verify.Verifier, theHosts, and aRuntimeRegistry. None of these seams is wired atcmd/kscore-server/cmd/kscore-moduleboot yet — until then the loader can parse/verify/policy-check/runtime-init only with explicitly supplied seams (unit/fake-tested), and a server cannot load+execute a published module end-to-end. - Landed (CLI run):
kscore-module run <dir|module.zip> [input-json]now loads, verifies (against--keytrusted keys +--sig, or--skip-verificationfor local dev), and executes a module end-to-end, printing the result. Newinternal/modulepackage supplies the production capability hosts (LocalHosts: FS=os, HTTP=net/http, Exec=os/exec, Logger=slog; Secrets nil → fails closed for a standalone CLI) + aBuildLoaderconstructor; the Epic 14 task-12 capability→Starlark builtin shims landed aspkg/module/runtime/starlark/capbuiltins(theBuiltinProviderexposingfs_read/fs_write/http_get/http_post/exec_run/secret_read/secret_write/kv_*/logover the scoped backends; host calls inherit the execution deadline via a newstarlark.ContextFromThread). For the CLI,Policyis nil (allow-all) — the security boundary is signature verification + the manifest-declared, scope-enforced capability layer. Still deferred: server-side module execution (no consumer exists — noModuleService/workflow calls the loader, so wiring one intocmd/kscore-serverwould be dead code; a server execution surface is net-new functionality, not boot-wiring), and with it theinternal/policyPolicyCheckeradapter, thesecrets.BrokerSecretsHostadapter,config.ModuleConfig, and resolving an installed registry ref (run vendor/pkg@verfrom the CAS). - Why deferred: same root as the other boot-wiring items — the seams are intentionally injected so
pkg/module/*stays dependency-light and unit-testable; production construction (trust policy from config, Hosts from the live secrets/exec/http stacks, theinternal/policy→PolicyCheckeradapter, the Starlark runtime registered formanifest.TypeStarlark) lands with the kscore-module CLI (task 14, which wires its own seams for the author/install flow) and the server-lifecycle boot integration, so all module wiring happens in one coherent place. - Acceptance for unblock:
kscore-module(task 14) constructs aModuleLoaderwith the real verifier+trust-policy, theinternal/policyPolicyCheckeradapter, realcapability.Hosts, and the Starlark runtime (task 11) registered — andkscore-module install→runloads, verifies, policy-checks, and executes a signed published module end-to-end; the same wiring is available tokscore-serverfor server-side module execution. - References:
pkg/module/loader(task 10, landed — seams);pkg/module/capabilityHosts(task 3);pkg/module/verifyTrustPolicy(task 4);internal/policy.Engine(Epic 12, thePolicyCheckerproduction impl); Epic 14 tasks 3/4/10, forward 11/12/14; companion of “Cluster gRPC services boot registration …”.
Cluster gRPC services boot registration (ClusterService/CoordinationService + mTLS listener)
- Priority: gate-v1.0
- What: Epic 13 task 12 ships
CoordinationGRPCServer(the server↔server NATS-down recovery channel; mTLS-only — rejects callers without a verified client cert) and task 13 ships the matchingCoordinationClient(per-peer pooled client + heartbeat liveness + retry/backoff). Neither is wired atcmd/kscore-serverboot, and task 15 now shipsClusterGRPCServer(internal/controlplane/grpc_cluster_server.go) + the cluster REST handler (pkg/api/cluster/handler.go) +internal/cluster/backup.go, none of which is wired either. CoordinationService specifically needs a dedicated mTLS server↔server listener (separate from the agent/operator surface), and the client needs its pool driven fromMembershipManager+ real mTLS creds + the “NATS down ⇒ use this channel” switchover.ClusterGRPCServerneeds registering on the operator/API gRPC surface and the REST handler needs its realClusterProviders(Status/Leader/Members/Rebalance/Backup) + theEvictorhook (aMembershipManager-backed RemoveMember) constructed at boot — until then the cluster REST routes return 503 and the gRPC server is unit/bufconn-tested only. Task 14’sGracefulShutdownorchestrator also needs hooking into the kscore-server SIGTERM/Stop path with a realStopAccepting(connection-manager/listener drain). Until wired, the implementations exist + are unit/bufconn-tested but peers cannot actually reach each other over this channel (NATS-down recovery falls back to slower etcd-only paths), the cluster API surface is dark, and shutdown is not driven on SIGTERM. - Partially landed (PR-A):
cmd/kscore-servernow constructs the clustering stack at boot (cluster.gostartCluster, gated oncluster.enabled) and registersClusterGRPCServeron the operator surface with liveLeader/Members/Rebalancer/ShardStore/LeaderWatchseams + theEtcdClient-backedEvictor(deletes the member’s etcd membership key directly); the REST handler gets realClusterProviders(Leader/Members/Rebalance/Backup), so cluster routes go 503→200 (Status excepted). - Landed (PR-B):
cmd/kscore-server/coordination.gonow starts a dedicated mTLS-onlyCoordinationServicelistener whencluster.coordination.listen_addris set (strictRequireAndVerifyClientCert, wired to the liveLeader/Members/Shards), plus the peer-dialingCoordinationClientwhose pool is reconciled fromMembershipManager(aMembershipObserveradds/removes peers; self excluded). A new client-side mTLS builder (identity.BuildClientTLSConfig) issues this node’s control-plane SVID for dialing and verifies peers by SPIFFE ID against the trust bundle (DNS-agnostic), with the same signing-CA rotation watcher as the server path. The channel is mTLS-only by contract → a listener configured withidentity.enabled=falsefails the boot loudly.Task 14’sGracefulShutdownis wired into the SIGTERM path withStopAcceptingdraining the coordination listener + client (LEAVING → leadership transfer → deregister, before the API server stops). - Landed (PR-C — health/status):
startClusternow constructs + starts theHealthMonitor(built-in etcd + heartbeat checkers, plus boot-supplied non-criticalstorage/natsping checkers frommain.go;cfg.cluster.healthtimings; closes Task 7’s “real DB/NATS checker wiring (boot)”). ItsStatus()/Quorum()back theClusterServiceHealthseam (GetClusterStatusreports member/healthy counts + quorum), theCoordinationServiceHealthseam (ClusterHealthanswers instead ofUnavailable), and a new RESTStatusprovider — soGET /cluster/statusgoes 503→200. The monitor is stopped in the gracefulStopAcceptinghook (beforeSetStatus(LEAVING)) so its status reconcile can’t race the LEAVING write. This entry is now fully landed. Remaining cluster boot-wiring lives in the sibling entries below — still deferred:FailoverManagerwiring. (TheFencingManagerin-flightDrainer+ write-path guard landed in PR-D — see the fencing entry; the coordinationNATSStatusNATS-reachability seam landed in a follow-up vianats.Manager.Connected()/Detail(), soNATSStatus/ClusterHealth.NatsHealthynow report real state.) - Why deferred: same root as the other cluster boot-wiring items — there is no cluster→kscore-server boot wiring yet (deferred since Epic 13 task 1). Server registration + the mTLS listener land with the server-lifecycle/boot integration (Epic 13 task 14 or a dedicated boot task) so all clustering wiring happens in one coherent place.
- Acceptance for unblock: kscore-server boot registers
CoordinationGRPCServeron a dedicated mTLS-only listener andClusterGRPCServer(task 15) on the operator/API surface; a 3-node cluster exchanges health/leader/recovery over CoordinationService with NATS down (verified by the HA NATS-failure E2E, task 17). The task-17 HA suite (test/e2e/ha/) already exercises this mechanism in-process — the realCoordinationGRPCServerover a real mTLS listener with NATS toggled down + non-mTLS rejection — so the remaining unblock is the boot registration itself plus the multi-process form (see “HA E2E multi-process / iptables-partition form”). - References:
internal/controlplane/grpc_coordination_server.go;internal/controlplane/coordination_client.go;internal/controlplane/grpc_cluster_server.go;pkg/api/cluster/handler.go(+pkg/api/server/handlers.gocall site);internal/cluster/backup.go;internal/cluster/shutdown.go(GracefulShutdown);api/proto/keystone/core/v1/{coordination,cluster}.proto; Epic 13 tasks 12/13/14/15; companions: “Cluster leader-check boot wiring (kscore-server)”, “Fencing guard wired around server write paths”.
Fencing guard wired around server write paths
- Priority: gate-v1.0
- What: Epic 13 task 11 ships
FencingManagerwithGuard(OpType) (release, error)+ValidEpoch(e)— the split-brain enforcement (minority/stale-epoch nodes block writes). The mechanism is built and reactive (HealthMonitor quorum signal + etcd epoch watch). - Landed (PR-D):
startClusterconstructs + starts theFencingManager(over the boot HealthMonitor + LeaderElector + etcd;cluster.fencing.mode, defaultread_only), andpkg/api/serverguards the request paths through it: a gRPC unary+stream interceptor callsGuardper RPC (a curated write-method set →OpWrite, everything else →OpRead; unknown ⇒ read so a minority node keeps serving reads, storage-quorum backstopping any unlisted write) and rejects a fenced write withcodes.Unavailable; an HTTP middleware fences the REST surface by verb (POST/PUT/PATCH/DELETE→ write → 503 when fenced;GET/HEAD/OPTIONS→ read). The guard is held for the request duration, andFencingManager.Drainis wired as theGracefulShutdownDrainerso in-flight guarded ops finish before deregister. Layering stays clean:pkg/api/serverexposes a tinyFencer interface { Guard(write bool) (func(), error) }andcmd/kscore-serverinjects an adapter over theFencingManager(nil ⇒ unfenced single-node path). CoordinationService is intentionally never fenced — it runs on its own listener and is the recovery channel. Still deferred: theValidEpoch-at-commit refinement (the narrow TOCTOU window where an op begins before a fence and commits after) —Guardalready rejects whileFenced()(covering both quorum loss and the deposed-leader epoch fence), so this is a finer per-handler commit-path check, not a boot-wiring gap. - Acceptance: write-bearing gRPC/REST handlers acquire
FencingManager.Guard(OpWrite)(release on completion); the HA E2E suite (task 17) verifies a minority partition rejects writes within 1s while the majority serves. Task 17’sTestHA_MinorityPartitionBlocksWritesWithin1s/SplitBrainNoDoubleLeaderverify the realFencingManager.Guardmechanism (writes fenced <1s, reads continue, auto-heal, no double-leader) against a controllable quorum seam; PR-D wiresGuardinto the real server write paths so a partitioned running server rejects write RPCs. - References:
internal/cluster/fencing.go(Guard/ValidEpoch/Fenced);pkg/api/server/fencing.go(interceptors + middleware +Fencerseam);cmd/kscore-server/cluster.go(fenceradapter + boot construction); Epic 13 task 11; companion of “Cluster leader-check boot wiring (kscore-server)”.
Cluster leader-check boot wiring (kscore-server)
- Priority: gate-v1.0
- What: Epic 13 tasks 6/8/9 expose leader-gate seams —
ShardManager.LeaderCheck,FailoverManager.LeaderCheck, and the canonicalSingletonTaskManager.LeaderCheck()— plus the long-standingWithRetentionLeaderCheckseam on theinternal/audit+internal/eventsRetentionEnforcers (defaultAlwaysLeader). The actual wiring (cfg.LeaderCheck = le.IsLeader/ registering the RetentionEnforcers asStartStopTasks on theSingletonTaskManager) is acmd/kscore-serverboot concern and depends on clustering being constructed at server boot, which is itself deferred (Epic 13 task 1 left boot wiring out). Until then a clustered deployment runs those leader-only side-effects on every node (audit/events retention deletes, shard rebalance writes, failover orchestration) — correctness-safe (idempotent / CAS-guarded) but duplicated work. - Partially landed (PR-A): boot now feeds the canonical
SingletonTaskManager.LeaderCheck()into theShardManager(only the leader persists reassignments) and into theinternal/eventsRetentionEnforcerviaWithRetentionLeaderCheck(only the leader prunes). Still deferred: theFailoverManagerleader-check — there is no productionAgentReassigner/JobReassigneryet, so gating a no-op would be meaningless; it rides with the agent/job-reassignment integration (the agent ownership-handoff boot-gated item — a control-plane shard-map change, not an agent transport reconnect). Theinternal/auditRetentionEnforceris not constructed at boot in v0.1, so there is nothing to gate there yet. - Why deferred: there is no cluster→kscore-server boot wiring yet; this lands with the server-lifecycle/boot integration (Epic 13 task 14 graceful-shutdown wires into the Epic 04 server lifecycle, or a dedicated boot task) so all clustering components are constructed and wired in one coherent place.
- Acceptance for unblock: kscore-server boot constructs the
LeaderElector+SingletonTaskManager;ShardManager/FailoverManagerconfigs receivestm.LeaderCheck(); the audit + eventsRetentionEnforcers are constructedWithRetentionLeaderCheck(stm.LeaderCheck())(or registered asStartStopTasks); a 3-node cluster runs each leader-only task on exactly one node (verified by the HA E2E suite, task 17). Task 17 exercises real election + leader-gatedShardManager/FailoverManagerin-process (single-leader invariant, leader-kill failover); the remaining unblock is the boot construction/wiring so the leader-only side-effects are deduplicated in a running multi-node deployment. - References:
internal/cluster/singleton.go(SingletonTaskManager.LeaderCheck);internal/cluster/{shardmanager,failover}.go(LeaderCheckseam);internal/audit/retention.go+internal/events/retention.go(WithRetentionLeaderCheck/AlwaysLeader); Epic 13 tasks 6/8/9.
HA E2E multi-process / iptables-partition form
- Priority: gate-v1.0
- What: Epic 13 task 17 ships the HA E2E suite (
test/e2e/ha/) as in-process, component-level tests against the realinternal/clusterstack + realCoordinationGRPCServerover a real mTLS listener + real embedded etcd + real Postgres. The literal forms in the epic text — a multi-process3×kscore-serverdeployment, realiptables/network-partition injection, killing realetcd/NATS/Postgresprocesses and asserting via a running server’s RPC surface — are deferred. They require the cluster→kscore-serverboot wiring (the three companiongate-v1.0entries) to exist first, plus a process/container orchestration harness (thetest/e2e/statedocker-compose precedent) and root/network-namespace control for iptables. - Why deferred: there is no cluster→kscore-server boot wiring yet (deferred since Epic 13 task 1), so there is no running clustered server to multi-process-test or partition; the in-process suite already proves every HA mechanism (election/failover/fencing/recovery/coordination/backup) against the real implementation. The multi-process form is integration packaging on top of boot wiring, not new clustering logic, and is gated behind the same boot-wiring milestone.
- Acceptance for unblock: with the boot-wiring entries landed, a containerised 3×
kscore-servercluster forms; aniptables/netem partition makes the minority reject writes within 1s while the majority serves; killing the leader process elects a new leader and completes failover; stopping the etcd/NATS/Postgres containers and restarting them recovers with zero data loss — all asserted through the running servers’ gRPC/REST surfaces and run in CI. The server-integrated SLO numbers also ride here: Epic 13 task 18 ships theslogate (make slo,test/e2e/ha/slo_test.go) verifying the §4.15 SLOs against the in-process mechanisms; the server-integrated SLO measurements (agent-reconnect latency through a running multi-process kscore-server, graceful-shutdown zero-disconnect timing) land with this multi-process form. - References:
test/e2e/ha/(task 17, landed — in-process form);test/e2e/ha/slo_test.go+make slo(task 18, landed — in-process SLO gate);test/e2e/state/(docker-compose harness precedent); companions: “Cluster gRPC services boot registration …”, “Fencing guard wired around server write paths”, “Cluster leader-check boot wiring (kscore-server)”; Epic 13 tasks 17/18.
Blueprint applied-runs store (durable)
- Priority: gate-v1.0
- What: Epic 15 task 5 ships the blueprint
Executorwith anAppliedStoreinterface + in-memory implementation only.kscorectl blueprint rollback <run-id>resolves the recorded apply (blueprint, version, namespace, resolved entrypoint, redacted params) from this store; with the in-memory impl a server restart loses every applied-run record, so rollback only works within the lifetime of the process that ran the apply. - Why deferred: matches the
pkg/sagaprecedent (ships in-memory, durable backend later). A durable applied-runs store is a real persistence surface (schema, migration, masking of sensitive params at rest, integration with the existing SQLite/state store) that is orthogonal to the executor logic itself; the v1.0 minimum is enough for the trial flow where apply + rollback happen in one server lifetime. - Acceptance for unblock: a blueprint applied on one server process is rollback-able from a restarted process via
kscorectl blueprint rollback <run-id>; sensitive (source: secret/sensitive: true) params are masked in the persisted record. - References:
internal/blueprint/applied_store.go(AppliedStore+NewMemoryAppliedStore);internal/blueprint/executor.go;pkg/saga/log_sqlite.go(durable-store precedent); Epic 15 task 5.
Remote / distributed blueprint apply wiring
- Priority: gate-v1.0
- What:
kscore-blueprint apply/rollback(Epic 15 task 10) driveinternal/blueprint.Executoragainst an injectedStateRunner.cmd/kscore-blueprintwires a local-host StateRunner (statemgmt + stdlib) so single-node apply works; a--targetselecting remote agents has no wired distributed StateRunner and returns a clear “remote apply not configured” error. Until wired,kscorectl blueprint apply demo --target id:agent-1works only for the local host, not a remote agent fleet. - Why deferred: same root as the cluster/module boot-wiring items — distributed apply needs the blueprint executor hooked to the remote-execution path (Epic 07) at
kscore-serverboot, which is the deferred server-lifecycle integration. The local path is enough for the v0.5 single-node trial. - Acceptance for unblock:
kscorectl blueprint apply <bp> --target id:<agent>applies the rendered state collection on a remote agent and records the AppliedRun; rollback targets the same fleet. - References:
internal/cli/blueprint/apply.go;cmd/kscore-blueprint/main.go;internal/blueprint/executor.go;pkg/api/blueprint/handler.go+internal/controlplane/grpc_blueprint_server.go(tasks 11/12 — REST routes + gRPCBlueprintServiceship with empty providers, return 503 /codes.Unavailableuntil this boot wiring suppliesBlueprintCatalog/BlueprintApplier); Epic 15 tasks 10/11/12; companion of the cluster boot-wiring entries.
Durable runbook execution store
- Priority: gate-v1.0
- What: Epic 15 task 10 ships
kscore-runbook status/list-executions/auditagainst anExecutionStoreinterface + in-memory implementation only. The runbook engine (task 7) keeps an in-memoryExecution; task 9 added audit/event emission but not a queryable run store. With the in-memory store,status/list-executionsonly see runs from the current process — a CLI invocation after the run cannot query it. - Why deferred: mirrors the
pkg/saga/ blueprint applied-store precedent (ships in-memory, durable backend later). A durable runbook execution store is a real persistence surface (schema, sensitive-input masking at rest, integration with the SQLite/state store) orthogonal to the engine; the v1.0 minimum is enough for the trial flow where execute + inspect happen in one process. - Acceptance for unblock: a runbook executed by one process is queryable via
kscore-runbook status <id>/list-executionsfrom a later process; sensitive inputs are masked in the persisted record. - References:
internal/cli/runbook/query.go;internal/runbook/engine.go(Execution);pkg/api/runbook/handler.go+internal/controlplane/grpc_runbook_server.go(tasks 11/12 — REST + gRPCRunbookServiceship with empty providers, return 503 /codes.Unavailableuntil this boot wiring suppliesRunbookCatalog/RunbookRunner/ExecutionStore);pkg/saga/log_sqlite.go(durable-store precedent); Epic 15 tasks 10/11/12.
Blueprint e2e real docker-compose convergence form
- Priority: gate-v1.0
- What: Epic 15 task 13 ships
test/e2e/blueprint/as a build-tagged in-process suite that drives the real Epic-15 engines against the real catalog with a recording State Runner (it asserts every resolved declaration is dispatched + hooks run + rollback reverts). It does not convergeproduction-cluster’s etcd/postgres/nats packages+services on real hosts/containers — that is not CI-safe with the v1.0 stdlib modules and is orthogonal to proving the integration wiring. - Why deferred: same shape as the Epic 13 “HA E2E multi-process / iptables-partition form” deferral — the in-process suite proves the mechanism; the multi-container form needs a docker-compose harness (cf. the shell-driven, v0.5-gated
test/e2e/state/) plus provider modules able to converge real services, which is a separate hardening surface (Epic 19). - Acceptance for unblock:
production-clusterapplied via the blueprint executor against a docker-compose topology brings up etcd + Postgres + a NATS cluster and the suite asserts the services are actually reachable/healthy, not just that declarations were dispatched. - References:
test/e2e/blueprint/e2e_test.go(in-process form, landed);test/e2e/state/docker-compose.yml+run.sh(compose-harness precedent); Epic 13 “HA E2E multi-process / iptables-partition form” (companion deferral); Epic 15 task 13; Epic 19 (E2E hardening).
GitOps webhook receiver boot registration (kscore-server)
- Priority: gate-v1.0
- What: Epic 16 tasks 1-4 ship
internal/gitops/webhook— the inbound webhook HTTPReceiver(its ownhttp.Server, default:8081/webhooks, body-capped), theHandlerinterface +Registry, the four v1.0 provider handlers (ArgoCD/Flux/GitHub/GitLab) with aNewDefaultRegistry, header-based source auto-detection (Registry.Detect), per-source authentication (Authenticator: HMAC-SHA256 / Bearer / None viaBuildAuthenticators), and normalization + best-effort re-emission on the Keystone event bus (ToKscoreEvent→gitops.<provider>.<subtype>;EventEmitterseam,events.CategoryGitopsadded to the closed taxonomy). The full receiver→auth→parse→emit pipeline is complete and httptest-verified with a fake emitter (acceptance 102/103 ticked).config.GitOpsConfiggates it (gitops.webhook.enableddefault off;gitops.webhook.sources.<provider>per-source auth), but theReceiveris not constructed in thecmd/kscore-serverboot sequence: theconfig.GitOpsSourceAuthConfig→webhook.AuthSpectranslation and the liveevents.EventPublisherwiring are not built at boot. Until then the receiver runs only when started directly; a running kscore-server does not expose:8081/webhooks. - Landed (Epic 19 task 2c):
cmd/kscore-server/gitops.gostartGitOpsconstructs theReceiverwhengitops.webhook.enabled— per-source authenticators fromgitops.webhook.sources(authSpecsFromConfig→BuildAuthenticators),NewDefaultRegistry, the liveevents.EventPublisheras emitter — started/stopped on the server lifecycle. A running kscore-server now exposes:8081/webhooksand re-emits authenticated webhooks asgitops.<provider>.*. Boot construction fully landed; the receiver→auth→parse→emit pipeline is httptest-verified (no separate E2E through a running server). This entry is complete. - Why deferred: same root as the Epic 13/15 dark-until-boot deferrals — boot registration (constructing the
Receiverfrom config: registry + auth specs fromgitops.webhook.sources+ the live event publisher, started/stopped on the server lifecycle) belongs with the server-lifecycle integration so all GitOps webhook wiring happens in one coherent place. The mechanism itself is fully built and tested; only the boot construction remains. - Acceptance for unblock: kscore-server boot constructs the
Receiverfromconfig.GitOpsConfigwith the default registry, the per-source authenticators built fromgitops.webhook.sources, and the live event publisher; a signed ArgoCD/GitHub webhook to a running server’s:8081/webhooksis authenticated, parsed, and re-emitted on the bus asgitops.<provider>.*. - References:
internal/gitops/webhook/{receiver,handler,detect,auth,auth_build,emit,register,event}.go;internal/config/gitops.go(GitOpsConfig+GitOpsSourceAuthConfig);internal/events/category.go(CategoryGitops); Epic 16 tasks 1-4 (landed), forward 10; companions: the cluster/blueprint/runbook boot-wiring entries.
K8s rollout-undo client-go adapter + GitOps rollback boot wiring
- Priority: gate-v1.0
- What: Epic 16 task 7 ships
internal/gitops/rollback— theExecutorinterface +Registry+ three executors (Git-revert, ArgoCD sync-to-revision, K8s rollout-undo) behind narrow client seams (GitClient/ArgoClient/K8sRolloutClient). The Git seam is implemented byrollback/gitexec(go-git v5) and the ArgoCD seam byrollback/argoexec(stdlib REST); both are real and tested. TheK8sRolloutClientconcrete adapter is deliberately not written — it would pullk8s.io/client-go+k8s.io/api+k8s.io/apimachinery(3-module version-lockstep) and no v1.0 acceptance line needs a live cluster at task 7. The K8s executor is real + fake-tested butRegisterAllis commonly called with a nilK8sseam, sokscore-gitops rollback … (k8s)returnsErrNotConfigureduntil the adapter exists. Task 8 landed the rollbackEngine(engine.go/state.go:pkg/statemachine-driven Pending→Approved/Rejected→InProgress→Completed/Failed→Verifying→Verified/VerificationFailed, optional approval gate,PostVerifierseam). Task 9 landed the durable stores:rollback.SQLiteStoreandverification.{Memory,SQLite}ResultStore, both viapkg/dbutilmirroring thepkg/sagaSQLite contract. Task 10 landed the user-facing surfaces: the §4.13 REST routes onpkg/api/gitopswired to aProviders(Rollback / Verifications) blueprint-style struct, plus thecmd/kscore-gitopsCLI (reachable askscorectl gitops …via the Epic-14 plugin dispatch) withverify <file>androllback/rollback approve|reject|get|list. Task 10 also fixed a T8/T9 gap surfaced in scope (Rollback.Confignow persisted via an additive SQLite column with aPRAGMA table_infoALTER guard, so approval-gated rollbacks resume with the originally-supplied executor configuration — acceptance 106 unblocked). The mechanism is complete + unit-tested, but it is not yet constructed atcmd/kscore-serverboot: the REST handler ships with an emptyProviders{}(routes return 503) until boot constructs the engine + the durableSQLiteStore+ the liveverification.ResultStore, the K8sclient-goadapter still needs to be written so--executor k8sworks against a real cluster, andGetLastKnownGoodis still provider-history best-effort (verification-confirmed signal needs the wiring at boot too). - Mostly landed (Epic 19 task 2c):
cmd/kscore-server/gitops.gostartGitOpsconstructs the rollbackEngineover the durablerollback.SQLiteStore+verification.SQLiteResultStore, registers the Git-revert executor, and passes both as thepkg/api/gitops.Providers(REST routes 503→200). Remaining (the only genuinely-deferred piece): theK8sRolloutClientclient-goadapter so--executor k8sworks against a real cluster — the heavyk8s.io/client-go+api+apimachinery3-module dependency, deliberately deferred and not acceptance-testable without a live cluster. (The ArgoCD executor stays v1.x — needs a real server;GetLastKnownGoodverification-store consultation also rides with the K8s wiring.) The engine/stores/REST boot wiring is landed — this entry now tracks only the K8s adapter. - Why deferred: same dependency-light / dark-until-boot root as the other Epic 16 entries — heavy clients stay behind seams so
internal/gitops/rollbackis unit-testable without a cluster; the concrete client-go adapter + the Engine/CLI boot construction land together at the GitOps boot-wiring step so all of it is wired in one coherent place, and go.mod stays free of the k8s.io trio until it is actually exercised. - Acceptance for unblock: a
K8sRolloutClientclient-goadapter (Deployment get + ReplicaSet revision history + rollout-undo) is written sokscore-gitops rollback --app web --strategy previous --executor k8sagainst a real cluster rolls a Deployment back;kscore-serverboot constructs the rollback Engine over arollback.SQLiteStore+ theverification.SQLiteResultStoreand passes them as thepkg/api/gitops.Providersso the REST routes light up (today: 503);GetLastKnownGoodconsults the task-9verification.ResultStorerather than raw provider history. - References:
internal/gitops/rollback/{executor,git,argocd,k8s,register,engine,state,store_sqlite}.go;internal/gitops/verification/{result_store,result_store_sqlite}.go;internal/gitops/rollback/gitexec(go-git, landed);internal/gitops/rollback/argoexec(REST, landed);pkg/api/gitops/handler.go(REST §4.13, landed);cmd/kscore-gitops+internal/cli/gitops(CLI, landed);pkg/statemachine(engine FSM);pkg/dbutil(SQLite stores); PROJECT-DETAILS §3.2 (K8s row —client-godeferred), §4.13 external-clients; Epic 16 tasks 7-10 (landed); companions: the GitOps webhook receiver + cluster/blueprint/runbook boot-wiring entries.
Outbound webhook Manager boot wiring (kscore-server)
- Priority: gate-v1.0
- What: Epic 16 tasks 11-18 ship
internal/webhook/outbound— push-drivenManager.Handle(ctx, events.Event)with glob filter + bounded fan-out viafilepath.Match+sync.WaitGroup; durableSubscriptionStore(memory + SQLite);HTTPDispatcher(HMAC-SHA256 signing + per-sub timeout); in-Manager retry loop with exp backoff + jitter; per-endpointCircuitBreaker(5/30s/2); §4.14 REST routes onpkg/api/webhooks(subscription CRUD +POST {id}/test+GET {id}/deliveries, secret masked on every non-create response per §4.14);cmd/kscore-webhook outboundCLI (list/create/show/delete/history/test); publicSign+Verifyhelpers; and a build-taggedtest/e2e/webhook/end-to-end suite (happy path / retry exhaustion / circuit-breaker) that drives the whole pipeline against a real httptest.Server receiver.config.WebhookOutboundConfigcarries the §4.14 keys plus the task-12 tunables (max_concurrent_deliveries,refresh_interval);webhook.outbound.enableddefaults off (opt-in, thegitops.webhookprecedent). It is not yet constructed atcmd/kscore-serverboot: nothing subscribesevents.EventSubscriberto routekscore.<cluster>.events.>deliveries intoManager.Handle, and the REST handler ships with emptyProviders{}(routes return 503) until boot wires the liveStore+Manager. Until boot lands, the package surface is unit + integration-tested only; a running kscore-server emits no outbound webhooks, and the/api/v1/webhooks/subscriptionsroutes are dark. - Landed (Epic 19 task 2c + breaker fix):
cmd/kscore-server/webhook.gostartOutboundWebhookconstructs theManagerover the SQLiteSubscriptionStorewith the config-drivenRetryPolicy, subscribesevents.EventSubscribertokscore.<cluster>.events.>routing each delivery toManager.Handle, and passesManager+Storeas thewebhooks.Providers(REST routes 503→200). The breaker fix closes the last gap: theHTTPDispatcheris now wrapped in the per-endpointCircuitBreaker(5/30s/2) at boot, as §4.14 requires — it had been dispatching bare (a boot test now guards the wrap). Fully landed. - Why deferred: same dependency-light / dark-until-boot root as the other Epic 16 boot entries — the Manager is push-driven precisely so the package stays free of the NATS / events subscriber API; the boot wiring (subscribe + route + select Memory vs SQLite store + construct Dispatcher + RetryPolicy + CircuitBreaker + pass to REST
Providers) lands in one coherent place. - Acceptance for unblock: kscore-server boot constructs the Manager over the SQLite
SubscriptionStorewith the task-14RetryPolicypopulated fromwebhook.outboundconfig and the Dispatcher wrapped by the task-15 circuit breaker, subscribesevents.EventSubscribertokscore.<cluster>.events.>and routes each delivery toManager.Handle; the same Manager + Store are passed aswebhooks.Providersso the §4.14 REST routes (today: 503) light up, and post-CRUDManager.Refreshpicks up new subscriptions immediately. - References:
internal/webhook/outbound/{types,store,store_sqlite,manager,dispatcher,retry,circuit_breaker,sign}.go;internal/config/webhook.go(WebhookOutboundConfig);pkg/api/webhooks/handler.go(REST §4.14, landed);cmd/kscore-webhook+internal/cli/webhook(CLI, landed);test/e2e/webhook/(build-tagged integration suite, landed);internal/events/subscriber.go(the subscriber API the boot path will consume); Epic 16 tasks 11-18 (landed, Epic 16 CLOSED 18/18 / 13/13 acceptance); companions: the GitOps webhook receiver + GitOps rollback boot-wiring entries.
Pre-rotation signing-CA retention in the bootstrap trust bundle
- Priority: gate-v1.0
- What: Epic 09 task 14’s
SVIDBootstrapIssuerpopulatesAgentCredentials.TrustBundlePEMfromprovider.GetTrustBundle().X509Authorities()— which in v0.1 is just the root cert.CAManager.RotateSigningCAdoesn’t retain the previous signing CA cert, so an agent that bootstrapped before a rotation but disconnects through it needs to refresh its bundle from the server before its cached SVID chain verifies (the leaf still chains to a now-discarded signing CA whose public material the bundle no longer advertises). Acceptable for v0.5 single-CP (rotation rare, agents reconnect fast); multi-CP HA needs a grace window of retained signing CAs in the bundle. - Why deferred: v0.5 single-CP single-server doesn’t experience the disconnect-across-rotation scenario at scale (rotation is hourly-poll + 50%-of-lifetime). Multi-CP HA (Epic 13) introduces wider clock skew + more aggressive rotation; that’s the natural place to land the retention window.
- Acceptance:
CAManager.RotateSigningCArecords prior signing CAs into a retention list;BuildTrustBundleincludes the active root + retained pre-rotation signing CAs; retention window is config-driven (default 24h, matching MaxSVIDTTL);SVIDBootstrapIssuerandBuildServerTLSConfigboth benefit transparently. - References:
internal/identity/ca.goCAManager;internal/identity/embedded_provider.gobuildBundle;internal/controlplane/bootstrap_jointoken.goSVIDBootstrapIssuer; Epic 09 task 14 (landed); Epic 13 (clustering).
Agent-side cancel propagation (SIGTERM to in-flight commands)
- Priority: gate-v1.0
- What: When
CancelBatchJobfires server-side, also signal the affected agents over NATS so any in-flightos/execprocess is SIGTERM’d. v1.0 cancel persists CANCELLED status server-side; agent-side in-flight processes keep running until they exit naturally. - Why deferred: Needs a new NATS cancel-command message type (subject scheme, envelope shape, signing) and an agent-side handler that maps inbound cancels to per-command context.CancelFuncs in the executor. Each piece is small but together they’re a meaningful surface. v0.x trial scope tolerates the gap — long-running commands time out via agent.ExecutorConfig.DefaultTimeout.
- Acceptance:
kscorectl exec cancel <id>mid-batch results in agent-side processes receiving SIGTERM (then SIGKILL after KillGrace); per-agent batch_agent_result rows record the cancelled state. - References: Epic 07 task 12 acceptance bullet (“agent receives SIGTERM”);
internal/agent/executor.gofor the SIGTERM-then-SIGKILL kill protocol that already exists.
Unified single + batch dispatch persistence
- Priority: gate-v1.0
- What: Today every batch agent dispatch creates both a
commandsrow (viaCommandDispatcher.Dispatch) and abatch_agent_resultsrow. At fleet scale that’s a 2× write cost per command. - Why deferred: Reusing
CommandDispatchergives us free retention + consistent signing for v1.0, but past a certain batch volume the duplication will show up in disk pressure / query cost. - Acceptance: Batch dispatches write to a single row family (TBD: extend batch_agent_results, OR keep commands and drop batch_agent_results, OR linked via a
commands.batch_idFK). Pick depends on observed query shape. - References: Epic 07 task 12 layering note;
internal/controlplane/nats_batch_executor.go.
Server-side target expression compile (proto extension)
- Priority: gate-v1.0
- What: Add
string target_expressiontov1.BatchExecuteCommandRequest(or a newTarget.expressionfield) plus a server-side resolver that runs the fullinternal/targetingshorthand —os:linux,arch:amd64,status:online,ip:10.0.0.0/8,id:web-*(glob),OR,NOT, parens — and returns the matching agent set. kscorectlexecsubcommands switch from client-sideParseTarget(limited to AND-of-labels-plus-hostname-glob) to sending the raw expression string. - Why deferred: v1.0 proto
Targetonly carriesagent_ids,labels,hostname_pattern— three AND’d dimensions. kscorectlexec11a parses the shorthand client-side and rejects (ErrTargetUnsupported) anything that doesn’t fit. Adding a string field is a proto change + regenerated stubs + GRPCServer wiring; we held it out of 11a to keep the CLI commit narrow. - Acceptance:
kscorectl exec run "uptime" --target "os:linux AND (role:web OR role:cache)"resolves correctly against a stretched fleet; the AND-of-labels-plus-hostname-glob path remains a valid (degenerate) subset. - References: Epic 07 task 11a;
internal/cli/exec/target.goParseTarget+ErrTargetUnsupported;internal/controlplane/agent_resolver.go(the existing server-side filter is the foundation).
Batch job retention (DeleteBatchJobsBefore + cascading FK)
- Priority: gate-v1.0
- What:
BatchJobStore.DeleteBatchJobsBefore(t)analogous toCommandStore.DeleteCommandsBefore, plusON DELETE CASCADEon thebatch_agent_results.batch_job_idFK so batch cleanup wipes the per-agent rows in lockstep. A retention loop onBatchDispatcher(runRetention-shaped, mirroring CommandDispatcher) drops batches older than the configured TTL. - Why deferred: Pre-1.0 deployments don’t accumulate enough batches to need retention before trial-readiness. The store surface and the retention loop are independent additions; either can land first.
- Acceptance:
BatchJobStore.DeleteBatchJobsBefore(t)deletes bothbatch_jobsrows and dependentbatch_agent_resultsrows in a single tx (or cascade);BatchDispatcherhas a configurable retention TTL and a periodic sweeper; CLI shows expected behavior after one TTL elapses. - References: Epic 07 task 10;
internal/state/store.goBatchJobStore;internal/controlplane/command_dispatcher.gorunRetentionas the pattern.
Replay protection on agent commands
- Priority: gate-v1.0
- What: Timestamp window + nonce dedup in
internal/agent.SecurityEnforcer.Validate. Today HMAC alone gates command execution. - Why deferred: HMAC covers v0.x trial scope. Nonce store needs a persistence layer + TTL eviction; not worth the complexity for the v1.0 ship date.
- Acceptance: Replayed
CommandRequest(same nonce within window) is rejected with a typed error and audit log; legitimate commands inside the window pass. - References: Epic 06 task 4
_(landed)_annotation; PROJECT-DETAILS §4.10;internal/agent/security.go.
Schema versioning via golang-migrate
- Priority: gate-v1.0
- What: Replace auto-DDL on startup with versioned migrations.
- Why deferred: v1.0 schema is new and stable; auto-DDL is fine until the first breaking change.
- Acceptance:
kscore-migrate up/downcycles cleanly across at least one breaking schema change; CI runs migrations against PostgreSQL + SQLite. - References: PROJECT-DETAILS §4.3 (line 301); Epic 02 scope-out.
Reactor engine + event lifecycle tracking
- Priority: gate-v1.0
- What: Filter→action chains with throttle/debounce/DLQ + lifecycle states (created/published/routed/processing/processed/failed/expired).
- Why deferred: v1.0 ships passive event system (emit/subscribe/query). Reactors need runbook + policy boundaries to be settled.
- Acceptance:
LogAction/EventAction/WebhookActionreactors fire on event match with throttle + DLQ semantics. - References: PROJECT-DETAILS §4.9 (lines 679-681).
Maintenance + Schedule gRPC
- Priority: gate-v1.0
- What: gRPC handlers for
MaintenanceServiceandScheduleService(REST is in v1.0). - Why deferred: gRPC + REST land together post-v1.0 per design.
- Acceptance: gRPC clients call
MaintenanceService.{Plan,Apply,Status}andScheduleService.{Create,List,Run}. - References: Epic 03 scope-out (line 33);
pkg/api/maintenance/handler.go:5.
Server-side heartbeat / metadata NATS subscriber → agent registry
- Priority: gate-v1.0
- What: kscore-server has
internal/controlplane.ConnectionManager.Heartbeat(ctx, id)andUpdateAgentplumbing, but nothing in v0.1 subscribes tokscore.{cluster}.agent.heartbeat/kscore.{cluster}.agent.{id}.stateand feeds those payloads into the registry. Agents publish heartbeats and metadata into the void; the server’s agent registry stays empty. - Why deferred: Epic 06 owns the agent side (publishing) — it shipped. The consumer side is a server-runtime concern that fits naturally with Epic 07 (remote execution targeting reads from the registry) or Epic 13 (clustering / agent registry HA). Either of those epics will land the bridge.
- Acceptance: Three Epic 06 acceptance bullets that gate on this consumer flip to ✓ once the bridge lands —
- “Agent registers with control plane on startup (visible in
kscorectl agents list)” - “Heartbeat every 30s; control plane marks stale after 3 missed”
- “Metadata published on startup + every 60s; visible via
kscorectl agents show <id>” Plus: a server-side integration test driving an agent → server registry visible round-trip.
- “Agent registers with control plane on startup (visible in
- References: Epic 06 task 12
_(landed)_surfaced the gap;internal/controlplane/connection_manager.go:218(Heartbeat method waiting for a caller); agent publishes atinternal/agent/agent.go(runHeartbeatLoop,runMetadataLoop).
state.apply.skip event taxonomy + wiring
- Priority: gate-v1.0
- What: Epic 08 task 6 ships a
RunObserver.Skipcallback for cascade-skipped declarations (an earlier failure aborted the run; subsequent decls don’t execute but are surfaced via the observer so external subscribers — alerting, audit, dashboards — see them). The statemgmt runner does not own event-subject naming; that’s Epic 11. The corresponding event typestate.apply.skipis therefore NOT yet listed in PROJECT-DETAILS §4.9’s event taxonomy (“agent 5 / job 4 / state 5 / system 3 / user 3 / policy 2 = 22 types”). It needs to be added when Epic 11 wires the runner observer to the NATS event bus. - Why deferred: The statemgmt task adds the observer hook; the event system itself lives in Epic 11. Updating §4.9’s taxonomy now would predict an event Epic 11 hasn’t shipped.
- Acceptance: PROJECT-DETAILS §4.9 lists
state.apply.skipalongside the existing 5state.*types (state.*becomes 6 types, total 23); Epic 11’s event publisher emitskscore.{cluster}.events.state.apply.skipfor each cascade-skipped decl; an integration test asserts the subject lands in JetStream. - References: Epic 08 task 6;
internal/statemgmt/runner.goRunObserver.Skip; PROJECT-DETAILS §4.9 event taxonomy; Epic 11 (event system).
Salt-faithful prereq direction in statemgmt resolver
- Priority: gate-v1.0
- What: The Epic 08 dependency resolver applies a uniform direction rule to all eight requisite keys:
<key>: [B]on A puts B before A;<key>_in: [B]puts A before B. Salt’s actualprereqsemantic is the opposite — Salt readsprereq: [B]on A as “A is a prerequisite for B” (A first). Keystone’s rule deviates so all eight keys teach the same way. - Why deferred: One rule is much easier to teach + remember than a per-key direction table. v0.x trial scope hasn’t surfaced a real workflow that needs Salt-faithful prereq; if it does, we add a per-key direction policy in the resolver and surface it on the DSL.
- Acceptance: Resolver applies a per-key direction policy where
prereqandprereq_inuse the Salt-faithful convention while keepingrequire/watch/onchanges(and their_invariants) on the existing uniform rule; docs in PROJECT-DETAILS §4.8 reflect the per-key directions explicitly. - References: Epic 08 task 5;
internal/statemgmt/resolve.gopackage comment; PROJECT-DETAILS §4.8.
Full RBAC role/permission CRUD
- Priority: gate-v1.0
- What:
Role/PermissionCRUD with per-resource permissions + dynamic policy. v1.0 ships fixed admin/operator/readonly. - Why deferred: 3-role model covers trial scope; full CRUD needs UI affordances we don’t have yet.
- Acceptance: Operators define custom roles via
kscorectl roles create; bindings to principals enforce on API calls. - References: Epic 09 scope-out (line 22); PROJECT-DETAILS §4.10 (line 737).
Multi-table transaction wrapper
- Priority: gate-v1.0
- What: A
state.Txtype that batches mutations across tables atomically. - Why deferred: v1.0 stores can serialize through caller-side coordination; full transaction wrapper needs careful error-handling design.
- Acceptance: Cross-table writes (agent + apikey + audit) commit/rollback atomically; tests verify rollback on mid-batch failure.
- References: Epic 02 scope-out (line 24);
internal/controlplane/bootstrap.go:270(API key issuance currently non-transactional).
Rotation orchestrator
- Priority: gate-v1.0
- What: Strategies (blue-green, rolling, canary, immediate) with health checks + auto-rollback.
- Why deferred: v1.0 secrets ship with manual rotation; orchestration is its own domain.
- Acceptance:
kscore-secrets rotateinvokes a strategy, runs health checks, rolls back on failure. - References: Epic 00 deferred list (line 77); PROJECT-DETAILS §4.11 (line 765).
Vault AWS IAM auth method
- Priority: gate-v1.0
- What: Epic 10 task 5 ships the Vault backend with Token / AppRole / Kubernetes / LDAP auth methods. AWS IAM auth (
vault/api/auth/aws) is detected via the config-validation rejection path but not wired, because that sub-package pulls ingithub.com/aws/aws-sdk-go-v2/credentials/stscredsfor the STS request-signing. AWS-EC2-hosted Vault deployments that prefer instance-profile auth are blocked on this. - Why deferred: aws-sdk-go-v2 weight is real — adding it to v1.0 ships a heavier binary for every operator regardless of whether they use AWS. Trial-scope deployments (the v0.5 target) typically use Token or AppRole; production AWS deployments are the natural v1.0 audience for IAM.
- Acceptance:
Auth.Method=awswith anAWSAuthConfig{Role, Region, IAMServerIDHeader, ...}round-trips against avault devserver with the AWS auth method enabled; the AWS SDK dep is brought in only when AWS auth is configured (build tag or explicit constructor). - References:
internal/secrets/vault/config.goAuthConfig (current four-method enum);github.com/hashicorp/vault/api/auth/aws(the SDK helper); Epic 10 task 5 (landed).
Encrypted-file backend master-key rotation tooling
- Priority: gate-v1.0
- What: Epic 10 task 4 ships the encrypted-file backend with
ResolveMasterKey(env / file / inline schemes). The on-disk envelope carries a key fingerprint so a wrong-key boot fails with a recognisable mismatch error, but there is no first-class way to rotate the master key without operator surgery: v0.1 procedure is stop server → decrypt with the old key via a one-off Go program → re-encrypt with the new key → start with new key. PROJECT-DETAILS §4.11 explicitly lists “Master key rotation breaks cache; mitigate via dual-key window” as a gotcha. Akscore-secrets rotate-master-key --from <old-source> --to <new-source>subcommand makes this a first-class operation. - Why deferred: v0.5 single-CP single-server deployments tolerate the manual procedure (rotation is uncommon at trial scale); the dual-key window the gotcha calls for needs the broker-side cache (task 8) to honour two fingerprints simultaneously during a rotation window, which is a wider design than task 4 covers. v1.0 SemVer stability should not lock in a master-key story that has no rotation path.
- Acceptance:
kscore-secrets rotate-master-keydecrypts the state file under the old key, re-encrypts under the new key, persists atomically via the existingwriteAtomic; the broker’s running backends pick up the new key without process restart; the cache holds entries under both fingerprints during a configurable grace window then evicts the old. - References:
internal/secrets/file/master_key.goResolveMasterKey;internal/secrets/file/storage.goenvelope (FingerprintLenkeyID); PROJECT-DETAILS §4.11 (“Master key rotation breaks cache; mitigate via dual-key window”); Epic 10 task 4 (landed); future Epic 10 task 8 cache.
Encryption at rest (KeyProvider)
- Priority: gate-v1.0
- What: Pluggable
KeyProvider(file/age/Vault/Cloud KMS) encrypts secrets + audit logs at rest. - Why deferred: PostgreSQL TDE + filesystem encryption cover v0.x trial. Full implementation gates on cloud KMS work.
- Acceptance:
cfg.storage.encryption.provider = age|vault|aws-kmsround-trips encrypted blobs; key rotation re-wraps. - References: Epic 02 scope-out (line 23); PROJECT-DETAILS §4.3 (line 303); §4.11.
File distribution: NATS Object Store + Git backends, mirror groups, conflict resolution
- Priority: gate-v1.0
- What: Multi-backend file dist with mirror groups, sync engine, conflict resolution strategies.
- Why deferred: v1.0 ships local FS + S3.
- Acceptance: Mirror group spans 3 backends; conflict resolution (newest-wins/largest-wins/primary-wins/manual) is operator-selectable.
- References: Epic 18 scope-out (lines 69-73).
Backup orchestration features
- Priority: gate-v1.0
- What: Automated scheduling, rolling upgrades, drift detection on self-config, self-healing, DR test harness.
- Why deferred: v1.0 ships manual
kscore-backuponly. - Acceptance:
kscore-backup schedule addregisters a cron; rolling upgrade verifies health between waves. - References: Epic 18 scope-out (lines 61-65).
Bootstrap phase handlers + durable checkpointer
- Priority: gate-v1.0
- What: Real
PhaseHandlerimplementations forkscore-bootstrap(host detect via/etc/os-release, config-file writing, binary install + systemd unit, blueprint engine invocation, /health verify) plus a SQLite-backedstatemachine.Checkpointerthat survives process restarts. - Why deferred: Epic 18 task 2 ships the FSM seam +
internal/selfmgmt.BootstrapManager+ an in-memory checkpointer + a recording test double. Real handlers depend on install paths, systemd helpers, and the blueprint runtime; they compose at the CLI layer (Epic 18 task 7). - Acceptance:
kscore-bootstrap --seed dev-seed.yamlruns end-to-end on a fresh Linux host and lands atStateVerified; SIGKILL mid-Configuring then restart resumes atStateConfiguring. - References: Epic 18 task 2 (this commit); Epic 18 task 7;
internal/selfmgmt/bootstrap.go.
Backup + restore component adapters (storage/JetStream/etcd/config/secrets/cluster)
- Priority: gate-v1.0
- What: Real adapter implementations behind the Epic-18 task-3 backup seams AND the task-6 restore seams: SQLite online-backup + Postgres
pg_dumpfor write, file-replace +pg_restorefor read; per-streamnats.go/jsmsnapshots both ways;clientv3.Snapshot()+ etcd snapshot restore; filesystemConfigCollector+ConfigRestore; encrypted-file backendCopy+ overwrite; MembershipManager+ShardStore-drivenClusterMetadata+ClusterRestore. Plus a realClusterDetectorwired from MembershipManager + the state store so the populated-cluster safety guard fires in production. - Why deferred: Each adapter is a non-trivial dep dance (
pg_dump/pg_restoreshell-out vs Go-native, NATS Manager threading, etcd client coupling, secrets backend internals). Epic 18 tasks 3 + 6 ship the orchestrator + 12 narrow seams + manifest + tar artifact + integrity verification + populated-cluster guard so adapters land independently and get wired by thekscore-backupCLI (Epic 18 task 7). - Acceptance:
kscore-backup createthenkscore-backup restoreround-trips populated state onto a fresh server; restore over a populated server fails without--force; partial restore (--config-only) writes only the config files. - References: Epic 18 tasks 3, 6, 7;
internal/backup/manager.go;internal/backup/restore.go.
Policy CRUD via gRPC (CreatePolicy/UpdatePolicy/DeletePolicy/activate/deactivate/remediate/monitor)
- Priority: gate-v1.0
- What: Server-side mutating endpoints for policies (today returns Unimplemented).
- Why deferred: Mutations gate on enforcement going GA so operators can author policies that actually run.
- Acceptance: gRPC clients author policies via the API; CLI subcommands
create|update|delete|activate|deactivate|remediate|monitorwire to the new endpoints. - References: Epic 03 task spec (line 16);
pkg/api/policy/handler.go:5;pkg/api/v1/policy.pb.go:3.
Batch dispatcher: no orphan-job recovery
- Priority: gate-v1.0
- What: Jobs in RUNNING state at process start are not auto-recovered; operators must inspect + retry.
- Why now: Orphan recovery needs ownership semantics (which CP node owns the job?) which settle once clustering has shipped — so the first post-v1.0 cleanup release.
- Acceptance for unblock: On startup the dispatcher reclaims/retries jobs it owns that were RUNNING; additive, no API change.
- References:
internal/controlplane/batch_dispatcher.go:72.
API key issuance: non-transactional
- Priority: gate-v1.0
- What:
internal/controlplane.BootstrapHandlerissues credentials via separate write paths (not a single transaction). - Why now: v1.0 doesn’t have a multi-table tx wrapper (the multi-table-tx backlog entry above) — this is a pure consumer of that, so it tracks that work. Internal refactor, no external surface change.
- References:
internal/controlplane/bootstrap.go:270.
Boostrap PSK consumption: in-memory tracking only
- Priority: gate-v1.0
- What: Used PSKs are tracked in-memory. Server restart re-permits a previously-used PSK.
- Why now: Persistence path needs the bootstrap-state-store work, which rides the bootstrap-hardening cycle. Bug-fix-shaped (makes restart behaviour correct), not a compatibility break.
- References:
internal/config/nats.go:53; Epic 05 task 9_(landed)_.
Server boot-integration foundation (kscore-server lifecycle wiring)
- Priority: gate-v1.0
- What: A single boot/server-lifecycle integration step in
cmd/kscore-serverthat constructs and registers the components every “boot registration” entry below depends on. Epic 13 task 1 deliberately left boot wiring out, so clustering, the module loader, the GitOps webhook receiver + rollback engine, the outbound webhook Manager, and the blueprint/runbook gRPC+REST surfaces are all built as injected seams that nothing wires at server start. This entry is that wiring: construct the components from config, register the gRPC services + mTLS listener(s), mount the REST handlers, subscribe the event consumers, and hang it all on the Epic 04 server start/stop lifecycle (graceful shutdown included). - Why deferred: the recurring dependency-light / dark-until-boot pattern — each subsystem ships its seam unit-tested, and production construction was held for one coherent boot task so the wiring is not scattered. It is the critical-path prerequisite for the cluster, module, GitOps, webhook, and blueprint/runbook boot-registration entries; landing it first unblocks the rest.
- Acceptance for unblock: kscore-server boot constructs + registers the clustering, module-loader, GitOps, outbound-webhook, and blueprint/runbook components from config on the server lifecycle; each companion boot-registration entry’s surface lights up (no longer 503 / Unimplemented) once its component is wired here.
- References:
cmd/kscore-server/main.go(service-registration block); Epic 04 server lifecycle; Epic 13 task 1 (boot wiring left out); companions: the cluster / module / GitOps / outbound-webhook / blueprint-runbook boot-registration entries.
Soak-test infrastructure for fd / connection / goroutine leaks
- Priority: gate-v1.0
- What: Epic 19 §Acceptance line 109 asks for “no goroutine / connection / fd leaks in 1-hour soak test” — a v1.0 gate bar. Task 6 (goleak in integration tests) and task 8 (hardening pass — per-component lifecycle audit in
docs/project/HARDENING-BASELINE.md) shipped the mechanisms; the 1-hour soak harness itself is the remaining gate-v1.0 deliverable. It adds: a soak target (make soakor similar) that runs the topology under representative load for N hours, captureslsofbaselines + diffs, snapshots goroutine count viaruntime.NumGoroutine, and asserts each is bounded. - Why deferred: The mechanism is in place (goleak under integration, lifecycle table for connections). Building the soak orchestration — load generator, time-boxed runner, lsof differ, goroutine snapshotter, CI integration with a long-running job — is dedicated scope.
- Acceptance:
make soakruns the docker-compose topology under sustained load for ≥1 hour, asserts: (a)runtime.NumGoroutineflat ±5%; (b) per-processlsofcount flat ±5%; (c) per-process RSS flat ±10%. CI runs it nightly. - References: epic 19 task 6
_(landed)_; epic 19 task 8_(landed)_;docs/project/HARDENING-BASELINE.md.
Sustained-load profiling baseline
- Priority: gate-v1.0
- What: Epic 19 task 8 captured a CPU + allocation baseline against the perf SLO workload (
make profile;docs/project/PROFILING-BASELINE.md). That workload is ~250 ms total — useful for catching obvious hot spots, insufficient for production-shape profiling. The v1.0 gate’s “performance + load test baseline passing” wants a sustained-load harness (10-minute-ish runs against a multi-agent topology with steady command + event + state-apply load) and a per-domain profile breakdown (state engine, blueprint, secrets, audit). - Why deferred: The initial baseline catches the bar the epic first asked for (>5% CPU / >50 MB alloc); the sustained-load form is the production-shape measurement the v1.0 gate needs on top.
- Acceptance:
make profile-sustained(or equivalent) runs the harness; per-domain profile files land next todocs/project/PROFILING-BASELINE.md’s baseline numbers; the doc gains a per-domain section. - References: epic 19 task 8
_(landed)_;docs/project/PROFILING-BASELINE.md.
Security baseline expansion
- Priority: gate-v1.0
- What: Epic 19 task 7 shipped the initial baseline pipeline:
make security-{secrets,vulns,sast,licenses}running gitleaks, govulncheck, gosec (HIGH-only, G115 excluded), go-licenses (strict). The v1.0 gate’s “security review complete” tightens the configuration + widens the scan set:- Re-enable G115 + per-site audit. Currently ~84 G115 (integer-overflow conversion) findings sit at proto<->Go field-width boundaries, parser-checked archive headers, and range-validated config. Re-enabling the rule requires a per-site audit +
//#nosec G115annotations (or, where the conversion is genuinely unbounded, arange check + return errorfix). - semgrep — cross-language SAST; catches patterns gosec doesn’t (SSRF, template injection, regexp DoS).
- trivy / grype — container-image + lockfile scanning; lands with the v1.0 release-image build.
- syft — SBOM generation; lands with epic 19’s release-artifact work.
- hadolint — Dockerfile linting (
test/e2e/single/Dockerfile.kscore+ the prod release Dockerfile when it lands). - gosec MEDIUM gate — the initial baseline gates HIGH only per epic 19 acceptance; graduate MEDIUM to a required floor after a one-time triage pass.
- Re-enable G115 + per-site audit. Currently ~84 G115 (integer-overflow conversion) findings sit at proto<->Go field-width boundaries, parser-checked archive headers, and range-validated config. Re-enabling the rule requires a per-site audit +
- Why deferred: The initial baseline ships the four scans the epic explicitly names; the broader suite is the v1.0 security-review expansion. The G115 re-enablement specifically requires a large triage PR with little signal-to-noise improvement for the current codebase.
- Acceptance: Each sub-bullet adds its own Make target + CI step + brief docs entry under
docs/project/SECURITY-GOVERNANCE.md“Security Baseline Pipeline.”make security(or equivalent umbrella) runs all. - References: epic 19 task 7
_(landed)_;Makefilesecurity-*targets;docs/project/SECURITY-GOVERNANCE.md“Security Baseline Pipeline” section.
Dependency posture re-audit
- Priority: gate-v1.0
- What: A periodic deep-audit of the direct + indirect Go dependency tree. The v0.x baseline audit (2026-05-23, Phase B3 of
PUBLIC-LAUNCH-CHECKLIST.md) flagged a small set of single-maintainer / lightly-maintained deps that are acceptable for v0.x but warrant explicit re-evaluation before v1.0 / v2.0. Specifically: (a)lib/pq→jackc/pgxgraduation — lib/pq is in maintenance-only mode; pgx is the community-active Postgres driver; (b)gobwas/globre-audit — minimal-scope glob matcher, feature-complete per maintainer, no known CVEs but worth re-validating; (c) single-maintainer-but-active deps re-check —modernc.org/sqlite,santhosh-tekuri/jsonschema/v6,shirou/gopsutil/v4. - Why deferred: No security signal warrants action at v0.x. The audit’s purpose is to keep the dependency profile honest before SemVer stability kicks in at v1.0.
- Acceptance: A timestamped re-audit lands in
docs/project/SECURITY-GOVERNANCE.md“Dependency posture” section with the new module count + license distribution + maintainer-health categorization. Any newly-introduced single-maintainer dep gets an explicit accept/replace decision. - References:
docs/project/SECURITY-GOVERNANCE.md“Dependency posture” section (2026-05-23 baseline);go.moddirect + indirect blocks.
Policy enforcement (Enforce + Warn modes)
- Priority: gate-v1.0
- What: Flip
policy.enforcement_enabled=true. v1.0 ships the policy engine in audit-mode-only; turning enforcement on is a v1.0 blocker that lands late in the path, after operators have built audit-mode confidence across the v0.x line. - Why deferred: A misconfigured policy blocks the fleet. Audit-mode lets operators build confidence first, which is why this lands late rather than early.
- Acceptance: Policy in Enforce mode blocks command exec; Warn mode logs and allows.
- References: Epic 03 task 6; Epic 12 (audit/policy); PROJECT-DETAILS §4.12 (line 859); companion: “Policy enforcement side-effects”.
Policy enforcement side-effects (Warn events + Enforce violation handlers)
- Priority: gate-v1.0
- What: Epic 12 task 10’s
policy.Enforcerships the audit-mode gate plus the allow/deny gate switch behindWithEnforcementEnabled(Audit/Warn → allow, Enforce → block on a denying verdict). PROJECT-DETAILS §4.12’s enforcement spec also requires the side-effects:Warnmode must emit a warn-level policy event;Enforcemode must invoke registered violation handlers before denying. The gate decision is in place; the side-effects still need infrastructure built (a policy-violation event type on the Epic 11 bus + a violation-handler registration/dispatch mechanism on the Enforcer). - Why deferred: the gate logic was testable on its own and proved the shape; the side-effects have no consumer until enforcement is turned on, so they land with the enforcement flip — the behavior-changing release the v1.0 path schedules late.
- Acceptance:
policy.enforcement_enabled=trueis operator-settable (config plumbing);Warnmode emits apolicy.warn(or §4.9-canonical) event through the events bus on a denying verdict and still allows;Enforcemode invokes each registered violation handler (newEnforcer.RegisterViolationHandlerseam) before returningAllowed=false; release notes call out the enforcement behavior change loudly. - References:
internal/policy/enforcer.go(Enforcegate switch — the// Side-effects ... deferredcomment); PROJECT-DETAILS §4.12 “enforcement changes”; Epic 12 task 10 (landed); companion: “Policy enforcement (Enforce + Warn modes)”.
RunbookGRPCServer not registered at boot
- Priority: gate-v1.0
- What:
internal/controlplane/grpc_runbook_server.go::RunbookGRPCServeris fully implemented (all four RPCs:ListRunbooks,GetRunbook,ExecuteRunbook,GetRunbookExecution) and has tests, butcmd/kscore-server/main.gonever callssrv.RegisterService(&v1.RunbookService_ServiceDesc, ...)for it. The service is dark at runtime — a client that callsRunbookService.ListRunbooksgetscodes.Unimplemented. Same shape as the Epic-13 ClusterService / CoordinationService boot-wiring entries already on the gate-v1.0 list. - Why deferred: surfaced during Phase E E-cleanup-2 dead-code audit. The implementation is there; what’s missing is the boot-wiring decision (RunbookGRPCServer needs a configured
Enginefrominternal/runbookto construct, and the production engine has no boot wiring yet). - Acceptance:
cmd/kscore-server/main.goconstructs arunbook.Enginefrom config + registersRunbookGRPCServer. Integration test verifiesRunbookService.ListRunbooksreturns 200 against a running server with a configured runbook catalog. The runbook-catalog config block (config.RunbookConfig?) is defined + validated like blueprint catalog is today. - References:
internal/controlplane/grpc_runbook_server.go;cmd/kscore-server/main.go(service registration block, ~line 474-543);internal/runbookpackage; existing peer entries “Cluster gRPC services boot registration” and “Coordination gRPC services boot registration” on the gate-v1.0 list.
Blueprint and runbook REST handler packages not mounted
- Priority: gate-v1.0
- What:
pkg/api/blueprint/handler.goandpkg/api/runbook/handler.goexist (landed by Epic 15 task 11) but are imported by nothing outside their own_test.gofiles.pkg/api/server/options.gonever wires them onto the HTTP mux. A reader browsingpkg/api/would reasonably expect these REST surfaces to work but they’re unreachable at runtime. The gRPCBlueprintGRPCServerIS registered; only the REST companion is dark. - Why deferred: surfaced during Phase E E-cleanup-2 dead-code audit. The runbook REST mount depends on the same RunbookGRPCServer wiring above (so this could fold into that entry). The blueprint REST mount is standalone — the gRPC server is wired, just the REST handler is orphaned.
- Acceptance:
pkg/api/server/options.go(or equivalent registration site) mounts bothpkg/api/blueprintandpkg/api/runbookhandlers alongside the existing domain handlers. The orphaned-import audit (this very entry) returns zero hits. End-to-end smoke test invokes one route per handler viacurlagainst a running server. - References:
pkg/api/blueprint/handler.go,pkg/api/runbook/handler.go,pkg/api/server/options.go(where other handlers register); related to “RunbookGRPCServer not registered at boot” above.
Zipkin tracing exporter: do not freeze into the v1.0 surface
- Priority: gate-v1.0
- What: Decide Zipkin’s status in the frozen v1.0 config surface before the v1.0 contract freeze.
go.opentelemetry.io/otel/exporters/zipkinis upstream-removed in early 2027; if v1.0 freezestracing.exporter: zipkinas a supported value, removing it later is a breaking config change that SemVer would force into a v2.0 — but upstream removal lands well before a plausible v2.0. The v0.x line is the only window where dropping a config option is free. So: either remove the Zipkin exporter during v0.x, or ship v1.0 withtracing.exporter: zipkinexplicitly marked deprecated/experimental (not a frozen contract) so it can be removed without a major bump. - Why deferred: v1.0 still names Zipkin (Epic 17 task 4); this is a contract-surface call that belongs at the v1.0 freeze, after the v0.x deprecation warning (companion below) has run for a while. The version-skew risk is real — a frozen, unmaintained exporter eventually stops compiling against newer OTel SDK releases, forcing a freeze-or-drop choice.
- Acceptance: by the v1.0 contract freeze,
tracing.exporter: zipkinis either removed (operators on OTLP) or documented as a non-frozen/experimental option exempt from the SemVer freeze; the OTLP exporters (gRPC + HTTP) are the supported, frozen tracing surface; release notes carry the migration note. - References:
internal/tracing/exporters.go;internal/config/tracing.go;docs/project/VERSIONING.md(v1.0 contract freeze); companion: “Zipkin tracing exporter: deprecation warning → OTLP”.
v0.x — desirable pre-v1.0 (no specific gate)
Changie configuration: replacements for reference-link maintenance
- Priority: v0.x
- What: With the
headerPathfix landed in the v0.1.0-followups PR,changie mergenow round-trips CHANGELOG.md without wiping the persistent top-of-file content. The remaining gap is the Markdown reference-links section at the bottom ([Unreleased]: …compare/v0.1.0...HEAD,[v0.1.0]: …,[v1.0.0]: …). Currently the link defs live INSIDE the oldest per-version file (.changes/v0.1.0.md) so they ship at the bottom of CHANGELOG.md after merge, but when v0.2.0 ships the[Unreleased]:compare-base needs hand-updating fromv0.1.0...HEADtov0.2.0...HEADand a new[v0.2.0]:line added — which means editing the immutable.changes/v0.1.0.md. Fix: add areplacements:block to.changie.yamlthat runs at batch time, finds the[Unreleased]:line via regex, and replaces it with[v<new>]:+ an updated[Unreleased]:so the maintenance is automatic. Changie supports this pattern; see changie replacements docs . - Why deferred (from v0.1.0-followups): scope. The main bug —
changie mergewiping the Changelog header — was the v0.1.0-followups PR’s headline fix. Reference-link automation is a nice-to-have that doesn’t block the v0.2.0 cut (manual edit at release time is acceptable for the v0.x line; the maintenance step is documented in.changie.yamlcomments). - Acceptance:
.changie.yamlhas areplacements:block that, whenmake changelog-batch VERSION=v0.2.0runs against a CHANGELOG.md currently pointing at[Unreleased]: …compare/v0.1.0...HEAD, produces a CHANGELOG.md whose bottom reads[v0.2.0]: …releases/tag/v0.2.0\n[Unreleased]: …compare/v0.2.0...HEAD\n[v0.1.0]: …\n[v1.0.0]: …; verified by a round-trip test. The reference-link maintenance note in.changie.yamlis removed when this lands. - References:
.changie.yamlheaderPath comment block;.changes/v0.1.0.mdend-of-file ref-link section; changie replacements docs .
Vault auto re-authentication on token expiry
- Priority: v0.x
- What: Epic 10 task 5’s
tokenRenewerkeeps the Vault auth token alive by callingauth/token/renew-selfshortly before expiry. If renewal fails (network glitch, Vault restart, policy change), the renewer logs WARN and the backend’sHealthreports unhealthy — operators must restart the server to recover, since the renewer doesn’t try to re-login from the stored credentials. For unattended trial deployments this means a transient Vault outage can require manual intervention. - Why deferred: re-login from stored credentials needs careful thought about credential lifecycle. AppRole SecretIDs can be single-use, in which case re-login fails the same way; Token auth credentials don’t change. The right behavior is method-aware and adds enough complexity to deserve its own task.
- Acceptance: On
auth/token/renew-selffailure, the renewer re-invokes the configured auth method to obtain a fresh token and resumes the renewal loop on success; method-specific lockout heuristics (skip re-login if the previous N attempts failed) prevent runaway login storms. - References:
internal/secrets/vault/token_renewer.go;internal/secrets/vault/client.goauthenticate; Epic 10 task 5 (landed).
Bootstrap protocol versioning
- Priority: v0.x
- What: Epic 09 task 14 extended
controlplane.AgentCredentialsfrom three JSON fields to six (addedcert_chain_pem,private_key_pem,trust_bundle_pem). All new fields useomitemptyso older agents reading the JSON ignore them, but a formalprotocol_versionfield would make compatibility explicit + give operators a clean knob for opt-in protocol upgrades down the line. - Why deferred: cosmetic. Both wire shapes (Epic 05 task 9 three-field form + Epic 09 task 14 six-field form) round-trip cleanly through
encoding/jsonand the agent runtime treats new fields as additive. Adding explicit versioning is forward-thinking but not v0.5-blocking. - Acceptance:
AgentCredentials.ProtocolVersion uint32field; bootstrap response carries the current version; agents check + log on mismatch. - References:
internal/controlplane/bootstrap.goAgentCredentials; Epic 09 task 14 (landed).
Direct signing-cert accessor on CAManager
- Priority: v0.x
- What: Epic 09 task 12’s
IdentityGRPCServer.ExportCA WHAT_SIGNINGandGetStatus SigningExpiresAtpaths currently issue a throwaway 1-minute probe cert to reach the signing CA’s cert (issued.Chain[1]). Works, but it’s awkward — every probe consumes a serial number + briefly bloats the on-disk metadata. AddCAManager.SigningCACert() *x509.Certificate(andSigningCAExpiresAt() time.Timefor the status path) so the server reads the in-memory state directly. - Why deferred: cosmetic. The probe-issue path is correct + tested; the cleanup is an internal refactor with no external behavior change.
- Acceptance:
IdentityGRPCServer.ExportCA WHAT_SIGNINGreturns the same PEM without invokingIssueCertificate;GetStatus.SigningExpiresAtreads fromSigningCAExpiresAt(); the throwaway-probe path is removed. - References:
internal/controlplane/grpc_identity_server.goExportCA + GetStatus;internal/identity/ca.goCAManager.
Join-token “any agent” mode (AgentID-less binding)
- Priority: v0.x
- What: Epic 09 task 8’s
JoinTokenAttestorREQUIRES the storedJoinToken.AgentIDto be non-empty — empty AgentID is rejected at attestation with av0.xmention. The §4.10 spec describesAgentIDas an optional bind (“Metadata”); v0.1 makes it required so the attestor always returns a deterministic SPIFFE ID. v0.x adds an “any-agent” mode where the new agent supplies its claimed ID in theAttestRequestand the attestor binds the claimed AgentID atMarkUsedtime. - Why deferred: avoids two interaction loops (the agent-claims-then-server-binds dance + what happens if two agents present the same token concurrently with different claims) that aren’t on the v0.5 gate critical path. Operators today either issue one token per agent or use the
MaxUses > 1mode with explicit AgentIDs per cohort. - Acceptance: a
kscore-identity token create --any-agentcreates a token with emptyAgentID; agents pass--agent-id agent-1on attestation; the attestor binds the claimed AgentID atomically atMarkUsedtime and returns the right SPIFFE ID; concurrent agents-with-different-claims see exactly one success perMaxUsesslot. - References:
internal/identity/join_token_attestor.goJoinTokenAttestor.Attest(the v0.1 empty-AgentID rejection); Epic 09 task 8 (landed); the §4.10 spec’s “optional bind” phrasing.
Live mid-execution stdout / stderr streaming
- Priority: v0.x
- What: Wire
BatchAgentOutputchunks into theBatchExecuteCommand(andCommandOutputChunks into theExecuteCommand) gRPC streams as the agent produces them, instead of one buffered chunk at completion. - Why deferred: v1.0 agents return stdout / stderr buffered inside
CommandResponse— no live mid-execution publish. The gRPC streaming protocol’s shape allows interleaved output chunks (per PROJECT-DETAILS §4.7), so callers won’t see kscorectl--follow-style live output until the agent grows an incremental publish path and the server-side response correlator forwards chunks before the terminalCommandResponse. - Acceptance: A long-running command (e.g.,
tail -F) streams partial stdout to a connectedkscorectl execover gRPC; agent disconnect mid-stream surfaces asAGENT_FAILEDwith the partial bytes already flushed. - References: Epic 07 task 9;
internal/controlplane/grpc_server.go;internal/agent/agent.gocommand handler.
Eval-time observability for malformed targeting patterns
- Priority: v0.x
- What: Slog hooks (or counters) for
match()calls that swallow a malformed glob or an unparseable IP-vs-CIDR comparison. Today both fall through tofalsesilently. - Why deferred: The match function lives in a hot path (one call per agent per term per batch). Per-Matcher logger threading needs either a closure-built program per dispatch or a context plumbed through
expr.Run; both add overhead the v0.x trial doesn’t need. Operators get a clear parse error fromCompile; the gap is only at runtime. - Acceptance: A target expression with a literal-asterisk pattern (
role:*-with-meta-on-empty-value) or a bad CIDR logs once per dispatch with the offending pattern, evaluation count, and target-expression raw form; no per-agent log spam. - References: Epic 07 task 3;
internal/targeting/match.gomatchValue.
git stdlib module — authentication, submodules, advanced clone
- Priority: v0.x
- What: Epic 08 task 11 ships the
gitmodule (statespresent/latest/absent) relying on whatever the agent’s existing git / SSH configuration provides for auth. Reserved for v0.x:- Deploy keys / per-repo SSH identity files, credential-helper configuration, token-in-URL rotation, SSH known-hosts management.
- Submodules (
--recurse-submodulesand submodule sync onlatest). - Sparse checkout, partial clone (
--filter), bare repos. - Branch tracking on
latest(v1.0 updates whatever ref is currently checked out to the fetched commit; it does not switch branches or maintain a named local tracking branch). - Reliable shallow clone + arbitrary-SHA checkout (v1.0 falls back to a full clone then
git checkout <sha>, which can fail on a shallow clone).
- Why deferred: Auth in particular is a security-sensitive surface (key-material handling, host-key TOFU policy) better designed alongside Epic 09/10; the other items are scope-trims to keep the v1.0 module reviewable. The Provider interface (
Inspect/RemoteSHA/Clone/Sync/Remove) is the extension point. - Acceptance: a
credentials:/identity_file:param flows an SSH key or credential helper into the clone/fetch;submodules: truerecurses; the integration test clones a private repo over SSH with a deploy key. - References: Epic 08 task 11
_(landed)_;internal/statemgmt/stdlib/git/git.gopackage comment;internal/statemgmt/stdlib/git/gitcli.go.
link stdlib module — relative-target normalisation
- Priority: v0.x
- What: Epic 08 task 11 ships the
linkmodule (symlink + hard link, statespresent/absent). Symlink targets are compared and stored verbatim — a relative target is not canonicalised against the link’s directory, sotarget: ../fooandtarget: /abs/fooare treated as distinct even when they resolve to the same path. v1.x: an opt-in canonicalising compare (resolve relative targets, optionally chase intermediate symlinks) and Windows link support. - Why deferred: Verbatim compare is unambiguous and matches what the operator wrote; canonicalisation has surprising corner cases (symlink chains, the link’s own directory not existing yet) that deserve their own design pass. Low impact — operators who want absolute behaviour write absolute targets.
- References: Epic 08 task 11
_(landed)_;internal/statemgmt/stdlib/link/link.gopackage comment.
cron stdlib module — per-field schedule, cron.d, env lines
- Priority: v0.x
- What: Epic 08 task 11 ships the
cronmodule (per-user crontab entries viacrontab(1), statespresent/absent, identified by a# keystone-cron: <name>marker comment). v1.0 takes oneschedulestring (five fields or an@-shortcut) and validates only its shape (field count / known shortcut). Reserved for v0.x:- Salt-style separate
minute/hour/day_of_month/month/day_of_weekparams. /etc/cron.ddrop-in mode (acron_d: trueswitch, or a separate module) — for now thefilemodule manages those.- Environment-variable lines (
KEY=value) in the crontab. - Deep cron-field syntax validation (ranges, steps, month/day names).
- Salt-style separate
- Why deferred: One
schedulestring covers the common case ergonomically; the rest are scope-trims to keep the v1.0 module reviewable. The marker-comment design and theProvider(Read/Write) seam accommodate all of it. - Acceptance:
minute: "*/5"etc. compose into a schedule;cron_d: truewrites/etc/cron.d/<name>with ausercolumn; anenv:map emitsKEY=valuelines above the entry; malformed fields are rejected at validate time. - References: Epic 08 task 11
_(landed)_;internal/statemgmt/stdlib/cron/cron.gopackage comment;internal/statemgmt/stdlib/cron/params.govalidateSchedule.
systemd_timer stdlib module — generated service, user timers, more [Timer] knobs
- Priority: v0.x
- What: Epic 08 task 11 ships the
systemd_timermodule — generates a.timerunit fromon_calendar(+ optionalpersistent) and manages its enabled/active state; the triggered.serviceis the operator’s job (compose withfile+service, or pointservice:at an existing unit). Reserved for v0.x:- Also generate the paired
.serviceunit (anexec_start:/user:/working_dir:param set). --user(per-user) timers.on_boot_sec/on_unit_active_sec/on_startup_sec/randomized_delay_secand other[Timer]directives (v1.0 takesOnCalendar+Persistent).- Calendar-expression validation (v1.0 lets
systemctl enablereject malformed expressions at apply time).
- Also generate the paired
- Why deferred: Composing with the
file/servicemodules is the Unix-y v1.0 path and keeps the module small; the[Timer]knob set and the generated-service feature each warrant their own design pass. TheProviderinterface (unit-file ops + the systemctl verbs) extends cleanly. - Acceptance:
exec_start:+user:generate a paired<name>.service;on_boot_sec:emitsOnBootSec=; auser: truetimer lands under~/.config/systemd/user/; a malformedon_calendaris rejected before any file is written. - References: Epic 08 task 11
_(landed)_;internal/statemgmt/stdlib/timer/timer.gopackage comment;internal/statemgmt/stdlib/timer/unit.gorenderTimerUnit.
config stdlib module — more formats, separators, uncomment-aware updates
- Priority: v0.x
- What: Epic 08 task 11 ships the
configmodule (one key/value in a config file, formatskeyvalue+ini, both=-delimited, case-sensitive keys, full-line comments only). Reserved for v0.x:- Case-insensitive key matching (a
case_insensitive: trueswitch) — INI and sshd-style configs often want it. - Configurable separator (
separator: " "for sshd-styleKey Value,": "for YAML-ish, etc.) — v1.0 is=-only. - Inline / trailing comments preservation (
key=value # note) — v1.0 treats# noteas part of the value unless#/;starts the line. - Uncomment-aware updates (
#PermitRootLogin yes→ set the real directive) — v1.0 just appends a new line. - Repeated-key directives (multiple
HostKeylines), multi-line values / continuation lines, indentation-aware insertion under[section]headers, duplicate[section]headers. - TOML / YAML / JSON / XML formats; creating parent directories for a new file.
- Case-insensitive key matching (a
- Why deferred:
keyvalue+inicover the bulk of day-to-day “set this one directive” needs; the rest each carry parsing/round-trip subtleties (especially inline comments and multi-line values) that warrant their own design pass. The line-orientedformat.gocore + theformatparam extend cleanly. - Acceptance:
case_insensitive: truematchesKey/key;separator: " "round-trips ansshd_configdirective; an inline# commentsurvives a value change; setting a directive that exists only as#directive ...rewrites the comment line in place. - References: Epic 08 task 11
_(landed)_;internal/statemgmt/stdlib/config/config.gopackage comment;internal/statemgmt/stdlib/config/format.go.
archive stdlib module — absent state, clean mode, safe symlinks, more formats
- Priority: v0.x
- What: Epic 08 task 11 ships the
archivemodule (extract tar/tar.gz/tar.bz2/zip intotarget,presentonly, idempotent via acreatespath or a size+mtime sentinel, path-escape defense, symlink/hardlink entries skipped). Reserved for v0.x:state: absent— needs an extraction manifest (which files came from the archive) to remove only those; for now usefile: <target>state: absent.clean: true— removetargetbefore extracting (Salt’sarchive.extractedclean mode).- Safe symlink / hardlink extraction (create them only when the link target stays within
target). .tar.xz/.tar.zst/.7zand other formats (need a third-party decompressor dep).- sha-based source identity instead of size+mtime (so a
touchof the archive doesn’t trigger a needless re-extract). owner/groupchown of the extracted tree; mtime preservation of extracted files.- Extraction size / entry-count limits (zip-bomb hardening) —
max_extracted_bytes/max_entries. skip_existing(don’t overwrite files already present intarget).
- Why deferred: “extract a release tarball once” — the v0.1 scope — is the dominant case, and
state: absentplus the security/format extensions each carry real design weight (manifest tracking, decompressor deps, symlink-resolution policy). Theextract.gocore + theformatparam + theProvider-less direct-fs shape extend cleanly. - Acceptance:
state: absentremoves exactly the previously-extracted entries;clean: truewipestargetfirst; a.tar.xzarchive extracts; a symlink entry pointing insidetargetis created while one pointing outside is rejected; a zip bomb is refused once it exceeds the configured cap. - References: Epic 08 task 11
_(landed)_;internal/statemgmt/stdlib/archive/archive.gopackage comment;internal/statemgmt/stdlib/archive/extract.go.
at stdlib module — replace-on-change, per-user queues, batch
- Priority: v0.x
- What: Epic 08 task 11 ships the
atmodule (one-shot scheduled jobs via theattoolchain, tagged with a# keystone-at: <name>marker comment,present/absent, matched by name only). Reserved for v0.x:- Replace-on-change — detect a queued job whose command or time differs from the declaration and re-queue it (v1.0 leaves an existing tagged job untouched; you change the declaration name or
atrmit first). - Per-user
atqueues — submit/list/remove as another user (via su); v1.0 manages the agent’s own queue. - The
batchlow-load variant (thebatchcommand, orat -b). - Queue-letter scoping of the queue scan (
atq -q <letter>); richer multi-line-script handling and submit-time-environment control.
- Replace-on-change — detect a queued job whose command or time differs from the declaration and re-queue it (v1.0 leaves an existing tagged job untouched; you change the declaration name or
- Why deferred: “queue this command once at a given time” is the v0.1 scope;
at’s fire-once model makes replace-on-change and recurring semantics genuinely ambiguous (re-resolving a relative time spec like “now + 1 hour” never equals the daemon’s frozen timestamp), so they want their own design pass. TheProvider(ListJobs/JobScript/Submit/Remove) and the marker-comment identity extend cleanly. - Acceptance: a
presentdeclaration whose command changed re-queues the job (old one removed); auser:param queues a job for that user;batch: truesubmits viabatch; the queue scan can be limited to one queue letter. - References: Epic 08 task 11
_(landed)_;internal/statemgmt/stdlib/at/at.gopackage comment;internal/statemgmt/stdlib/at/provider_linux.go.
x509 stdlib module — combined PEM, encrypted keys, more issuance options
- Priority: v0.x
- What: Epic 08 task 11 ships the
x509module (Go packagepki; manage a TLS cert + private-key pair with crypto/x509, self-signed or CA-signed, RSA/ECDSA/Ed25519,present/absent). Reserved for v0.x:- Combined cert+key PEM in a single file (HAProxy-style) — v1.0 requires
key_path≠ the cert path. - OpenSSL-style SAN prefixes (
IP:/DNS:/email:/URI:) — v1.0 auto-detects IP vs DNS and never emits email/URI SANs. - More Subject fields (Country, Locality, State, OU, …); encrypted (passphrase-protected) private keys.
- Explicit key/cert file mode + owner params (v1.0: new key 0600, new cert 0644; rewrites preserve the mode).
- Key reuse policy on regeneration (v1.0 keeps a still-valid key); CRL / OCSP / AIA / Name-Constraints extensions;
MaxPathLenfor CA certs. - PKCS#12 (
.p12/.pfx) bundles; CSR generation (x509.private_key_managed+x509.csrsplit à la Salt); ACME / external issuer integration.
- Combined cert+key PEM in a single file (HAProxy-style) — v1.0 requires
- Why deferred: “generate a server cert for this host (self- or CA-signed) and renew it before it expires” is the dominant case and the v0.1 scope; the rest each carry their own format/protocol weight (combined PEM round-tripping, encrypted-key passphrase handling, PKCS#12, ACME). The pure-Go
cert.gocore (generateKey/loadPrivateKey/loadCertificate/checkState/buildTemplate/signCert) and the param set extend cleanly. - Acceptance: a combined cert+key file round-trips; a
key_passphrasedecrypts/encrypts the key on disk;country: USetc. land in the Subject; a.p12bundle is produced; a CSR is emitted for an external CA. - References: Epic 08 task 11
_(landed)_;internal/statemgmt/stdlib/pki/x509.gopackage comment;internal/statemgmt/stdlib/pki/cert.go.
mount stdlib module — remount-on-change, escaping, swap, crypttab
- Priority: v0.x
- What: Epic 08 task 11 ships the
mountmodule (manage an /etc/fstab entry + the live mount via /proc/mounts + mount(8)/umount(8); statesmounted/present/unmounted/absent). Reserved for v0.x:- Remount-on-change —
mount -o remountwhen the fstab options change for an already-mounted filesystem; reconcile a live device change (v1.0 updates fstab but doesn’t touch a stale live mount, and doesn’t re-verify the live device against the declaration because the kernel resolves UUID=/LABEL= to a real device). - fstab
\040escaping for whitespace in mount points / devices / options — v1.0 rejects whitespace in those fields. findmnt-based inspection (richer than /proc/mounts);noauto/nofailawareness so amounteddeclaration on anoautoentry isn’t reported drifted just because it isn’t mounted at the moment.- swap-type fstab entries (
swapis theswapmodule’s job); loop-device / encrypted (crypttab) coordination; per-mount fsck/dump heuristics for the defaultpass/dump. unmountedwithpersist: true(also drop the fstab entry — v1.0: useabsent); bind/move/rbind helpers beyond puttingbindinopts.
- Remount-on-change —
- Why deferred: “ensure this device is mounted here with these options, and the fstab agrees” is the v0.1 scope; remount-on-change and the live-device reconciliation need the UUID/LABEL-resolution problem solved (resolve the declared identifier to a kernel device before comparing), and fstab escaping / crypttab / swap each carry their own format weight. The pure
fstab.goeditor and theProvider(Lookup/Mount/Unmount) extend cleanly. - Acceptance: changing
optson amounteddeclaration triggers amount -o remount; a mount point with a space round-trips through fstab and /proc/mounts; anoautomountedentry isn’t flagged drifted while down; an encrypted device coordinates with crypttab. - References: Epic 08 task 11
_(landed)_;internal/statemgmt/stdlib/mount/mount.gopackage comment;internal/statemgmt/stdlib/mount/fstab.go.
swap stdlib module — UUID sources, resize, fallocate, custom opts
- Priority: v0.x
- What: Epic 08 task 11 ships the
swapmodule (manage a swapfile/partition + its fstab entry + its live swapon state; stateson/present/off/absent; a not-yet-existing swapfile is created withdd). Reserved for v0.x:UUID=/LABEL=swap sources — v1.0 requires the source to be an absolute path (swapfile or device).- Enforcing/changing the size of an existing swapfile (
size:only governs creation today);fallocate-based fast swapfile creation (v1.0 usesdd, slow for large files). - Custom fstab options for swap (
nofail,discard, …) beyonddefaults+pri=N;mkswap -L <label>/-f. - Not rotating a swap partition’s UUID when re-activating (v1.0 runs
mkswapbeforeswaponon a not-active source, which re-initialises a pre-existing swap area); btrfs (NOCOW) swapfiles; zram / dphys-swapfile flavours.
- Why deferred: “create a swapfile (or use a partition), enable it, put it in fstab” is the v0.1 scope;
UUID=/LABEL=need the same identifier-resolution work asmount, swapfile resize needs a swapoff/recreate/mkswap/swapon cycle, and themkswap-on-re-activate UUID-rotation concern wants anIsSwapAreaprobe (blkid/swaplabel) to skipmkswapwhen the area is already valid. The purefstab.goeditor and theProvider(Lookup/MakeSwap/SwapOn/SwapOff/CreateSwapfile) extend cleanly. - Acceptance: a
UUID=…swap source resolves and is managed; changingsize:on an existing swapfile resizes it (swapoff → recreate → mkswap → swapon);fallocatecreates a large swapfile instantly;nofaillands in the fstab opts; re-activating a partition doesn’t change its UUID. - References: Epic 08 task 11
_(landed)_;internal/statemgmt/stdlib/swap/swap.gopackage comment;internal/statemgmt/stdlib/swap/fstab.go.
ssh stdlib module — key validation, options-set compare, whole-file management
- Priority: v0.x
- What: Epic 08 task 11 ships the
sshmodule (manage one entry in a user’s ~/.ssh/authorized_keys; statespresent/absent; matched by key material). Reserved for v0.x:- Validating the key blob as a well-formed SSH public key (via
golang.org/x/crypto/ssh.ParseAuthorizedKey) — v1.0 only charset-checks the<keytype> <blob>pair, to avoid promotinggolang.org/x/cryptoto a direct dependency. - Quote-aware comma-splitting / set-comparison of the options field (v1.0 compares it verbatim, after collapsing whitespace) — so
no-pty,no-X11-forwardingandno-X11-forwarding,no-ptyaren’t treated as equal. - An
authorized_keys2/ custom-path override; whole-file management (ssh_auth_file.managed-style — declare the complete key set, removing unmanaged keys). AuthorizedKeysCommand/ SSH certificate (*-cert-v01@openssh.com) handling; per-linefrom=/environment=/tunnel=helpers;ssh_known_hostsmanagement; sshd_config tweaks (overlaps with theconfigmodule today).
- Validating the key blob as a well-formed SSH public key (via
- Why deferred: “ensure this public key is in
’s authorized_keys (with these options/comment)” is the v0.1 scope; full key parsing wants the x/crypto dep promoted (a deliberate hold), and options-set comparison needs a quote-aware splitter, and whole-file management is a different (and more dangerous — it removes keys) operation. The pure authkeys.goline editor and thekey/options/commentparams extend cleanly. - Acceptance: a malformed key blob is rejected at validate time;
options: [no-pty, no-X11-forwarding]matches an existing line with those options in any order; amanage_file: truedeclaration replaces the whole file; a cert-type key (ssh-ed25519-cert-v01@openssh.com) is handled. - References: Epic 08 task 11
_(landed)_;internal/statemgmt/stdlib/ssh/ssh.gopackage comment;internal/statemgmt/stdlib/ssh/authkeys.go.
iptables stdlib module — structured rules, ordering, both-family, distro persistence
- Priority: v0.x
- What: Epic 08 task 11 ships the
iptablesmodule (manage a single iptables/ip6tables rule viaiptables -C/-A/-I/-D; statespresent/absent; optionalsave: <path>foriptables-saveoutput). Reserved for v0.x:family: both(apply the rule to both iptables and ip6tables, requiring it in both); structured rule params (proto/dport/source/jump/… à la Salt) as an alternative to the rawrulestring; rule position / ordering management (re-place a rule that exists but is in the wrong spot — v1.0 never moves an existing rule).- Quote-aware rule parsing (
--comment "with spaces"— v1.0 splits on whitespace, so multi-word comments need the list form or a single word). - Distro-aware persistence (Debian
netfilter-persistent//etc/iptables/rules.v4, RHEL theiptablesservice //etc/sysconfig/iptables) and whole-rules-file management, beyond the plainsave: <path>;iptables-nftvsiptables-legacybackend selection; chain creation (-N) and chain policy (-P). nftablesis its own separate module.
- Why deferred: “ensure this one rule is (not) in this chain” is the v0.1 scope;
family: bothdoubles the bookkeeping, structured params explode the surface (iptables has dozens of match options), ordering management needsiptables -S/handle bookkeeping, and persistence is genuinely distro-divergent. TheProvider(HasRule/AddRule/DeleteRule/Save, parameterised by family) and theruleparsing extend cleanly. - Acceptance:
family: bothkeeps the rule in both ruleset families;proto: tcp, dport: 22, jump: ACCEPTbuilds the rule; a rule that exists at the wrong position is moved when aposition:is declared;--comment "two words"round-trips; on Debian the rule survives a reboot vianetfilter-persistent; aniptables-nfthost is detected. - References: Epic 08 task 11
_(landed)_;internal/statemgmt/stdlib/iptables/iptables.gopackage comment;internal/statemgmt/stdlib/iptables/params.go.
nftables stdlib module — structured rules, ordering, table/chain management, comment matching
- Priority: v0.x
- What: Epic 08 task 11 ships the
nftablesmodule (manage a single nft rule in an existing table/chain: list the chain withnft --handle list chain, match on the rule’s canonical text,nft add rule/nft insert rule … index N/nft delete rule … handle N; statespresent/absent; optionalsave: <path>fornft list rulesetoutput). Reserved for v0.x:- Structured rule params (
proto/dport/saddr/jump/… à la Salt) as an alternative to the rawruleexpression; rule ordering / re-placement (re-place a rule that exists but is in the wrong spot — v1.0 never moves an existing rule); matching a rule by itscomment "…"or by handle instead of by canonical text (so a rule whose body nft has re-normalised — service names, abbreviations — still matches and isn’t re-added). - Quote-aware rule parsing (
comment "with spaces"— v1.0 splits on whitespace, so a multi-word comment needs the list form with the quotes embedded). - Managing tables and chains themselves (
nft add table|chain, base-chaintype … hook … priority …; policy …;), named sets / maps / flowtables, and atomic whole-ruleset file management /nftables.servicepersistence beyond the plainsave: <path>. iptablesis its own separate module.
- Structured rule params (
- Why deferred: “ensure this one rule is (not) in this chain” is the v0.1 scope; structured params explode the surface (nft’s expression grammar is large), ordering management needs handle/index bookkeeping, and table/chain/set management is a meaningfully different (and more destructive) operation. The
Provider(ListRuleHandles/AddRule/DeleteRule/SaveRuleset) and theruleparsing extend cleanly; the canonical-text matcher is the obvious place to add comment/handle matching. - Acceptance:
proto: tcp, dport: 22, jump: acceptbuilds the rule; a rule that exists at the wrong index is moved when anindex:is declared;tcp dport ssh acceptmatches the storedtcp dport 22 accept(canonicalised before comparison);comment "two words"round-trips; amanage_table: true/manage_chain: truedeclaration creates the table/chain with the declared hook+policy; a saved ruleset is reloadable vianft -f. - References: Epic 08 task 11
_(landed)_;internal/statemgmt/stdlib/nftables/nftables.gopackage comment;internal/statemgmt/stdlib/nftables/params.go.
langpkg stdlib module — per-user/per-project installs, lockfiles, semver ranges, more ecosystems
- Priority: v0.x
- What: Epic 08 task 11 ships the
langpkgmodule (one package per declaration viamanager: pip|npm|gem, with optional strict-equalityversion:pin; global / system-wide installs only —pip install,npm install -g,gem install). Reserved for v0.x:- Per-user / per-project installs: pip
--user+ venv + pipx; npm non-global /--prefix; gem--user-install+ Bundler. Aworking_dir:param + auser:param give operators the per-project / per-user mode they need without leaving the abstraction. - PEP-668
--break-system-packagesopt-in for pip on modern Debian / Ubuntu / etc. where system-wide pip is blocked by default. v1.0 lets pip’s own error surface to the operator; the eventual flag (pep668_override: true?) keeps the override explicit and decl-local. - Lockfile-driven installs:
npm ciagainst package-lock.json;pip install -r requirements.txt;bundle install --deploymentagainst Gemfile.lock. These would be a separate op (“install from lockfile”) in the same module, mutually exclusive withname:. - Semver / version-range matching:
>= 1.2,~> 2.5,^3.0.0. v1.0 is strict equality — a manifest likeversion: ">=2"is rejected at validate. - Manager-options pass-through (
mkfs_options-style list): arbitrary install flags —--index-url,--registry,--no-cache,--registry-mirror, env vars (NPM_CONFIG_REGISTRY=…), proxy configuration. - Additional managers:
cargo(Rust),composer(PHP),mvn/gradle(Java),go install(Go binaries — somewhat self-referential but useful),cpan/cpanm(Perl).
- Per-user / per-project installs: pip
- Why deferred: “install / remove this package globally” is the v0.1 scope; per-project mode opens working-directory + cwd + permission concerns; lockfile installs need a different
name-free op shape; semver ranges need a per-manager constraint parser (or a vendored library); the catalog expansion is mostly more Provider methods. The Provider (3 methods × 3 managers) extends cleanly along both axes. - Acceptance: a
working_dir: /opt/app+user: appdecl installsgunicorninto the app user’s venv; alockfile: package-lock.jsondecl runsnpm ciand is idempotent against the lockfile hash;version: ">=2,<3"matches any 2.x version;manager_options: ["--index-url=https://internal.pypi/"]round-trips;manager: cargoinstalls a Rust binary viacargo install. - References: Epic 08 task 11
_(landed)_;internal/statemgmt/stdlib/langpkg/langpkg.gopackage comment;internal/statemgmt/stdlib/langpkg/params.go.
Percentage-based / rolling batch execution
- Priority: v0.x
- What:
kscorectl exec run --rolling 25%for staged rollouts. - Why deferred: v1.0 ships full-fanout + concurrency-cap; rolling needs progress-pause/resume semantics.
- Acceptance: A 100-target batch with
--rolling 10%runs in 10 waves; failure-rate threshold halts the rollout. - References: Epic 07 scope-out (line 29).
Active dial-time circuit breaker eviction
- Priority: v0.x
- What: Skip OPEN endpoints when nats.go picks the next reconnect target.
- Why deferred: Requires replacing nats.go’s native multi-URL failover with a per-endpoint dial loop — substantial refactor for marginal v1.0 benefit.
- Acceptance: An endpoint with breaker OPEN is skipped during reconnect attempts until the breaker half-opens.
- References: Epic 05 task 7
_(landed)_;internal/nats/breaker.go:20.
Glob matching: no ** (double-star)
- Priority: v0.x
- What:
internal/agent.SecurityEnforcerusespath.Match(single-star only). gobwas/glob with double-star semantics is reserved for v1.x. - Why now: stdlib
path.Matchcovers the v1.0 command-allowlist use cases. Small, self-contained; folded into the linting/capabilities work. Note: broadening allowlist matching is a behaviour change to watch in review, but not a compatibility break. - References:
internal/agent/security.go:59.
Migration journal: no per-table checkpoint resume
- Priority: v0.x
- What:
kscore-migraterecords per-table checkpoints in the txlog but recovery from a partial migration restarts from the last full-table boundary, not the row-level checkpoint. - Why now: Row-level resume needs a transactional checkpoint protocol that v1.0’s
state.Tx(deferred post-v1.0) would unlock. Improvement to recovery only; no compatibility break. - References:
internal/state/migrate_txlog.go:25.
Rename source/v1x-backlog label to drop the version pin
- Priority: v0.x
- What: The umbrella label was renamed
v1x-backlog→roadmap-backlogto drop the version pin (the underlying file was renamedV1X-BACKLOG.md→docs/project/ROADMAP.mdearlier). The paired source labelsource/v1x-backlogstill carries the legacyv1xname. Rename it consistently — likelysource/v1x-backlog→source/roadmap-backlog(or another version-neutral name) — so the provenance enum no longer references a retired version line. - Why deferred: surfaced during the pre-public-launch trackerctl provisioning sweep. Renaming the source label touches
tools/trackerctl/issues.go::labelNamesFor,tools/trackerctl/config/labels.yaml,tools/trackerctl/labels_test.go,tools/trackerctl/reconcile_test.go,docs/project/ISSUE-TRACKING.md, the README; and once issues exist that carry the label, a Forgejo-side migration step (relabel-by-API or sync-labels delete+recreate) is needed. The umbrella rename happened in the same sweep because zero issues existed yet; if more issues land before this entry is taken, the rename gets more expensive. - Acceptance:
source/v1x-backlogis gone fromtools/trackerctl/config/labels.yaml,issues.go::labelNamesFor, both test files,ISSUE-TRACKING.md§source/v1x-backlogtable row, and the README.trackerctl sync-labels --applyre-converges Codeberg to the new name. Existing issues (if any) carry the new label and not the old one.roadmap-backlogandsource/v1x-backlogno longer coexist in the label set. - References:
tools/trackerctl/config/labels.yaml;tools/trackerctl/issues.go::labelNamesFor;docs/project/ISSUE-TRACKING.md§source/v1x-backlog; the umbrella rename’s commit (look forroadmap-backlogin the trackerctl/ history).
PROJECT-DETAILS.md lint cleanup
- Priority: v0.x
- What:
PROJECT-DETAILS.mdcarries ~210 pre-existing markdownlint errors: MD032 blanks-around-lists (124), MD029 ordered-list-prefix (45), MD031 blanks-around-fences (23), MD022 blanks-around-headings (14), plus a few MD049/MD028/MD012 stragglers. Currently carved out of the lint glob in.markdownlint-cli2.yamlso CI doesn’t fail. The errors are mechanical (whitespace + ordered-list numbering) but the file is large and the MD029 cluster needs a style decision (one-based-incrementalvssingle-zero— the codebase hasn’t picked one because the rule never enforced). - Why deferred: surfaced during the pre-public-launch lint hygiene sweep. The other 4 large root-level files (FEATURES.md, AGENTS.md, RELEASE-PLAYBOOK.md, CONTRIBUTING.md) all got cleaned + brought into the lint glob in the same sweep; PROJECT-DETAILS.md is the largest holdout and the only one needing a style call before the cleanup can be mechanical.
- Acceptance:
markdownlint-cli2 PROJECT-DETAILS.mdreturns 0 errors. The carve-out is removed from.markdownlint-cli2.yaml’signoresblock.make docs-lint(with the file no longer carved out) stays clean. - References:
PROJECT-DETAILS.md;.markdownlint-cli2.yaml(the carve-out comment to delete); MD029 doc at https://github.com/DavidAnson/markdownlint/blob/main/doc/md029.md for the style call.
Operations runbooks accuracy sweep
- Priority: v0.x
- What:
docs/runbooks/carries pre-v1.0 operational content (bootstrap-new-cluster.md,backup-restore.md,capacity-scaling.md,security-incident.md). Sweep them for accuracy — current binary names, current CLI flags, current config keys, current REST/gRPC endpoints — and graduate them into the Hugodocs/content/en/docs/operations/tree (after the gate-v0.5 Hugo site lands). - Why deferred: The runbooks are correct in shape (procedures, decision trees, recovery steps) but reference interfaces that shifted during reconstruction. A sweep keeps them runnable for v0.5 external testers; a lighter re-verify happens again against the frozen v1.0 contracts before v1.0.
- Acceptance: Each runbook step is verified against a fresh
make e2e-uptopology; commands run as documented; outputs match. - References: epic 19 task 9
_(landed)_;docs/runbooks/*.
Error-message docs URLs
- Priority: v0.x
- What: Many error messages would benefit from a docs URL (
See https://keystone-core.io/docs/errors/<slug>style). Epic 19 §Hardening calls this out; epic 19 task 8 deferred it because the Hugo docs site was post-v1.0 at the time. With Hugo pulled forward to gate-v0.5, this becomes actionable in the v0.x line — it can land any time after the Hugo site is live + the per-error slug pages exist. - Why deferred: URLs to a non-existent docs site would rot. It waits on the gate-v0.5 Hugo site + the per-error slug pages; once those exist, the helper-plus-emitter work is small and serves the v0.5 tester audience directly.
- Acceptance: A
pkg/api/apierror(or similar) helper produces “. See ” strings; the docs site has the matching slug pages; key user-facing errors (config validation, secrets read, command exec failures) carry the URLs. - References: epic 19 task 8
_(landed)_; the Hugo docs ROADMAP entry under gate-v0.5;keystone-core.iodomain provisioning.
kscore-events query subcommand (CEL post-filter)
- Priority: v0.x
- What: PROJECT-DETAILS §4.9 lists
queryalongsidelist. The intended distinction was a CEL-filtered post-fetch query —kscore-events query --filter "tags.role == 'web' && severity.at_least('warn')"runs ListEvents with the indexed filters, then applies the CEL filter client-side, then emits matching events. v1.0 task 7 shipslist(structural filters) +subscribe --replay <window>(CEL on the streaming path); the standalonequerysubcommand is subsumed bysubscribe --replay <large-window>for ad-hoc work. - Why deferred: 80%+ of
query’s use cases are covered bysubscribe --replay. A dedicated subcommand only adds value when (1) operators want a non-streaming, bounded-output exit-when-done flow against CEL, or (2) CEL push-down to SQL is implemented so the filter narrows the query rather than running post-fetch. Pulled to v0.x because it needs no new server surface (client-side CEL over existing ListEvents). - Acceptance:
kscore-events query --filter "..."iterates ListEvents pages, applies CEL filter client-side, emits matching events as JSON lines, exits when the bound is exhausted. CEL syntax:severity.at_least('warn')form per Epic 11 task 5. - References: PROJECT-DETAILS §4.9 CLI list;
internal/events/filter.go(CompileFilter, landed); Epic 11 task 7 (kscore-events CLI; landed).
kscore-audit export --redaction-config file
- Priority: v0.x
- What: Epic 12 task 15 ships
kscore-audit exportwith redaction driven by repeatable flags (--redact-key,--redact-pattern,--redact-user,--redact-replacement). Add a--redaction-config <file>that unmarshals a YAML/JSONaudit.RedactionConfigInputso a vetted redaction policy is reusable + reviewable in source control rather than retyped as flags per invocation. - Why deferred: the flag form satisfies the §4.12 acceptance bar (
--redact-pattern 'password=\S+') and the export path already takes a*RedactionConfig(the file would just be a second constructor of the same struct). §4.12’s risk note (“redaction regex must be reviewed before prod”) makes a checked-in config the prod-grade ergonomic; pulled to v0.x because it is a tiny additive convenience over a complete redaction path. - Acceptance:
kscore-audit export --redaction-config redaction.yamlloads{redact_metadata_keys, redact_patterns, redact_user, replacement}intoaudit.NewRedactionConfig; flags, when also given, layer over / override the file; a malformed file or bad regex fails loudly before the first entry is written. - References:
internal/cli/audit/export.go(flag-drivenRedactionConfigInput);internal/audit/redaction.goNewRedactionConfig(landed); Epic 12 task 15 (landed).
kscore-cluster watch subcommand
- Priority: v0.x
- What: A
kscore-cluster watchlive membership/leadership tail over the existingWatchMembership/WatchLeadershipserver-streams (landed in Epic 13 task 15, not yet CLI-exposed). Split out of thekscore-cluster-backup scheduleentry because it needs no new server surface — pure CLI sugar over RPCs that already exist. - Why deferred: not in the FEATURES/acceptance CLI list; the same stream-CLI gap deferred for kscore-events/kscore-audit. Cheap to add for operators who want a live cluster tail.
- Acceptance:
kscore-cluster watch [--leadership]consumesClusterServiceClient.WatchMembership/WatchLeadershipand renders events until interrupted, likekscore-events watch. - References:
pkg/api/v1ClusterServiceClient.WatchMembership/WatchLeadership(landed in Epic 13 task 15);internal/cli/cluster; Epic 13 task 16 (landed).
kscore-audit search subcommand
- Priority: v0.x
- What:
kscore-audit search— richer ad-hoc filtered query over the audit log. Subsumed bylog’s filter flags for v1.0, but a dedicated search surface is cheap (no new server surface) and friendly for operators. Split out of thekscore-audit analyze/timeline/watchentry because, unlike those, it needs nothing new server-side. - Why deferred:
searchis sugar overloguntil a distinct query model is demanded; pulled to v0.x because it is the cheap, no-new-surface half of the original four-subcommand entry. - Acceptance:
kscore-audit search <filters>runs an ad-hoc filtered query over the audit store and renders matches; degenerates to thelogfilter set where they overlap. - References: PROJECT-DETAILS §4.12
kscore-auditCLI list;internal/cli/audit(log/report/stats landed); Epic 12 task 14 (landed).
Backup destinations: Backblaze B2 documentation + smoke test
- Priority: v0.x
- What: Document Backblaze B2 as a first-class S3-compatible destination — it works today via
s3://+Endpoint: s3.<region>.backblazeb2.comwith no code change — and add an integration smoke test once an account is provisioned. Split out of the broader backup-destinations entry because, unlike SFTP/GCS/Azure, it needs no new code — only docs + a test. - Why deferred: Backblaze B2 is the next-priority compatible service per operator direction; pulled to v0.x because it is the no-code half (docs + smoke) of the original entry.
- Acceptance: B2 documented in
internal/backup/dest/dest.gopackage comment + README; a smoke test exercises a real B2 bucket via thes3://path + B2 endpoint. - References: Epic 18 Scope § (lines 26-28);
internal/backup/dest/.
Zipkin tracing exporter: deprecation warning → OTLP
- Priority: v0.x
- What:
go.opentelemetry.io/otel/exporters/zipkinis upstream-deprecated with planned removal in early 2027. Start the operator migration now: maketracing.exporter: zipkinvalidate with a deprecation warning pointing at OTLP (HTTP or gRPC), and steer the tracing docs toward OTLP as the recommended exporter. OTLP is the OTel-blessed wire format and most backends (including Zipkin via a collector) accept it natively. This is the cheap half — no exporter removal yet. - Why deferred (from v0.1): Epic 17 task 4 shipped Zipkin support because the epic explicitly listed it; the upstream deprecation surfaced afterward. Emitting the warning early gives operators the maximum runway to migrate before the gate-v1.0 freeze decision (companion above).
- Acceptance:
tracing.exporter=zipkinlogs a deprecation warning at config-validation time naming OTLP as the replacement; tracing docs recommend OTLP; the exporter still works (no removal). - References:
internal/config/tracing.go(Validate);internal/tracing/exporters.go;internal/tracing/doc.go; companion: “Zipkin tracing exporter: do not freeze into the v1.0 surface”.
Phase E1: required signed commits on main branch protection
- Priority: v0.x — target v0.8 (the signing batch)
- What:
docs/project/PUBLIC-LAUNCH-CHECKLIST.mdE1 calls for a signed-commit requirement on themainbranch protection rule (alongside the DCO check and required status checks). DCO landed as a CI gate (.forgejo/workflows/ci-fast.ymldco-checkjob, status-check-required onmain) and the 11ci-fast.ymljobs are required, but the “require signed commits” toggle on themainrule is left off for v0.1.x. - Why deferred: solo-maintainer v0.1.x posture. Mandatory signed commits requires a documented contributor-key onboarding flow (GPG/SSH key generation, Codeberg upload, verification cross-check) before it’s reasonable to gate merges on it.
RELEASE-PLAYBOOK.mdalready covers signed release tags, which is the higher-value signing surface for v0.1.x. Deferred 2026-05-27 with maintainer approval during the E1 close-out; briefly re-bucketed to gate-v0.5 (2026-05-31) then returned to v0.x (2026-06-19) once the v0.5 gate checklist was confirmed to carry no signing item. Retargeted to v0.8 (2026-06-22): all signing work — release signing, signed commits, hosted-repo signing — is deferred to one v0.8 supply-chain batch, so v0.2–v0.7 (incl. the v0.5 external-tester milestone) ship unsigned perRELEASE-PLAYBOOK.md§6. - Acceptance: contributor-key onboarding flow documented (probably in
CONTRIBUTING.mdor a newdocs/project/SIGNING-KEYS.md); themainbranch protection rule on Codeberg hasrequire_signed_commits: true; this ROADMAP entry removed and the E1 landed-note in PUBLIC-LAUNCH-CHECKLIST.md updated to mark signed-commit enforcement as live. - References:
docs/project/PUBLIC-LAUNCH-CHECKLIST.mdE1;docs/project/CODEBERG-SETTINGS-AUDIT.md;RELEASE-PLAYBOOK.mdsigning ceremony.
Release signing ceremony — signed tags + checksums + SBOMs
- Priority: v0.x — target v0.8 (the signing batch)
- What: v0.1.0 shipped as a one-time carve-out with no signed tag, no signed checksums, no signed SBOM. Trust model collapsed to TLS-to-codeberg.org + manual
sha256sum -c. A v0.x release lands the full single-signer signing flow perRELEASE-PLAYBOOK.md§6 (Signing) + §9 (Publication): signing-key generation ceremony (RELEASE-PLAYBOOK §2 v0.x simplification),tag.gpgsign truewired into git config,goreleaserconfigured to emit.sigsidecars for archives + packages + checksums + SBOM, RELEASE-PLAYBOOK §6 carve-out removed, CHANGELOG verification section updated to the signed flow. - Why deferred (from v0.1.0): soft-launch posture for v0.1.0 (per
docs/project/GOVERNANCE.md§ Launch Posture +PUBLIC-LAUNCH-CHECKLIST.mdF1) tolerated the unsigned trust gap; signing-key setup + the per-platform key-distribution story were not blocking the curious-operator audience. Recorded as the v0.1.0-only carve-out during v0.1.0 release prep on 2026-05-27; briefly re-bucketed to gate-v0.5 (2026-05-31), returned to v0.x (2026-06-19), then retargeted to v0.8 (2026-06-22): the §6 carve-out was extended to cover the whole v0.1–v0.7 line so the unsigned-vs-signing-onboarding tradeoff is made once, as a v0.8 supply-chain batch, rather than gating the v0.5 external-tester cut. - Acceptance: signing key generated per
RELEASE-PLAYBOOK.md§2 v0.x simplification;git config tag.gpgsign truewired into the release workstation;make releaseemits.sigfiles forchecksums.txt+ every SBOM (and ideally per-archive sidecars pergoreleasersigns:block);RELEASE-PLAYBOOK.md§6 carve-out removed;SECURITY.md“Supply chain security & release verification” section updated; this ROADMAP entry removed when v0.8 ships signed. - References:
RELEASE-PLAYBOOK.md§2 + §6 + §9 (v0.x single-signer);CHANGELOG.mdv0.1.0 Verification section (unsigned trust-model callout);SECURITY.md“Supply chain security” subsection;.goreleaser.yaml(currently nosigns:block).
Native package repositories — APT, DNF/YUM
- Priority: v0.x — unsigned repos land at v0.5; signatures at v0.8 (the signing batch). Build + multi-version publish tooling landed (#220).
- What: v0.1.0 ships
.deband.rpmpackages attached as direct downloads on the Codeberg Release page; operatorsdpkg -i/rpm -ithe file by hand. A hosted package-repo experience —apt/dnfindices onrepos.keystone-core.io,apt-get install kscore-cli/dnf install kscore-serverworking out of the box. Split by signing posture (2026-06-22): the hosted repos themselves ship at v0.5 unsigned (apt [trusted=yes]/dnf repo_gpgcheck=0— same trust level as the unsigned direct downloads, just with install-tool convenience); GPG-signed metadata (aptRelease.gpg/InRelease+ dnfrepomd.xml.asc, repo-signing key onboarded parallel to the release-signing ceremony) is added at v0.8. Remaining for v0.8: the signing key + flipping the templates tosigned-by/repo_gpgcheck=1;docs/project/GETTING-STARTED.mdupdated to use the repo-install path as the primary recipe. - Why deferred (from v0.1.0): soft-launch audience tolerates direct-download
dpkg -i/rpm -i; a later v0.x release is when the convenience step pays off. Decision 2026-05-27 during v0.1.0 release prep; briefly bucketed to gate-v0.5 (2026-05-31), returned to v0.x (2026-06-19), retargeted to v0.8 (2026-06-22) to pair with the signing batch, then split (2026-06-22): unsigned repos pulled to v0.5, signing kept at v0.8. The build + multi-version publish tooling landed in #220 (scripts/repo/,make repo-build/repo-smoke/repo-publishincl.REPO_SIGN=unsigned,deploy/repos/); what remains is the live host going up. - Acceptance: apt repo serving
.deb+ dnf/yum repo serving.rpmfromrepos.keystone-core.io(unsigned at v0.5, GPG-signed at v0.8); install recipe indocs/project/GETTING-STARTED.mduses the repo path as the primary, with the direct-download path documented as a fallback;RELEASE-PLAYBOOK.md§9 “Publication” updated to include the publish-to-repo step; CHANGELOG entry on the release that lands it; this ROADMAP entry removed (signed-repo follow-up tracked with the signing batch). - References:
scripts/repo/+deploy/repos/(tooling, #220);.goreleaser.yamlnfpms block (produces.deb+.rpm);RELEASE-PLAYBOOK.md§9 Publication; companion: “Release signing ceremony” (shared key-onboarding work).
v1.x — post-v1.0 feature additions
Logging: context-aware threading of deep helpers
- Priority: v1.x
- What: 122
slog.Info/Warn/Error/Debug(non-context) calls live in deep helpers (collectors, init paths, shutdown helpers) that don’t havectxin scope. They drop the correlation ID that the request-scoped path threads vialogging.WithCorrelationID. v1.x graduates each site toslog.*Contextby threadingctxthrough the relevant call chains (or accepting a logger that carries the correlation ID). - Why deferred: Each site is small but the threading is cross-cutting; bundling it into task 8’s hardening pass would balloon scope. Request-scoped logging already carries the correlation ID — the gap is only in deep helpers.
- Acceptance:
tools/logaudit(orgrepaudit) reports zero non-contextslog.*calls outside an explicit allowList; allowList entries each name why the site can’t take a ctx (e.g., process-wide init). - References: epic 19 task 8
_(landed)_;docs/project/HARDENING-BASELINE.md“Logging audit” section.
Release dry-run expansion
- Priority: v1.x
- What: Epic 19 task 13 shipped
make release-dry-run: goreleaser snapshot +scripts/release-smoke.shasserting checksum, archive content, linux-binary--version, deb/rpm content (viadebian:12-slim), and opt-in deb/rpm install smoke in freshdebian:12-slim+rockylinux:9containers. v1.x rounds out the smoke surface:- SBOM generation + verification.
.goreleaser.yamlnotes SBOM (CycloneDX + SPDX) is out-of-scope for v1.0; v1.x adds it via a separatemake security-sbompath + a smoke assertion that an SBOM file is present and parseable. - Cross-arch install smoke. v1.0 ships arm64 packages but the install smoke only exercises the host-arch package. v1.x adds
qemu-user-static/binfmt-miscregistration to run arm64 install smoke on amd64 hosts. systemd-analyze verifyon the installed unit files. Requiressystemdin the smoke container, inflating pull size; deferred until SBOM lands so the trade-off is made once.
- SBOM generation + verification.
- Why deferred: The v1.0 smoke covers every artifact-shape failure mode that has shown up in practice (the task 13 implementation surfaced 15 binaries missing
--versionwiring + 6 regex anchor bugs in the smoke script itself; both fixed inline). The remaining gaps are belt-and-suspenders, not v1.0 blockers. - Acceptance: Each sub-bullet adds its own check function in
scripts/release-smoke.sh(or the container companion) and a row indocs/project/SECURITY-GOVERNANCE.md“Release Dry-Run Smoke.” - References: epic 19 task 13
_(landed)_;scripts/release-smoke.sh;docs/project/SECURITY-GOVERNANCE.md“Release Dry-Run Smoke” section.
Rate-limit: Retry-After HTTP-date format alternative
- Priority: v1.x
- What: RFC 7231 allows
Retry-Afterto carry either a delta-seconds integer or an HTTP-date. v1.0 ships delta-seconds only (internal/ratelimit/middleware/http.go::writeRejected429). v1.x adds an opt-in HTTP-date variant for proxies / caches that prefer the absolute form. - Why deferred: Delta-seconds is the more widely-supported shape and satisfies the v1.0 acceptance line. HTTP-date is operator preference; no concrete demand yet.
- Acceptance: An operator flag selects between formats; tests cover both round-trips through a representative client.
- References:
internal/ratelimit/middleware/http.go; RFC 7231 §7.1.3.
Rate-limit: configurable gRPC retry-after-ms trailer key
- Priority: v1.x
- What: The gRPC interceptor emits a
retry-after-mstrailer with the suggested retry delay (internal/ratelimit/middleware/grpc.go::setRetryAfterTrailer). The key name is hard-coded; some operator environments use different conventions (e.g.grpc-retry-pushback-ms). v1.x exposes the trailer key as a constructor option. - Why deferred: There is no formal gRPC standard for retry-after; the v1.0 key is the convention this codebase ships. Renaming is mechanical when an operator needs it.
- Acceptance: Interceptor constructor accepts a trailer-key option; default is unchanged; tests cover override.
- References:
internal/ratelimit/middleware/grpc.go.
Rate-limit: Auditor hook for rejections
- Priority: v1.x
- What: The file-distribution transport.Service has an optional
Auditor func(principal, op, path, reason error)callback that fires on ACL denials (internal/files/transport/service.go). The rate-limit middleware has no analogous hook today; rejections are countable via metrics but not auditable. v1.x adds a symmetricAuditorseam so Epic 12’s audit store can record rate-limit denials with the inbound key + the configured limit. - Why deferred: The metric (
kscore_ratelimit_rejected_total) already gives operators rejection visibility for v1.0 dashboards. Auditing rejections is a compliance-shop ask; bundle it with the post-v1.0 rate-limit expansion (per-namespace quotas, per-route rules). - Acceptance: New
WithAuditor(fn)option on the HTTP middleware + gRPC interceptor; auditor fires synchronously on each deny carrying key + reason + observed RPS. - References:
internal/ratelimit/middleware/http.go;internal/files/transport/service.go::Auditor(the symmetry target).
File distribution: PUT-side resume
- Priority: v1.x
- What: Resume a chunked PUT after a network interrupt — client retries with the same transfer-id, server picks up at the last received chunk. v1.0 ships GET-side resume only (
internal/files/transport.GetOptions.FromChunk); a partial PUT today must restart from chunk 0. - Why deferred: PUT resume needs durable server-side scratch state — the service must remember which chunks of which in-flight transfer it has received across crashes. v1.0’s
internal/files/transport.Servicekeeps in-flight state in memory only; the durable layer (JetStream-backed chunk inbox, or a SQLite scratch table) is meaningful design surface. GET resume is mechanically simple (re-read from the backend at the requested chunk offset) and satisfies Epic 18’s “Resume after network interrupt works” acceptance line for the common direction. - Acceptance: A PUT interrupted at chunk K resumed by the client with
FromChunk=Kcompletes without re-uploading chunks 0..K-1; server-side scratch state survives restart; per-transfer scratch is reaped on a configurable TTL. - References:
internal/files/transport/service.go::handlePut;internal/files/transport/doc.go(Resume §); Epic 18 task 11.
Backup destinations: SFTP + GCS + Azure Blob + advanced S3 auth
- Priority: v1.x
- What: (a) Three NEW destination backends behind the Epic-18 task 5
Destinationseam: SFTP (golang.org/x/crypto/ssh + go-sftp), Google Cloud Storage, Azure Blob. (b) Advanced S3 auth flavors not wired in v1.0 (IRSA, instance profiles, assumed roles, web identity) — reachable through minio-go’s IAM credentials provider but not exposed viaConfiguntil needed. (Backblaze B2 — which works today vias3://with no code change — was split out as a v0.x doc + smoke-test entry.) - Why deferred: v1.0 ships local filesystem + S3-compatible (AWS / MinIO / B2 / Wasabi / R2 / DigitalOcean Spaces via the same
s3://scheme + endpoint). SFTP / GCS / Azure each need a new SDK + auth surface; bundling them now would balloon v1.0’s dep graph. Epic 18 Scope § lists “SFTP, GCS, Azure for v1.5”. - Acceptance:
kscore-backup create --dest sftp://host/path/foo.tarsucceeds; same forgs://andazblob://;AWS_WEB_IDENTITY_TOKEN_FILEproduces a valid S3 client on EKS. - References: Epic 18 Scope § (lines 26-28);
internal/backup/dest/.
Backup encryption: AWS KMS + Vault key providers
- Priority: v1.x
- What: KeyProvider adapters that wrap AWS KMS Encrypt/Decrypt and Vault Transit operations so
kscore-backupcan encrypt with cloud-managed keys instead of operator-managed age private keys. The age envelope stays as the on-disk format; KMS/Vault wrap the age recipient material (envelope-of-envelope). - Why deferred: v1.0 ships file-backed age keys only (
internal/backup/age.LoadIdentityFile/LoadRecipientsFile). The KMS path needs the cloud-credential surface that arrives with the v1.x cloud-identity work; mixing the two now would force partial cloud-credential code into v1.0. Epic 18 Risks § names “v1.5 integrates KMS”. - Acceptance:
kscore-backup create --key-provider=aws-kms:arn:...writes an artifact whose age recipients are wrapped under the KMS key; restore unwraps via the same KMS. Same forvault:transit:keystone-backup. - References: Epic 18 Risks §; PROJECT-DETAILS §4.20 (“master key from env or KMS”);
internal/backup/age/.
kscore-secrets backends subcommand
- Priority: v1.x
- What: Epic 10 task 10 ships the
kscore-secretsCLI with seven subcommand groups (get / put / delete / list / leases / transit / leases-leaf-cmds). PROJECT-DETAILS §4.11 also listsbackends— list configured backends + capabilities — which v1.0 defers because the gRPC service has noListBackendsRPC yet. Operators today can read this from the server’s YAML config or the/api/statuspayload, but a CLI surface would be friendlier. - Why deferred: needs either a new gRPC method (
SecretsService.ListBackends() → BackendInfo[]) or an extension of/api/status’s payload. v1.0 trial workflows can read the YAML directly. - Acceptance:
kscore-secrets backendsprints one row per configured backend with name + type + capabilities; gRPC method (or REST endpoint) populates the data. - References: PROJECT-DETAILS §4.11 CLI list;
api/proto/keystone/core/v1/secrets.proto(no ListBackends); Epic 10 task 10 (landed).
kscore-secrets audit subcommand
- Priority: v1.x
- What:
kscore-secrets auditqueries the secrets-domain audit log — everysecret.accessevent the broker fires on every op. v1.0 emits events via the slog-backedLogAuditor(Epic 10 task 3); Epic 12 ships the SQLiteAuditStore+AuditServicegRPC that this subcommand queries. - Why deferred: depends on Epic 12’s audit-query API which is a separate epic.
- Acceptance:
kscore-secrets audit --action=get_secret --since=1hreturns matching audit rows; flags include--principal,--path,--allowed,--limit,--page-token. - References: PROJECT-DETAILS §4.11 + §4.12; Epic 12 (Audit & Policy); Epic 10 task 10 (landed).
kscore-secrets dynamic subcommand
- Priority: v1.x
- What:
kscore-secrets dynamic <path>issues a dynamic credential through the broker. v1.0 lacks anIssueDynamicSecretRPC — operators can drive this through the in-process API but not over gRPC. - Why deferred: requires adding a new proto RPC
IssueDynamicSecret(path, role, ttl, params) → Secret. Additive but uses proto-breaking-pipeline time + needs careful thought on theparamsshape (map<string, string> vs google.protobuf.Struct for arbitrary types like PKIalt_names). - Acceptance: proto adds
IssueDynamicSecretRPC; CLI subcommand exists; ENG round-trips an actual Vault DB credential round-trip in the integration test. - References:
api/proto/keystone/core/v1/secrets.proto;internal/secrets/backend.goIssueDynamicSecret; Epic 10 task 10 (landed).
kscore-secrets cache subcommand
- Priority: v1.x
- What:
kscore-secrets cache statsreports hit-rate / entry count / memory bytes;kscore-secrets cache cleardrops every entry (operator-driven post-rotation). Epic 10 task 8’sSecretCacheexposes both internally; v1.0 doesn’t expose them over the wire. - Why deferred: needs new proto RPCs
GetCacheStats+ClearCache(or a REST/api/v1/secrets/cacheendpoint). Operator demand at trial scale is low;kscore-serverlog lines emit cache stats on a regular cadence. - Acceptance:
kscore-secrets cache statsprints hit/miss/eviction counters;kscore-secrets cache clearempties the cache + the next operator request observes a cache miss. - References:
internal/secrets/secret_cache.goSecretCache.{Stats,Clear}; Epic 10 task 8 (landed); Epic 10 task 10 (landed).
kscore-secrets template subcommand
- Priority: v1.x
- What: Consul-template-style config rendering. Operators write a config template with
{{ secret "kv/app/db" "password" }}placeholders; the subcommand fetches every referenced secret and renders the output. Common pattern for app-config integration where the app doesn’t natively talk to the secrets API. - Why deferred: substantial feature — needs a template language pick (Go text/template + custom funcmap is the obvious starting point), file output / atomic-rewrite semantics, optional watch mode that re-renders on secret changes (depends on Epic 11 events), and security review of what placeholder syntax is permitted.
- Acceptance:
kscore-secrets template -i /etc/app/config.tmpl -o /etc/app/configrenders the template, atomically replaces the output, exits 0; missing-secret references fail-fast with a clear error;--watchre-renders on a SIGUSR1 (or Epic 11 event when ready). - References: PROJECT-DETAILS §4.11 CLI list; analogous to
consul-template/vault agent; Epic 10 task 10 (landed); Epic 11 (Events) for the watch path.
kscore-events retention subcommand
- Priority: v1.x
- What: PROJECT-DETAILS §4.9 lists
retentionin the kscore-events CLI surface — operator-driven manual application of retention policies + inspection of recent retention runs. The v1.0 retention enforcer (Epic 11 task 8) runs hourly on the cluster leader; operators have no CLI surface to trigger an ad-hoc run, inspect the current policy table, or audit recent retention deletions. - Why deferred: needs new gRPC RPCs
ApplyRetention(policies) → (deleted_count)+GetRetentionPolicy() → []RetentionPolicy+GetRetentionHistory(limit) → []RetentionRun. The retention enforcer itself (Epic 11 task 8) ships the scheduling; the CLI is the operator-facing manual hook on top. - Acceptance:
kscore-events retention showprints the active policy table;kscore-events retention apply --type agent.heartbeat --max-age 24h --max-count 10000invokes a one-shot retention pass against that policy;kscore-events retention history --limit 10shows recent runs with timestamp + rows-deleted. - References: PROJECT-DETAILS §4.9 CLI list;
api/proto/keystone/core/v1/event.proto; Epic 11 task 7 (kscore-events CLI; landed); Epic 11 task 8 (retention enforcer; pending).
kscore-events analyze subcommand
- Priority: v1.x
- What: PROJECT-DETAILS §4.9 lists
analyzeas an operator analysis tool over the event stream. The spec is fuzzy — likely something like top-N event types by frequency, error-rate over a window, source-distribution histograms, or anomaly detection. Defer until we know the concrete operator pain points it should solve. - Why deferred: spec is intentionally fuzzy in §4.9 (“analyze” without acceptance criteria). v1.0’s
statssubcommand covers the simple total + per-type + per-severity case; everything else needs operator demand data to scope.query(deferred separately) covers ad-hoc filtered exploration. - Acceptance: TBD — gather operator pain points from v0.x trial deployments first; likely lands as one or more sub-subcommands under
kscore-events analyze. - References: PROJECT-DETAILS §4.9 CLI list; Epic 11 task 7 (kscore-events CLI; landed).
kscore-policy check / test subcommands
- Priority: v1.x
- What: PROJECT-DETAILS §4.12’s CLI v1.0 list includes
checkandtestalongsidelist|validate|show|eval|compliance|violations. Epic 12 task 14 shipped the latter six;checkandtestwere deferred.checkis plausibly a fold ofvalidate+ a dry-run against a sample input (overlapseval);testis a policy unit-test harness (a testdata directory of input→expected-verdict cases, à laopa test). Both are policy-authoring conveniences with no acceptance criteria in the epic. - Why deferred: §4.12 gives neither an acceptance line nor a concrete shape (mirrors the kscore-events
analyzedeferral). v1.0’svalidate(compile-check) +eval(run against--input) already cover the authoring loop;check/testonly earn their keep once operators define the testdata/assertion format they want (likelyopa test-compatible for the OPA case). Scoping needs trial-deployment demand. - Acceptance: TBD — gather authoring-workflow pain from v0.x trials;
testlikely lands askscore-policy test <dir>running a fixtures directory of{input, expect_allowed, expect_violations}cases with a non-zero exit on mismatch;checklikely folds intovalidate --input. - References: PROJECT-DETAILS §4.12 CLI v1.0 list;
internal/cli/policy(validate/eval landed); Epic 12 task 14 (landed).
kscore-audit analyze / timeline / watch subcommands
- Priority: v1.x
- What: PROJECT-DETAILS §4.12’s
kscore-auditv1.0 list islog|report|export|stats|search|analyze|timeline|watch. Epic 12 task 14 shippedlog|report|stats;exportis owned by task 15;searchwas split out as a v0.x entry. The remaining three: analyze (fuzzy operator-analysis tool, no acceptance criteria — same shape as the deferred kscore-eventsanalyze); timeline (all evaluations for a single resource over time — maps topolicy.ReportGenerator.ResourceAuditTrail, which has no gRPC RPC on PolicyService); watch (live audit tail — needs an audit-tail streaming RPC + theaudit.BufferedAuditorexposed over the wire; neither exists in v1.0). - Why deferred:
analyzeis fuzzy-spec (gather demand first);timelineandwatchrequire new server surface (aResourceAuditTrailunary RPC and an audit-tail server-stream respectively) that is out of task 14’s scope and not on the v1.0 critical path. - Acceptance:
timeline— addPolicyService.GetResourceAuditTrail(or an audit-service RPC) wrappingReportGenerator.ResourceAuditTrail, thenkscore-audit timeline <resource-type> --since …renders it oldest-first.watch— add an audit-tail server-streaming RPC fed byaudit.BufferedAuditor, thenkscore-audit watchtails it likekscore-events watch.analyze— TBD from trial demand. - References: PROJECT-DETAILS §4.12
kscore-auditCLI list;internal/policy/compliance.goResourceAuditTrail(landed, no RPC);internal/auditBufferedAuditor(landed, not wire-exposed); Epic 12 task 14 (landed).
kscore-cluster-backup schedule subcommand
- Priority: v1.x
- What: Epic 13 task 16 shipped
kscore-cluster-backup(backup|restore|list|verify). Theschedulesubcommand (automated periodic snapshots) is deferred — the epic explicitly tags backup scheduling/automation as a future release (epic lines 47/60). (kscore-cluster watchwas split out as a separate v0.x entry.) - Why deferred:
scheduleis out of v1.0 epic scope by the epic’s own wording (no acceptance line; needs a scheduler/retention design — overlaps the general backup-automation deferral). The acceptance-critical surface (status,backup --output,restore --input --force) all shipped. - Acceptance: TBD from trial demand; likely
kscore-cluster-backup schedule add --cron … --output-dir …registering a periodic snapshot job with retention. - References:
internal/cli/cluster(landed: the 13 shipped subcommands); Epic 13 task 16 (landed); companion: the general backup automation/scheduling deferral.
Strict audit-on-access via Auditor.Emit error return
- Priority: v1.x
- What: §4.11’s “failure to log = bug” invariant currently can’t fail the in-flight secrets op when the events-bridge audit publish fails —
secrets.Auditor.Emithas no error return (deliberately, per the v1.0 contract “fire and forget; never error back to the caller”). The Epic 11 task 10 bridge (cmd/kscore-server/audit_bridge.go) logs publish failures at WARN and bumpsevents.AuditEmitter.FailedPublishes, but the broker continues regardless. Stronger consistency requires the Auditor interface to surface errors so the broker can fail the op (or fail-open with a configurable policy). - Why deferred: changing
secrets.Auditor.Emitto return an error is a broad refactor — every existing implementation (LogAuditor, BufferedAuditor, SamplingAuditor, MultiAuditor, NoopAuditor, every test fake) updates; the broker grows policy code for “what to do when emit fails” (fail-open vs fail-closed, configurable). Operator demand at v0.x trial scale is “log + alert when it fails,” which the current bridge already provides via the FailedPublishes counter + ERROR slog line. - Acceptance:
secrets.Auditor.Emitreturnserror; the broker has anAuditFailurePolicyconfig (fail_open/fail_closed); the bridge propagates publish errors; integration test confirms a denied publish underfail_closedcauses the secrets op to error. - References: PROJECT-DETAILS §4.11 “failure to log = bug”;
internal/secrets/audit.goAuditor interface;cmd/kscore-server/audit_bridge.go; Epic 11 task 10 (landed).
Secrets transit batch API on the wire
- Priority: v1.x
- What: Epic 10 task 7’s
TransitBackendinterface supports batch ops (each method takesItems []…Input), but the v1.0api/proto/keystone/core/v1/secrets.protoEncryptRequest/DecryptRequest/SignRequest/VerifyRequestare singleton-only (one plaintext per call). Operators with bulk-transit workloads can drive the in-processTransitBackenddirectly but can’t reach batch over gRPC/REST. - Why deferred: a proto change is the right tool — adding a
BatchEncryptRPC with arepeated EncryptItemshape is additive but needs the buf-breaking pipeline run and the gRPC service stubs regenerated. Operator demand for batch over the wire isn’t pressing at trial scale; the in-process API serves the only known caller. - Acceptance:
api/proto/keystone/core/v1/secrets.protoaddsBatchEncrypt/BatchDecrypt/BatchSign/BatchVerifyRPCs withrepeated …Itemrequests and per-item-error responses; the gRPC service implementation routes them throughsecrets.TransitBackend.{Encrypt,Decrypt,Sign,Verify}directly (no item-by-item loop on the server side); REST handlers gainPOST /api/v1/transit/{op}/batchendpoints. - References:
internal/secrets/transit.goTransitBackend;api/proto/keystone/core/v1/secrets.proto; Epic 10 task 9 (landed).
Secrets transit GenerateDataKey on the wire
- Priority: v1.x
- What: Epic 10 task 7 added
TransitBackend.GenerateDataKey(envelope encryption — returns a plaintext key + the Vault-wrapped form). v1.0’s proto + REST surface doesn’t expose it; operators can only reach it via the in-process API. - Why deferred: envelope encryption is a specialist pattern; the v1.0 audience (
Encrypt/Decrypt/Sign/Verifyfor app-layer integration) doesn’t depend on it. Adding a proto RPC + REST endpoint is straightforward but uses proto-breaking pipeline time. - Acceptance:
api/proto/keystone/core/v1/secrets.protoaddsGenerateDataKeyRPC with aModeenum field (PLAINTEXT/WRAPPED) +bits+context; the gRPC service forwards toTransitBackend.GenerateDataKey;POST /api/v1/transit/datakey/{plaintext|wrapped}/{key}REST endpoint mirrors it. - References:
internal/secrets/transit.goTransitBackend.GenerateDataKey;api/proto/keystone/core/v1/secrets.proto; Epic 10 task 9 (landed).
Encrypted-file per-secret TTL expiry
- Priority: v1.x
- What: Epic 10 task 9’s gRPC + REST
WriteSecretaccepts attl_secondsfield (per the proto declared in Epic 03). The current behavior stores the value asMetadata["ttl_seconds"]so backends that honor it natively (Vault KV v2 via lease TTLs on dynamic mounts) can read it, but the encrypted-file backend (task 4) doesn’t auto-expire entries past their TTL — the metadata is observational only. - Why deferred: per-secret expiry on the file backend needs an additional disk sweep alongside the existing CAS / atomic-rewrite logic; not a v0.5 / v1.0 gating need. Operators that want per-secret TTL today use the cache TTL (Epic 10 task 8) or the Vault backend.
- Acceptance: encrypted-file backend reaps entries whose
Metadata["ttl_seconds"]has elapsed sinceUpdatedAt;GetSecreton an expired path returnsErrSecretNotFoundeven without an explicitDeleteSecret; a configurable background sweep runs at a default 1m cadence; rewrite-on-expiry is atomic. - References:
internal/secrets/file/backend.go;api/proto/keystone/core/v1/secrets.protoWriteSecretRequest.ttl_seconds; Epic 10 task 9 (landed).
kscore-identity federation subcommands
- Priority: v1.x
- What: Epic 09 task 12’s
kscore-identityCLI ships seven subcommands (token {create,list,revoke,cleanup}+ca {info,rotate-signing,export}+status). PROJECT-DETAILS §4.10 names a fourth top-level group —federation {add-domain, list, fetch-bundle}— for cross-trust-domain operation. v0.1 explicitly defers it (the spec marks it post-v1.0). The CLI + gRPC service + EmbeddedProvider would gainFederatedTrustDomainrecords, a bundle-endpoint exposure, and per-domain refresh hints; lands alongside the Provider trust-federation work itself. - Why deferred: federation needs a wire protocol for bundle distribution + cross-domain trust policy + a federated
Attestpath — its own design pass. v0.1 ships single-domain only. - Acceptance:
kscore-identity federation add-domain spiffe://peer.example.org/registers + fetches the peer bundle;kscore-identity federation listshows registered domains + last-refresh timestamps;kscore-identity federation fetch-bundle <domain>retrieves it ad-hoc; the existing trust-federation v1.x ROADMAP entry covers the underlying provider work. - References: Epic 09 scope-out (line 23);
internal/cli/identity(current CLI surface); the existing “Trust federation (cross-domain bundle endpoint)” entry below.
Trust federation (cross-domain bundle endpoint)
- Priority: v1.x
- What: SPIFFE federation — fetch + verify trust bundles from peer domains.
- Why deferred: v1.0 ships single-domain embedded CA; federation needs bundle distribution protocol.
- Acceptance:
kscore-identity federation add-domain/list/fetch-bundleworks against a second cluster. - References: Epic 09 scope-out (line 23); PROJECT-DETAILS §4.10.
WASM module runtime
- Priority: v1.x
- What: Wazero-based module execution alongside the v1.0 Starlark runtime.
- Why deferred: Starlark covers v1.0 module needs; WASM adds a second runtime to maintain.
- Acceptance:
pkg/plugin/runtime/wasmloads + runs a signed wasm module via the sameRuntimeinterface as Starlark. - References: Epic 00 deferred list (line 69).
Module signing: cosign keyless / Rekor transparency / encrypted cosign keyfile interop
- Priority: v1.x
- What: Epic 14 task 4 ships
pkg/module/verifyas a pure-stdlib, cosign-compatible keyed detached-blob verifier (RSA/ECDSA/Ed25519, KeyID-indexed trust policy) — explicitly Option C, nosigstore/cosigndependency. Deferred: (a) cosign keyless signing/verification (Fulcio-issued short-lived certs + OIDC identity); (b) Rekor transparency-log inclusion proofs; (c) reading the encrypted cosign keyfile format thatcosign generate-key-pairemits (scrypt + nacl/secretbox), sokscore-module signcould consume an existingcosign.key. v1.0kscore-module signuses a plain PKCS8/SEC1 PEM (“local.key”). - Why deferred: the epic’s v0.1 decision is “Cosign-only verification, no SumDB transparency log” and OCI is v1.1; keyless/Rekor pull the full
sigstore/cosign+ Fulcio/Rekor/TUF dependency tree (Kubernetes-scale) for capability that is entirely post-v1.0 scope. Keyed verification + a TLS-trusted registry is the v1.0 supply-chain baseline; the stdlib verifier is forward-compatible (the trust policy /Signaturetypes extend to a keyless issuer without an API break). - Acceptance: a module signed by the real
cosignCLI in keyless mode verifies through an extended trust policy with a Rekor inclusion check;kscore-module signcan load a cosign-encryptedcosign.key. - References:
pkg/module/verify(task 4, landed — keyed/stdlib); epic 14 “Scope (out)” (SumDB v1.2); companion of “WASM module runtime”.
Module registry publish authentication
- Priority: v1.x
- What: Epic 14 task 9 ships
cmd/kscore-registrywith an unauthenticatedPOST /publish(multipart manifest + module ZIP). Deferred: authenticating publishers (API key / mTLS client cert / signed-publish token) and per-namespace publish authorization so only an owner can publish undervendor/*. - Why deferred: §4.18’s v1.0 trust model is the TLS-trusted registry transport + Cosign verification at load time (the loader/task-4’s job), and the v1.0 filesystem registry runs on a trusted boundary (local / trusted network); the epic gives no publish-auth acceptance criterion. Read integrity is already covered (content-addressed hashes + signature verification at install). Publisher identity is an additive concern that does not change the artifact format.
- Acceptance:
POST /publishrequires a valid credential (API key or mTLS); an unauthenticated publish is rejected 401/403; a publish under a namespace the credential does not own is rejected; existing read endpoints are unaffected. - References:
pkg/module/registry/handler.go(publish),cmd/kscore-registry(task 9, landed — unauthenticated); companion of “Module signing: cosign keyless / Rekor transparency / encrypted cosign keyfile interop”.
Starlark hard heap-bytes cap
- Priority: v1.x
- What: Epic 14 task 11’s Starlark runtime bounds a module by an execution-step cap (
thread.SetMaxExecutionSteps), a wall-clock timeout (thread.Cancel), and Starlark’s intrinsic recursion / no-whilestrictness. §4.18 also lists “memory heap limits”; go.starlark.net has no public allocation/heap-bytes ceiling API, so v1.0 approximates it via the step+time bounds. Add a true per-execution heap-bytes cap once upstream exposes one (the long-proposedthread.SetMaxAllocs/allocation accounting) or via an out-of-process / cgroup memory bound for module execution. - Why deferred: the step + wall-clock bounds already prevent unbounded CPU and runaway loops (the practical DoS vectors); a precise heap ceiling needs runtime support that go.starlark.net does not currently provide, and re-implementing allocation accounting is out of v1.0 scope. The manifest already carries
limits.memory(parsed, task 1) so the contract is forward-compatible — only the enforcement is deferred. - Acceptance: a module exceeding
limits.memoryis terminated with a typed error before exhausting host memory; verified by a high-allocation test module. - References:
pkg/module/runtime/starlark(task 11, landed — step+time bounds);pkg/module/manifestLimits.Memory(task 1, parsed); go.starlark.netThread; companion of “WASM module runtime”.
Per-capability-call context propagation in the Starlark SDK
- Priority: v1.x
- What: Epic 14 task 12’s
modules/sdk/starlarkcapability builtins call their backends withcontext.Background()— Starlark builtins receive a*starlark.Thread, not acontext.Context, so the per-call ctx (deadline/cancellation) is not threaded into individual capability invocations (e.g. anhttp.getdoes not inherit a caller deadline beyond the module-wide bound). Add propagation of the module-execution context (and any per-call deadline) into each capability call. - Why deferred: the task-11 thread watchdog (
thread.Cancelon timeout/ctx-cancel) already aborts the entire Starlark execution mid-call, so a runaway or hung capability call is still bounded at the module level for v1.0; fine-grained per-call ctx only changes which call observes the cancellation first, not whether execution is bounded. Threading ctx requires either a thread-local convention (thread.SetLocal) set by the runtime beforeCallor a signature change, and is a clean follow-up rather than a v1.0 correctness gap. - Acceptance: a capability call (e.g.
http.get) observes the module execution context’s deadline/cancellation directly (verified by cancelling mid-call and asserting the backend received a cancelled ctx), with the module-level watchdog unchanged. - References:
modules/sdk/starlark/sdk.go(guardusescontext.Background());pkg/module/runtime/starlark(task-11 thread watchdog); Epic 14 task 12; companion of “Starlark hard heap-bytes cap”.
Module test framework: injectable record/replay http/exec/secrets test hosts
- Priority: v1.x
- What: Epic 14 task-15
pkg/module/testingwires os-backedfs+ a discardLoggerfor test runs but leaveshttp/exec/secretshosts nil (fail-closed + audited if a module both requested and uses them). Add injectable, deterministic record/replay hosts so a module’s unit tests can exercise itshttp/exec/secretscapability paths without real network/process/secret-store side effects. - Why deferred: network/process/secret side effects in unit tests are an anti-pattern; for v1.0 a module’s pure logic +
kv/log/fs-scope behaviour is unit-testable, and the nil-host fail-closed path is itself testable and demonstrates the security contract. Record/replay fixtures are an ergonomics follow-up, not a v1.0 correctness gap (Options.Hostsalready lets a caller inject fakes today — this is about a built-in fixture format). - Acceptance:
kscore-module testcan run a module whose tests callhttp.get/exec/secrets.readagainst a declared fixture (recorded responses), with the capability scoping + audit trail unchanged. - References:
pkg/module/testing/hosts.go(defaultHosts),pkg/module/testing/runner.go(Options.Hosts); Epic 14 task 15.
Module test framework: multi-file module tests via Starlark load()
- Priority: v1.x
- What: the task-15 runner uses the single-entrypoint module model — a
*_test.starreaches the module’s functions via a merged predeclared environment, with no Starlarkload()support. Add a module-awareload()so a multi-file module (helpers split across.starfiles) can be tested with the same import graph the runtime would use. - Why deferred: v1.0 modules are single-entrypoint (the manifest declares one
entrypoint); the merged-predeclared approach gives tests full access to module-level definitions for that model.load()requires a threadLoadresolver + cycle handling that only matters once multi-file modules are supported (itself post-v1.0). - Acceptance: a module split across multiple
.starfiles, loaded viaload(), is testable end-to-end throughkscore-module testwith deterministic, cycle-safe resolution. - References:
pkg/module/testing/runner.go(mergeDicts, entrypoint exec); Epic 14 task 15; companion of “Module test framework: injectable record/replay http/exec/secrets test hosts”.
TUI monitor (kscore-monitor)
- Priority: v1.x
- What: Bubble Tea single-pane-of-glass — 8 base views (dashboard/agents/events/state/policy/jobs/logs/metrics), 13 with enhancements (cluster/secrets/leases/schedules/runbooks/webhooks).
- Why deferred: v1.0 ships Grafana dashboards + CLI. TUI needs gRPC-multiplex client + NATS subscriber. Not blocking trial.
- Acceptance:
kscore-monitoropens, navigates between views, refreshes live. - References: PROJECT-DETAILS §4.16 (line 1123); Epic 17.
SPIRE-backed identity provider
- Priority: v1.x
- What: Swap embedded CA for external SPIRE server + agent socket.
- Why deferred: Embedded CA covers v1.0; SPIRE needs operational tooling.
- Acceptance:
IdentityProviderinterface backed by SPIRE socket; SVID rotation is automatic. - References: Epic 09 scope-out (line 24); PROJECT-DETAILS §4.10 (line 695).
K8s operator
- Priority: v1.x
- What: CRDs + reconciler for declarative Keystone Core management.
- Why deferred: K8s is one deployment target; v1.0 ships standalone-binary baseline.
- Acceptance:
Cluster,Agent,BlueprintCRDs reconcile against a live cluster. - References: Epic 00 deferred list (line 73).
Weighted endpoint load distribution + K8s endpoint discovery
- Priority: v1.x
- What:
cfg.NATS.Endpoints[].Weightactually distributes load; K8s service-name endpoint discovery. - Why deferred: v1.0 uses priority-only endpoint selection; weight is reserved in the schema. K8s discovery is part of the operator work.
- Acceptance: Load measurably distributes proportionally to weights;
nats.urls = ["k8s://service-name"]resolves through discovery. - References:
internal/config/nats.go:113;internal/nats/subject.go:135.
AWS decorrelated jitter for fleet-scale reconnect storms
- Priority: v1.x
- What: Replace symmetric exp-jitter (
reconnectDelayininternal/nats/backoff.go) with AWS decorrelated jitter —delay = min(max, random(base, prev_delay * 3)). Better herd-avoidance properties at >500-agent scale. - Why deferred: Symmetric jitter is adequate for v0.x trial fleets (≤500 agents per design). Decorrelated needs careful per-call state tracking and is harder to test deterministically.
- Acceptance:
reconnectDelay(or a sibling) implements decorrelated jitter; benchmark/sim shows tighter reconnect-time distribution at 1000+ agent scale; opt-in via a config knob (reconnectjitterstrategy: symmetric|decorrelated). - References: Epic 06 task 10
_(landed)_;internal/nats/backoff.go.
Telemetry gateway
- Priority: v1.x
- What:
kscore-telemetry-gateway— standalone collector for logs/metrics/traces over NATS subjects. - Why deferred: v1.0 emits OTel/Prom directly to operator-supplied backends.
- Acceptance: Gateway aggregates from N agents; forwards to Loki/Prom/Jaeger.
- References: PROJECT-DETAILS §4.16 (line 1152); Epic 17.
Output archival to object storage cold-tier
- Priority: v1.x
- What: Long-running batch results overflow to S3/GCS/Azure cold tier.
- Why deferred: v1.0 keeps results in PostgreSQL with size cap.
- Acceptance: Outputs > N MiB archive to operator-configured bucket;
kscorectl exec showfetches from cold tier. - References: Epic 07 scope-out (line 30).
Saga/checkpoint advanced features
- Priority: v1.x
- What: Checkpoint-resume (re-load a crashed saga and continue from the last completed step with the pre-step Data restored, persisting per-step transition rows); cross-state compensation graphs (compensation by dependency graph instead of strict reverse-linear); richer multi-failure aggregation reporting; gRPC
StateService.RollbackStateSaga(saga-driven rollback, distinct from today’s whole-run rollback CLI). The durablepkg/saga/log_sqliteLogitself is NO LONGER deferred — it landed in Epic 15 task 2 (v1.0 scope per PROJECT-DETAILS §4.17 “in-memory or SQLite log” / FEATURES “no checkpoint resume”); only the per-step checkpoint-resume protocol on top of it remains v1.x. - Why deferred: v0.1 ships the minimal scaffold per Epic 08 task 12 — forward-execute steps, on first error walk completed steps in reverse with aggregate-and-continue compensation semantics, in-memory and (Epic 15 task 2) durable SQLite log,
Runner.RunSaga(History)integration that re-applies the most recent priorStateRunRecordper decl. Resume-from-checkpoint is a distinct durability surface beyond a persistent log (atomic per-step transitions, recovery-on-startup hook, integration withkscore-serverlifecycle); cross-state compensation graphs reshape the algorithm; the v0.1 minimum is enough for the trial use case where a saga either finishes or is restartable from the top. - Acceptance: A 10-step saga survives a crash mid-step 5, resumes from checkpoint, completes the remainder; a cross-state compensation graph compensates step 3 before step 2 when the graph dictates;
kscorectl state rollback-saga <run-id>reconstructs the saga from history and walks the compensation graph. - References: PROJECT-DETAILS §4.17 (saga coordinator block, v1.x line 1197); Epic 08 task 12 (landed); Epic 15 task 2 (durable SQLite log landed);
pkg/saga/doc.gopackage overview.
Air-gap baseline
- Priority: v1.x
- What: Offline registry, bootstrap packages, upgrade archives, full security scanning suite, signed module bundles, signing ceremony.
- Why deferred: v1.0 assumes online package repos.
- Acceptance: Operator installs Keystone Core in a fully air-gapped network; module updates flow through offline registry.
- References: PROJECT-DETAILS §6.2 (line 1496).
Cloud workload identity (AWS IRSA, GCP WI, Azure MI)
- Priority: v1.x
- What: Cloud-native identity binding without static credentials.
- Why deferred: v1.0 ships embedded CA + JWT/PSK; cloud bindings need per-provider integration.
- Acceptance: Agent on EC2 with IRSA receives identity from instance metadata.
- References: Epic 09 scope-out (line 25).
gRPC-gateway annotation-driven REST + OpenAPI auto-gen
- Priority: v1.x
- What: REST + OpenAPI generated from proto annotations.
- Why deferred: v1.0 hand-codes both for control and simplicity (the v0 reset traced its complexity in part to grpc-gateway tooling friction).
- Acceptance: A new gRPC method automatically gets REST + OpenAPI without hand-edits.
- References: Epic 03 scope-out (line 30); PROJECT-DETAILS §4.5 (line 426).
kscore-agent service start|stop subcommands
- Priority: v1.x
- What: PROJECT-DETAILS §4.6 lists
service install|uninstall|start|stop|status. v1.0 shipsinstall|uninstall|statusonly. - Why now:
systemctl start kscore-agent/systemctl stop kscore-agentare universally known by Linux operators; wrapping them adds maintenance for zero ergonomic value. Picked up post-v1.0 because the Windows agent’s SCM integration needs the same command shape and makes the wrapper worthwhile. - Acceptance for unblock:
kscore-agent service start|stopexist and proxy to the platform service manager; documented as the cross-platform form. Additive — no change to existing subcommands. - References: Epic 06 task 9
_(landed)_; PROJECT-DETAILS §4.6 (line 473).
Bootstrap: demo mode only (TUI + non-interactive)
- Priority: v1.x
- What: Both
kscore-agent bootstrappaths (TUI wizard from task 7 and--non-interactiveflags from task 8) accept all three modes structurally butbootstrap.ValidateForV10rejects production / enterprise with a v1.x deferral message before the Engine reaches Validate. - Why now: Production mode needs TLS cert collection (gates on Identity/cert tooling); enterprise mode needs blueprint selection. Production-mode unblock lands with the SPIRE/cert-rotation cycle; enterprise/blueprint pieces (the wizard screens) trail into the blueprint-runtime cycle — see that entry. Lifting the gate is purely additive for existing demo-mode users.
- Acceptance for unblock: TUI screens for cert paths + cert-generation toggle (production); equivalent
--generate-certsCLI flag wired to the non-interactive path. Drop or no-opbootstrap.ValidateForV10for production. - References: Epic 06 tasks 7 + 8
_(landed)_;internal/agent/bootstrap/configure.go(searchValidateForV10);cmd/kscore-agent/main.go(searchbuildConfigurer).
Bootstrap CLI flags dropped from v1.0 surface
- Priority: v1.x
- What: The original Epic 06 task 8 spec listed
--postgres-*,--nats-*beyond--join/--join-token,--generate-certs, and--apply-blueprint. v1.0 ships without them. - Why now:
--postgres-*is server-only and belongs in the future unifiedkscore-bootstrapbinary (the agent doesn’t run a database). Extra--nats-*flags are unnecessary because v1.0 agents are external-mode only — embedded NATS is v2.x+.--generate-certsre-appears with the cert tooling;--apply-blueprinttrails to the blueprint runtime. All additive flag additions. - Acceptance for unblock: Per-flag — when its blocking work lands, add the flag with appropriate plumbing. The
--state-pathflag added in task 8 stays. - References: Epic 06 task 8
_(landed)_;cmd/kscore-agent/main.go(searchregisterBootstrapFlags).
Bootstrap auto-installs systemd unit (production mode)
- Priority: v1.x
- What: When demo-only mode-gate lifts, the bootstrap Engine’s Install phase should call
systemd.Installautomatically — operator runskscore-agent bootstraponce, gets both config and unit installed/enabled. - Why now: Downstream of production mode (above); rides the same SPIRE/cert-rotation cycle. Convenience chaining, additive — the explicit two-step flow still works.
- Acceptance for unblock: When
bootstrap.ValidateForV10lifts for production,bootstrap.NewDefaultInstaller(or a production-mode wrapper) chainssystemd.Installafter the YAML render. - References: Epic 06 task 9
_(landed)_;internal/agent/bootstrap/install.go;internal/agent/systemd/install.go.
Type=notify systemd integration (sd_notify)
- Priority: v1.x
- What: v1.0 unit uses
Type=exec.Type=notifywould let systemd track agent readiness viasd_notify("READY=1")calls — useful forWants=kscore-agent.serviceordering and reliable health checks. - Why now: Requires the daemon to call into
coreos/go-systemd/v22/daemon’sSdNotify, the dep explicitly skipped for v1.0. The telemetry-gateway work is the first real consumer of agent readiness signalling. New unit + daemon ship together, so no break for existing installs. - Acceptance for unblock: Daemon calls
SdNotify("READY=1")after NATS connect + initial heartbeat publishes; unit flips toType=notify;systemctl is-activereportsactivatinguntil ready. - References: Epic 06 task 9
_(landed)_;internal/agent/systemd/unit.go(searchType=exec).
Bootstrap wizard: storage backend + blueprint selection screens
- Priority: v1.x
- What: PROJECT-DETAILS §4.6 + Epic 06 task 7 originally envisioned the agent wizard collecting storage backend and applying blueprints. Both were dropped from the v1.0 agent surface — storage is server-only (the future unified
kscore-bootstrapbinary’s concern); blueprint apply gates on the blueprint runtime. - Why now: Blueprint apply needs the plugin/module system + the full blueprint catalogue, which is the blueprint-runtime cycle’s headline. Storage screens still wait on the unified binary (no committed release) and may slip further. Additive wizard screens — existing wizard flows unchanged.
- Acceptance for unblock: For blueprints — the blueprint runtime lands, then a “select blueprints to apply” screen feeds an installer-side blueprint apply step. For storage —
cmd/kscore-bootstrap(unified server+agent binary) gains the storage screens. - References: Epic 06 task 7
_(landed)_; PROJECT-DETAILS §4.6 (line 451).
Bootstrap: no rollback / transactional revert
- Priority: v1.x
- What: Bootstrap engine resumes from checkpoint but doesn’t revert side effects (config files, systemd units) on failure.
- Why now: Rollback semantics need install-step inversion + snapshot tracking. Slotted with the compliance/scan-scheduler cycle’s general hardening; new
--rollbackflag is additive. - Acceptance: Failed bootstrap re-runs cleanly; for true rollback, operator runs
kscore-agent bootstrap --rollback. - References: Epic 06 task 6;
internal/agent/bootstrap/doc.go:19.
Dedicated kscore system user auto-creation
- Priority: v1.x
- What: v1.0 systemd unit defaults to root;
--user/--groupflags let operators run as a dedicated user, but for theservice installpath the user must already exist (no auto-create). The server.deb/.rpmpostinst already creates thekscoresystem user — this entry covers extending that to theservice installflow. - Why now: User creation belongs in package-mgmt territory (rpm/deb post-install via
useradd --system), and packaging is part of the air-gap / supply-chain work. The default-user flip is handled inside the package post-install (not as a surprise to in-place tarball upgrades), so it stays non-breaking. - Acceptance for unblock: The
service installpath can default--user kscore --group kscore(the user the server package already creates) and update the rendered ReadWritePaths to match. - References: Epic 06 task 9
_(landed)_; the air-gap / supply-chain packaging work.
Auto-rotation of in-memory NATS creds
- Priority: v1.x
- What: Agent rotates NATS credentials without restart.
- Why now: Gates on the SPIRE provider (post-v1.0). v1.0 rotation = restart.
- Acceptance: Agent re-issues NATS creds on cert rotation event without dropping the connection.
- References: Epic 06 scope-out (line 33); PROJECT-DETAILS §4.6 (line 482).
v2.x+ — architectural post-v1.0
Adaptive sampling tied to error metrics
- Priority: v2.x+
- What: OTel
Samplerimplementation that rebalances sample rate on the observed error rate fromkscore_*_failed_totalmetrics. PROJECT-DETAILS §4.16 line 1126 namesadaptive (rebalances on observed error rate)as part of the v1.0 sampler enum, but Epic 17 Scope-out (line 82) defers it to v2.0 with the rationale “refinement”. Epic 17 task 4 ships the other four samplers (always_on/off,probabilistic,parent_based,rate_limiting);internal/config/TracingConfig.Validaterejectssampler=adaptivewith a pointer to this entry. - Why deferred: needs a stable kscore metric set + an error-rate-feedback path that doesn’t exist in v1.0. The other four samplers cover every v1.0 use case (probabilistic at 0.1 is the documented default).
- Acceptance:
tracing.sampler: adaptivevalidates and constructs a sampler that increases sample rate on error spikes (>1% per-route 5xx) and decays back to baseline; integration test with a stub metric source. - References: Epic 17 Scope-out (line 82); PROJECT-DETAILS §4.16 line 1126;
internal/config/tracing.goValidateadaptive branch.
Embedded NATS / hybrid mode / leaf node / endpoint advertiser / supercluster / WebSocket
- Priority: v2.x+
- What: Agents run embedded nats-servers; cluster forms via leaf-node mesh; reverse-leaf NAT traversal for agents behind firewalls; WebSocket transport for browser-side ops.
- Why deferred: v1.0 ships agent-as-NATS-client only; hybrid topology + WebSocket multiply test surface.
- Acceptance: Agent runs embedded NATS; reverse-leaf publishes its endpoint; supercluster gateway federates two clusters.
- References: Epic 05 scope-out (lines 38-42); Epic 06 scope-out (lines 27-28).
Config: no per-endpoint TLS overrides
- Priority: v2.x+
- What:
cfg.NATS.Endpoints[]use the cluster-wide TLS config; per-endpoint overrides are reserved schema fields. - Why now: v1.0 has one TLS strategy per cluster; mixed-TLS topologies become relevant alongside the v2.x+ multi-region work, but the override fields already exist and wiring them is additive (existing configs keep working), so it slots into later platform-polish rather than waiting for that v2.x+ work.
- References:
internal/config/nats.go:126.
Cluster-wide HMAC secret (vs per-agent)
- Priority: v2.x+
- What: All agents share one HMAC secret in v1.0. Per-agent keys derived from the bootstrap exchange replace it.
- Why now: Breaking change — it changes the agent↔server authentication model and needs a key-distribution mechanism still being designed; lands with the v2.x+ auth/security infra changes (cloud KMS, federation).
- Acceptance for unblock: Bootstrap exchange establishes a per-agent key; server authenticates inbound by agent identity; cluster-wide secret removed (or relegated to a legacy compatibility window decided at design time).
- References: Epic 06 task 6
_(landed)_;internal/agent/security.go:71;internal/config/security.go:15.
Windows agent (native service)
- Priority: v2.x+
- What: kscore-agent on Windows with
service install/uninstall/start/stop/statussubcommands and SCM integration. - Why deferred: v1.0 platform target = Linux amd64+arm64. Windows needs separate stdlib (
win_feature,win_firewall, etc.) and user-switching —internal/agent/exec_user_windows.go:15is a stub returningnot supported in v1.0. Re-bucketed from v1.x to v2.x+ on 2026-05-31 when the v1.0 platform scope was clarified to be linux-only end-to-end (server + agent + bootstrap), making non-linux agent work explicitly post-v1.0 architectural change rather than a v1.x feature add. - Acceptance: kscore-agent runs as a Windows service; SIGTERM equivalent triggers clean shutdown; user switching works via
LogonUser/CreateProcessAsUser. - References: PROJECT-DETAILS §4.6 (line 487), §4.8 (line 619);
internal/agent/exec_user_windows.go.
macOS agent
- Priority: v2.x+
- What: kscore-agent on macOS with launchd integration.
- Why deferred: Linux is v1.0 target; macOS adds
launchdstdlib + Keychain integration. Re-bucketed from v1.x to v2.x+ on 2026-05-31 alongside the Windows agent entry. - Acceptance: kscore-agent runs under launchd; agent identity stored in Keychain.
- References: PROJECT-DETAILS §4.6 (line 487), §4.8 (line 619).