My linux config and rc files https://git.sommerfeld.dev/dotfiles/atom/etc/mkinitcpio.d/linux-hardened.preset?h=master 2026-05-29T10:18:16Z 2026-05-29T10:18:16Z sommerfeld sommerfeld@sommerfeld.dev 2026-05-29T10:18:16Z urn:sha1:a6f1e9026e5d19b50c7c522e30e6f216fc9f8180 Promotes linux-hardened to the sole primary kernel and replaces linux with linux-lts as the safety-net fallback. Rationale: - linux and linux-hardened track the same upstream major version and ship within days of each other, so 'linux' was a poor fallback for the regression class that historically takes out the hardened kernel on this hardware (e.g. checkpoint 026 wake-from-suspend panic). linux-lts lags by weeks/months and is almost always known-good when hardened breaks. - Drop etc/mkinitcpio.d/linux.preset, add linux-lts.preset. Hardened preset header + bootstrap.sh efibootmgr instructions updated accordingly (hardened registered first so it's the default; lts registered as the on-demand fallback). - Also add mkinitcpio-firmware (AUR) to silence the spurious 'missing firmware' warnings during initramfs builds. Manual host-side steps after deploy: paru -S linux-lts linux-lts-headers mkinitcpio-firmware sudo pacman -Rsn linux # or via 'just pkg-apply' undeclared flow sudo rm -f /etc/mkinitcpio.d/linux.preset # chezmoi-deployed, not pkg-owned sudo mkinitcpio -P sudo efibootmgr # add the Arch LTS entries, drop the stock linux ones Note: meta/nvidia.txt still lists 'linux-headers' for nvidia-dkms. That's a per-host concern; flagged for follow-up if any nvidia host moves to the linux-lts world. 2026-05-29T10:18:15Z sommerfeld sommerfeld@sommerfeld.dev 2026-05-29T10:18:15Z urn:sha1:6e0c5c33438e5e898bd075c33a45b3abf9d1b26b Confirmed root cause: this hardware's S3 (deep) firmware path triggers a fatal wake-from-suspend hang only on linux-hardened. INIT_ON_FREE + slab hardening + tighter locking turn a latent driver race that stock linux gets away with into an unrecoverable panic so early the journal isn't even flushed. mem_sleep_default=s2idle bypasses the BIOS S3 path entirely (s0ix is a pure-kernel low-power state) and suspends/resumes reliably under hardened. This is a widespread Lenovo S3 firmware issue across post-2018 ThinkPads (see Ubuntu T560, X1C9/10/11 reports). Lenovo themselves moved newer firmwares to s2idle-only. Not a linux-hardened bug per se; just hardened being a strict enough kernel to make the bug fatal. Keep: * mem_sleep_default=s2idle in etc/kernel/cmdline-linux-hardened.tmpl (only the hardened UKI; stock linux keeps unchanged shared cmdline) Revert (all the diagnostic / speculative scaffolding from the last few commits): * MODULES=(intel_lpss_pci) → MODULES=() — Arch wiki touchpad fix was not the cause here * nmi_watchdog=panic softlockup_panic=1 panic=10 — only needed to auto-reboot during diagnosis * no_console_suspend — diagnostic-only * etc/systemd/logind.conf.d/20-no-suspend.conf — masking workaround * sleep-target masking block in run_onchange_after_deploy-etc.sh.tmpl, replaced with a one-shot cleanup that removes any leftover /dev/null symlinks from systems that ran the previous version * systemd-pstore.service from systemd-units/system.txt — added only to catch the diagnostic panic * diagnose-suspend.sh helper (and its .gitignore/.chezmoiignore entries) * sway suspend → lock-session keybind workaround * power-menu.sh Suspend entry restoration * KEYBINDS.md docs 2026-05-29T10:18:14Z sommerfeld sommerfeld@sommerfeld.dev 2026-05-29T10:18:14Z urn:sha1:be5f8a2e6be3af4963399bb7f994f76d76b3a239 Symptoms (Intel CPU + linux-hardened + blinking caps lock + hard hang on resume from S3) are a direct match for the Arch wiki entry: https://wiki.archlinux.org/title/Power_management/Suspend_and_hibernate#Touchpad_causes_a_kernel_panic_on_resume https://bbs.archlinux.org/viewtopic.php?id=231881 When intel_lpss_pci is loaded late (via udev after userspace is up), the touchpad/I2C controller it parents can be torn down by suspend before the module's resume callback is registered, leading to a NULL-deref panic during resume. The kernel never makes it far enough to flush logs — which matches our 'PM: suspend entry (deep)' being the last journal line. Fix: load intel_lpss_pci from the initramfs so it's available before the suspend/resume code path runs. Why this only bites linux-hardened: the hardening config enables INIT_ON_FREE, slab freelist hardening, page poisoning, and stricter pointer validation, which turn what's a silent UAF on stock linux into an immediate panic on hardened. Stock 'just works' by accident. Also drop the speculative init_on_free=0 from the hardened cmdline now that we have a targeted hypothesis. Keep nmi_watchdog=panic + softlockup_panic=1 + panic=10 as belt-and-braces: if this fix is wrong, the next hang will auto-reboot with a usable panic log in 'journalctl -b -1 -k' instead of needing the power button again. 2026-05-29T10:18:14Z sommerfeld sommerfeld@sommerfeld.dev 2026-05-29T10:18:14Z urn:sha1:e2a7a2fdb9ba66e777ec1a8c0d3c9301cc21bdab Split the hardened UKI cmdline off the shared etc/kernel/cmdline.tmpl so we can carry workarounds without poking the stock linux build. Daily-driving linux-hardened on this hardware has reliably hung on resume from S3: black screen, blinking caps-lock + power LED, only the power button helps. The kernel journal stops at 'PM: suspend entry (deep)' with nothing after, so the freeze is below the level where logs can flush — characteristic of a hard hang inside a device driver's suspend/resume callback rather than a userspace bug. linux-hardened defaults init_on_free=1, which zeroes pages on free. On Intel + iwlwifi/i915/nvme stacks this routinely surfaces latent UAFs as suspend hangs that are invisible on stock linux. Drop that knob to 0 for the hardened cmdline as the working hypothesis. Add nmi_watchdog=panic, softlockup_panic=1, panic=10 so if the next attempt still wedges, a stuck CPU self-panics and auto-reboots within ~10s, giving us a 'journalctl -b -1 -k' trace to look at instead of having to force-power-off blindly. Stock linux is untouched. 2026-05-29T10:18:12Z sommerfeld sommerfeld@sommerfeld.dev 2026-05-29T10:18:12Z urn:sha1:03898ea972794117607791260e8715460f5c9469 Keeping the fallback after all — leaves the door open to dropping the stock 'linux' package entirely once linux-hardened is proven as a daily driver. Without hardened-fallback, that future single-kernel config would have zero autodetect recovery path. This reverts commit c0c9183. 2026-05-29T10:18:12Z sommerfeld sommerfeld@sommerfeld.dev 2026-05-29T10:18:12Z urn:sha1:0f8df9f0a7b6032861606c20712f750d7a1b5484 Stock linux-fallback already covers the 'autodetect missed a module' recovery scenario, regardless of which kernel you tried to boot. hardened being opt-in means a hardened-default failure naturally falls back to stock — no need for hardened-fallback as a second safety net. Saves ESP space and mkinitcpio regen time on each linux-hardened update. 2026-05-29T10:18:12Z sommerfeld sommerfeld@sommerfeld.dev 2026-05-29T10:18:12Z urn:sha1:8ebe3f106e53dc4032428a2e3435c4feea969087 Installs linux-hardened + linux-hardened-headers alongside the stock linux kernel. Stock kernel remains the default; linux-hardened is opt-in via efibootmgr --bootnext after the EFI entry is registered (one-time host-side step, documented in the preset). After first 'just pkg-apply', mkinitcpio auto-builds /boot/EFI/Linux/arch-linux-hardened.efi from the new preset (sharing etc/kernel/cmdline.tmpl with the stock UKI — same LUKS root, no kernel-specific cmdline knobs). Host-side EFI entry registration: sudo efibootmgr --create --disk /dev/nvme0n1 --part 1 \ --label 'Arch Hardened' --loader '\\EFI\\Linux\\arch-linux-hardened.efi' Roll back any time by removing both packages and the preset file; the stock kernel and its UKI are untouched.