This ticket addresses my message in this email thread:
T40169e012e2a14db-Md6af70f253bdd34a76609032
We need to see if we can Do Better on machine shutdown with hardware VMs (BHYVE primarily, but if KVM isn’t already doing something better, it should as well). The thread also mentions a known-and-deployed workaround:
triton-cn-shutdown
which either could be imported whole-cloth, or have its underlying principles and methods integrated better into brand-specific paths of zones shutdown paths.
Dan McDonald commented on 2025-08-18T21:35:00.197-0400:
As of today (Mon Aug 18) I’ve made brand-specific modifications to svc-zones that for LX just run zoneadm -z <LX-UUID> shutdown and for KVM and BHYVE peform operations that will send an ACPI power-off to the guest (SIGTERM for BHYVE and a qmp message for KVM). It does NOT currently solve invocations of halt , poweroff, or reboot, but it should help shutdown by having svcadm disable -t zones do the right thing on a per-brand basis.
I’m not sure if it’s the full solution, never mind the correct one, but I’d be interested in folks trying out what I have.
Dan McDonald commented on 2025-08-19T15:12:49.365-0400:
After discussions with @Carlos Neira I decided to modify KVM (illumos-kvm-cmd) to accept SIGTERM as an ACPI power off the same way bhyve does. This simplifies the changes to svc-zones substantially without sacrificing functionality.
Dan McDonald commented on 2025-08-19T16:09:20.365-0400:
Testing notes:
Timed before/after of svcadm disable -t -s zones. Slight gain (92sec vs. 93sec) in post-this-change using the heavier-weight JS method of KVM shutting down on a medium-sized CN:
UUID TYPE RAM STATE ALIAS
d3872d67-abba-4616-9450-baa6ef32a523 OS 128 running nat-e370529f-7559-4936-87ec-da7eda882f5b
f5cbba59-3a54-41ab-ae26-1936de3a2753 OS 128 running nat-1669c9b7-6478-4dbd-bf29-c97d0e4a26f5
fa7130e6-9907-4851-8980-72d2f4ed9588 OS 128 running nat-0e061e43-d350-48fe-b102-d04702133519
0a934a2d-6ad8-e60c-d06b-b1d995f6c675 OS 256 running neil
0e471e14-8d49-e236-ed53-e73a16507ee1 OS 256 running edward
24be8bd2-9389-ee4d-e1df-ca3a5eb08adf OS 256 running mike
2ea8f2fb-f64f-c95e-a666-98628004fde1 OS 256 running eddie
35000f72-eb50-61f7-b2d0-9953e20a5831 OS 256 running underlay-ipf-test2
37028d33-c79b-60a8-9005-fc66e800240f OS 256 running public-ipf-test2
4ef62b2f-22b2-4ad1-bb48-afcb982a8741 OS 256 running vxlnat-test
5dca0c7d-3565-e5a3-e6a6-b2ebe8c3c467 OS 256 running michael
64ed095b-d0ea-616a-adae-beec737cfe45 OS 256 running jon
7940335a-a403-e6f3-af1e-ac33c4d63654 OS 256 running admin-ipf-test2
7ab7231e-a31f-40dd-f9b5-ad6357aa6729 OS 256 running public-ipf-test
7fe42e9a-934a-c007-b477-ef1b616ce3fc OS 256 running alex
8599a1cd-6f37-670b-faa0-9ac1ab4d7d7f OS 256 running natworthy
9d972a10-fd3d-ea16-ec20-914d130e4080 OS 256 running chris
bae7721f-4894-cd24-f21e-cf385b3c1328 OS 256 running admin-ipf-test
e24c305a-6903-c9fa-8cfb-8ef45236101a OS 256 running underlay-ipf-test
82d4edf2-98a0-4cb0-9efb-576f410c43d0 LX 512 running ubuntu18-pre
93c0ec62-af01-4070-aacd-9e42a0410000 LX 512 running void-lx-test
ad51bb39-da2d-c466-e726-f66e6bed7747 LX 512 running alpine
d5b509f3-fb2a-4db3-9bb3-72c3ca65c996 LX 512 running ubuntu16
dd1c1805-c95e-4612-833f-98ca6dcbb573 LX 512 running counterparts
f02da1ad-841c-c9c9-f587-f9f2f5819cac LX 512 running alpine-38
ff0d7c86-7f76-4377-b680-70ee28623389 LX 512 running alpine-latest
61a7a718-3927-e254-8d0a-fade03a8c738 OS 1024 running cfwlogt1
8b212382-c799-6a71-c2a7-9e4c14f3f159 OS 1024 running cfwlogt0
a34f6c4a-4f9e-eede-dff7-c0e7d1e670de OS 1024 running test2
b011bcd9-dc09-40b6-8784-00c518d995f4 KVM 1024 running drama
b0dc6415-a524-e07d-87cc-ad0353253b92 LX 1024 running mike-mongodb-1
aebda76e-4937-4352-98b8-c80db1ecf13f KVM 2048 running presto
9a379e13-f759-4b31-81e9-5379d7b2f30c BHYV 4096 running ubuntu-bhyve
e309d3a6-ca31-4ae7-8104-dfd7c1385478 OS 4096 running ipxe-build
d4e73847-52fa-426b-ac29-d290142215e7 OS 16384 running smartos-build-2
e8518b9c-8cf9-49df-814f-3ba8d9977e0f OS 16384 running smartos-build-2024Interactive before/after testing of pkill -TERM -z KVM-UUID qemu on a FreeBSD guest demonstrated post-this change SIGTERM will give the guest an ACPI power off.
vmadm console monitoring during shutdown -y -s6 -g0demonstrated a clean poweroff on both a BHYVE guest and a KVM guest.
Some LX zones (the alpine ones and ubuntu18-pre ) seem to be a bit resistant to the aforementioned shutdown run but they hit the takes-too-long-path in the end.
Dan McDonald commented on 2025-08-25T09:21:02.562-0400:
To further improve things, illumos-kvm-cmd now handles SIGTERM identically to bhyve. I will re-run the prior shutdown test.
Dan McDonald commented on 2025-08-25T09:22:31.319-0400:
Additionally, I have this from the person from the email thread:
That was amazing! Before I knew it all my zones (I mean all of them) were stopped. I monitored my OpenBSD BHYVE zones and they shut down perfectly. Aside from the Synapse one. I gotta figure out what that one is doing. Not SmartOS's fault. It did get the signal and started shutting down though, I say that is a success on the hypervisor side. This is an amazing improvement that is going to be very welcome.On Aug 20, 2025, at 9:01 PM, Dan McDonald <danmcd@edgecast.io> wrote:
>
> How many zones are you running?
About 15 zones. My box isn’t all that powerfulDan McDonald commented on 2025-08-25T23:56:44.750-0400 (edited 2025-08-25T23:57:02.419-0400):
Re-timed shutdown with the KVM accepting SIGTERM, and a less-forky way of getting zoneadm list -p output into shell variables, and shaved the shutdown time above to 91 seconds instead.
Dan McDonald commented on 2025-08-26T00:13:58.242-0400:
One last before/after test. I’m going to show vmadm console outputs on ubuntu-22 running on bhyve and FreeBSD running on KVM before-and-after this fix handling whatever shutdown -y -g0 -i6 does.
Oh
Before (ubuntu-22 on bhyve):
kebe(~)[0]% ssh root@nuc
(root@nuc) Password:
SmartOS (build: 20250821T002042Z)
[root@nuc ~]# vmadm list
UUID TYPE RAM STATE ALIAS
833586e6-8b37-4e24-b496-06dcf172a74f LX 256 running lx-test
170e57f0-4ffe-4fad-af11-c235d04ee16d BHYV 8192 running ubuntu-22
7d5f8a32-ed89-4d90-837a-ffb26428e8f1 KVM 8192 running freebsd
c5fc4aa5-3fac-455f-a95e-b685be69a78f OS 16384 running nuc-build-2024
d00f8241-409b-4b1c-8de6-546d4c16410f OS 16384 running nuc-build
[root@nuc ~]# vmadm console 170e57f0-4ffe-4fad-af11-c235d04ee16d
[Connected to zone '170e57f0-4ffe-4fad-af11-c235d04ee16d' console]
ubuntu login: Broadcast Message from root (pts/2) on nuc Tue Aug 26 04:00:02...
THE SYSTEM nuc IS BEING SHUT DOWN NOW ! ! !
Log off now or risk your files being damaged
[NOTICE: Zone halted]
Connection to nuc closed by remote host.
Connection to nuc closed.
kebe(~)[255]% Before (FreeBSD on KVM):
kebe(~)[0]% ssh root@nuc
(root@nuc) Password:
SmartOS (build: 20250821T002042Z)
[root@nuc ~]# vmadm console 7d5f8a32-ed89-4d90-837a-ffb26428e8f1
FreeBSD/amd64 (freebsd) (cuau0)
login: Broadcast Message from root (pts/2) on nuc Tue Aug 26 04:00:02...
THE SYSTEM nuc IS BEING SHUT DOWN NOW ! ! !
Log off now or risk your files being damaged
[root@nuc ~]# Connection to nuc closed by remote host.
Connection to nuc closed.
kebe(~)[255]% After (ubuntu-22 on bhyve):
kebe(~)[255]% ssh root@nuc
(root@nuc) Password:
SmartOS (build: 20250825T190650Z)
[root@nuc ~]# vmadm console 170e57f0-4ffe-4fad-af11-c235d04ee16d
[Connected to zone '170e57f0-4ffe-4fad-af11-c235d04ee16d' console]
ubuntu login: Broadcast Message from root (pts/2) on nuc Tue Aug 26 04:12:31...
THE SYSTEM nuc IS BEING SHUT DOWN NOW ! ! !
Log off now or risk your files being damaged
[ OK ] Removed slice system-modprobe.slice - Slice /system/modprobe.
[ OK ] Stopped target cloud-init.target - Cloud-init target.
[ OK ] Stopped target graphical.target - Graphical Interface.
[ OK ] Stopped target nss-lookup.target - Host and Network Name Lookups.
[ OK ] Stopped target rpc_pipefs.target.
[ OK ] Stopped target rpcbind.target - RPC Port Mapper.
[ OK ] Stopped target timers.target - Timer Units.
[ OK ] Stopped apt-daily-upgrade.timer - …y apt upgrade and clean activities.
[ OK ] Stopped apt-daily.timer - Daily apt download activities.
[ OK ] Stopped dpkg-db-backup.timer - Daily dpkg database backup timer.
[ OK ] Stopped e2scrub_all.timer - Period…Metadata Check for All Filesystems.
[ OK ] Stopped fstrim.timer - Discard unused filesystem blocks once a week.
[ OK ] Stopped logrotate.timer - Daily rotation of log files.
[ OK ] Stopped man-db.timer - Daily man-db regeneration.
[ OK ] Stopped motd-news.timer - Message of the Day.
[ OK ] Stopped systemd-tmpfiles-clean.tim…y Cleanup of Temporary Directories.
[ OK ] Stopped update-notifier-download.t…hat failed at package install time.
[ OK ] Stopped update-notifier-motd.timer… a new version of Ubuntu available.
[ OK ] Stopped target time-set.target - System Time Set.
[ OK ] Closed cloud-init-hotplugd.socket - cloud-init hotplug hook socket.
[ OK ] Closed lvm2-lvmpolld.socket - LVM2 poll daemon socket.
[ OK ] Closed systemd-rfkill.socket - Loa…ll Switch Status /dev/rfkill Watch.
Unmounting run-rpc_pipefs.mount - RPC Pipe File System...
Stopping blk-availability.service - Availability of block devices...
[ OK ] Stopped cloud-final.service - Cloud-init: Final Stage.
[ OK ] Stopped target multi-user.target - Multi-User System.
[ OK ] Stopped target getty.target - Login Prompts.
Stopping apport.service - automatic crash report generation...
Stopping autofs.service - Automounts filesystems on demand...
[ OK ] Stopped cloud-config.service - Cloud-init: Config Stage.
[ OK ] Stopped target cloud-config.target - Cloud-config availability.
Stopping cron.service - Regular ba…ground program processing daemon...
Stopping finalrd.service - Create …time dir for shutdown pivot root...
Stopping getty@tty1.service - Getty on tty1...
Stopping rsyslog.service - System Logging Service...
Stopping serial-getty@ttyS0.service - Serial Getty on ttyS0...
Stopping systemd-logind.service - User Login Management...
Stopping systemd-random-seed.service - Load/Save OS Random Seed...
[ OK ] Stopped triton-guest.service - Triton HVM Guest Tools.
[ OK ] Stopped cron.service - Regular background program processing daemon.
[ OK ] Stopped serial-getty@ttyS0.service - Serial Getty on ttyS0.
[ OK ] Stopped rsyslog.service - System Logging Service.
[ OK ] Stopped getty@tty1.service - Getty on tty1.
[ OK ] Stopped systemd-logind.service - User Login Management.
[ OK ] Unmounted run-rpc_pipefs.mount - RPC Pipe File System.
[ OK ] Stopped blk-availability.service - Availability of block devices.
[ OK ] Removed slice system-getty.slice - Slice /system/getty.
[ OK ] Removed slice system-serial\x2dget…slice - Slice /system/serial-getty.
Stopping systemd-user-sessions.service - Permit User Sessions...
[ OK ] Stopped systemd-random-seed.service - Load/Save OS Random Seed.
[ OK ] Stopped systemd-user-sessions.service - Permit User Sessions.
[ OK ] Stopped apport.service - automatic crash report generation.
[ OK ] Stopped autofs.service - Automounts filesystems on demand.
[ OK ] Stopped target basic.target - Basic System.
[ OK ] Stopped target network-online.target - Network is Online.
[ OK ] Stopped target network.target - Network.
[ OK ] Stopped target paths.target - Path Units.
[ OK ] Stopped target remote-fs.target - Remote File Systems.
[ OK ] Stopped target remote-fs-pre.targe…reparation for Remote File Systems.
[ OK ] Stopped target nfs-client.target - NFS client services.
[ OK ] Stopped target slices.target - Slice Units.
[ OK ] Removed slice user.slice - User and Session Slice.
[ OK ] Stopped target sockets.target - Socket Units.
[ OK ] Closed iscsid.socket - Open-iSCSI iscsid Socket.
[ OK ] Closed ssh.socket - OpenBSD Secure Shell server socket.
[ OK ] Closed syslog.socket - Syslog Socket.
[ OK ] Closed uuidd.socket - UUID daemon activation socket.
[ OK ] Stopped target sysinit.target - System Initialization.
[ OK ] Stopped target cryptsetup.target - Local Encrypted Volumes.
[ OK ] Stopped systemd-ask-password-conso…equests to Console Directory Watch.
[ OK ] Stopped systemd-ask-password-wall.…d Requests to Wall Directory Watch.
[ OK ] Stopped target integritysetup.targ… Local Integrity Protected Volumes.
[ OK ] Stopped target swap.target - Swaps.
[ OK ] Stopped target veritysetup.target - Local Verity Protected Volumes.
[ OK ] Stopped cloud-init.service - Cloud-init: Network Stage.
Stopping systemd-binfmt.service - Set Up Additional Binary Formats...
[ OK ] Stopped systemd-networkd-wait-onli… Wait for Network to be Configured.
Stopping systemd-networkd.service - Network Configuration...
Stopping systemd-resolved.service - Network Name Resolution...
Stopping systemd-timesyncd.service - Network Time Synchronization...
Stopping systemd-update-utmp.servi…ord System Boot/Shutdown in UTMP...
[ OK ] Stopped systemd-binfmt.service - Set Up Additional Binary Formats.
[ OK ] Stopped systemd-timesyncd.service - Network Time Synchronization.
[ OK ] Stopped systemd-resolved.service - Network Name Resolution.
[ OK ] Stopped systemd-networkd.service - Network Configuration.
[ OK ] Unset automount proc-sys-fs-binfmt…ormats File System Automount Point.
[ OK ] Stopped target network-pre.target - Preparation for Network.
[ OK ] Closed systemd-networkd.socket - Network Service Netlink Socket.
[ OK ] Stopped cloud-init-local.service -…ud-init: Local Stage (pre-network).
[ OK ] Stopped systemd-sysctl.service - Apply Kernel Variables.
[ OK ] Stopped systemd-modules-load.service - Load Kernel Modules.
[ OK ] Stopped systemd-update-utmp.servic…ecord System Boot/Shutdown in UTMP.
[ OK ] Stopped systemd-tmpfiles-setup.ser…ate Volatile Files and Directories.
[ OK ] Stopped finalrd.service - Create f…untime dir for shutdown pivot root.
[ OK ] Stopped target local-fs.target - Local File Systems.
Unmounting boot-efi.mount - /boot/efi...
[ OK ] Unmounted boot-efi.mount - /boot/efi.
Unmounting boot.mount - /boot...
[ OK ] Stopped systemd-fsck@dev-disk-by\x…eck on /dev/disk/by-uuid/7571-A262.
[ OK ] Unmounted boot.mount - /boot.
[ OK ] Reached target umount.target - Unmount All Filesystems.
[ OK ] Stopped systemd-fsck@dev-disk-by\x…c4ca57-27cc-4d54-9b4d-a576da646758.
[ OK ] Removed slice system-systemd\x2dfs…slice - Slice /system/systemd-fsck.
[ OK ] Stopped target local-fs-pre.target…Preparation for Local File Systems.
Stopping lvm2-monitor.service - Mo…ing dmeventd or progress polling...
Stopping multipathd.service - Devi…pper Multipath Device Controller...
[ OK ] Stopped systemd-tmpfiles-setup-dev…Create Static Device Nodes in /dev.
[ OK ] Stopped systemd-tmpfiles-setup-dev…ic Device Nodes in /dev gracefully.
[ OK ] Stopped multipathd.service - Devic…Mapper Multipath Device Controller.
[ OK ] Stopped systemd-remount-fs.service…mount Root and Kernel File Systems.
[ OK ] Stopped lvm2-monitor.service - Mon…using dmeventd or progress polling.
[ OK ] Reached target shutdown.target - System Shutdown.
[ OK ] Reached target final.target - Late Shutdown Services.
[ OK ] Finished systemd-poweroff.service - System Power Off.
[ OK ] Reached target poweroff.target - System Power Off.
[ 250.137034] reboot: Power down
[NOTICE: Zone halted]
Connection to nuc closed by remote host.
Connection to nuc closed.
kebe(~)[255]% After (FreeBSD on KVM):
kebe(~)[255]% ssh root@nuc
(root@nuc) Password:
SmartOS (build: 20250825T190650Z)
[root@nuc ~]# vmadm console 7d5f8a32-ed89-4d90-837a-ffb26428e8f1
FreeBSD/amd64 (freebsd) (ttyu0)
login: Broadcast Message from root (pts/2) on nuc Tue Aug 26 04:12:31...
THE SYSTEM nuc IS BEING SHUT DOWN NOW ! ! !
Log off now or risk your files being damaged
Stopping cron.
Waiting for PIDS: 653.
Stopping sshd.
Waiting for PIDS: 643.
Stopping ntpd.
Waiting for PIDS: 614.
Stopping devd.
Waiting for PIDS: 347.
Writing entropy file:.
Writing early boot entropy file:.
.
Terminated
Aug 26 04:12:32 freebsd syslogd: exiting on signal 15
Waiting (max 60 seconds) for system process `vnlru' to stop... done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining... 0 done
Waiting (max 60 seconds) for system thread `bufdaemon' to stop... done
Waiting (max 60 seconds) for system thread `bufspacedaemon-0' to stop... done
Waiting (max 60 seconds) for system thread `bufspacedaemon-1' to stop... done
Waiting (max 60 seconds) for system thread `bufspacedaemon-2' to stop... done
Waiting (max 60 seconds) for system thread `bufspacedaemon-3' to stop... done
All buffers synced.
Swap device [file] removed.
Uptime: 3m55s
acpi0: Powering system off
[root@nuc ~]# Connection to nuc closed by remote host.
Connection to nuc closed.
kebe(~)[255]%