OS-6620: bhyve reboot should reuse existing process


Issue Type:Improvement
Priority:5 - Low
Created at:2018-02-14T20:21:02.814Z
Updated at:2018-08-17T15:15:37.364Z


Created by:Jerry Jelinek [X]
Reported by:Jerry Jelinek [X]
Assigned to:Hans Rosenfeld [X]


Fixed: A fix for this issue is checked into the tree and tested.
(Resolution Date: 2018-08-17T15:15:37.350Z)

Fix Versions

2018-08-30 Zolom Swamp (Release Date: 2018-08-30)

Related Issues

Related Links




Currently when a reboot is issued from within a bhyve VM we have to tear down the zone and reboot the entire zone. It would be better if we could re-use the init process (zhyve) we already have. Normally this could be done in the restart_init function which can be called when the zone's init exits, but for zhyve we currently get an EFAULT out of exec_init when it's called by restart_init. This can only happen when we try the copyout to construct the stack. If we can fix this we could re-use the existing process and hang on to the memory we've already allocated for the VM.


Comment by Jerry Jelinek [X]
Created at 2018-02-14T21:05:13.279Z

Marking as low priority since this is not a blocker for releasing bhyve.

Comment by Mike Gerdts [X]
Created at 2018-02-15T22:48:20.855Z

I don't think we need to necessarily reuse the same process - reusing the zone would be sufficient. The goal is to avoid tearing down the guest memory so as to avoid the long teardown and setup time, as well as the risk that the memory isn't available due to someone else allocating memory.

Comment by John Levon [X]
Created at 2018-03-01T14:22:16.623Z

It's perhaps worth pointing out that this means we wouldn't pick up any new zonecfg changes across this kind of reboot. Not sure if that'll be a problem?

Comment by Mike Gerdts [X]
Created at 2018-03-05T22:50:49.631Z

I was thinking that we probably need two forms of reboot - hard and soft for lack of better terms.  A hard reboot will do a full teardown and fresh setup of the zone.  This can only be initiated from the global zone (or tooling that acts in the global zone).  That's ok, because you have to be in the global zone to change the config too.  Any reboot that is initiated in the guest is a soft reboot.

This implies that a complete implementation may also need to have new flags on zoneadm reboot and zoneadm shutdown -r to distinguish between hard and soft reboots.

Comment by Jerry Jelinek [X]
Created at 2018-03-05T22:57:09.611Z
Updated at 2018-03-05T22:58:05.152Z

I was thinking along similar lines, where a reboot initiated from within the VM kept the same process, but an external reboot behaved as we do now. I don't really see a need for additional zoneadm reboot options at this time, since there is no infrastructure to leverage that.

Comment by Patrick Mooney [X]
Created at 2018-05-03T19:10:58.746Z

I believe this was addressed in OS-6612?

Comment by Jerry Jelinek [X]
Created at 2018-05-03T19:28:28.049Z

No, OS-6612 just allows the zone to be rebooted from within the VM, but we currently have to exit the old bhyve process and start a new one from scratch.

Comment by Patrick Mooney [X]
Created at 2018-05-03T19:38:06.181Z

Pardon me. We are leaving the memory resources allocated though, right?

Comment by Jerry Jelinek [X]
Created at 2018-05-03T20:06:53.730Z

No, we reboot the zone when init exits, so all of the bhyve resources are deallocated and then reallocated. Preserving all of the resources is the whole point of this ticket.

Comment by Hans Rosenfeld [X]
Created at 2018-08-13T11:39:23.479Z

Preventing the EFAULT when re-exec'ing init was handled in OS-7089. The remaining changes to bhyve to allow VM reboots reusing the allocated resources are handled here.

Testing: I've been using this change for a while. I can reboot VMs with various guest OSs from within and the already allocated resources will be reused. There remains an issue (OS-7110) with the UEFI firmware dying early in reboot due to an unexpected exception, but for now this worked around by clearing the lower 1G of VM memory.

Comment by Hans Rosenfeld [X]
Created at 2018-08-17T14:19:05.961Z

Additional testing: John asked for this to be tested in conjunction with GPGPU passthru. I didn't run the load tests, but the NVIDIA samples run fine before and after a in-VM reboot. It is no different than before.

Comment by Jira Bot
Created at 2018-08-17T15:14:41.762Z

illumos-joyent commit a72f92c76cef0e035c3ae9af38ea6789c0676527 (branch master, by Hans Rosenfeld)

OS-6620 bhyve reboot should reuse existing process
Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
Reviewed by: John Levon <john.levon@joyent.com>
Reviewed by: Jerry Jelinek <jerry.jelinek@joyent.com>
Approved by: Patrick Mooney <patrick.mooney@joyent.com>