OS-7696: bhyve should avoid all-cpu cross calls


Issue Type:Improvement
Priority:4 - Normal
Created at:2019-04-03T16:14:51.233Z
Updated at:2019-04-08T18:39:45.366Z


Created by:Patrick Mooney [X]
Reported by:Patrick Mooney [X]
Assigned to:Patrick Mooney [X]


Fixed: A fix for this issue is checked into the tree and tested.
(Resolution Date: 2019-04-08T18:39:45.350Z)

Fix Versions

2019-04-11 Pete Hornberger (Release Date: 2019-04-11)

Related Links




As part of the effort to understand the implication of shared resources for Hyperthreaded CPUs, facilities which issue cross calls to all CPUs are being scrutinized. In bhyve, this is exposed via the smp_rendezvous glue interface. Previously it was used for VMX initialization in addition to cautious EPT invalidation when a VM is created. With the VMX initialization taken over by HMA, only the invept use-case remains. This too could be supplanted by an HMA helper for the task (as well as integration with the register_cpu_setup_func handler).


Comment by Patrick Mooney [X]
Created at 2019-04-05T18:39:40.588Z

To test this, I booted the updated PI on a bhyve-capable CN and looked for several things:
1. Did hma_vmx_capabs contained the expected bits for the hardware
2. While toggling a CPU with psradm -f <#> && psradm -n <#> did dtrace on hma_vmx_do_invept show calls from the correct CPU id (and INVEPT type)
3. When rebooting a VM to induce single-context INVEPT on its eptp, did dtrace on hma_vmx_do_invept show calls on the correct CPUs (with single-context type)
4. With HVC_INVEPT_ONE artificially removed from hma_vmx_capabs, did that same VM reboot test result in similar hma_vmx_do_invept calls, only with the all-context type
5. With one or more CPUs administratively disabled with psradm, did the same VM reboot tests skip the offline CPUs when doing their INVEPT operations

The test PI was successful in all of these tests.

Comment by Patrick Mooney [X]
Created at 2019-04-08T17:59:49.292Z

With @john.levon's suggested changes regarding CPU_INTR_ON/CPU_ON implemented, I booted an updated PI and traced hma_vmx_cpu_setup and hma_vmx_invept_xcall calls when toggling the state of a CPU. I observed calls with the appropriate events (CPU_ON) and the VMX xcall running on the expected CPU. This on top of the fact that VMX came up as expected suggests that it is functioning as desired.

Comment by Jira Bot
Created at 2019-04-08T18:31:22.359Z

illumos-joyent commit 919d37f67476ae6f9911035197d34891582d50e6 (branch master, by Patrick Mooney)

OS-7696 bhyve should avoid all-cpu cross calls
Reviewed by: Robert Mustacchi <rm@joyent.com>
Reviewed by: John Levon <john.levon@joyent.com>
Approved by: John Levon <john.levon@joyent.com>