OS-6865: bhyve could be lazy about FPU state

Details

Issue Type:Improvement
Priority:4 - Normal
Status:Resolved
Created at:2018-04-02T22:32:58.641Z
Updated at:2018-04-05T23:54:36.462Z

People

Created by:Patrick Mooney [X]
Reported by:Patrick Mooney [X]
Assigned to:Patrick Mooney [X]

Resolution

Fixed: A fix for this issue is checked into the tree and tested.
(Resolution Date: 2018-04-05T23:54:36.452Z)

Fix Versions

2018-04-12 Promised Land (Release Date: 2018-04-12)

Related Links

Labels

bhyve

Description

The FPU save/restore logic from upstream performs a swap of host/guest FPU state surrounding every call to VMRUN.  This is presumably to ensure that such state is manipulated inside the critical_enter/critical_exit section, where it cannot be preempted.  Since illumos has the convenience of savectx/restorectx handlers available, we could choose to be lazy about when we do these FPU state swaps.

Comments

Comment by Patrick Mooney [X]
Created at 2018-04-03T18:57:01.081Z

I wrote up a dtrace script for quantifying the scope of this FPU-state-saving elision:

BEGIN
{
        secs = 0;
}

vmm:vm_run:entry
{
        self->vmx = 0;
        self->fpu = 0;
}

vmm:vmx_run:entry
{
        self->vmx++;
}

vmm:restore_guest_fpustate:entry
{
        self->fpu++;
}

vmm:vm_run:return
{
        @ = quantize(self->vmx - self->fpu);
        @total = quantize(self->vmx - self->fpu);
        @summed = sum(self->vmx - self->fpu);
}

tick-1s
{
        printa(@);
        trunc(@);
        secs++;
}

END
{
        printf("TOTAL:\n");
        printa(@total);
        printa("FPU ops saved: %@u\n", @summed);
        if (secs > 0) {
                normalize(@summed, secs);
                printa("(%@u/s)\n", @summed);
        }
}

Running this with a varied workload, including CPU-bound and network-bound tasks, the results seem like they're promising:

  8      2                             :END TOTAL:


           value  ------------- Distribution ------------- count
              -1 |                                         0
               0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 58000
               1 |                                         15
               2 |                                         9
               4 |                                         11
               8 |                                         19
              16 |                                         26
              32 |                                         30
              64 |                                         19
             128 |                                         19
             256 |                                         123
             512 |                                         66
            1024 |                                         18
            2048 |                                         14
            4096 |                                         4
            8192 |                                         5
           16384 |                                         1
           32768 |                                         0

FPU ops saved: 258410

  8      2                             :END (6984/s)

Further data capture on real-world workloads would be valuable.


Comment by Jira Bot
Created at 2018-04-05T23:50:51.473Z

illumos-joyent commit 6a175f35f25ea47a4b116ad2dd1a0600fdf5a2bc (branch master, by Patrick Mooney)

OS-6864 bhyve should guard against going off-cpu
OS-6865 bhyve could be lazy about FPU state
Reviewed by: Jerry Jelinek <jerry.jelinek@joyent.com>
Reviewed by: John Levon <john.levon@joyent.com>
Approved by: John Levon <john.levon@joyent.com>