OS-7170: bhyve should support AMD

Details

Issue Type:Improvement
Priority:4 - Normal
Status:Resolved
Created at:2018-08-23T19:16:37.389Z
Updated at:2019-03-15T15:37:07.003Z

People

Created by:Patrick Mooney [X]
Reported by:Patrick Mooney [X]
Assigned to:Patrick Mooney [X]

Resolution

Fixed: A fix for this issue is checked into the tree and tested.
(Resolution Date: 2019-03-15T15:36:55.754Z)

Fix Versions

2019-03-28 Operation Righteous Cowboy Lightning (Release Date: 2019-03-28)

Related Links

Labels

bhyve

Description

When the bhyve porting effort was underway, all effort was focused on the Intel side, ignoring AMD support. With Epyc coming down the pipeline, it would be nice to finish the porting effort so that bhyve works there as well.

Comments

Comment by Patrick Mooney [X]
Created at 2019-03-07T04:27:19.463Z

A community member on IRC (jubal) tried a test PI on their relatively old Opteron system (CPU part: OS2439YDS6DGN) and reported that they were able to boot a centos-7 VM without any issues.


Comment by Ryan Kitchen
Created at 2019-03-13T21:33:52.330Z
Updated at 2019-03-13T22:30:12.639Z

The two goals of this test were to a) stress the EPYC hardware enough to create instability b) to get some baseline benchmark performance numbers.

The lab machine odyssey has 1.4TB disk, 256gigs of ram, and 32 cores (64 virtual cpus).

I created 6 VM's with varying disk, cpu, and ram  that consumed 2/3 of total system memory, 56 of the 64 vcpus, and approximately half of the 1.4 TB of disk space.
I ran benchmarks on a single system at a time for optimal baselines as seen below from a 16cpu / 32gb ram system.

I ran a variety of  load generation to stress the overall system and see if anything fell over. Nothing did.
If there was more time I would have liked to generate a more "organic" application based load. I'll work on tooling up automation to deploy something quickly, so that in the future we can make better tests in short time frames.

In terms of baselines the cpu performance in particular looks quite good.

Raw results:

#sysbench cpu

[root@befce8ba-ce72-4e2c-bee8-baf5b38130c2 ~]# rm -f /rent/result.txt && for i in {1..10}; do sysbench cpu --threads=8 run | grep "events per second"; cat result1.txt | awk '{ sum += $1 } END { if (NR > 0) print sum / NR }'

Result: 10380.8

#sysbench ram

rm -f /rent/result1.txt && for i in {1..10};do sysbench memory --threads=8 run | grep "MiB/sec" | awk '{print $4}' | sed -e 's/(//g' >> /rent/result1.txt; done;cat result1.txt | awk '{ sum += $1 } END { if (NR > 0) print sum / NR }'

Result: 1122.03

#fio disk

JobnameRead BWRead IOPSWrite BWWrite IOPS
readblock128ksize5gfiooutput162974112732.3593400
writeblock128ksize5gfiooutput00134778410529.56298
rwblock128ksize5gfiooutput2854072229.7473872854632230.182927
randreadblock8kfile1gfiooutput404965062.06310600
randwriteblock8kfile1gfiooutput001006310125788.8676
randrwblock8kfile1gfiooutput288153601.896514286593582.492874
randreadblock64kfile1gfiooutput2462023846.91242100
randwriteblock64kfile1gfiooutput00108660716978.23834
randrwblock64kfile1gfiooutput1109491733.5882731118191747.185044
randreadblock128kfile1gfiooutput2999352343.24942800
randwriteblock128kfile1gfiooutput0012038769405.281286
randrwblock128kfile1gfiooutput1517561185.597191552031212.529274
randreadblock256kfile1gfiooutput5271872059.32629500
randwriteblock256kfile1gfiooutput0015022575868.194842
randrwblock256kfile1gfiooutput227555888.888889240349938.866577
randreadblock8kfile5gfiooutput414075175.92424400
randwriteblock8kfile5gfiooutput0077845297306.60728
randrwblock8kfile5gfiooutput281533519.142633281193514.956692
randreadblock64kfile5gfiooutput2295283586.37597400
randwriteblock64kfile5gfiooutput00158013224689.57203
randrwblock64kfile5gfiooutput1071831674.7386481069161670.573342
randreadblock128kfile5gfiooutput2956062309.42715400
randwriteblock128kfile5gfiooutput009845787692.018779
randrwblock128kfile5gfiooutput1463361143.2559181463651143.479232
randreadblock256kfile5gfiooutput4157381623.97906600
randwriteblock256kfile5gfiooutput0011459844476.502732
randrwblock256kfile5gfiooutput167820655.547035167231653.246421
randrwratio80block8ksize5gfiooutput337614220.24341784681058.574777
randrwratio80block64ksize5gfiooutput1850752891.79763446378724.660074
randrwratio80block128ksize5gfiooutput2368961850.75302959948468.35013
randrwratio80block256ksize5gfiooutput3347091307.45838784855331.466069

Comment by Patrick Mooney [X]
Created at 2019-03-14T20:53:47.855Z

With vmx_call_trap rearranged to vmm_call_trap, I wanted to make sure NMI handling was still ok, which it was:

fffffe007aeed440 fffffffff79488ff ()
fffffe007aeed470 unix:av_dispatch_nmivect+34 ()
fffffe007aeed480 unix:nmiint+152 ()
fffffe007aeed570 vmm:vmm_call_trap+3e ()
fffffe007aeed670 vmm:vmx_run+7d2 ()
fffffe007aeed760 vmm:vm_run+21f ()
fffffe007aeedc20 vmm:vmmdev_do_ioctl+775 ()
fffffe007aeedcc0 vmm:vmm_ioctl+12f ()
fffffe007aeedd00 genunix:cdev_ioctl+39 ()
fffffe007aeedd50 specfs:spec_ioctl+60 ()
fffffe007aeedde0 genunix:fop_ioctl+55 ()
fffffe007aeedf00 genunix:ioctl+9b ()
fffffe007aeedf10 unix:brand_sys_syscall+21a ()

Comment by Patrick Mooney [X]
Created at 2019-03-14T21:02:59.451Z
Updated at 2019-03-14T21:07:42.260Z

Using the typical variety of guests (Linux, Windows, FreeBSD, illumos) on an Intel machine, operation seemed normal as expected. The PI was also run nested under KVM on a Threadripper machine where it seemed to function fine.


Comment by Jira Bot
Created at 2019-03-15T15:37:07.003Z

illumos-joyent commit 2453029c010976e95241a5f5244e86d44dc6194c (branch master, by Patrick Mooney)

OS-7170 bhyve should support AMD
Reviewed by: John Levon <john.levon@joyent.com>
Reviewed by: Hans Rosenfeld <hans.rosenfeld@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>