OS-6760: bhyve passthru devices should use physical path

Details

Issue Type:Improvement
Priority:4 - Normal
Status:Resolved
Created at:2018-03-12T19:50:03.006Z
Updated at:2018-05-15T15:44:32.108Z

People

Created by:Mike Gerdts [X]
Reported by:Mike Gerdts [X]
Assigned to:John Levon

Resolution

Fixed: A fix for this issue is checked into the tree and tested.
(Resolution Date: 2018-05-15T09:44:25.146Z)

Fix Versions

2018-05-24 Sector 7 (Release Date: 2018-05-24)

Related Issues

Related Links

Labels

bhyve

Description

PCI pass through (PPT) devices are generic names that are allocated sequentially.  In order to delegate these devices to a bhyve zone, one of two approaches is needed:

  1. Have tooling outside of the zone that ensures that the /dev/ppt<N> maps to the device that is intended for the zone.
  2. Configure the zone with the bus:slot:fn that is being delegated and leave it to zoneadmd to create the PPT device and add the newly created /dev/ppt<N> to the zone's device profile.

The second approach fits better with the historical approach to zones.  That is, once a zone is configured and installed, it should be able to boot on its own without dependencies on software that is not delivered as part of illumos.

Comments

Comment by Mike Gerdts [X]
Created at 2018-03-12T20:01:42.865Z

Implementation of this becomes much easier with RFD 127.


Comment by John Levon
Created at 2018-03-14T21:45:29.671Z

Embedding /dev/ppt as is into the zonecfg is insufficient as well as problematic. The pptX corresponds to the device instance, but these - and ##/etc/driver_aliases## - are not persisted across reboots. Thus, if we set match in the zonecfg, in general, we'll need to potentially update all such zone configs on boot once we get our new device instance to driver mapping.

In fact this would essentially consist of removing all the ppt device entries and re-populating them, since the zone config doesn't identify the driver alias in the /dev/ppt scheme (I think the BDF pci-slot setting is what the guest sees).

All this kind of implies to me that we should have the driver alias as part of the zone config, have zoneadmd or a boot hook do the ppt set up namely: locating the device corresponding to the alias, making sure everything is detached from it, attaching a new ppt instance, and finally adding the corresponding /dev/pptX to both the env vars passed to zhyve argv and the dev profile.


Comment by John Levon
Created at 2018-04-03T13:46:26.202Z
Updated at 2018-04-11T16:18:11.657Z

As discussed in RFD 114, the rough plan is to specify model=passthru devices with the physical /devices/... path that identifies a ppt-attached instance. zoneadmd, when it sees a model=passthru device, will look up the corresponding /dev/pptX link and provide this in the environment passed to the bhyve boot hook.

This obviously presumes these devices have been created/attached already. The mechanism for this is /etc/ppt_aliases. This can be added as a boot module ala http://dtrace.org/blogs/wesolows/2013/12/28/anonymous-tracing-on-smartos/ and is processed just before /etc/driver_aliases, and should avoid any issues with other drivers binding before ppt gets a chance.


Comment by John Levon
Created at 2018-05-08T09:18:00.389Z

To test this (both on its own and with the vmadm changes), I picked an unused device on a test system and modified the USB grub configuration to mark it as a PPT device, and also another device type in ppt_matches. I tested the various forms of ppt_matches explicitly as well.

I ran pptadm list in its various forms (including under libumem to look for leaks).

I assigned the PPT device to a zone and made sure it all looked good (note the actual boot will fail as it's not really a PPT-able device, but I verified that the bhyve args are all as expected).


Comment by Jira Bot
Created at 2018-05-15T09:42:50.606Z

illumos-joyent commit e847b91c79ac0a60cba1e1852929eb37e96cf21c (branch master, by John Levon)

OS-6760 bhyve passthru devices should use physical path
Reviewed by: Robert Mustacchi <rm@joyent.com>
Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
Approved by: Patrick Mooney <patrick.mooney@joyent.com>