|Priority:||4 - Normal|
|Created by:||Mike Gerdts [X]|
|Reported by:||Mike Gerdts [X]|
|Assigned to:||John Levon|
Fixed: A fix for this issue is checked into the tree and tested.
(Resolution Date: 2018-05-15T09:44:25.146Z)
2018-05-24 Sector 7 (Release Date: 2018-05-24)
PCI pass through (PPT) devices are generic names that are allocated sequentially. In order to delegate these devices to a bhyve zone, one of two approaches is needed:
/dev/ppt<N>maps to the device that is intended for the zone.
bus:slot:fnthat is being delegated and leave it to
zoneadmdto create the PPT device and add the newly created
/dev/ppt<N>to the zone's device profile.
The second approach fits better with the historical approach to zones. That is, once a zone is configured and installed, it should be able to boot on its own without dependencies on software that is not delivered as part of illumos.
Implementation of this becomes much easier with RFD 127.
Embedding /dev/ppt as is into the zonecfg is insufficient as well as problematic. The pptX corresponds to the device instance, but these - and ##/etc/driver_aliases## - are not persisted across reboots. Thus, if we set match in the zonecfg, in general, we'll need to potentially update all such zone configs on boot once we get our new device instance to driver mapping.
In fact this would essentially consist of removing all the ppt device entries and re-populating them, since the zone config doesn't identify the driver alias in the /dev/ppt scheme (I think the BDF pci-slot setting is what the guest sees).
All this kind of implies to me that we should have the driver alias as part of the zone config, have zoneadmd or a boot hook do the ppt set up namely: locating the device corresponding to the alias, making sure everything is detached from it, attaching a new ppt instance, and finally adding the corresponding /dev/pptX to both the env vars passed to zhyve argv and the dev profile.
As discussed in RFD 114, the rough plan is to specify model=passthru devices with the physical
/devices/... path that identifies a ppt-attached instance. zoneadmd, when it sees a model=passthru device, will look up the corresponding
/dev/pptX link and provide this in the environment passed to the bhyve boot hook.
This obviously presumes these devices have been created/attached already. The mechanism for this is
/etc/ppt_aliases. This can be added as a boot module ala http://dtrace.org/blogs/wesolows/2013/12/28/anonymous-tracing-on-smartos/ and is processed just before
/etc/driver_aliases, and should avoid any issues with other drivers binding before ppt gets a chance.
To test this (both on its own and with the vmadm changes), I picked an unused device on a test system and modified the USB grub configuration to mark it as a PPT device, and also another device type in ppt_matches. I tested the various forms of ppt_matches explicitly as well.
I ran pptadm list in its various forms (including under libumem to look for leaks).
I assigned the PPT device to a zone and made sure it all looked good (note the actual boot will fail as it's not really a PPT-able device, but I verified that the bhyve args are all as expected).
OS-6760 bhyve passthru devices should use physical path
Reviewed by: Robert Mustacchi <email@example.com>
Reviewed by: Patrick Mooney <firstname.lastname@example.org>
Approved by: Patrick Mooney <email@example.com>