OS-7026: Add DDI support for Upgradable Firmware Modules

Details

Issue Type:New Feature
Priority:4 - Normal
Status:Resolved
Created at:2018-06-15T16:00:20.821Z
Updated at:2019-06-19T23:44:49.454Z

People

Created by:Former user
Reported by:Former user
Assigned to:Former user

Resolution

Fixed: A fix for this issue is checked into the tree and tested.
(Resolution Date: 2019-06-19T23:44:49.438Z)

Fix Versions

2019-07-04 Verdukianism (Release Date: 2019-07-04)

Related Issues

Description

For background, please refer to the section "UFM Visibility" in RFD 89:

https://github.com/joyent/rfd/blob/master/rfd/0089/README.md

This ticket covers the following changes to:

A separate ticket (OS-7014) was filed to cover userland changes to extend libtopo such that UFM information can be exposed in the topology.

Comments

Comment by Former user
Created at 2018-08-01T23:38:51.154Z

To prove out the new DDI interfaces, I plan to update both the mpt_sas and i40e drivers to report firmware information via the DDI UFM interfaces.


Comment by Former user
Created at 2019-06-13T20:21:52.869Z
Updated at 2019-06-19T23:40:06.059Z

Testing

DDI UFM functionality

In order to both prove out and exercise the new DDI UFM interfaces, both the mptsas and i40e drivers have been modified to be consumers of the new framework. So a big part of the testing was running a PI with these changes and verifying that the drivers we're able to successfully register with the DDI UFM subsystem - verified by examining the kernel state in mdb (see output below).

> ufm_handles::walk avl | ::print -t ddi_ufm_handle_t
ddi_ufm_handle_t {
    kmutex_t ufmh_lock = {
        void *[1] _opaque = [ 0 ]
    }
    char [1024] ufmh_devpath = [ "/pci@14,0/pci8086,2030@0/pci8086,37c0@0/pci8086,37c5@3/pci15d9,37d2@0" ]
    ddi_ufm_ops_t *ufmh_ops = i40e_ufm_ops
    void *ufmh_arg = 0xfffffe25b2c21000
    uint_t ufmh_state = 0x3
    uint_t ufmh_version = 0x1
    struct ddi_ufm_image *ufmh_images = 0xfffffe25aea6a180
    uint_t ufmh_nimages = 0x1
    ddi_ufm_cap_t ufmh_caps = 0x1 (DDI_UFM_CAP_REPORT)
    nvlist_t *ufmh_report = 0xfffffe2607bcfdc0
    avl_node_t ufmh_link = {
        struct avl_node *[2] avl_child = [ 0, 0 ]
        uintptr_t avl_pcb = 0xfffffe25b144a5f9
    }
}
ddi_ufm_handle_t {
    kmutex_t ufmh_lock = {
        void *[1] _opaque = [ 0 ]
    }
    char [1024] ufmh_devpath = [ "/pci@14,0/pci8086,2030@0/pci8086,37c0@0/pci8086,37c5@3/pci15d9,37d2@0,1" ]
    ddi_ufm_ops_t *ufmh_ops = i40e_ufm_ops
    void *ufmh_arg = 0xfffffe25b39cf000
    uint_t ufmh_state = 0x3
    uint_t ufmh_version = 0x1
    struct ddi_ufm_image *ufmh_images = 0xfffffe25d1a3fb80
    uint_t ufmh_nimages = 0x1
    ddi_ufm_cap_t ufmh_caps = 0x1 (DDI_UFM_CAP_REPORT)
    nvlist_t *ufmh_report = 0xfffffe25ae1a6608
    avl_node_t ufmh_link = {
        struct avl_node *[2] avl_child = [ 0xfffffe25a63b1c78, 0xfffffe25a63c7ab8 ]
        uintptr_t avl_pcb = 0x2
    }
}
ddi_ufm_handle_t {
    kmutex_t ufmh_lock = {
        void *[1] _opaque = [ 0 ]
    }
    char [1024] ufmh_devpath = [ "/pci@7e,0/pci8086,2030@0/pci15d9,808@0" ]
    ddi_ufm_ops_t *ufmh_ops = mptsas_ufm_ops
    void *ufmh_arg = 0xfffffe25b2b51000
    uint_t ufmh_state = 0x3
    uint_t ufmh_version = 0x1
    struct ddi_ufm_image *ufmh_images = 0xfffffe2605e83a00
    uint_t ufmh_nimages = 0x1
    ddi_ufm_cap_t ufmh_caps = 0x1 (DDI_UFM_CAP_REPORT)
    nvlist_t *ufmh_report = 0xfffffe25cf9bd780
    avl_node_t ufmh_link = {
        struct avl_node *[2] avl_child = [ 0, 0xfffffe25a7bc04b8 ]
        uintptr_t avl_pcb = 0xfffffe25b144a5fe
    }
}
ddi_ufm_handle_t {
    kmutex_t ufmh_lock = {
        void *[1] _opaque = [ 0 ]
    }
    char [1024] ufmh_devpath = [ "/pseudo/ufmtest@0" ]
    ddi_ufm_ops_t *ufmh_ops = 0xffffffffc0128f60
    void *ufmh_arg = 0
    uint_t ufmh_state = 0x3
    uint_t ufmh_version = 0x1
    struct ddi_ufm_image *ufmh_images = 0
    uint_t ufmh_nimages = 0
    ddi_ufm_cap_t ufmh_caps = 0 (0)
    nvlist_t *ufmh_report = 0
    avl_node_t ufmh_link = {
        struct avl_node *[2] avl_child = [ 0, 0 ]
        uintptr_t avl_pcb = 0xfffffe25a63c7abd
    }
}
>

To exercise the ioctls for the ufm(7d) driver, I wrote a the following test tool, which I ran to retrieve UFM information from the mptsas, i40e and ufmtest drivers:

https://github.com/joyent/smartos-dev-tools/tree/master/ddi/ufm-ioctl

See output below - exercising the ioctl's from both 32 and 64-bit userland processes:

# ./ufm-ioctl-32 -d /pci@14,0/pci8086,2030@0/pci8086,37c0@0/pci8086,37c5@3/pci15d9,37d2@0 -i getcaps
CAPS: 1
# ./ufm-ioctl-32 -d /pci@14,0/pci8086,2030@0/pci8086,37c0@0/pci8086,37c5@3/pci15d9,37d2@0 -i reportsz
Report Size: 444 bytes
# ./ufm-ioctl-32 -d /pci@14,0/pci8086,2030@0/pci8086,37c0@0/pci8086,37c5@3/pci15d9,37d2@0 -i report
Report Size: 444 bytes
nvlist version: 0
	ufm-images = (array of embedded nvlists)
	(start ufm-images[0])
	nvlist version: 0
		ufm-image-description = Firmware
		ufm-image-slots = (array of embedded nvlists)
		(start ufm-image-slots[0])
		nvlist version: 0
			ufm-slot-attributes = 0x4
			ufm-slot-version = 3.1
			ufm-slot-misc = (embedded nvlist)
			nvlist version: 0
				firmware-build = cc7d
				api-version = 1.5
			(end ufm-slot-misc)

		(end ufm-image-slots[0])

	(end ufm-images[0])
# ./ufm-ioctl-64 -d /pci@14,0/pci8086,2030@0/pci8086,37c0@0/pci8086,37c5@3/pci15d9,37d2@0 -i getcaps
CAPS: 1
# ./ufm-ioctl-64 -d /pci@14,0/pci8086,2030@0/pci8086,37c0@0/pci8086,37c5@3/pci15d9,37d2@0 -i reportsz
Report Size: 444 bytes
# ./ufm-ioctl-64 -d /pci@14,0/pci8086,2030@0/pci8086,37c0@0/pci8086,37c5@3/pci15d9,37d2@0 -i report
Report Size: 444 bytes
nvlist version: 0
	ufm-images = (array of embedded nvlists)
	(start ufm-images[0])
	nvlist version: 0
		ufm-image-description = Firmware
		ufm-image-slots = (array of embedded nvlists)
		(start ufm-image-slots[0])
		nvlist version: 0
			ufm-slot-attributes = 0x4
			ufm-slot-version = 3.1
			ufm-slot-misc = (embedded nvlist)
			nvlist version: 0
				firmware-build = cc7d
				api-version = 1.5
			(end ufm-slot-misc)

		(end ufm-image-slots[0])

	(end ufm-images[0])
# ./ufm-ioctl-32 -d /pci@7e,0/pci8086,2030@0/pci15d9,808@0 -i getcaps
CAPS: 1
# ./ufm-ioctl-32 -d /pci@7e,0/pci8086,2030@0/pci15d9,808@0 -i reportsz
Report Size: 312 bytes
# ./ufm-ioctl-32 -d /pci@7e,0/pci8086,2030@0/pci15d9,808@0 -i report
Report Size: 312 bytes
nvlist version: 0
	ufm-images = (array of embedded nvlists)
	(start ufm-images[0])
	nvlist version: 0
		ufm-image-description = IOC Firmware
		ufm-image-slots = (array of embedded nvlists)
		(start ufm-image-slots[0])
		nvlist version: 0
			ufm-slot-attributes = 0x4
			ufm-slot-version = 15.0.3.0
		(end ufm-image-slots[0])

	(end ufm-images[0])

# ./ufm-ioctl-64 -d /pci@7e,0/pci8086,2030@0/pci15d9,808@0 -i getcaps
CAPS: 1
# ./ufm-ioctl-64 -d /pci@7e,0/pci8086,2030@0/pci15d9,808@0 -i reportsz
Report Size: 312 bytes
# ./ufm-ioctl-64 -d /pci@7e,0/pci8086,2030@0/pci15d9,808@0 -i report
Report Size: 312 bytes
nvlist version: 0
	ufm-images = (array of embedded nvlists)
	(start ufm-images[0])
	nvlist version: 0
		ufm-image-description = IOC Firmware
		ufm-image-slots = (array of embedded nvlists)
		(start ufm-image-slots[0])
		nvlist version: 0
			ufm-slot-attributes = 0x4
			ufm-slot-version = 15.0.3.0
		(end ufm-image-slots[0])

	(end ufm-images[0])
# ./ufm-ioctl-32 -d /pseudo/ufmtest@0 -i getcaps
CAPS: 1
# ./ufm-ioctl-32 -d /pseudo/ufmtest@0 -i reportsz
Report Size: 1164 bytes
# ./ufm-ioctl-32 -d /pseudo/ufmtest@0 -i report
Report Size: 1164 bytes
nvlist version: 0
	ufm-images = (array of embedded nvlists)
	(start ufm-images[0])
	nvlist version: 0
		ufm-image-description = fw image 1
		ufm-image-slots = (array of embedded nvlists)
		(start ufm-image-slots[0])
		nvlist version: 0
			ufm-slot-attributes = 0x4
			ufm-slot-version = 1.0
		(end ufm-image-slots[0])

	(end ufm-images[0])
	(start ufm-images[1])
	nvlist version: 0
		ufm-image-description = fw image 2
		ufm-image-slots = (array of embedded nvlists)
		(start ufm-image-slots[0])
		nvlist version: 0
			ufm-slot-attributes = 0x4
			ufm-slot-version = 1.0
		(end ufm-image-slots[0])
		(start ufm-image-slots[1])
		nvlist version: 0
			ufm-slot-attributes = 0x0
			ufm-slot-version = 1.1
		(end ufm-image-slots[1])

	(end ufm-images[1])
	(start ufm-images[2])
	nvlist version: 0
		ufm-image-description = fw image 3
		ufm-image-slots = (array of embedded nvlists)
		(start ufm-image-slots[0])
		nvlist version: 0
			ufm-slot-attributes = 0x4
			ufm-slot-version = 1.0
		(end ufm-image-slots[0])
		(start ufm-image-slots[1])
		nvlist version: 0
			ufm-slot-attributes = 0x0
			ufm-slot-version = 1.1
		(end ufm-image-slots[1])
		(start ufm-image-slots[2])
		nvlist version: 0
			ufm-slot-attributes = 0x8
		(end ufm-image-slots[2])

	(end ufm-images[2])
# ./ufm-ioctl-64 -d /pseudo/ufmtest@0 -i getcaps
CAPS: 1
# ./ufm-ioctl-64 -d /pseudo/ufmtest@0 -i reportsz
Report Size: 1164 bytes
# ./ufm-ioctl-64 -d /pseudo/ufmtest@0 -i report
Report Size: 1164 bytes
nvlist version: 0
	ufm-images = (array of embedded nvlists)
	(start ufm-images[0])
	nvlist version: 0
		ufm-image-description = fw image 1
		ufm-image-slots = (array of embedded nvlists)
		(start ufm-image-slots[0])
		nvlist version: 0
			ufm-slot-attributes = 0x4
			ufm-slot-version = 1.0
		(end ufm-image-slots[0])

	(end ufm-images[0])
	(start ufm-images[1])
	nvlist version: 0
		ufm-image-description = fw image 2
		ufm-image-slots = (array of embedded nvlists)
		(start ufm-image-slots[0])
		nvlist version: 0
			ufm-slot-attributes = 0x4
			ufm-slot-version = 1.0
		(end ufm-image-slots[0])
		(start ufm-image-slots[1])
		nvlist version: 0
			ufm-slot-attributes = 0x0
			ufm-slot-version = 1.1
		(end ufm-image-slots[1])

	(end ufm-images[1])
	(start ufm-images[2])
	nvlist version: 0
		ufm-image-description = fw image 3
		ufm-image-slots = (array of embedded nvlists)
		(start ufm-image-slots[0])
		nvlist version: 0
			ufm-slot-attributes = 0x4
			ufm-slot-version = 1.0
		(end ufm-image-slots[0])
		(start ufm-image-slots[1])
		nvlist version: 0
			ufm-slot-attributes = 0x0
			ufm-slot-version = 1.1
		(end ufm-image-slots[1])
		(start ufm-image-slots[2])
		nvlist version: 0
			ufm-slot-attributes = 0x8
		(end ufm-image-slots[2])

	(end ufm-images[2])


DDI UFM Robustness

A set of of automated tests which primarily focus on negative test cases was also developed and is included in this integration. Sample results are below:

# ./ufm-test 
2019-06-13T18:51:16Z TEST ufm_open_negative_001: Open /dev/ufm in write-only mode
2019-06-13T18:51:16Z TEST ufm_open_negative_002: Open /dev/ufm in read-write mode
2019-06-13T18:51:16Z TEST ufm_open_negative_003: Open /dev/ufm in exclusive mode
2019-06-13T18:51:16Z TEST ufm_open_negative_004: Open /dev/ufm in non-block mode
2019-06-13T18:51:16Z TEST ufm_open_negative_005: Open /dev/ufm in ndelay mode
2019-06-13T18:51:16Z TEST ufm_getcaps_negative_001: Bad DDI UFM version (too low)
2019-06-13T18:51:16Z TEST ufm_getcaps_negative_002: Bad DDI UFM version (too high)
2019-06-13T18:51:16Z TEST ufm_getcaps_negative_003: Bad devpath (empty)
2019-06-13T18:51:16Z TEST ufm_getcaps_negative_004: Bad devpath (not a device)
2019-06-13T18:51:16Z TEST ufm_getcaps_negative_005: Bad devpath (not UFM device)
2019-06-13T18:51:16Z TEST ufm_getcaps_negative_006: Bad devpath (no NUL term)
2019-06-13T18:51:16Z TEST ufm_getcaps_negative_007: Bad devpath (not ascii str)
2019-06-13T18:51:16Z TEST ufm_reportsz_negative_001: Bad DDI UFM version (too low)
2019-06-13T18:51:16Z TEST ufm_reportsz_negative_002: Bad DDI UFM version (too high)
2019-06-13T18:51:16Z TEST ufm_reportsz_negative_003: Bad devpath (empty)
2019-06-13T18:51:16Z TEST ufm_reportsz_negative_004: Bad devpath (not a device)
2019-06-13T18:51:16Z TEST ufm_reportsz_negative_005: Bad devpath (not UFM device)
2019-06-13T18:51:16Z TEST ufm_reportsz_negative_006: Bad devpath (no NUL term)
2019-06-13T18:51:16Z TEST ufm_reportsz_negative_007: Bad devpath (not ascii str)
2019-06-13T18:51:16Z TEST ufm_report_negative_001: Bad DDI UFM version (too low)
2019-06-13T18:51:16Z TEST ufm_report_negative_002: Bad DDI UFM version (too high)
2019-06-13T18:51:16Z TEST ufm_report_negative_003: Bad devpath (empty)
2019-06-13T18:51:16Z TEST ufm_report_negative_004: Bad devpath (not a device)
2019-06-13T18:51:16Z TEST ufm_report_negative_005: Bad devpath (not UFM device)
2019-06-13T18:51:16Z TEST ufm_report_negative_006: Bad devpath (no NUL term)
2019-06-13T18:51:16Z TEST ufm_report_negative_007: Bad devpath (not ascii str)
2019-06-13T18:51:16Z TEST ufm_report_negative_008: bad bufsz (too small)
2019-06-13T18:51:16Z TEST ufm_report_negative_009: bad buf (NULL pointer)
2019-06-13T18:51:16Z TEST ufm_callback_negative_001: ddi_ufm_op_getcaps fails
2019-06-13T18:51:16Z TEST ufm_callback_negative_002: ddi_ufm_op_getcaps fails
2019-06-13T18:51:16Z TEST ufm_callback_negative_003: ddi_ufm_op_nimages fails
2019-06-13T18:51:16Z TEST ufm_callback_negative_004: ddi_ufm_op_fill_image fails
2019-06-13T18:51:16Z TEST ufm_callback_negative_005: ddi_ufm_op_fill_slot fails
2019-06-13T18:51:16Z Number of Tests Run: 33
2019-06-13T18:51:16Z Number of Passes:    33
2019-06-13T18:51:16Z Number of Fails :    0

Memory Leaks

After both running the automated tests and exercising the subsystem via the ufm-ioctl utility, I forced a crash dump and then ran findleaks on the resulting dump to verify there were no kernel leaks from the UFM subsystem or the i40e and mptsas drivers.

Build Cleanliness

I ran full non-debug and debug builds and verified there were no warnings. I also verified that the build was smatch and cstyle clean. Finally, to ensure there were no missed git adds, I applied the patch for this change to a clean illumos-joyent repo and verified that it would build.


Comment by Jira Bot
Created at 2019-06-19T23:42:02.748Z

smartos-live commit e7352686c10ff7d34d5ffd1abdd035761ba7eca5 (branch master, by Rob Johnston)

OS-7026 Add DDI support for Upgradable Firmware Modules
Reviewed by: Robert Mustacchi <robert.mustacchi@joyent.com>
Approved by: Jerry Jelinek <jerry.jelinek@joyent.com>


Comment by Jira Bot
Created at 2019-06-19T23:43:02.609Z

illumos-joyent commit 285d665c1bfb19b7a0d31074cbb554aae649ca56 (branch master, by Rob Johnston)

OS-7014 Add libtopo support for Upgradable Firmware Modules
OS-7026 Add DDI support for Upgradable Firmware Modules
Reviewed by: Robert Mustacchi <robert.mustacchi@joyent.com>
Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
Approved by: Jerry Jelinek <jerry.jelinek@joyent.com>