Issue Type: | Improvement |
---|---|
Priority: | 4 - Normal |
Status: | Resolved |
Created at: | 2019-04-10T14:41:14.177Z |
Updated at: | 2019-04-15T22:34:36.217Z |
Created by: | Former user |
---|---|
Reported by: | Former user |
Assigned to: | Former user |
Fixed: A fix for this issue is checked into the tree and tested.
(Resolution Date: 2019-04-15T22:34:36.204Z)
2019-04-25 Queen of Jordan (Release Date: 2019-04-25)
As part of ongoing work on the viona subsystem of bhyve, it has become apparent that the ability to impose a barrier on the mac RX path would be valuable. This barrier functionality would ensure that after a call to the function (mac_rx_barrier
), any threads executing RX callbacks for the client, both via mac_rx_set
and mac_promisc_add
, will have completed those calls. It does not prevent new entry to the RX callbacks, but does promise that such entries will have been totally completed or begin fresh, after the barrier returns.
The pending wad for OS-7447, which inspired the need for mac_rx_barrier
, was the primary test-bed for this change. A CN was booted up on a platform containing a combination of both proposed changes. A bhyve instance was started and loaded with test network traffic (via iperf3). Using bhyvectl --wrlock-cycle
, the viona instances serving the guest were repeatedly forced to relinquish their driver lease in bhyve, requiring mac_rx_barrier
calls to ensure all MAC callbacks would either finish or observe the changes to viona state. Visibility of these events was tracked with dtrace, verifying that callback info state like mcbi_barrier_cnt
was as expected.
The second focus was to ensure that the new limits on MAC callback walker entry (threads entering the callback list must wait until pending deletes or barrier requests have finished) are functioning as expected. Catching such an occurrence is difficult given the tight timing of promisc callbacks and the relatively slow pace of events which manipulate those lists. I used the following dtrace to force such circumstances to be visible:
dls_rx_promisc:entry { chill(50000) } mac_callback_walker_enter:entry { self->t = timestamp; self->w = args[0]; } mac_callback_walker_enter:return { @ = quantize(timestamp - self->t); self->t = 0 } cv_wait:entry /self->t/ { stack(); print(*self->w) }
With unfiltered promisc enabled in a zone, I repeatedly started and stopped snoop(1)
:
while true; do snoop -r -d net0 -c 5; done
This caused promisc callbacks to be created and deleted, each time the snoop
command ran and exited, resulting in eventual firing of the cv_wait
probe, showing the non-zero mcbi_del_cnt
:
9 25074 cv_wait:entry mac`mac_callback_walker_enter+0x33 mac`mac_promisc_dispatch+0x47 mac`mac_rx_common+0x45 mac`mac_rx+0xb6 mac`mac_rx_ring+0x2b igb`igb_intr_rx_work+0x5c igb`igb_intr_rx+0x15 apix`apix_dispatch_by_vector+0x8c apix`apix_dispatch_lowlevel+0x25 unix`switch_sp_and_call+0x13 mac_cb_info_t { kmutex_t *mcbi_lockp = 0xfffffe59ff4f7c40 kcondvar_t mcbi_cv = { ushort_t _opaque = 0x1 } uint_t mcbi_del_cnt = 0x1 uint_t mcbi_walker_cnt = 0x1 uint_t mcbi_barrier_cnt = 0 } ^C 9 25074 cv_wait:entry mac`mac_callback_walker_enter+0x33 mac`mac_promisc_dispatch+0x47 mac`mac_rx_common+0x45 mac`mac_rx+0xb6 mac`mac_rx_ring+0x2b igb`igb_intr_rx_work+0x5c igb`igb_intr_rx+0x15 apix`apix_dispatch_by_vector+0x8c apix`apix_dispatch_lowlevel+0x25 unix`switch_sp_and_call+0x13 mac_cb_info_t { kmutex_t *mcbi_lockp = 0xfffffe59ff4f7c40 kcondvar_t mcbi_cv = { ushort_t _opaque = 0x1 } uint_t mcbi_del_cnt = 0x1 uint_t mcbi_walker_cnt = 0x1 uint_t mcbi_barrier_cnt = 0 } value ------------- Distribution ------------- count 1024 | 0 2048 |@@@@@@@@@ 579 4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1718 8192 |@@@@ 243 16384 | 8 32768 | 4 65536 | 0
Using a DEBUG kernel, I hammered several of the test conditions and then forced a dump, checking ::findleaks
once it had completed. The result was clean.
illumos-joyent commit 39324c3c3e700905593cf884cb6987cf0db01381 (branch master, by Patrick Mooney)
OS-7727 want mac rx barrier function
Reviewed by: Robert Mustacchi <rm@joyent.com>
Reviewed by: Ryan Zezeski <rpz@joyent.com>
Approved by: Ryan Zezeski <rpz@joyent.com>