OS-6778: MAC loopback traffic should avoid cksum work

Details

Issue Type:Improvement
Priority:4 - Normal
Status:Resolved
Created at:2018-03-15T22:08:04.762Z
Updated at:2019-11-14T18:29:10.990Z

People

Created by:Former user
Reported by:Former user
Assigned to:Former user

Resolution

Fixed: A fix for this issue is checked into the tree and tested.
(Resolution Date: 2018-10-19T13:40:10.103Z)

Fix Versions

2018-11-08 EGOT (Release Date: 2018-11-08)

Related Issues

Description

MAC has sweeping support for allowing clients to offload checksum work onto the hardware itself by setting flags requesting that calculation be done on their behalf. When traffic from those clients ends up looping back within MAC, a soft checksum calculation is forced to ensure that the destination entity receives a packet with a properly populated checksum field. Since this traffic isn't leaving the box (or even the specific MAC instance), we could instead simply set the HCKSUM_OK flag when loopback traffic requests hardware checksumming. That would eliminate quite a bit of unnecessary work in the datapath.

Comments

Comment by Former user
Created at 2018-04-04T18:55:10.379Z
Updated at 2018-04-04T19:12:56.089Z

The system has a moderately complex mechanism for propagating HW checksum capabilities throughout the stack so that the various checksum's can be calculated, or not, depending on what can be done in the NIC HW itself.

When a netstack is created (e.g. at zone boot time), the associated ill_t's ill_hcksum_capab->ill_hcksum_txflags is populated with the capabilities from the underlying NIC. This value is then used upstack (e.g. in ip_output_cksum_v4) to determine if the IP checksum must be calculated, or perhaps a partial checksum must be calculated.

Within MAC, when we're looping back a packet within the box (e.g. onto another netstack for a different zone), the code must handle the case where all of the checksums were not calculated, due to the anticipated HW calculation(s), and compute those checksums (see mac_fix_cksum) since the packet will never be sent down to the NIC. This means that both MAC and the upper IP code both duplicate the capability to calculate all of the checksums.

In order to avoid the checksum calculation throughout the stack when a packet is looping back, the code first needs to know if the packet actually will loop back or go out on the wire. This is known only in the MAC code in the places where mac_fix_cksum is called (e.g. mac_tx_send). There is no easy way to expose this information up to the IP code, since there is a bunch of intervening DLD code involved in the packet's flow through the stack.

However, because MAC already has the ability to calculate the checksums when a packet is looping back, it makes sense to invert the current logic, so that MAC always calculates all of the checksums when a packet is going out on the wire (and the HW cannot calculate the checksum). Then, when a packet is looping back, that work can be skipped in MAC and the mblk can be tagged so that the receiving side knows that the packet never left the box and thus has no checksums.

This implies that much of the code for propagating the HW checksum capabilities upstack is unnecessary. As a first step, we will always set HCKSUM_IPHDRCKSUM | HCKSUM_INET_FULL_V4 | HCKSUM_INET_FULL_V6 on the ill_hcksum_capab->ill_hcksum_txflags so that the IP code never calculates any SW checksum, then we will modify the MAC code so it uses the HW capabilities combined with its knowledge of the packet's ultimate destination (wire or loopback) to calculate the checksums when necessary.

On the receive side, if we see a packet was looped back, but then is passing into a guest VM (e.g. in vnd) then we can calculate the necessary checksums at that time.

Once this is working as expected, we can then clean up the rest of the code to remove everything that is propagating HW checksum capabilities throughout the stack.


Comment by Former user
Created at 2018-10-18T15:06:58.281Z

See OS-2340 for test notes.


Comment by Jira Bot
Created at 2018-10-19T05:02:17.091Z

illumos-joyent commit 104c53876a87e773ef729efa9419a70fe24933cb (branch master, by Ryan Zezeski)

OS-2340 vnics should support LSO
OS-6778 MAC loopback traffic should avoid cksum work
OS-6794 want LSO support in viona
Reviewed by: Robert Mustacchi <rm@joyent.com>
Reviewed by: Dan McDonald <danmcd@joyent.com>
Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
Reviewed by: Hans Rosenfeld <hans.rosenfeld@joyent.com>
Reviewed by: Jorge Schrauwen <jorge@blackdot.be>
Approved by: Dan McDonald <danmcd@joyent.com>


Comment by Jira Bot
Created at 2018-11-01T17:59:57.945Z

illumos-joyent commit e6151003c8efce19988229b39ba3dbb5551dc6f1 (branch master, by Ryan Zezeski)

OS-2340 vnics should support LSO
OS-6778 MAC loopback traffic should avoid cksum work
OS-6794 want LSO support in viona
OS-7319 dangling ref in mac_sw_cksum()
OS-7331 mac_sw_cksum() drops valid UDP traffic
Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
Reviewed by: Dan McDonald <danmcd@joyent.com>
Approved by: Patrick Mooney <patrick.mooney@joyent.com>


Comment by Jira Bot
Created at 2019-06-20T12:08:44.137Z

illumos-joyent commit 87738edeea3a17bfc0f19c6e1c3a597f3970e943 (branch master, by Ryan Zezeski)

OS-7520 OS-6778 broke IPv4 forwarding
OS-6878 mac_fix_cksum is incomplete
OS-7806 cannot move link from NGZ to GZ
Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
Reviewed by: Robert Mustacchi <rm@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>


Comment by Jira Bot
Created at 2019-11-14T18:29:10.990Z

illumos-joyent commit dff0473c792fc5c3ef7c3d862e557fe201a5d671 (branch master, by Joyent Automation)

OS-8027 reinstate mac-loopback hardware emulation on Tx (undo OS-6778)

Reviewed by: Dan McDonald <danmcd@joyent.com>
Reviewed by: Jason King <jason.king@joyent.com>
Approved by: Dan McDonald <danmcd@joyent.com>