OS-2340: vnics should support LSO


Issue Type:Improvement
Priority:3 - Elevated
Created at:2013-06-25T01:30:04.000Z
Updated at:2018-11-01T18:05:05.223Z


Created by:Former user
Reported by:Former user
Assigned to:Former user


Fixed: A fix for this issue is checked into the tree and tested.
(Resolution Date: 2018-10-19T13:39:50.625Z)

Fix Versions

2018-11-08 EGOT (Release Date: 2018-11-08)

Related Issues


Currently vnics do not have LSO support at all. It's not clear per se why this is the case, if it's just an oversight, a major architectural issue, or something in between. Currently vnic_m_getcapab doesn't even think about LSO. So working from there and doing similar things to hardware checksum is probably what we want.


Comment by Former user
Created at 2018-10-18T14:47:01.353Z
Updated at 2018-10-18T15:13:08.250Z

Testing Notes

All benchmarks use the default iperf3 settings (128K application buffer) and run for 15 seconds. Don't read too much into the exact perf numbers shown here. These tests were run on my home workstation which tends to show slower perf than our production environments. For example, the iperf3 tests between BHYVE VMs over an external network showed 9 Gbps throughput over 1500 MTU (compared to my measly 5 Gbps). The iperf tests can also vary fairly significantly across runs -- I believe depending on core placement. The important thing is that all throughput numbers are better than the current release (with the exception of KVM which is bottlenecked by VND).

External Network, non-VLAN, 1500 MTU

GZ primary -> GZ primary6.58 GbpsGoes thru DLS b/c of LLDP binding
GZ -> GZ7.16 Gbps
GZ -> NGZ7.63 Gbps
NGZ -> GZ6.52 Gbps
GZ -> LX7.27 Gbps
LX -> GZ8.52 Gbps
GZ -> BHYVE4.58 Gbps
BHYVE -> GZ7.67 Gbps
GZ -> KVM905 Mbps
KVM -> GZ619 Mbps
BHYVE -> BHYVE5.58 Gbps
BHYVE -> KVM932 Mbps
KVM -> BHYVE572 Mbps
BHYVE -> NGZ6.53 Gbps
NGZ -> BHYVE4.45 Gbps
KVM -> NGZ606 Mbps
NGZ -> KVM918 Mbps
NGZ -> NGZ7.04 Gbps

External Network, VLAN, 1500 MTU

GZ VLAN -> GZ VLAN5.76 Gbpsdladm create-vlan
GZ -> GZ7.72 Gbps
GZ -> NGZ7.52 Gbps
NGZ -> GZ6.65 Gbps
GZ -> LX7.08 Gbps
LX -> GZ6.43 Gbps
GZ -> BHYVE4.98 Gbps
BHYVE -> GZ6.60 Gbps
GZ -> KVM957 Mbps
KVM -> GZ563 Mbps
BHYVE -> BHYVE5.35 Gbps
BHYVE -> KVM1.01 Gbps
KVM -> BHYVE499 Mbps
BHYVE -> NGZ7.69 Gbps
NGZ -> BHYVE4.86 Gbps
NGZ -> KVM939 Mbps
KVM -> NGZ565 Mbps
NGZ -> NGZ7.67 Gbps

MAC-loopback, non-VLAN, 1500 MTU

GZ -> GZ19.2 Gbps
GZ -> NGZ21.3 Gbps
NGZ -> GZ6.28 Gbps
GZ -> LX24.1 Gbps
LX -> GZ21.3 Gbps
GZ -> BHYVE10.5 Gbps
BHYVE -> GZ11.3 Gbps
GZ -> KVM902 Mbps
KVM -> GZ545 Mbps
BHYVE -> BHYVE13.1 Gbps
BHYVE -> KVM922 Mbps
KVM -> BHYVE538 Mbps
BHYVE -> NGZ13.3 Gbps
NGZ -> BHYVE7.90 Gbps
NGZ -> KVM543 Mbps
KVM -> NGZ559 Mbps
NGZ -> NGZ5.88 Gbps

MAC-loopback, VLAN, 1500 MTU

GZ -> GZ19.3 Gbps
GZ -> NGZ23.0 Gbps
NGZ -> GZ7.67 Gbps
GZ -> LX21.9 Gbps
LX -> GZ7.67 Gbps
GZ -> BHYVE10.9 Gbps
BHYVE -> GZ5.03 Gbps
GZ -> KVM924 Mbps
KVM -> GZ435 Mbps
BHYVE -> BHYVE12.1 Gbps
BHYVE -> KVM911 Mbps
KVM -> BHYVE463 Mbps
BHYVE -> NGZ4.87 Gbps
NGZ -> BHYVE8.73 Gbps
NGZ -> KVM625 Mbps
KVM -> NGZ486 Mbps
NGZ -> NGZ5.34 Gbps

External, non-VLAN, 9000 MTU aggr of ixgbe, 1500 MTU guest/VNIC

GZ -> GZ7.44
GZ -> BHYVE5.33 Gbps
BHYVE -> GZ6.36 Gbps
GZ -> KVM845 Mbps
KVM -> GZ626 Mbps
BHYVE -> BHYVE5.11 Gbps
BHYVE -> KVM888 Mbps
KVM -> BHYVE508 Mbps

External, VLAN, 9000 MTU aggr of ixgbe, 1500 MTU guest/VNIC

GZ -> GZ7.40 Gbps
GZ -> BHYVE5.15 Gbps
BHYVE -> GZ6.39 Gbps
GZ -> KVM880 Mbps
KVM -> GZ559 Mbps
BHYVE -> BHYVE5.35 Gbps
BHYVE -> KVM808 Mbps
KVM -> BHYVE463 Mbps

Etherstub, non-VLAN

GZ -> GZ9.43 Gbps9000 MTU VNIC
GZ -> BHYVE2.37 Gbps1500 MTU guest
BHYVE -> GZ2.36 Gbps1500 MTU guest
GZ -> BHYVE5.46 Gbps9000 MTU guest
BHYVE -> GZ8.37 Gbps9000 MTU guest
GZ -> KVM738 Mbps1500 MTU guest
KVM -> GZ505 Mbps1500 MTU guest
BHYVE -> KVM1.04 Gbps1500 MTU guest
KVM -> BHYVE523 Mbps1500 MTU guest
BHYVE -> KVM2.87 Gbps9000 MTU guest
KVM -> BHYVE1.87 Gbps9000 MTU guest
BHYVE -> BHYVE2.11 Gbps1500 MTU guest
BHYVE -> BHYVE6.35 Gbps9000 MTU guest

Etherstub, VLAN

GZ -> GZ8.89 Gbps9000 MTU VNIC
GZ -> BHYVE2.12 Gbps1500 MTU guest
BHYVE -> GZ1.99 Gbps1500 MTU guest
GZ -> BHYVE5.37 Gbps9000 MTU guest
BHYVE -> GZ8.14 Gbps9000 MTU guest
GZ -> KVM802 Mbps1500 MTU guest
KVM -> GZ425 Mbps1500 MTU guest
BHYVE -> KVM1.07 Gbps1500 MTU guest
KVM -> BHYVE512 Mbps1500 MTU guest
BHYVE -> KVM3.00 Gbps9000 MTU guest
KVM -> BHYVE1.70 Gbps9000 MTU guest
BHYVE -> BHYVE1.94 Gbps1500 MTU guest
BHYVE -> BHYVE5.55 Gbps9000 MTU guest

Bridge loopback

VLAN VNICs created via create-vnic don't seem to work with bridges. I had to use create-vlan. I think this might be a bug in bridge.

GZ -> GZ14.6 Gbps

Bridge non-loopback, non-VLAN

In this scenario I have 4 distinct ixgbe ports: ixgbe{0..3}. ixgbe0 and ixgbe2 are connect back-to-back, as well as ixgbe1 and ixgbe3. Then I created a bridge between ixgbe0 and ixgbe1, connecting the two L2 segments. Then I place VNICs on the ixgbe2/ixgbe3 ports and send traffic between them. The only way this works is if the traffic successfully crosses the bridge made between ixgbe0/ixgbe1.

I also had to run create-vlan on ixgbe{0,1} in order to pass the traffic for the VLAN VNICs on ixgbe{2,3}. I'm not sure if this is by design or a bug in the bridge code; but it's the same situation in master.

There was no reason to do all the different combinations here as this is really no different than sending over a NIC. I just ran these as a sanity check.

GZ -> GZ6.15 Gbps
GZ -> BHYVE4.22 Gbps
BHYVE -> GZ6.22 Gbps
BHYVE -> BHYVE4.33 Gbps

Bridge non-loopback, VLAN

GZ -> GZ4.92 Gbps
GZ -> BHYVE4.65 Gbps
BHYVE -> GZ5.50 Gbps
BHYVE -> BHYVE5.00 Gbps

Other tests

Comment by Former user
Created at 2018-10-18T16:36:23.132Z

It's also worth noting a few other things related to testing:

Comment by Jira Bot
Created at 2018-10-19T05:02:15.412Z

illumos-joyent commit 104c53876a87e773ef729efa9419a70fe24933cb (branch master, by Ryan Zezeski)

OS-2340 vnics should support LSO
OS-6778 MAC loopback traffic should avoid cksum work
OS-6794 want LSO support in viona
Reviewed by: Robert Mustacchi <rm@joyent.com>
Reviewed by: Dan McDonald <danmcd@joyent.com>
Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
Reviewed by: Hans Rosenfeld <hans.rosenfeld@joyent.com>
Reviewed by: Jorge Schrauwen <jorge@blackdot.be>
Approved by: Dan McDonald <danmcd@joyent.com>

Comment by Former user
Created at 2018-10-23T20:14:50.506Z

This is being backed out temporarily due to panics in the networking stack. See OS-7319.

Comment by Jira Bot
Created at 2018-10-23T21:53:18.775Z

illumos-joyent commit 92b1263de0f0e8df3856a798e95a142d3bb7547b (branch master, by Ryan Zezeski)

Revert "OS-2340 vnics should support LSO" [breaks COAL]

This reverts commit 104c53876a87e773ef729efa9419a70fe24933cb.

Comment by Jira Bot
Created at 2018-11-01T17:59:56.383Z

illumos-joyent commit e6151003c8efce19988229b39ba3dbb5551dc6f1 (branch master, by Ryan Zezeski)

OS-2340 vnics should support LSO
OS-6778 MAC loopback traffic should avoid cksum work
OS-6794 want LSO support in viona
OS-7319 dangling ref in mac_sw_cksum()
OS-7331 mac_sw_cksum() drops valid UDP traffic
Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
Reviewed by: Dan McDonald <danmcd@joyent.com>
Approved by: Patrick Mooney <patrick.mooney@joyent.com>