OS-6719: aggr needs support for multiple pseudo rx groups


Issue Type:Improvement
Priority:4 - Normal
Created at:2018-03-01T17:00:36.687Z
Updated at:2018-03-28T16:23:42.779Z


Created by:Former user
Reported by:Former user
Assigned to:Former user


Fixed: A fix for this issue is checked into the tree and tested.
(Resolution Date: 2018-03-28T16:23:42.763Z)

Fix Versions

2018-03-29 Old Man's House (Release Date: 2018-03-29)


We need multiple pseudo Rx groups for aggr in order to better utilize the underlying hardware.

MAC groups are an abstraction used to group and program hardware rings. Groups are where we place unicast and VLAN filters. The more groups a MAC provider supports, the more MAC clients (e.g. VNICs) it can hardware accelerate through the use of hardware filtering. This hardware filtering relieves the MAC framework from performing software classification and allows the SRS to poll the MAC's hardware rings. In effect, network performance is better when a client has a reserved MAC group with hardware filtering.

An aggr is both a MAC client and a MAC provider. It's a client to the underlying MAC/NIC (aggr_port_t) and a provider of the aggregation (aggr_grp_t). As a provider it must support the groups abstraction to allow clients of the aggr to make use of hardware classification. It does this by creating a "pseudo group": an abstraction that combines 1 hardware group from each port. E.g., if we have two ixgbe ports with groups of 4 rings, then 1 pseudo group will contain a mapping to one group on each ixgbe port, and contain 8 pseudo rings.

The problem is that aggr currently only creates one pseudo group – no matter what the underlying hardware may provide. So, once again, if we aggregate two ixgbe NICs, each with 32 groups, the aggr will only make use of one group and the other 31 will go to waste. The upshot of this is that an aggr can provide hardware classification for only one client. The moment there are two or more clients all traffic going over the aggr must be software classified and performance improvements like polling mode are lost.

The purpose of this ticket is to track the work of adding multiple Rx pseudo group support to aggr.


Comment by Former user
Created at 2018-03-28T03:33:22.716Z

Testing notes:

On my local workstation running a DEBUG kernel I ran the following tests. Each of these tests was run in 3 scenarios: 1) two ixgbe aggrs with active LACP and default (1500) MTU, 2) two ixgbe aggrs with active LACP and 9000 MTU, 3) mixed aggr of ixgbe/igb with LACP off.

I also ran a test where I would continously ping for a bogus IP to generate L2 multicast traffic. I would do this while booting a server with aggrs and make sure that the broadcast traffic didn't interfere with the creation of the aggr like we saw in OS-6697.

Finally, I ran several tests on a JPC CN. I booted a JPC CN on these aggr bits and then created two VMs: one KVM (KVM_A), one SmartOS container (SOS_A). I put both VMs on the external and fabric (overlay) networks. I then created a SmartOS container (SOS_B) on another CN. Then I ran the following tests:

Comment by Jira Bot
Created at 2018-03-28T16:17:28.236Z

illumos-joyent commit 4328d0a48dfc91c96c425a121cc858f78e9365d2 (branch master, by Ryan Zezeski)

OS-6719 aggr needs support for multiple pseudo rx groups
Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
Reviewed by: Jerry Jelinek <jerry.jelinek@joyent.com>
Reviewed by: Robert Mustacchi <rm@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>