OS-7329: Want pluggable TCP congestion control algorithms

Details

Issue Type:Bug
Priority:4 - Normal
Status:Resolved
Created at:2018-10-25T21:01:02.529Z
Updated at:2019-10-28T20:41:19.655Z

People

Created by:Former user
Reported by:Former user
Assigned to:Former user

Resolution

Fixed: A fix for this issue is checked into the tree and tested.
(Resolution Date: 2019-08-09T22:55:17.506Z)

Fix Versions

2019-08-15 YouFace (Release Date: 2019-08-15)

Related Issues

Description

As part of TRITON-885, we will want to port the following commits from Delphix in order to get pluggable TCP congestion control algorithms:

This will give us three different congestion control algorithms:

Comments

Comment by Former user
Created at 2018-12-10T20:21:04.084Z
To compare the difference in behaviour between these algorithms, I set up an instance in eu-ams-1 and ran iperf there:

cpm@enlil ~ % ssh nehalennia 
Enter passphrase for key '/home/cpm/.ssh/id_ecdsa': 
   __        .                   .
 _|  |_      | .-. .  . .-. :--. |-
|_    _|     ;|   ||  |(.-' |  | |
  |__|   `--'  `-' `;-| `-' '  ' `-'
                   /  ; Instance (minimal-64 18.3.0)
                   `-'  https://docs.joyent.com/images/smartos/minimal

[root@nehalennia ~]# 
[root@nehalennia ~]# 
[root@nehalennia ~]# 
[root@nehalennia ~]# iperf3 -s -p 11845
-----------------------------------------------------------
Server listening on 11845
-----------------------------------------------------------

I then set up an LX zone in staging on a PI with the TRITON-885 changes.

Running cubic:

root@5a0d3532-1b43-4459-f8e4-b5fc36a7cd26:~# iperf3 -C cubic -c 37.153.108.173 -p 11845 -t 120
Connecting to host 37.153.108.173, port 11845
[  4] local 172.26.7.5 port 35815 connected to 37.153.108.173 port 11845
<snip>
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-120.00 sec   742 MBytes  51.9 Mbits/sec    0             sender
[  4]   0.00-120.00 sec   742 MBytes  51.9 Mbits/sec                  receiver

iperf Done.
root@5a0d3532-1b43-4459-f8e4-b5fc36a7cd26:~# iperf3 -C cubic -c 37.153.108.173 -p 11845 -t 120
Connecting to host 37.153.108.173, port 11845
[  4] local 172.26.7.5 port 43952 connected to 37.153.108.173 port 11845
<snip>
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-120.00 sec   733 MBytes  51.2 Mbits/sec    0             sender
[  4]   0.00-120.00 sec   733 MBytes  51.2 Mbits/sec                  receiver

iperf Done.

In a separate window, I ran connstat so that I could see how the congestion window changed over time (note that the first TCP connection shown here is the iperf control connection, used primarily for exchanging test parameters):

root@5a0d3532-1b43-4459-f8e4-b5fc36a7cd26:~# /native/usr/bin/connstat  -o laddr,lport,raddr,rport,rto,rtt,cwnd,retransbytes -i 2 -F rport=11845,state=established
<snip>
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  50396  37.153.108.173  11845      582   146029        8688             0
     172.26.7.5  35815  37.153.108.173  11845      577   141839     4019816       1259344
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  50396  37.153.108.173  11845      582   146029        8688             0
     172.26.7.5  35815  37.153.108.173  11845      579   143791     3706240       1285408
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  50396  37.153.108.173  11845      582   146029        8688             0
     172.26.7.5  35815  37.153.108.173  11845      576   140837     3706240       1285408
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  50396  37.153.108.173  11845      582   146029        8688             0
     172.26.7.5  35815  37.153.108.173  11845      575   140171     4175448       1312920
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  50396  37.153.108.173  11845      582   146029        8688             0
     172.26.7.5  35815  37.153.108.173  11845      575   140712     4460648       1320160
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  50396  37.153.108.173  11845      582   146029        8688             0
     172.26.7.5  35815  37.153.108.173  11845      579   143357     1873128       1391112
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  50396  37.153.108.173  11845      582   146029        8688             0
     172.26.7.5  35815  37.153.108.173  11845      578   142766     2971352       1474400
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  50396  37.153.108.173  11845      582   146029        8688             0
     172.26.7.5  35815  37.153.108.173  11845      577   142108     3236392       1530872
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  50396  37.153.108.173  11845      582   146029        8688             0
     172.26.7.5  35815  37.153.108.173  11845      575   140121     2759360       1598928
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  50396  37.153.108.173  11845      582   146029        8688             0
     172.26.7.5  35815  37.153.108.173  11845      578   142629     2577202       1606168
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  50396  37.153.108.173  11845      582   146029        8688             0
     172.26.7.5  35815  37.153.108.173  11845      578   142966     3342736       1669880
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  50396  37.153.108.173  11845      582   146029        8688             0
     172.26.7.5  35815  37.153.108.173  11845      576   140972      523536       1682216
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  50396  37.153.108.173  11845      582   146029        8688             0
     172.26.7.5  35815  37.153.108.173  11845      577   142122     1042072       1771296
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  50396  37.153.108.173  11845      582   146029        8688             0
     172.26.7.5  35815  37.153.108.173  11845      592   153909     2164816       1811840
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  50396  37.153.108.173  11845      582   146029        8688             0
     172.26.7.5  35815  37.153.108.173  11845      575   140043     3111056       1854584
<snip>
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  53231  37.153.108.173  11845      582   146074        8688             0
     172.26.7.5  43952  37.153.108.173  11845      575   140360     3405112       2668808
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  53231  37.153.108.173  11845      582   146074        8688             0
     172.26.7.5  43952  37.153.108.173  11845      574   139796     3405112       2668808
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  53231  37.153.108.173  11845      582   146074        8688             0
     172.26.7.5  43952  37.153.108.173  11845      575   140435     3476760       2694872
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  53231  37.153.108.173  11845      582   146074        8688             0
     172.26.7.5  43952  37.153.108.173  11845      573   139120     3473224       2707904
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  53231  37.153.108.173  11845      582   146074        8688             0
     172.26.7.5  43952  37.153.108.173  11845      576   140926     3703456       2733968
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  53231  37.153.108.173  11845      582   146074        8688             0
     172.26.7.5  43952  37.153.108.173  11845      576   141463     1874576       2743408
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  53231  37.153.108.173  11845      582   146074        8688             0
     172.26.7.5  43952  37.153.108.173  11845      574   139984     2509440       2760784
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  53231  37.153.108.173  11845      582   146074        8688             0
     172.26.7.5  43952  37.153.108.173  11845      576   140865     1052168       2803528
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  53231  37.153.108.173  11845      582   146074        8688             0
     172.26.7.5  43952  37.153.108.173  11845      576   141182     2833904       2936104
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  53231  37.153.108.173  11845      582   146074        8688             0
     172.26.7.5  43952  37.153.108.173  11845      592   154087     1317152       2979544
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  53231  37.153.108.173  11845      582   146074        8688             0
     172.26.7.5  43952  37.153.108.173  11845      589   151397     1665175       2985336
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  53231  37.153.108.173  11845      582   146074        8688             0
     172.26.7.5  43952  37.153.108.173  11845      576   141202     1926704       3028080
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  53231  37.153.108.173  11845      582   146074        8688             0
     172.26.7.5  43952  37.153.108.173  11845      574   139290      704592       3041112
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  53231  37.153.108.173  11845      582   146074        8688             0
     172.26.7.5  43952  37.153.108.173  11845      576   140970     2995328       3087504
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  53231  37.153.108.173  11845      582   146074        8688             0
     172.26.7.5  43952  37.153.108.173  11845      598   152554     2960632       3123008

Running newreno:

root@5a0d3532-1b43-4459-f8e4-b5fc36a7cd26:~# iperf3 -C newreno -c 37.153.108.173 -p 11845 -t 120
Connecting to host 37.153.108.173, port 11845
[  4] local 172.26.7.5 port 39532 connected to 37.153.108.173 port 11845
<snip>
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-120.00 sec   341 MBytes  23.8 Mbits/sec    0             sender
[  4]   0.00-120.00 sec   341 MBytes  23.8 Mbits/sec                  receiver

iperf Done.
root@5a0d3532-1b43-4459-f8e4-b5fc36a7cd26:~# iperf3 -C newreno -c 37.153.108.173 -p 11845 -t 120
Connecting to host 37.153.108.173, port 11845
[  4] local 172.26.7.5 port 43926 connected to 37.153.108.173 port 11845
<snip>
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-120.00 sec   159 MBytes  11.1 Mbits/sec    0             sender
[  4]   0.00-120.00 sec   159 MBytes  11.1 Mbits/sec                  receiver

iperf Done.

And we can see that its congestion window isn't getting as large as it did for cubic:

root@5a0d3532-1b43-4459-f8e4-b5fc36a7cd26:~# /native/usr/bin/connstat  -o laddr,lport,raddr,rport,rto,rtt,cwnd,retransbytes -i 2 -F rport=11845,state=established
<snip>
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  57185  37.153.108.173  11845      582   145770        8688             0
     172.26.7.5  39532  37.153.108.173  11845      575   140487      137560        343176
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  57185  37.153.108.173  11845      582   145770        8688             0
     172.26.7.5  39532  37.153.108.173  11845      574   139703      159280        343176
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  57185  37.153.108.173  11845      582   145770        8688             0
     172.26.7.5  39532  37.153.108.173  11845      582   145792      179552        343176
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  57185  37.153.108.173  11845      582   145770        8688             0
     172.26.7.5  39532  37.153.108.173  11845      579   143680      199824        343176
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  57185  37.153.108.173  11845      582   145770        8688             0
     172.26.7.5  39532  37.153.108.173  11845      574   139704      309872        347520
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  57185  37.153.108.173  11845      582   145770        8688             0
     172.26.7.5  39532  37.153.108.173  11845      574   139984      330144        347520
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  57185  37.153.108.173  11845      582   145770        8688             0
     172.26.7.5  39532  37.153.108.173  11845      577   141950      350416        347520
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  57185  37.153.108.173  11845      582   145770        8688             0
     172.26.7.5  39532  37.153.108.173  11845      576   141308      370688        347520
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  57185  37.153.108.173  11845      582   145770        8688             0
     172.26.7.5  39532  37.153.108.173  11845      575   140222      390960        347520
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  57185  37.153.108.173  11845      582   145770        8688             0
     172.26.7.5  39532  37.153.108.173  11845      577   142147      214304        348968
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  57185  37.153.108.173  11845      582   145770        8688             0
     172.26.7.5  39532  37.153.108.173  11845      577   141908      234576        348968
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  57185  37.153.108.173  11845      582   145770        8688             0
     172.26.7.5  39532  37.153.108.173  11845      574   139818      256296        348968
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  57185  37.153.108.173  11845      582   145770        8688             0
     172.26.7.5  39532  37.153.108.173  11845      575   140727      276568        348968
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  57185  37.153.108.173  11845      582   145770        8688             0
     172.26.7.5  39532  37.153.108.173  11845      575   140296      157832        350416
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  57185  37.153.108.173  11845      582   145770        8688             0
     172.26.7.5  39532  37.153.108.173  11845      575   140052      178104        350416
<snip>
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  48925  37.153.108.173  11845      582   145713        8688             0
     172.26.7.5  43926  37.153.108.173  11845      573   139101      111496         81840
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  48925  37.153.108.173  11845      582   145713        8688             0
     172.26.7.5  43926  37.153.108.173  11845      573   138994      133216         81840
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  48925  37.153.108.173  11845      582   145713        8688             0
     172.26.7.5  43926  37.153.108.173  11845      580   144224      153488         81840
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  48925  37.153.108.173  11845      582   145713        8688             0
     172.26.7.5  43926  37.153.108.173  11845      574   139411      233184         84736
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  48925  37.153.108.173  11845      582   145713        8688             0
     172.26.7.5  43926  37.153.108.173  11845      573   139074      267184         94176
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  48925  37.153.108.173  11845      582   145713        8688             0
     172.26.7.5  43926  37.153.108.173  11845      573   138985      287456         94176
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  48925  37.153.108.173  11845      582   145713        8688             0
     172.26.7.5  43926  37.153.108.173  11845      573   139049      309176         94176
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  48925  37.153.108.173  11845      582   145713        8688             0
     172.26.7.5  43926  37.153.108.173  11845      574   139651      551048        107208
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  48925  37.153.108.173  11845      582   145713        8688             0
     172.26.7.5  43926  37.153.108.173  11845      590   152651      208512        107208
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  48925  37.153.108.173  11845      582   145713        8688             0
     172.26.7.5  43926  37.153.108.173  11845      573   139022      230232        107208
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  48925  37.153.108.173  11845      582   145713        8688             0
     172.26.7.5  43926  37.153.108.173  11845      573   139182      258496        137616
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  48925  37.153.108.173  11845      582   145713        8688             0
     172.26.7.5  43926  37.153.108.173  11845      573   138916      278768        137616
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  48925  37.153.108.173  11845      582   145713        8688             0
     172.26.7.5  43926  37.153.108.173  11845      573   138973      300488        137616
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  48925  37.153.108.173  11845      582   145713        8688             0
     172.26.7.5  43926  37.153.108.173  11845      581   145221      320760        137616
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  48925  37.153.108.173  11845      582   145713        8688             0
     172.26.7.5  43926  37.153.108.173  11845      573   138859      175208        139064

Running with sunreno, which we'd expect to behave pretty similarly to the newreno module:

root@5a0d3532-1b43-4459-f8e4-b5fc36a7cd26:~# iperf3 -C sunreno -c 37.153.108.173 -p 11845 -t 120
Connecting to host 37.153.108.173, port 11845
[  4] local 172.26.7.5 port 45699 connected to 37.153.108.173 port 11845
<snip>
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-120.00 sec   238 MBytes  16.6 Mbits/sec    0             sender
[  4]   0.00-120.00 sec   238 MBytes  16.6 Mbits/sec                  receiver

iperf Done.
root@5a0d3532-1b43-4459-f8e4-b5fc36a7cd26:~# iperf3 -C sunreno -c 37.153.108.173 -p 11845 -t 120
Connecting to host 37.153.108.173, port 11845
[  4] local 172.26.7.5 port 51117 connected to 37.153.108.173 port 11845
<snip>
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-120.00 sec   543 MBytes  37.9 Mbits/sec    0             sender
[  4]   0.00-120.00 sec   543 MBytes  37.9 Mbits/sec                  receiver

iperf Done.

And looking at its congestion window:

root@5a0d3532-1b43-4459-f8e4-b5fc36a7cd26:~# /native/usr/bin/connstat  -o laddr,lport,raddr,rport,rto,rtt,cwnd,retransbytes -i 2 -F rport=11845,state=established
<snip>
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  58124  37.153.108.173  11845      582   146046        8688             0
     172.26.7.5  45699  37.153.108.173  11845      582   146080      737784        108600
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  58124  37.153.108.173  11845      582   146046        8688             0
     172.26.7.5  45699  37.153.108.173  11845      575   140742      375032        110048
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  58124  37.153.108.173  11845      582   146046        8688             0
     172.26.7.5  45699  37.153.108.173  11845      575   140015      198376        111496
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  58124  37.153.108.173  11845      582   146046        8688             0
     172.26.7.5  45699  37.153.108.173  11845      576   141151      208512        111496
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  58124  37.153.108.173  11845      582   146046        8688             0
     172.26.7.5  45699  37.153.108.173  11845      576   141283      218648        111496
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  58124  37.153.108.173  11845      582   146046        8688             0
     172.26.7.5  45699  37.153.108.173  11845      574   139849      228784        111496
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  58124  37.153.108.173  11845      582   146046        8688             0
     172.26.7.5  45699  37.153.108.173  11845      583   146549      348272        115840
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  58124  37.153.108.173  11845      582   146046        8688             0
     172.26.7.5  45699  37.153.108.173  11845      575   140008      358408        115840
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  58124  37.153.108.173  11845      582   146046        8688             0
     172.26.7.5  45699  37.153.108.173  11845      577   141656      368544        115840
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  58124  37.153.108.173  11845      582   146046        8688             0
     172.26.7.5  45699  37.153.108.173  11845      575   140312      378680        115840
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  58124  37.153.108.173  11845      582   146046        8688             0
     172.26.7.5  45699  37.153.108.173  11845      574   139869      388816        115840
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  58124  37.153.108.173  11845      582   146046        8688             0
     172.26.7.5  45699  37.153.108.173  11845      578   142863      400400        115840
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  58124  37.153.108.173  11845      582   146046        8688             0
     172.26.7.5  45699  37.153.108.173  11845      581   144957      410536        115840
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  58124  37.153.108.173  11845      582   146046        8688             0
     172.26.7.5  45699  37.153.108.173  11845      578   142558      349024        120184
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  58124  37.153.108.173  11845      582   146046        8688             0
     172.26.7.5  45699  37.153.108.173  11845      580   144717      263536        167968
<snip>
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      575   140452      863872        196984
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      577   142088     1049800        225944
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      576   141583     1049800        275176
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      575   140555     1049800        292552
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      575   140614     1049800        292552
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      576   140851     1049800        302688
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      577   141991     1049800        315720
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      575   140332      529968        317168
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      575   140138      393856        325856
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      575   140095      403992        325856
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      576   140983      416328        328752
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      576   140988      318560        330200
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      575   140495      598832        351920
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      576   141412      879048        363504
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      576   141068      638680        370744
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      593   154635     1049800        379432
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      599   153676     1049800        395360
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      578   142629     1049800        395360
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      575   140113     1049800        395360
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      575   140507     1049800        406944
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      580   144550      519832        422176
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      576   141581      787072        425072
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      576   140862     1035432        444648
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      595   153950     1049800        459128
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      575   140258     1142640        484496
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      576   140862     1049800        546760
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      577   142154      990544        689416
          LADDR  LPORT           RADDR  RPORT      RTO      RTT        CWND  RETRANSBYTES
     172.26.7.5  34746  37.153.108.173  11845      582   145700        8688             0
     172.26.7.5  51117  37.153.108.173  11845      57

Comment by Former user
Created at 2018-12-11T00:28:13.211Z

When there's little packet loss between the source host and Amsterdam, all of the algorithms behave pretty similarly, and move towards a similar window size. Using high -P values to create multiple flows along the same path helps with observing how they adjust their windows over time:

[root@test1 ~/iperf/src]# ./.libs/iperf3-static -C cubic -c 37.153.108.173 -p 11845 -t 120 -P 32
<snip>
[SUM]   0.00-120.00 sec  24.6 GBytes  1.76 Gbits/sec                  sender
[SUM]   0.00-120.00 sec  24.6 GBytes  1.76 Gbits/sec                  receiver

iperf Done.
[root@test1 ~/iperf/src]# ./.libs/iperf3-static -C cubic -c 37.153.108.173 -p 11845 -t 120 -P 32
<snip>
[SUM]   0.00-120.00 sec  22.2 GBytes  1.59 Gbits/sec                  sender
[SUM]   0.00-120.00 sec  22.2 GBytes  1.59 Gbits/sec                  receiver

iperf Done.
[root@test1 ~/iperf/src]# ./.libs/iperf3-static -C sunreno -c 37.153.108.173 -p 11845 -t 120 -P 32
<snip>
[SUM]   0.00-120.00 sec  17.3 GBytes  1.24 Gbits/sec                  sender
[SUM]   0.00-120.00 sec  17.3 GBytes  1.24 Gbits/sec                  receiver

iperf Done.
[root@test1 ~/iperf/src]# ./.libs/iperf3-static -C sunreno -c 37.153.108.173 -p 11845 -t 120 -P 32
<snip>
[SUM]   0.00-120.00 sec  15.2 GBytes  1.09 Gbits/sec                  sender
[SUM]   0.00-120.00 sec  15.2 GBytes  1.09 Gbits/sec                  receiver

iperf Done.
[root@test1 ~/iperf/src]# ./.libs/iperf3-static -C newreno -c 37.153.108.173 -p 11845 -t 120 -P 32
<snip>
[SUM]   0.00-120.00 sec  18.8 GBytes  1.34 Gbits/sec                  sender
[SUM]   0.00-120.00 sec  18.8 GBytes  1.34 Gbits/sec                  receiver

iperf Done.
[root@test1 ~/iperf/src]# ./.libs/iperf3-static -C newreno -c 37.153.108.173 -p 11845 -t 120 -P 32
<snip>
[SUM]   0.00-120.00 sec  19.2 GBytes  1.37 Gbits/sec                  sender
[SUM]   0.00-120.00 sec  19.2 GBytes  1.37 Gbits/sec                  receiver

iperf Done.

Running a finisher cubic:

[root@test1 ~/iperf/src]# ./.libs/iperf3-static -C cubic -c 37.153.108.173 -p 11845 -t 120 -P 32
<snip>
[SUM]   0.00-120.00 sec  23.3 GBytes  1.67 Gbits/sec                  sender
[SUM]   0.00-120.00 sec  23.3 GBytes  1.67 Gbits/sec                  receiver

iperf Done.

Comment by Former user
Created at 2019-08-09T21:02:59.172Z

I redid the above testing in the lab, and got similar results. The cubic_ticks algorithm below is a module I built for testing based on an earlier patchset to help show that the switch made during review from measuring elapsed time in ticks tonanoseconds didn't break anything.

Using -t 60 -P 16

AlgorithmRun 1Run 2Run 3Run 4Run 5Average
cubic_ticks926 Mbits/sec937 Mbits/sec915 Mbits/sec884 Mbits/sec908 Mbits/sec 914 Mbits/sec
cubic900 Mbits/sec918 Mbits/sec877 Mbits/sec920 Mbits/sec935 Mbits/sec 910 Mbits/sec
sunreno905 Mbits/sec786 Mbits/sec796 Mbits/sec801 Mbits/sec771 Mbits/sec 631.8 Mbits/sec
newreno613 Mbits/sec634 Mbits/sec707 Mbits/sec692 Mbits/sec525 Mbits/sec 634.2 Mbits/sec

Using -t 120 -P 32

AlgorithmRun 1Run 2Run 3Run 4Run 5Average
cubic_ticks1.35 Gbits/sec1.14 Gbits/sec1.48 Gbits/sec1.59 Gbits/sec1.49 Gbits/sec 1.41 Gbits/sec
cubic1.46 Gbits/sec1.42 Gbits/sec1.53 Gbits/sec1.54 Gbits/sec1.59 Gbits/sec 1.508 Gbits/sec
sunreno1.31 Gbits/sec1.26 Gbits/sec1.40 Gbits/sec1.50 Gbits/sec 1.37 Gbits/sec 1.368 Gbits/sec
newreno1.40 Gbits/sec1.41 Gbits/sec1.26 Gbits/sec 1.48 Gbits/sec 1.27 Gbits/sec 1.364 Gbits/sec

Comment by Former user
Created at 2019-08-09T21:06:29.882Z
Updated at 2019-08-09T21:08:33.897Z

You can see that ipadm allows listing and updating the algorithm used:

[root@test1 ~/iperf]# ipadm show-prop tcp -p congestion_control
PROTO PROPERTY              PERM CURRENT      PERSISTENT   DEFAULT      POSSIBLE
tcp   congestion_control    rw   sunreno      sunreno      sunreno      sunreno,newreno,
                                                                        cubic
[root@test1 ~/iperf]# ipadm set-prop tcp -p congestion_control=cubic
[root@test1 ~/iperf]# ipadm show-prop tcp -p congestion_control
PROTO PROPERTY              PERM CURRENT      PERSISTENT   DEFAULT      POSSIBLE
tcp   congestion_control    rw   cubic        cubic        sunreno      sunreno,newreno,
                                                                        cubic
[root@test1 ~/iperf]# ipadm set-prop tcp -p congestion_control=newreno -t
[root@test1 ~/iperf]# ipadm show-prop tcp -p congestion_control
PROTO PROPERTY              PERM CURRENT      PERSISTENT   DEFAULT      POSSIBLE
tcp   congestion_control    rw   newreno      cubic        sunreno      sunreno,newreno,
                                                                        cubic

After using modload to load the cubic_ticks module that I prepared for testing:

[root@test1 ~/iperf]# ipadm show-prop tcp -p congestion_control
PROTO PROPERTY              PERM CURRENT      PERSISTENT   DEFAULT      POSSIBLE
tcp   congestion_control    rw   newreno      cubic        sunreno      sunreno,newreno,
                                                                        cubic,
                                                                        cubic_ticks

Comment by Jira Bot
Created at 2019-08-09T22:54:46.157Z

illumos-joyent commit 9cba20af356f81a5f469ca8aa89ed17ee827b6a6 (branch master, by Sebastien Roy)

OS-7329 Want pluggable TCP congestion control algorithms
Reviewed by: Dan McDonald <danmcd@joyent.com>
Reviewed by: Robert Mustacchi <robert.mustacchi@joyent.com>
Approved by: Robert Mustacchi <robert.mustacchi@joyent.com>