OS-7334: Track TCP round-trip time in nanoseconds

Details

Issue Type:Bug
Priority:4 - Normal
Status:Resolved
Created at:2018-10-30T16:44:10.227Z
Updated at:2019-04-01T16:10:25.277Z

People

Created by:Former user
Reported by:Former user
Assigned to:Former user

Resolution

Fixed: A fix for this issue is checked into the tree and tested.
(Resolution Date: 2019-03-29T20:59:00.484Z)

Fix Versions

2019-04-11 Pete Hornberger (Release Date: 2019-04-11)

Description

As part of porting Delphix's connstat(1M) changes in OS-7327, we'll want to pull in their updates to TCP to use nanosecond timestamps for RTT instead of microseconds. This includes changes in DLPX-42721 and DLPX-43064.

This code initially had ifdef checks for KERNEL_32. As far as I can tell, this was never defined, even before illumos#8685, so I've removed them.

It's also worth calling out that this change will make TCP stop using echoed timestamps for calculating RTT. When upstreamed, this can be used to close out illumos#8725.

Comments

Comment by Former user
Created at 2019-03-27T18:50:35.658Z

You can see that the RTT is now measured in nanoseconds by using connstat(1M) to report the current RTT for the connection (RTT column is in nanoseconds, and RTO column is in milliseconds):

[root@smartos /var/tmp]# connstat -4 -i 1 -o raddr,rport,rtt,rto | grep -v -e 127.0.0.1 -e 0.0.0.0
RADDR                     RPORT RTT     RTO
165.225.164.14            80    446584  1360
10.88.88.2                42936 3620    404
RADDR                     RPORT RTT     RTO
165.225.164.14            80    446584  1360
10.88.88.2                42936 5076    406
RADDR                     RPORT RTT     RTO
165.225.164.14            80    446584  1360
10.88.88.2                42936 2054    402
RADDR                     RPORT RTT     RTO
165.225.164.14            80    446584  1360
10.88.88.2                42936 1037    401
RADDR                     RPORT RTT     RTO
165.225.164.14            80    446584  1360
10.88.88.2                42936 712     400
RADDR                     RPORT RTT     RTO
165.225.164.14            80    446584  1360
10.88.88.2                42936 667     400
RADDR                     RPORT RTT     RTO
165.225.164.14            80    446584  1360
10.88.88.2                42936 963     401
RADDR                     RPORT RTT     RTO
165.225.164.14            80    446584  1360
10.88.88.2                42936 594     400
RADDR                     RPORT RTT     RTO
165.225.164.14            80    446584  1360
10.88.88.2                42936 502     400
RADDR                     RPORT RTT     RTO
165.225.164.14            80    446584  1360
10.88.88.2                42936 566     400
RADDR                     RPORT RTT     RTO
165.225.164.14            80    446584  1360
10.88.88.2                42936 482     400

The 165.225.164.14 connection is a background curl to Manta, and has an SRTT of 446.584 milliseconds. The 10.88.88.2 connection is my SSH connection from my laptop into my COAL (running on the same laptop), and has an SRTT as low as 0.482 milliseconds at one point, a sub-millisecond RTT that would have previously been unrepresentable except as 0 or 1.


Comment by Former user
Created at 2019-03-27T21:19:01.089Z

I stood up a COAL with these changes. I was able to SSH in to it, curl a file from Manta, and download a new CNAPI image from updates.joyent.com with sdcadm, all without issue.


Comment by Former user
Created at 2019-03-29T01:11:06.304Z

These changes allow for measuring a more precise RTT, but that shouldn't have any impact on the performance. To verify this, I ran some iperf tests between two systems with 40GiB Chelsio NICs. On a recent PI (joyent_20190328T010321Z):

[root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120
Connecting to host 10.200.200.202, port 11845
[  5] local 10.200.200.201 port 63240 connected to 10.200.200.202 port 11845
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   425 MBytes  3.56 Gbits/sec
[  5]   1.00-2.00   sec   693 MBytes  5.81 Gbits/sec
[  5]   2.00-3.00   sec   450 MBytes  3.78 Gbits/sec
[  5]   3.00-4.00   sec   658 MBytes  5.52 Gbits/sec
[  5]   4.00-5.00   sec   455 MBytes  3.82 Gbits/sec
[  5]   5.00-6.00   sec   450 MBytes  3.78 Gbits/sec
[  5]   6.00-7.00   sec  1.91 GBytes  16.4 Gbits/sec
[  5]   7.00-8.00   sec   452 MBytes  3.79 Gbits/sec
[  5]   8.00-9.00   sec   478 MBytes  4.01 Gbits/sec
[  5]   9.00-10.00  sec   450 MBytes  3.77 Gbits/sec
[  5]  10.00-11.00  sec  1.17 GBytes  10.0 Gbits/sec
[  5]  11.00-12.00  sec   635 MBytes  5.33 Gbits/sec
[  5]  12.00-13.00  sec   572 MBytes  4.79 Gbits/sec
<snip>
[  5] 100.00-101.00 sec  1.94 GBytes  16.6 Gbits/sec
[  5] 101.00-102.00 sec  1.93 GBytes  16.5 Gbits/sec
[  5] 102.00-103.00 sec  1.94 GBytes  16.6 Gbits/sec
[  5] 103.00-104.00 sec  1.94 GBytes  16.6 Gbits/sec
[  5] 104.00-105.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 105.00-106.00 sec  1.93 GBytes  16.6 Gbits/sec
[  5] 106.00-107.00 sec  1.93 GBytes  16.6 Gbits/sec
[  5] 107.00-108.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 108.00-109.00 sec  1.93 GBytes  16.6 Gbits/sec
[  5] 109.00-110.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 110.00-111.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 111.00-112.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 112.00-113.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 113.00-114.00 sec  1.93 GBytes  16.6 Gbits/sec
[  5] 114.00-115.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 115.00-116.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 116.00-117.00 sec  1.91 GBytes  16.4 Gbits/sec
[  5] 117.00-118.00 sec  1.91 GBytes  16.4 Gbits/sec
[  5] 118.00-119.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 119.00-120.00 sec  1.93 GBytes  16.6 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-120.00 sec   157 GBytes  11.2 Gbits/sec                  sender
[  5]   0.00-120.00 sec   157 GBytes  11.2 Gbits/sec                  receiver

iperf Done.
[root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120
Connecting to host 10.200.200.202, port 11845
[  5] local 10.200.200.201 port 53375 connected to 10.200.200.202 port 11845
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   421 MBytes  3.53 Gbits/sec
[  5]   1.00-2.00   sec  1.17 GBytes  10.1 Gbits/sec
[  5]   2.00-3.00   sec   844 MBytes  7.08 Gbits/sec
[  5]   3.00-4.00   sec   516 MBytes  4.33 Gbits/sec
[  5]   4.00-5.00   sec   653 MBytes  5.47 Gbits/sec
[  5]   5.00-6.00   sec   712 MBytes  5.97 Gbits/sec
[  5]   6.00-7.00   sec  1.92 GBytes  16.5 Gbits/sec
[  5]   7.00-8.00   sec  1.96 GBytes  16.8 Gbits/sec
[  5]   8.00-9.00   sec  1.96 GBytes  16.8 Gbits/sec
[  5]   9.00-10.00  sec  1.95 GBytes  16.8 Gbits/sec
[  5]  10.00-11.00  sec  1.97 GBytes  16.9 Gbits/sec
[  5]  11.00-12.00  sec  1.97 GBytes  17.0 Gbits/sec
[  5]  12.00-13.00  sec  1.98 GBytes  17.0 Gbits/sec
[  5]  13.00-14.00  sec  1.98 GBytes  17.0 Gbits/sec
<snip>
[  5] 100.00-101.00 sec  1.97 GBytes  16.9 Gbits/sec
[  5] 101.00-102.00 sec  1.97 GBytes  16.9 Gbits/sec
[  5] 102.00-103.00 sec  1.97 GBytes  16.9 Gbits/sec
[  5] 103.00-104.00 sec  1.97 GBytes  16.9 Gbits/sec
[  5] 104.00-105.00 sec  1.97 GBytes  16.9 Gbits/sec
[  5] 105.00-106.00 sec  1.97 GBytes  17.0 Gbits/sec
[  5] 106.00-107.00 sec  1.97 GBytes  16.9 Gbits/sec
[  5] 107.00-108.00 sec  1.95 GBytes  16.8 Gbits/sec
[  5] 108.00-109.00 sec  1.96 GBytes  16.9 Gbits/sec
[  5] 109.00-110.00 sec  1.96 GBytes  16.9 Gbits/sec
[  5] 110.00-111.00 sec  1.97 GBytes  17.0 Gbits/sec
[  5] 111.00-112.00 sec  1.98 GBytes  17.0 Gbits/sec
[  5] 112.00-113.00 sec  1.96 GBytes  16.9 Gbits/sec
[  5] 113.00-114.00 sec  1.95 GBytes  16.8 Gbits/sec
[  5] 114.00-115.00 sec  1.95 GBytes  16.8 Gbits/sec
[  5] 115.00-116.00 sec  1.95 GBytes  16.8 Gbits/sec
[  5] 116.00-117.00 sec  1.95 GBytes  16.8 Gbits/sec
[  5] 117.00-118.00 sec  1.95 GBytes  16.8 Gbits/sec
[  5] 118.00-119.00 sec  1.95 GBytes  16.8 Gbits/sec
[  5] 119.00-120.00 sec  1.96 GBytes  16.8 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-120.00 sec   228 GBytes  16.3 Gbits/sec                  sender
[  5]   0.00-120.00 sec   228 GBytes  16.3 Gbits/sec                  receiver

iperf Done.
[root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120 -P 4
Connecting to host 10.200.200.202, port 11845
[  5] local 10.200.200.201 port 64322 connected to 10.200.200.202 port 11845
[  7] local 10.200.200.201 port 35299 connected to 10.200.200.202 port 11845
[  9] local 10.200.200.201 port 51000 connected to 10.200.200.202 port 11845
[ 11] local 10.200.200.201 port 36272 connected to 10.200.200.202 port 11845
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   637 MBytes  5.34 Gbits/sec
[  7]   0.00-1.00   sec   835 MBytes  7.01 Gbits/sec
[  9]   0.00-1.00   sec   417 MBytes  3.50 Gbits/sec
[ 11]   0.00-1.00   sec   630 MBytes  5.28 Gbits/sec
[SUM]   0.00-1.00   sec  2.46 GBytes  21.1 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   659 MBytes  5.53 Gbits/sec
[  7]   1.00-2.00   sec   734 MBytes  6.15 Gbits/sec
[  9]   1.00-2.00   sec   655 MBytes  5.50 Gbits/sec
[ 11]   1.00-2.00   sec   612 MBytes  5.13 Gbits/sec
[SUM]   1.00-2.00   sec  2.60 GBytes  22.3 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   626 MBytes  5.25 Gbits/sec
[  7]   2.00-3.00   sec   693 MBytes  5.81 Gbits/sec
[  9]   2.00-3.00   sec   620 MBytes  5.20 Gbits/sec
[ 11]   2.00-3.00   sec   864 MBytes  7.25 Gbits/sec
[SUM]   2.00-3.00   sec  2.74 GBytes  23.5 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
<snip>
[  5] 118.00-119.00 sec   616 MBytes  5.17 Gbits/sec
[  7] 118.00-119.00 sec   655 MBytes  5.49 Gbits/sec
[  9] 118.00-119.00 sec   602 MBytes  5.05 Gbits/sec
[ 11] 118.00-119.00 sec   736 MBytes  6.17 Gbits/sec
[SUM] 118.00-119.00 sec  2.55 GBytes  21.9 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5] 119.00-120.00 sec   614 MBytes  5.16 Gbits/sec
[  7] 119.00-120.00 sec   688 MBytes  5.78 Gbits/sec
[  9] 119.00-120.00 sec   599 MBytes  5.03 Gbits/sec
[ 11] 119.00-120.00 sec   764 MBytes  6.42 Gbits/sec
[SUM] 119.00-120.00 sec  2.60 GBytes  22.4 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-120.00 sec  71.0 GBytes  5.08 Gbits/sec                  sender
[  5]   0.00-120.00 sec  71.0 GBytes  5.08 Gbits/sec                  receiver
[  7]   0.00-120.00 sec  80.6 GBytes  5.77 Gbits/sec                  sender
[  7]   0.00-120.00 sec  80.6 GBytes  5.77 Gbits/sec                  receiver
[  9]   0.00-120.00 sec  70.4 GBytes  5.04 Gbits/sec                  sender
[  9]   0.00-120.00 sec  70.4 GBytes  5.04 Gbits/sec                  receiver
[ 11]   0.00-120.00 sec  89.2 GBytes  6.38 Gbits/sec                  sender
[ 11]   0.00-120.00 sec  89.2 GBytes  6.38 Gbits/sec                  receiver
[SUM]   0.00-120.00 sec   311 GBytes  22.3 Gbits/sec                  sender
[SUM]   0.00-120.00 sec   311 GBytes  22.3 Gbits/sec                  receiver

iperf Done.
[root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120 -P 4
Connecting to host 10.200.200.202, port 11845
[  5] local 10.200.200.201 port 41045 connected to 10.200.200.202 port 11845
[  7] local 10.200.200.201 port 51532 connected to 10.200.200.202 port 11845
[  9] local 10.200.200.201 port 55819 connected to 10.200.200.202 port 11845
[ 11] local 10.200.200.201 port 49547 connected to 10.200.200.202 port 11845
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   740 MBytes  6.21 Gbits/sec                  
[  7]   0.00-1.00   sec   582 MBytes  4.88 Gbits/sec                  
[  9]   0.00-1.00   sec   554 MBytes  4.65 Gbits/sec                  
[ 11]   0.00-1.00   sec   408 MBytes  3.42 Gbits/sec                  
[SUM]   0.00-1.00   sec  2.23 GBytes  19.2 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   721 MBytes  6.05 Gbits/sec                  
[  7]   1.00-2.00   sec   584 MBytes  4.90 Gbits/sec                  
[  9]   1.00-2.00   sec   724 MBytes  6.07 Gbits/sec                  
[ 11]   1.00-2.00   sec   490 MBytes  4.11 Gbits/sec                  
[SUM]   1.00-2.00   sec  2.46 GBytes  21.1 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
<snip>
[  5] 118.00-119.00 sec   525 MBytes  4.41 Gbits/sec                  
[  7] 118.00-119.00 sec   900 MBytes  7.55 Gbits/sec                  
[  9] 118.00-119.00 sec   459 MBytes  3.85 Gbits/sec                  
[ 11] 118.00-119.00 sec   818 MBytes  6.86 Gbits/sec                  
[SUM] 118.00-119.00 sec  2.64 GBytes  22.7 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5] 119.00-120.00 sec   502 MBytes  4.21 Gbits/sec                  
[  7] 119.00-120.00 sec   915 MBytes  7.68 Gbits/sec                  
[  9] 119.00-120.00 sec   430 MBytes  3.61 Gbits/sec                  
[ 11] 119.00-120.00 sec   863 MBytes  7.24 Gbits/sec                  
[SUM] 119.00-120.00 sec  2.65 GBytes  22.7 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-120.00 sec  63.6 GBytes  4.56 Gbits/sec                  sender
[  5]   0.00-120.00 sec  63.6 GBytes  4.56 Gbits/sec                  receiver
[  7]   0.00-120.00 sec  98.2 GBytes  7.03 Gbits/sec                  sender
[  7]   0.00-120.00 sec  98.2 GBytes  7.03 Gbits/sec                  receiver
[  9]   0.00-120.00 sec  54.8 GBytes  3.92 Gbits/sec                  sender
[  9]   0.00-120.00 sec  54.8 GBytes  3.92 Gbits/sec                  receiver
[ 11]   0.00-120.00 sec  93.8 GBytes  6.71 Gbits/sec                  sender
[ 11]   0.00-120.00 sec  93.8 GBytes  6.71 Gbits/sec                  receiver
[SUM]   0.00-120.00 sec   310 GBytes  22.2 Gbits/sec                  sender
[SUM]   0.00-120.00 sec   310 GBytes  22.2 Gbits/sec                  receiver

iperf Done.

On a PI built with the changes up to OS-7327:

[root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120
Connecting to host 10.200.200.202, port 11845
[  5] local 10.200.200.201 port 45276 connected to 10.200.200.202 port 11845
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   755 MBytes  6.34 Gbits/sec
[  5]   1.00-2.00   sec  1.09 GBytes  9.37 Gbits/sec
[  5]   2.00-3.00   sec   451 MBytes  3.78 Gbits/sec
[  5]   3.00-4.00   sec   451 MBytes  3.79 Gbits/sec
[  5]   4.00-5.00   sec   450 MBytes  3.77 Gbits/sec
[  5]   5.00-6.00   sec  1.16 GBytes  10.0 Gbits/sec
[  5]   6.00-7.00   sec   859 MBytes  7.20 Gbits/sec
[  5]   7.00-8.00   sec   619 MBytes  5.19 Gbits/sec
[  5]   8.00-9.00   sec   984 MBytes  8.26 Gbits/sec
[  5]   9.00-10.00  sec   735 MBytes  6.17 Gbits/sec
[  5]  10.00-11.00  sec   916 MBytes  7.68 Gbits/sec
[  5]  11.00-12.00  sec  1.59 GBytes  13.6 Gbits/sec
[  5]  12.00-13.00  sec   974 MBytes  8.17 Gbits/sec
[  5]  13.00-14.00  sec   759 MBytes  6.37 Gbits/sec
[  5]  14.00-15.00  sec  1.74 GBytes  14.9 Gbits/sec
[  5]  15.00-16.00  sec  1.16 GBytes  9.93 Gbits/sec
[  5]  16.00-17.00  sec   453 MBytes  3.80 Gbits/sec
[  5]  17.00-18.00  sec   952 MBytes  7.99 Gbits/sec
[  5]  18.00-19.00  sec   546 MBytes  4.58 Gbits/sec
[  5]  19.00-20.00  sec  1.18 GBytes  10.1 Gbits/sec
[  5]  20.00-21.00  sec  1.41 GBytes  12.1 Gbits/sec
[  5]  21.00-22.00  sec  1.98 GBytes  17.0 Gbits/sec
[  5]  22.00-23.00  sec  1.99 GBytes  17.1 Gbits/sec
[  5]  23.00-24.00  sec  1.98 GBytes  17.0 Gbits/sec
[  5]  24.00-25.00  sec  1.98 GBytes  17.0 Gbits/sec
[  5]  25.00-26.00  sec  1.99 GBytes  17.1 Gbits/sec
<snip>

[  5] 100.00-101.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 101.00-102.00 sec  1.93 GBytes  16.6 Gbits/sec
[  5] 102.00-103.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 103.00-104.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 104.00-105.00 sec  1.93 GBytes  16.5 Gbits/sec
[  5] 105.00-106.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 106.00-107.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 107.00-108.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 108.00-109.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 109.00-110.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 110.00-111.00 sec  1.94 GBytes  16.7 Gbits/sec
[  5] 111.00-112.00 sec  1.94 GBytes  16.7 Gbits/sec
[  5] 112.00-113.00 sec  1.94 GBytes  16.7 Gbits/sec
[  5] 113.00-114.00 sec  1.94 GBytes  16.7 Gbits/sec
[  5] 114.00-115.00 sec  1.94 GBytes  16.7 Gbits/sec
[  5] 115.00-116.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 116.00-117.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 117.00-118.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 118.00-119.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 119.00-120.00 sec  1.92 GBytes  16.5 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-120.00 sec   202 GBytes  14.5 Gbits/sec                  sender
[  5]   0.00-120.00 sec   202 GBytes  14.5 Gbits/sec                  receiver

iperf Done.
[root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120
Connecting to host 10.200.200.202, port 11845
[  5] local 10.200.200.201 port 33259 connected to 10.200.200.202 port 11845
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   506 MBytes  4.25 Gbits/sec
[  5]   1.00-2.00   sec  1.14 GBytes  9.79 Gbits/sec
[  5]   2.00-3.00   sec   640 MBytes  5.37 Gbits/sec
[  5]   3.00-4.00   sec   450 MBytes  3.78 Gbits/sec
[  5]   4.00-5.00   sec   515 MBytes  4.32 Gbits/sec
[  5]   5.00-6.00   sec   689 MBytes  5.78 Gbits/sec
[  5]   6.00-7.00   sec   486 MBytes  4.07 Gbits/sec
[  5]   7.00-8.00   sec   659 MBytes  5.53 Gbits/sec
[  5]   8.00-9.00   sec   573 MBytes  4.81 Gbits/sec
[  5]   9.00-10.00  sec  1017 MBytes  8.53 Gbits/sec
[  5]  10.00-11.00  sec   453 MBytes  3.80 Gbits/sec
[  5]  11.00-12.00  sec   460 MBytes  3.86 Gbits/sec
[  5]  12.00-13.00  sec  1.58 GBytes  13.6 Gbits/sec
[  5]  13.00-14.00  sec  1.10 GBytes  9.46 Gbits/sec
[  5]  14.00-15.00  sec  1.97 GBytes  16.9 Gbits/sec
[  5]  15.00-16.00  sec  1.97 GBytes  16.9 Gbits/sec
[  5]  16.00-17.00  sec  1.96 GBytes  16.9 Gbits/sec
[  5]  17.00-18.00  sec  1.96 GBytes  16.9 Gbits/sec
[  5]  18.00-19.00  sec  1.97 GBytes  16.9 Gbits/sec
[  5]  19.00-20.00  sec  1.96 GBytes  16.9 Gbits/sec
<snip>
[  5] 100.00-101.00 sec  1.97 GBytes  16.9 Gbits/sec
[  5] 101.00-102.00 sec  1.98 GBytes  17.0 Gbits/sec
[  5] 102.00-103.00 sec  1.98 GBytes  17.0 Gbits/sec
[  5] 103.00-104.00 sec  1.98 GBytes  17.1 Gbits/sec
[  5] 104.00-105.00 sec  1.98 GBytes  17.0 Gbits/sec
[  5] 105.00-106.00 sec  1.99 GBytes  17.1 Gbits/sec
[  5] 106.00-107.00 sec  1.99 GBytes  17.1 Gbits/sec
[  5] 107.00-108.00 sec  1.96 GBytes  16.9 Gbits/sec
[  5] 108.00-109.00 sec  1.95 GBytes  16.8 Gbits/sec
[  5] 109.00-110.00 sec  1.96 GBytes  16.9 Gbits/sec
[  5] 110.00-111.00 sec  1.96 GBytes  16.8 Gbits/sec
[  5] 111.00-112.00 sec  1.95 GBytes  16.8 Gbits/sec
[  5] 112.00-113.00 sec  1.95 GBytes  16.7 Gbits/sec
[  5] 113.00-114.00 sec  1.96 GBytes  16.8 Gbits/sec
[  5] 114.00-115.00 sec  1.97 GBytes  16.9 Gbits/sec
[  5] 115.00-116.00 sec  1.97 GBytes  16.9 Gbits/sec
[  5] 116.00-117.00 sec  1.96 GBytes  16.8 Gbits/sec
[  5] 117.00-118.00 sec  1.96 GBytes  16.8 Gbits/sec
[  5] 118.00-119.00 sec  1.96 GBytes  16.8 Gbits/sec
[  5] 119.00-120.00 sec  1.96 GBytes  16.8 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-120.00 sec   218 GBytes  15.6 Gbits/sec                  sender
[  5]   0.00-120.00 sec   218 GBytes  15.6 Gbits/sec                  receiver

iperf Done.
[root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120 -P 4
Connecting to host 10.200.200.202, port 11845
[  5] local 10.200.200.201 port 57228 connected to 10.200.200.202 port 11845
[  7] local 10.200.200.201 port 43997 connected to 10.200.200.202 port 11845
[  9] local 10.200.200.201 port 48400 connected to 10.200.200.202 port 11845
[ 11] local 10.200.200.201 port 57597 connected to 10.200.200.202 port 11845
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   621 MBytes  5.21 Gbits/sec
[  7]   0.00-1.00   sec   557 MBytes  4.67 Gbits/sec
[  9]   0.00-1.00   sec   610 MBytes  5.11 Gbits/sec
[ 11]   0.00-1.00   sec   636 MBytes  5.34 Gbits/sec
[SUM]   0.00-1.00   sec  2.37 GBytes  20.3 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   688 MBytes  5.77 Gbits/sec
[  7]   1.00-2.00   sec   612 MBytes  5.13 Gbits/sec
[  9]   1.00-2.00   sec   669 MBytes  5.61 Gbits/sec
[ 11]   1.00-2.00   sec   699 MBytes  5.86 Gbits/sec
[SUM]   1.00-2.00   sec  2.61 GBytes  22.4 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   686 MBytes  5.76 Gbits/sec
[  7]   2.00-3.00   sec   614 MBytes  5.15 Gbits/sec
[  9]   2.00-3.00   sec   672 MBytes  5.63 Gbits/sec
[ 11]   2.00-3.00   sec   697 MBytes  5.85 Gbits/sec
[SUM]   2.00-3.00   sec  2.61 GBytes  22.4 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
<snip>
[  5] 118.00-119.00 sec   687 MBytes  5.76 Gbits/sec
[  7] 118.00-119.00 sec   618 MBytes  5.19 Gbits/sec
[  9] 118.00-119.00 sec   668 MBytes  5.61 Gbits/sec
[ 11] 118.00-119.00 sec   718 MBytes  6.02 Gbits/sec
[SUM] 118.00-119.00 sec  2.63 GBytes  22.6 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5] 119.00-120.00 sec   681 MBytes  5.71 Gbits/sec
[  7] 119.00-120.00 sec   613 MBytes  5.14 Gbits/sec
[  9] 119.00-120.00 sec   663 MBytes  5.56 Gbits/sec
[ 11] 119.00-120.00 sec   713 MBytes  5.98 Gbits/sec
[SUM] 119.00-120.00 sec  2.61 GBytes  22.4 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-120.00 sec  80.3 GBytes  5.75 Gbits/sec                  sender
[  5]   0.00-120.00 sec  80.3 GBytes  5.75 Gbits/sec                  receiver
[  7]   0.00-120.00 sec  71.8 GBytes  5.14 Gbits/sec                  sender
[  7]   0.00-120.00 sec  71.8 GBytes  5.14 Gbits/sec                  receiver
[  9]   0.00-120.00 sec  78.1 GBytes  5.59 Gbits/sec                  sender
[  9]   0.00-120.00 sec  78.1 GBytes  5.59 Gbits/sec                  receiver
[ 11]   0.00-120.00 sec  82.8 GBytes  5.92 Gbits/sec                  sender
[ 11]   0.00-120.00 sec  82.8 GBytes  5.92 Gbits/sec                  receiver
[SUM]   0.00-120.00 sec   313 GBytes  22.4 Gbits/sec                  sender
[SUM]   0.00-120.00 sec   313 GBytes  22.4 Gbits/sec                  receiver

iperf Done.
[root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120 -P 4
Connecting to host 10.200.200.202, port 11845
[  5] local 10.200.200.201 port 59691 connected to 10.200.200.202 port 11845
[  7] local 10.200.200.201 port 60754 connected to 10.200.200.202 port 11845
[  9] local 10.200.200.201 port 39711 connected to 10.200.200.202 port 11845
[ 11] local 10.200.200.201 port 57824 connected to 10.200.200.202 port 11845
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   413 MBytes  3.47 Gbits/sec
[  7]   0.00-1.00   sec   734 MBytes  6.16 Gbits/sec
[  9]   0.00-1.00   sec   397 MBytes  3.33 Gbits/sec
[ 11]   0.00-1.00   sec   830 MBytes  6.97 Gbits/sec
[SUM]   0.00-1.00   sec  2.32 GBytes  19.9 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   459 MBytes  3.85 Gbits/sec
[  7]   1.00-2.00   sec   841 MBytes  7.06 Gbits/sec
[  9]   1.00-2.00   sec   445 MBytes  3.73 Gbits/sec
[ 11]   1.00-2.00   sec   856 MBytes  7.18 Gbits/sec
[SUM]   1.00-2.00   sec  2.54 GBytes  21.8 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   463 MBytes  3.88 Gbits/sec
[  7]   2.00-3.00   sec   845 MBytes  7.09 Gbits/sec
[  9]   2.00-3.00   sec   448 MBytes  3.76 Gbits/sec
[ 11]   2.00-3.00   sec   864 MBytes  7.24 Gbits/sec
[SUM]   2.00-3.00   sec  2.56 GBytes  22.0 Gbits/sec
<snip>
[  5] 118.00-119.00 sec   535 MBytes  4.49 Gbits/sec
[  7] 118.00-119.00 sec   794 MBytes  6.67 Gbits/sec
[  9] 118.00-119.00 sec   515 MBytes  4.32 Gbits/sec
[ 11] 118.00-119.00 sec   448 MBytes  3.76 Gbits/sec
[SUM] 118.00-119.00 sec  2.24 GBytes  19.2 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5] 119.00-120.00 sec   537 MBytes  4.51 Gbits/sec
[  7] 119.00-120.00 sec   803 MBytes  6.74 Gbits/sec
[  9] 119.00-120.00 sec   516 MBytes  4.33 Gbits/sec
[ 11] 119.00-120.00 sec   453 MBytes  3.80 Gbits/sec
[SUM] 119.00-120.00 sec  2.26 GBytes  19.4 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-120.00 sec  61.6 GBytes  4.41 Gbits/sec                  sender
[  5]   0.00-120.00 sec  61.6 GBytes  4.41 Gbits/sec                  receiver
[  7]   0.00-120.00 sec  93.6 GBytes  6.70 Gbits/sec                  sender
[  7]   0.00-120.00 sec  93.6 GBytes  6.70 Gbits/sec                  receiver
[  9]   0.00-120.00 sec  59.3 GBytes  4.24 Gbits/sec                  sender
[  9]   0.00-120.00 sec  59.3 GBytes  4.24 Gbits/sec                  receiver
[ 11]   0.00-120.00 sec  63.5 GBytes  4.54 Gbits/sec                  sender
[ 11]   0.00-120.00 sec  63.5 GBytes  4.54 Gbits/sec                  receiver
[SUM]   0.00-120.00 sec   278 GBytes  19.9 Gbits/sec                  sender
[SUM]   0.00-120.00 sec   278 GBytes  19.9 Gbits/sec                  receiver

iperf Done.
[root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120 -P 4
Connecting to host 10.200.200.202, port 11845
[  5] local 10.200.200.201 port 46703 connected to 10.200.200.202 port 11845
[  7] local 10.200.200.201 port 57035 connected to 10.200.200.202 port 11845
[  9] local 10.200.200.201 port 54622 connected to 10.200.200.202 port 11845
[ 11] local 10.200.200.201 port 51194 connected to 10.200.200.202 port 11845
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   482 MBytes  4.04 Gbits/sec
[  7]   0.00-1.00   sec  1.08 GBytes  9.30 Gbits/sec
[  9]   0.00-1.00   sec   346 MBytes  2.90 Gbits/sec
[ 11]   0.00-1.00   sec   817 MBytes  6.85 Gbits/sec
[SUM]   0.00-1.00   sec  2.69 GBytes  23.1 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   505 MBytes  4.23 Gbits/sec
[  7]   1.00-2.00   sec  1.15 GBytes  9.85 Gbits/sec
[  9]   1.00-2.00   sec   367 MBytes  3.08 Gbits/sec
[ 11]   1.00-2.00   sec   877 MBytes  7.36 Gbits/sec
[SUM]   1.00-2.00   sec  2.85 GBytes  24.5 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   514 MBytes  4.31 Gbits/sec
[  7]   2.00-3.00   sec  1.15 GBytes  9.88 Gbits/sec
[  9]   2.00-3.00   sec   364 MBytes  3.05 Gbits/sec
[ 11]   2.00-3.00   sec   857 MBytes  7.19 Gbits/sec
[SUM]   2.00-3.00   sec  2.84 GBytes  24.4 Gbits/sec
<snip>
[  5] 118.00-119.00 sec   517 MBytes  4.34 Gbits/sec
[  7] 118.00-119.00 sec  1.10 GBytes  9.47 Gbits/sec
[  9] 118.00-119.00 sec   362 MBytes  3.03 Gbits/sec
[ 11] 118.00-119.00 sec   845 MBytes  7.09 Gbits/sec
[SUM] 118.00-119.00 sec  2.79 GBytes  23.9 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5] 119.00-120.00 sec   510 MBytes  4.28 Gbits/sec
[  7] 119.00-120.00 sec  1.09 GBytes  9.35 Gbits/sec
[  9] 119.00-120.00 sec   364 MBytes  3.05 Gbits/sec
[ 11] 119.00-120.00 sec   860 MBytes  7.21 Gbits/sec
[SUM] 119.00-120.00 sec  2.78 GBytes  23.9 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-120.00 sec  60.3 GBytes  4.31 Gbits/sec                  sender
[  5]   0.00-120.00 sec  60.3 GBytes  4.31 Gbits/sec                  receiver
[  7]   0.00-120.00 sec   133 GBytes  9.49 Gbits/sec                  sender
[  7]   0.00-120.00 sec   133 GBytes  9.49 Gbits/sec                  receiver
[  9]   0.00-120.00 sec  42.7 GBytes  3.06 Gbits/sec                  sender
[  9]   0.00-120.00 sec  42.7 GBytes  3.06 Gbits/sec                  receiver
[ 11]   0.00-120.00 sec   100 GBytes  7.19 Gbits/sec                  sender
[ 11]   0.00-120.00 sec   100 GBytes  7.19 Gbits/sec                  receiver
[SUM]   0.00-120.00 sec   336 GBytes  24.0 Gbits/sec                  sender
[SUM]   0.00-120.00 sec   336 GBytes  24.0 Gbits/sec                  receiver

iperf Done.

Both platforms are getting the same transfer rates.

The single thread runs take a while on both platforms to get up to ~16 Gbits/sec, which I at first thought was due to the congestion window growing slowly, but was revealed by connstat to actually be due to the RTT starting out at around 2000-3000 microseconds and then dropping after a while to 400 microseconds. Once it got there it would usually stay there. I don't currently have a theory I'm confident in about why this is happening, but I'm able to observe it on both the current, stock platform, and on one with these changes.


Comment by Jira Bot
Created at 2019-03-29T20:57:30.762Z

illumos-joyent commit 7c6c8467e3896e00fd70cb263268a1d5bc5ac661 (branch master, by Sebastien Roy)

OS-7334 Track TCP round-trip time in nanoseconds
Portions contributed by: Cody Peter Mello <cody.mello@joyent.com>
Portions contributed by: Brandon Baker <bbaker@delphix.com>
Reviewed by: Jason King <jason.king@joyent.com>
Reviewed by: Robert Mustacchi <rm@joyent.com>
Reviewed by: Dan McDonald <danmcd@joyent.com>
Approved by: Dan McDonald <danmcd@joyent.com>