OS-7327: Want connstat(1M) command to display per-connection TCP statistics

Details

Issue Type:Bug
Priority:4 - Normal
Status:Resolved
Created at:2018-10-24T20:43:52.415Z
Updated at:2019-04-01T16:09:56.060Z

People

Created by:Former user
Reported by:Former user
Assigned to:Former user

Resolution

Fixed: A fix for this issue is checked into the tree and tested.
(Resolution Date: 2019-03-29T20:59:06.633Z)

Fix Versions

2019-04-11 Pete Hornberger (Release Date: 2019-04-11)

Related Issues

Description

Delphix created a connstat(1M) command, which allows fetching statistics about individual TCP connections. As part of the TRITON-885 porting, we'll want to pull it in. The relevant Delphix commits are:

Comments

Comment by Former user
Created at 2019-02-13T00:04:21.826Z
Updated at 2019-02-13T00:27:27.828Z

Filtering on the STATE field works:

[root@headnode (coal) ~]# connstat -F state=listen
          LADDR  LPORT           RADDR  RPORT        STATE
     10.99.99.7   2021         0.0.0.0      0       LISTEN
        0.0.0.0    111         0.0.0.0      0       LISTEN
        0.0.0.0    111         0.0.0.0      0       LISTEN
        0.0.0.0     22         0.0.0.0      0       LISTEN
      127.0.0.1   4307         0.0.0.0      0       LISTEN
     10.99.99.7   5309         0.0.0.0      0       LISTEN
     10.99.99.7   9163         0.0.0.0      0       LISTEN
      127.0.0.1   9090         0.0.0.0      0       LISTEN
      127.0.0.1     25         0.0.0.0      0       LISTEN
      127.0.0.1    587         0.0.0.0      0       LISTEN
      127.0.0.1   8080         0.0.0.0      0       LISTEN
     10.99.99.7  64473         0.0.0.0      0       LISTEN
             ::    111              ::      0       LISTEN
             ::     22              ::      0       LISTEN
            ::1     25              ::      0       LISTEN

Show only the IPv4 connections:

[root@headnode (coal) ~]# connstat -F state=listen -4
          LADDR  LPORT           RADDR  RPORT        STATE
     10.99.99.7   2021         0.0.0.0      0       LISTEN
        0.0.0.0    111         0.0.0.0      0       LISTEN
        0.0.0.0    111         0.0.0.0      0       LISTEN
        0.0.0.0     22         0.0.0.0      0       LISTEN
      127.0.0.1   4307         0.0.0.0      0       LISTEN
     10.99.99.7   5309         0.0.0.0      0       LISTEN
     10.99.99.7   9163         0.0.0.0      0       LISTEN
      127.0.0.1   9090         0.0.0.0      0       LISTEN
      127.0.0.1     25         0.0.0.0      0       LISTEN
      127.0.0.1    587         0.0.0.0      0       LISTEN
      127.0.0.1   8080         0.0.0.0      0       LISTEN
     10.99.99.7  64473         0.0.0.0      0       LISTEN

And only the IPv6 connections:

[root@headnode (coal) ~]# connstat -F state=listen -6
          LADDR  LPORT           RADDR  RPORT        STATE
             ::    111              ::      0       LISTEN
             ::     22              ::      0       LISTEN
            ::1     25              ::      0       LISTEN

Print stats three times in a row separated by two seconds with a timestamp:

[root@headnode (coal) ~]# connstat -F state=listen -6 -c 3 -i 2 -T d
February 13, 2019 at 12:00:06 AM UTC
          LADDR  LPORT           RADDR  RPORT        STATE
             ::    111              ::      0       LISTEN
             ::     22              ::      0       LISTEN
            ::1     25              ::      0       LISTEN
February 13, 2019 at 12:00:08 AM UTC
          LADDR  LPORT           RADDR  RPORT        STATE
             ::    111              ::      0       LISTEN
             ::     22              ::      0       LISTEN
            ::1     25              ::      0       LISTEN
February 13, 2019 at 12:00:10 AM UTC
          LADDR  LPORT           RADDR  RPORT        STATE
             ::    111              ::      0       LISTEN
             ::     22              ::      0       LISTEN
            ::1     25              ::      0       LISTEN

Same thing, but this time parsable and showing the connection's MSS:

= February 13, 2019 at 12:25:34 AM UTC
::,111,1220
::,22,1220
::1,25,1220
= February 13, 2019 at 12:25:36 AM UTC
::,111,1220
::,22,1220
::1,25,1220
= February 13, 2019 at 12:25:38 AM UTC
::,111,1220
::,22,1220
::1,25,1220

Adding -L removes the loopback address:

[root@headnode (coal) ~]# connstat -L
          LADDR  LPORT           RADDR  RPORT        STATE
     10.99.99.7  60824     10.99.99.20   5672  ESTABLISHED
     10.99.99.7   2021         0.0.0.0      0       LISTEN
     10.99.99.7  51664     10.99.99.25   2020  ESTABLISHED
        0.0.0.0    111         0.0.0.0      0       LISTEN
        0.0.0.0      0         0.0.0.0      0         IDLE
        0.0.0.0    111         0.0.0.0      0       LISTEN
        0.0.0.0      0         0.0.0.0      0         IDLE
        0.0.0.0     22         0.0.0.0      0       LISTEN
     10.99.99.7   5309         0.0.0.0      0       LISTEN
     10.99.99.7   9163         0.0.0.0      0       LISTEN
   10.88.88.200     22      10.88.88.2  36358  ESTABLISHED
     10.99.99.7  49601     10.99.99.26     80  ESTABLISHED
     10.99.99.7  64817     10.99.99.26     80  ESTABLISHED
     10.99.99.7  53308     10.99.99.26     80  ESTABLISHED
     10.99.99.7  64634     10.99.99.26     80  ESTABLISHED
     10.99.99.7  51329     10.99.99.10     80  ESTABLISHED
     10.99.99.7  33125     10.99.99.10     80  ESTABLISHED
     10.99.99.7  37813     10.99.99.10     80  ESTABLISHED
     10.99.99.7  56423     10.99.99.31     80    TIME_WAIT
     10.99.99.7  53490     10.99.99.31     80    TIME_WAIT
     10.99.99.7  64473         0.0.0.0      0       LISTEN
             ::    111              ::      0       LISTEN
             ::      0              ::      0         IDLE
             ::     22              ::      0       LISTEN

Using -e only shows established:

[root@headnode (coal) ~]# connstat -L -e
          LADDR  LPORT           RADDR  RPORT        STATE
     10.99.99.7  60824     10.99.99.20   5672  ESTABLISHED
     10.99.99.7  51664     10.99.99.25   2020  ESTABLISHED
   10.88.88.200     22      10.88.88.2  36358  ESTABLISHED
     10.99.99.7  57275     10.99.99.10     80  ESTABLISHED
     10.99.99.7  45090     10.99.99.10     80  ESTABLISHED
     10.99.99.7  47524     10.99.99.10     80  ESTABLISHED
     10.99.99.7  38091     10.99.99.26     80  ESTABLISHED
     10.99.99.7  57278     10.99.99.26     80  ESTABLISHED
     10.99.99.7  41935     10.99.99.26     80  ESTABLISHED
     10.99.99.7  54091     10.99.99.26     80  ESTABLISHED

Trying to combine that with a filter on STATE produces an error:

[root@headnode (coal) ~]# connstat -L -e -F state=listen
Ambiguous filter provided. The "state" field appears more than once.
usage: connstat [-eLP] [-4|-6] [-T d|u] [-F <filter>]
               [-i <interval> [-c <count>]] [-o <field>[,...]]
<snip further usage>

Comment by Former user
Created at 2019-03-27T21:32:41.054Z

This change does add new fields to struct tcpConnEntryInfo_s, which is consumed by netstat, so I ran the netstat command to show connection statistics to make sure things still looked right:

[root@headnode (coal) ~]# netstat -nv -f inet -P tcp | head -n 30

TCP: IPv4
Local/Remote Address Swind  Snext     Suna   Rwind  Rnext     Rack    Rto   Mss     State
-------------------- ----- -------- -------- ----- -------- -------- ----- ----- -----------
127.0.0.1.56689     
127.0.0.1.9090       32975 28985f6c 28985f6c 1048576 fa16dceb fa16dceb  1125  8180 ESTABLISHED
127.0.0.1.9090      
127.0.0.1.56689      1055200 fa16dceb fa16dceb 1055220 28985f6c 28985f6c  1125  8180 ESTABLISHED
10.99.99.7.52466    
10.99.99.25.2020     1049792 5a565a94 5a565a94 1049800 7ee818f0 7ee818f0  1572  1448 ESTABLISHED
127.0.0.1.40545     
127.0.0.1.9090       32975 41e87b23 41e87b23 1048576 994e4b12 994e4b12  1125  8180 ESTABLISHED
127.0.0.1.9090      
127.0.0.1.40545      1055200 994e4b12 994e4b12 1055220 41e87b23 41e87b23  1125  8180 ESTABLISHED
10.88.88.200.22     
10.88.88.2.49894     182272 478f9880 478f9880 1049800 35f2a7f1 35f2a7f1   401  1448 ESTABLISHED
127.0.0.1.48127     
127.0.0.1.9090       32975 9ecd6812 9ecd6812 1048576 b0e32dd5 b0e32dd5  1125  8180 ESTABLISHED
127.0.0.1.9090      
127.0.0.1.48127      1055200 b0e32dd5 b0e32dd5 1055220 9ecd6812 9ecd6812  1125  8180 ESTABLISHED
127.0.0.1.36867     
127.0.0.1.9090       32975 9818d798 9818d798 1048576 38d41088 38d41088  1125  8180 ESTABLISHED
127.0.0.1.9090      
127.0.0.1.36867      1055200 38d41088 38d41088 1055220 9818d798 9818d798  1125  8180 ESTABLISHED
127.0.0.1.39982     
127.0.0.1.9090       32975 35be07c0 35be07c0 1048576 d7f3f79a d7f3f79a  1125  8180 ESTABLISHED
127.0.0.1.9090      
127.0.0.1.39982      1055200 d7f3f79a d7f3f79a 1055220 35be07c0 35be07c0  1125  8180 ESTABLISHED
10.99.99.7.52968    
10.99.99.26.80       32806 1f306587 1f306587 1049800 93c0153d 93c0153d   517  1448 ESTABLISHED

Comment by Jira Bot
Created at 2019-03-29T20:57:28.382Z

illumos-joyent commit 583897b77026a5a95bd0c9a9f5c0bc1524da3a9f (branch master, by Sebastien Roy)

OS-7327 Want connstat(1M) command to display per-connection TCP statistics
Portions contributed by: Cody Peter Mello <cody.mello@joyent.com>
Portions contributed by: Ahmed G <ahmedg@delphix.com>
Reviewed by: Jason King <jason.king@joyent.com>
Reviewed by: Robert Mustacchi <rm@joyent.com>
Reviewed by: Dan McDonald <danmcd@joyent.com>
Approved by: Dan McDonald <danmcd@joyent.com>


Comment by Jira Bot
Created at 2019-03-30T06:23:28.169Z

illumos-joyent commit 8604ad92dd1e3e4f766b9ff23008dfff4853c251 (branch master, by Cody Peter Mello)

OS-7327 Want connstat(1M) command to display per-connection TCP statistics (fix timestamp fallthrough)
Reviewed by: Rob Johnston <rob.johnston@joyent.com>
Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
Approved by: Patrick Mooney <patrick.mooney@joyent.com>