OS-7980: lx getsockopt should return EOPNOTSUPP when TCP options are used with AF_UNIX

Details

Issue Type:Bug
Priority:4 - Normal
Status:Resolved
Created at:2019-09-05T16:32:41.931Z
Updated at:2019-09-17T17:24:32.105Z

People

Created by:Michael Zeller
Reported by:Michael Zeller
Assigned to:Michael Zeller

Resolution

Fixed: A fix for this issue is checked into the tree and tested.
(Resolution Date: 2019-09-17T17:24:32.092Z)

Fix Versions

2019-09-26 Bubble Boy (Release Date: 2019-09-26)

Description

I happened to catch on irc that users are seeing issues with newer versions of gitlab community edition.  It appears to be related to getsockopt and TCP_CORK.  More info can be seen here: https://gitlab.com/gitlab-org/gitlab-ce/issues/59484

Based on the bug report above I wrote a small C program that shows the different behavior on lx vs a real linux host.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <strings.h>
#include <sys/un.h>
#include <unistd.h>
#include <netinet/tcp.h>
#include <sys/types.h>

#define _GNU_SOURCE
#include <sys/socket.h>

#define SOCK_PATH "/tmp/gitlabce"

int
main()
{
	int sfd;
	int optval = 1;
	socklen_t optlen = sizeof (optval);
	struct sockaddr_un saddr;

	if ((sfd = socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC, 0)) < 0) {
		perror("socket");
		exit(1);
	}

	bzero(&saddr, sizeof (saddr));
	saddr.sun_family = AF_UNIX;
	strncpy(saddr.sun_path, SOCK_PATH, sizeof (saddr.sun_path) -1);

	if (bind(sfd, (const struct sockaddr *) &saddr, sizeof (saddr)) !=0) {
		perror("bind");
		close(sfd);
		exit(1);
	}

	if (listen(sfd, 128) != 0) {
		perror("listen");
		close(sfd);
		unlink(SOCK_PATH);
		exit(1);
	}

	if (getsockopt(sfd, SOL_TCP, TCP_CORK, &optval, &optlen) < 0) {
		perror("getsockopt");
		close(sfd);
		unlink(SOCK_PATH);
		exit(1);
	}

	close(sfd);
	unlink(SOCK_PATH);

}

On linux:

$ ./a.out
getsockopt: Operation not supported

On lx:

$ ./a.out
getsockopt: Protocol not available

What's not clear is if the ruby code in gitlab actually handles the EOPNOTSUPP.

Comments

Comment by Michael Zeller
Created at 2019-09-05T16:59:20.649Z

Downstack of where things fall apart in lx

# ./OS-7980.d
dtrace: script './OS-7980.d' matched 91641 probes
CPU FUNCTION
 23  -> lx_getsockopt
              lx_brand`lx_syscall_enter+0x1aa
              unix`sys_syscall+0x145

 23   | lx_getsockopt:entry
 23    -> copyin
 23    <- kcopy                               Returns 0x0
 23    -> getsonode
 23      -> getf
 23        -> getf_gen
 23          -> set_active_fd
 23          <- set_active_fd                 Returns 0xfffffede96632820
 23          -> set_active_fd
 23          <- set_active_fd                 Returns 0x0
 23        <- getf_gen                        Returns 0xfffffedece6ebd70
 23      <- getf                              Returns 0xfffffedece6ebd70
 23    <- getsonode                           Returns 0xfffffee00a929be0
 23    -> lx_getsockopt_tcp
 23      -> lx_sockopt_lookup
 23      <- lx_sockopt_lookup                 Returns 0x1
 23      -> socket_getsockopt
 23        -> sotpi_getsockopt
 23          -> so_lock_single
 23          <- so_lock_single                Returns 0x10
 23          -> soallocproto3
 23            -> soallocproto1
 23              -> soallocproto
 23                -> allocb_cred
 23                  -> allocb
 23                    -> kmem_cache_alloc
 23                    <- kmem_cache_alloc    Returns 0xfffffede53a94e80
 23                  <- allocb                Returns 0xfffffef6a9be5640
 23                  -> crhold
 23                  <- crhold                Returns 0xfffffef6a9be5640
 23                <- allocb_cred             Returns 0xfffffef6a9be5640
 23              <- soallocproto              Returns 0xfffffef6a9be5640
 23            <- soallocproto1               Returns 0xfffffef6a9be5640
 23            -> soappendmsg
 23            <- soappendmsg                 Returns 0xfffffede53a94f0c
 23            -> soappendmsg
 23            <- soappendmsg                 Returns 0xfffffede53a94f10
 23          <- soallocproto3                 Returns 0xfffffef6a9be5640
 23          -> kstrputmsg
 23            -> audit_getstate
 23            <- audit_getstate              Returns 0x0
 23            -> strput
 23              -> canputnext
 23              <- canputnext                Returns 0x1
 23              -> strmakedata
 23              <- strmakedata               Returns 0x0
 23              -> putnext
 23                -> tl_wput
 23                  -> tl_optmgmt
 23                    -> svr4_optcom_req
 23                      -> mi_offset_param
 23                      <- mi_offset_param   Returns 0xfffffede53a94f00
 23                      -> proto_opt_check
 23                        -> proto_opt_lookup
 23                        <- proto_opt_lookup Returns 0x0
 23                      <- proto_opt_check   Returns 0xfffffffe
 23                      -> optcom_err_ack
 23                        -> mi_tpi_err_ack_alloc
 23                          -> tpi_ack_alloc
 23                            -> reallocb
 23                            <- reallocb    Returns 0xfffffef6a9be5640
 23                          <- tpi_ack_alloc Returns 0xfffffef6a9be5640
 23                        <- mi_tpi_err_ack_alloc Returns 0xfffffef6a9be5640
 23                        -> qreply
 23                          -> putnext
 23                            -> strrput
 23                              -> strsock_proto
 23                                -> audit_getstate
 23                                <- audit_getstate Returns 0x0
 23                                -> soqueueack
 23                                <- soqueueack Returns 0x0
 23                              <- strsock_proto Returns 0x0
 23                            <- strrput     Returns 0x0
 23                          <- putnext       Returns 0x0
 23                        <- qreply          Returns 0x0
 23                      <- optcom_err_ack    Returns 0x0
 23                    <- svr4_optcom_req     Returns 0x0
 23                  <- tl_optmgmt            Returns 0x0
 23                <- tl_wput                 Returns 0x0
 23              <- putnext                   Returns 0x0
 23            <- strput                      Returns 0x0
 23            -> freemsg
 23            <- freemsg                     Returns 0xfffffede1beec000
 23          <- kstrputmsg                    Returns 0x0
 23          -> sowaitprim
 23            -> sowaitack
 23            <- sowaitack                   Returns 0x0
 23            -> proto_tlitosyserr
 23            <- proto_tlitosyserr           Returns 0x63
 23            -> freemsg
 23              -> dblk_lastfree
 23                -> crfree
 23                <- crfree                  Returns 0x6f
 23                -> kmem_cache_free
 23                <- kmem_cache_free         Returns 0xfffffedea2d00840
 23              <- dblk_lastfree             Returns 0xfffffedea2d00840
 23            <- freemsg                     Returns 0xfffffedea2d00840
 23          <- sowaitprim                    Returns 0x63
 23          -> so_unlock_single
 23          <- so_unlock_single              Returns 0xfffffee00a929e70
 23        <- sotpi_getsockopt                Returns 0x63
 23      <- socket_getsockopt                 Returns 0x63
 23    <- lx_getsockopt_tcp                   Returns 0x63
 23    -> snprintf
 23      -> vsnprintf
 23      <- vsnprintf                         Returns 0x10
 23    <- snprintf                            Returns 0x10
 23    -> lx_unsupported
 23    <- lx_unsupported                      Returns 0xfffffedf58ae3260
 23    -> copyout
 23    <- kcopy                               Returns 0x0
 23    -> releasef
 23      -> clear_active_fd
 23      <- clear_active_fd                   Returns 0xfffffede96632820
 23    <- releasef                            Returns 0xfffffede96632820
 23    -> set_errno
 23    <- set_errno                           Returns 0x63
 23  <- lx_getsockopt                         Returns 0x63

Comment by Michael Zeller
Created at 2019-09-05T22:13:46.223Z

I did some testing on Linux, illumos, and FreeBSD with the attached program to see what was returned on each platform when using an AF_UNIX socket.

illumos: ENOPROTOOPT
FreeBSD: EOPNOTSUPP
Linux: EINVAL

To add more background, this seems to come from the kgio ruby module.
http://bogomips.org/kgio.git/tree/ext/kgio/autopush.c?id=b04f162393355c67f453a99ec7682832fb8468e1#n215

	if (getsockopt(fd, IPPROTO_TCP, KGIO_NOPUSH, &corked, &optlen) != 0) {
		if (errno != EOPNOTSUPP)
			rb_sys_fail("getsockopt(TCP_CORK/TCP_NOPUSH)");
		errno = 0;
		state = AUTOPUSH_STATE_ACCEPTOR_IGNORE;
	}

KGIO_NOPUSH is defined as TCP_CORK (linux) or TCP_NOPUSH (freebsd).
It looks like the code will explicitly look for EOPNOTSUPP and then avoid setting the sockopt in other places. What's odd is this is likely broken for FreeBSD too since it seems FreeBSD will return EINVAL.

In the case of lx we should probably return EOPNOTSUPP when we attempt to get/set the sockopt on a AF_UNIX socket just to remain consistent with what Linux is doing.


Comment by Michael Zeller
Created at 2019-09-17T16:32:50.751Z

To further test this change I modified the test program above to loop over each TCP_XXX option and attempt to call getsockopt on the socket to ensure that lx and linux produced the same output.

#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <strings.h>
#include <sys/un.h>
#include <unistd.h>
#include <netinet/tcp.h>
#include <sys/types.h>

#define _GNU_SOURCE
#include <sys/socket.h>

#define SOCK_PATH "/tmp/gitlabce"

int
main()
{
	int i, sfd;
	int optval = 1;
	socklen_t optlen = sizeof (optval);
	struct sockaddr_un saddr;

	int options[] = {TCP_CONGESTION, TCP_CORK, TCP_DEFER_ACCEPT, TCP_INFO,
	    TCP_KEEPCNT, TCP_KEEPIDLE, TCP_KEEPINTVL, TCP_LINGER2, TCP_MAXSEG,
	    TCP_NODELAY, TCP_QUICKACK, TCP_SYNCNT, TCP_USER_TIMEOUT,
	    TCP_WINDOW_CLAMP};

	//int optionslen = sizeof (options) / sizeof (int);
	int optionslen = *(&options +1) - options;

	if ((sfd = socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC, 0)) < 0) {
		perror("socket");
		exit(1);
	}

	bzero(&saddr, sizeof (saddr));
	saddr.sun_family = AF_UNIX;
	strncpy(saddr.sun_path, SOCK_PATH, sizeof (saddr.sun_path) -1);

	if (bind(sfd, (const struct sockaddr *) &saddr, sizeof (saddr)) !=0) {
		perror("bind");
		goto done;
	}

	if (listen(sfd, 128) != 0) {
		perror("listen");
		goto done;
	}


	/*
	 * Not all of the optvals are ints, however this is just a quick check
	 * to make sure all the options return the same errno with AF_UNIX.
	 * So, for now just look past what their actual values should be set to.
	 */
	for (i = 0; i < optionslen; i++) {
		if (getsockopt(sfd, SOL_TCP, options[i], &optval, &optlen)
		    == -1) {
			if (errno != EOPNOTSUPP) {
				perror("getsockopt errno not EOPNOTSUPP");
			}
		}
	}


done:
	close(sfd);
	unlink(SOCK_PATH);

}

The results are as follows:

== linux ==

socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC, 0) = 3
bind(3, {sa_family=AF_UNIX, sun_path="/tmp/gitlabce"}, 110) = 0
listen(3, 128)                          = 0
getsockopt(3, SOL_TCP, TCP_CONGESTION, 0x7ffcbd02586c, [4]) = -1 EOPNOTSUPP (Operation not supported)
getsockopt(3, SOL_TCP, TCP_CORK, 0x7ffcbd02586c, [4]) = -1 EOPNOTSUPP (Operation not supported)
getsockopt(3, SOL_TCP, TCP_DEFER_ACCEPT, 0x7ffcbd02586c, [4]) = -1 EOPNOTSUPP (Operation not supported)
getsockopt(3, SOL_TCP, TCP_INFO, 0x7ffcbd02586c, [4]) = -1 EOPNOTSUPP (Operation not supported)
getsockopt(3, SOL_TCP, TCP_KEEPCNT, 0x7ffcbd02586c, [4]) = -1 EOPNOTSUPP (Operation not supported)
getsockopt(3, SOL_TCP, TCP_KEEPIDLE, 0x7ffcbd02586c, [4]) = -1 EOPNOTSUPP (Operation not supported)
getsockopt(3, SOL_TCP, TCP_KEEPINTVL, 0x7ffcbd02586c, [4]) = -1 EOPNOTSUPP (Operation not supported)
getsockopt(3, SOL_TCP, TCP_LINGER2, 0x7ffcbd02586c, [4]) = -1 EOPNOTSUPP (Operation not supported)
getsockopt(3, SOL_TCP, TCP_MAXSEG, 0x7ffcbd02586c, [4]) = -1 EOPNOTSUPP (Operation not supported)
getsockopt(3, SOL_TCP, TCP_NODELAY, 0x7ffcbd02586c, [4]) = -1 EOPNOTSUPP (Operation not supported)
getsockopt(3, SOL_TCP, TCP_QUICKACK, 0x7ffcbd02586c, [4]) = -1 EOPNOTSUPP (Operation not supported)
getsockopt(3, SOL_TCP, TCP_SYNCNT, 0x7ffcbd02586c, [4]) = -1 EOPNOTSUPP (Operation not supported)
getsockopt(3, SOL_TCP, TCP_USER_TIMEOUT, 0x7ffcbd02586c, [4]) = -1 EOPNOTSUPP (Operation not supported)
getsockopt(3, SOL_TCP, TCP_WINDOW_CLAMP, 0x7ffcbd02586c, [4]) = -1 EOPNOTSUPP (Operation not supported)
close(3)                                = 0
unlink("/tmp/gitlabce")                 = 0
exit_group(0)                           = ?
+++ exited with 0 +++

== lx ==

socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC, 0) = 3
bind(3, {sa_family=AF_LOCAL, sun_path="/tmp/gitlabce"}, 110) = 0
listen(3, 128)                          = 0
getsockopt(3, SOL_TCP, TCP_CONGESTION, 0x7fffffeff48c, 0x7fffffeff490) = -1 EOPNOTSUPP (Operation not supported)
getsockopt(3, SOL_TCP, TCP_CORK, 0x7fffffeff48c, 0x7fffffeff490) = -1 EOPNOTSUPP (Operation not supported)
getsockopt(3, SOL_TCP, TCP_DEFER_ACCEPT, 0x7fffffeff48c, 0x7fffffeff490) = -1 EOPNOTSUPP (Operation not supported)
getsockopt(3, SOL_TCP, TCP_INFO, 0x7fffffeff48c, 0x7fffffeff490) = -1 EOPNOTSUPP (Operation not supported)
getsockopt(3, SOL_TCP, TCP_KEEPCNT, 0x7fffffeff48c, 0x7fffffeff490) = -1 EOPNOTSUPP (Operation not supported)
getsockopt(3, SOL_TCP, TCP_KEEPIDLE, 0x7fffffeff48c, 0x7fffffeff490) = -1 EOPNOTSUPP (Operation not supported)
getsockopt(3, SOL_TCP, TCP_KEEPINTVL, 0x7fffffeff48c, 0x7fffffeff490) = -1 EOPNOTSUPP (Operation not supported)
getsockopt(3, SOL_TCP, TCP_LINGER2, 0x7fffffeff48c, 0x7fffffeff490) = -1 EOPNOTSUPP (Operation not supported)
getsockopt(3, SOL_TCP, TCP_MAXSEG, 0x7fffffeff48c, 0x7fffffeff490) = -1 EOPNOTSUPP (Operation not supported)
getsockopt(3, SOL_TCP, TCP_NODELAY, 0x7fffffeff48c, 0x7fffffeff490) = -1 EOPNOTSUPP (Operation not supported)
getsockopt(3, SOL_TCP, TCP_QUICKACK, 0x7fffffeff48c, 0x7fffffeff490) = -1 EOPNOTSUPP (Operation not supported)
getsockopt(3, SOL_TCP, TCP_SYNCNT, 0x7fffffeff48c, 0x7fffffeff490) = -1 EOPNOTSUPP (Operation not supported)
getsockopt(3, SOL_TCP, TCP_USER_TIMEOUT, 0x7fffffeff48c, 0x7fffffeff490) = -1 EOPNOTSUPP (Operation not supported)
getsockopt(3, SOL_TCP, TCP_WINDOW_CLAMP, 0x7fffffeff48c, 0x7fffffeff490) = -1 EOPNOTSUPP (Operation not supported)
close(3)                                = 0
unlink("/tmp/gitlabce")                 = 0
exit_group(0)                           = ?
+++ exited with 0 +++

Additionally I ran ltp which produced the following:

root@8f59784f-0d32-ed60-d148-fadf0e7eda0a:/opt/ltp# grep -i tfail /tmp/test.log
recv01      5  TFAIL  :  recv01.c:147: invalid MSG_ERRQUEUE flag set ; returned 0 (expected -1), errno 0 (expected 11)
recvfrom01    7  TFAIL  :  recvfrom01.c:170: invalid MSG_ERRQUEUE flag set ; returned 0 (expected -1), errno 0 (expected 11)
recvmsg01   10  TFAIL  :  recvmsg01.c:235: invalid MSG_ERRQUEUE flag set ; returned 0 (expected -1), errno 0 (expected 11)

The above errors look the same from previous runs of ltp a month or so ago.


Comment by Jira Bot
Created at 2019-09-17T17:17:30.715Z

illumos-joyent commit 8f77bed5b9e3f96e1fcf344bef8e85a338e9da6b (branch master, by Mike Zeller)

OS-7980 lx getsockopt should return EOPNOTSUPP when TCP options are used with AF_UNIX
Reviewed by: Dan McDonald <danmcd@joyent.com>
Reviewed by: Jerry Jelinek <jerry.jelinek@joyent.com>
Approved by: Jerry Jelinek <jerry.jelinek@joyent.com>