OS-6720: uart-ignorant guest sends bhyve into a spin

Details

Issue Type:Bug
Priority:4 - Normal
Status:Resolved
Created at:2018-03-01T18:51:06.461Z
Updated at:2018-03-23T19:31:15.474Z

People

Created by:Patrick Mooney [X]
Reported by:Patrick Mooney [X]
Assigned to:Mike Gerdts [X]

Resolution

Fixed: A fix for this issue is checked into the tree and tested.
(Resolution Date: 2018-03-23T17:21:46.717Z)

Fix Versions

2018-03-29 Old Man's House (Release Date: 2018-03-29)

Related Links

Labels

bhyve

Description

While trying to setup an ubuntu 17.10 guest using our KVM image, I stumbled on a situation where bhyve would spin relentlessly in the uart tty thread. This would occur after attempting to input data on the zone console attached to COM1. At that point, the zhyve process would start performing approximately 2.7 million syscalls/s, spinning on pollsys:

              libc.so.1`__pollsys+0xa
              libc.so.1`poll+0x56
              zhyve`uart_tty_thread+0x39
              libc.so.1`_thrp_setup+0x8a
              libc.so.1`_lwp_start
          2419414

Looking at the related source code, there appears to be a possible situation: The uart thread is polling for input on the zconsole to pass into the guest. When there is input, it checks to see if there's space in the RX fifo. If there is, it'll read the data from the console and place it in the fifo. If there is not space, however, it'll drop back into the poll loop (which will continue to yield POLLIN since the data has not been read from the zconsole yet).

Comments

Comment by Mike Gerdts [X]
Created at 2018-03-01T21:07:04.823Z

To reproduce, boot any linux VM while attached to the console with zlogin -C.  From the grub menu, edit the linux line and remove console=ttyS0,115200n8.  Boot (^X), then give more than 16 bytes of input.


Comment by Mike Gerdts [X]
Created at 2018-03-03T02:47:22.709Z

In FreeBSD source, this problem is not present through the use of the kqueues via the mevent interfaces.  There are many other parts of bhyve that have disabled the use of mevent because mevent.c relies on interfaces not present in SmartOS or illumos.

It is rather straight-forward to implement the mevent interfaces using event ports.  With this implemented and the mevent code enabled in uart_emul.c, this problem goes away.  This is the approach that will be taken to fix this bug.  Follow-on fixes can enable the mevent interfaces in other files.


Comment by Jira Bot
Created at 2018-03-23T16:26:38.667Z

illumos-joyent commit 7892cb3dceef9f2cbcd853e66fb05601a94bbba2 (branch master, by Mike Gerdts)

OS-6720 uart-ignorant guest sends bhyve into a spin
OS-6715 extraneous rxfifo_available() in uart_sock_drain()
Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
Reviewed by: Jerry Jelinek <jerry.jelinek@joyent.com>
Reviewed by: John Levon <john.levon@joyent.com>
Approved by: John Levon <john.levon@joyent.com>


Comment by Jira Bot
Created at 2018-03-23T19:31:15.474Z

illumos-joyent commit d5eb108ff4a94b03a014e90e6213bf7bf4faa374 (branch master, by Mike Gerdts)

OS-6820 bhyve provisioning times out after OS-6720
Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
Reviewed by: Jerry Jelinek <jerry.jelinek@joyent.com>
Approved by: Dan McDonald <danmcd@joyent.com>