OS-5441: dlmgmtd deadlock when forking

Details

Issue Type:Bug
Priority:4 - Normal
Status:Resolved
Created at:2016-06-02T17:15:10.000Z
Updated at:2016-06-06T16:15:32.000Z

People

Created by:Former user
Reported by:Former user
Assigned to:Former user

Resolution

Duplicate: The problem is a duplicate of an existing issue.
(Resolution Date: 2016-06-02T18:10:42.000Z)

Related Issues

Description

Okay, so we have a hung shrimp. Let's see what's going on. We have a lot
of threads stuck in a door call. For example, here are our stacks:

> 0t21::pid2proc | ::walk thread | ::stacks
THREAD           STATE    SOBJ                COUNT
ffffff431d98d3c0 SLEEP    SHUTTLE              2501
                 swtch+0x141
                 shuttle_swtch+0x203
                 door_return+0x214
                 doorfs32+0x180
                 sys_syscall32+0x14a

ffffff431d98d020 STOPPED  <NONE>               1364
                 swtch+0x141
                 stop+0x386
                 issig_forreal+0x3e4
                 issig+0x25
                 door_return+0x3ef
                 doorfs32+0x180
                 sys_syscall32+0x14a

ffffff43611bdb60 SLEEP    CV                     61
                 swtch+0x141
                 cv_wait_sig_swap_core+0x1b9
                 cv_wait_sig_swap+0x17
                 cv_waituntil_sig+0xbd
                 lwp_park+0x15e
                 syslwp_park+0x63
                 sys_syscall32+0x14a

ffffff448cb5c3e0 SLEEP    CV                     12
                 swtch+0x141
                 cv_timedwait_sig_hires+0x39d
                 cv_waituntil_sig+0xfa
                 nanosleep+0x19f
                 sys_syscall32+0x14a

ffffff4470e63140 STOPPED  <NONE>                  6
                 swtch+0x141
                 stop+0x386
                 issig_forreal+0x3e4
                 issig+0x25
                 cv_timedwait_sig_hires+0x20e
                 cv_waituntil_sig+0xfa
                 nanosleep+0x19f
                 sys_syscall32+0x14a

ffffff4486a45400 STOPPED  <NONE>                  5
                 swtch+0x141
                 stop+0x386
                 issig_forreal+0x3e4
                 issig+0x25
                 cv_wait_sig_swap_core+0x303
                 cv_wait_sig_swap+0x17
                 cv_waituntil_sig+0xbd
                 lwp_park+0x15e
                 syslwp_park+0x63
                 sys_syscall32+0x14a

ffffffd4cd763b00 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait_sig+0x185
                 lwp_suspend+0xa4
                 syslwp_suspend+0x48
                 sys_syscall32+0x14a

ffffff431de2cb60 SLEEP    CV                      1
                 swtch+0x141          
                 cv_wait_sig_swap_core+0x1b9
                 cv_wait_sig_swap+0x17
                 pause+0x45
                 sys_syscall32+0x14a

ffffff4743dcc040 SLEEP    RWLOCK                  1
                 swtch+0x141
                 turnstile_block+0x21a
                 rw_enter_sleep+0x19b
                 vnic_dev_delete+0x43
                 vnic_ioc_delete+0x28
                 drv_ioctl+0x1e4
                 cdev_ioctl+0x39
                 spec_ioctl+0x60
                 fop_ioctl+0x55
                 ioctl+0x9b
                 sys_syscall32+0x14a

Okay, we have a thread in vnic_dev_delete, a classic.

> ffffff4743dcc040::findstack -v
stack pointer for thread ffffff4743dcc040: ffffff01f0bb09e0
[ ffffff01f0bb09e0 _resume_from_idle+0x112() ]
  ffffff01f0bb0a10 swtch+0x141()
  ffffff01f0bb0ab0 turnstile_block+0x21a(ffffff4590ad33a0, 0, ffffffffc011b270, fffffffffbc08ce0, 0, 0)
  ffffff01f0bb0b20 rw_enter_sleep+0x19b(ffffffffc011b270, 0)
  ffffff01f0bb0b90 vnic_dev_delete+0x43(405cc, 0, ffffff5c53992968)
  ffffff01f0bb0bd0 vnic_ioc_delete+0x28(ffffff44ef93f958, 7ae58d04, 100003, ffffff5c53992968, ffffff01f0bb0e58)
  ffffff01f0bb0c70 drv_ioctl+0x1e4(1200000000, 1710002, 7ae58d04, 100003, ffffff5c53992968, ffffff01f0bb0e58)
  ffffff01f0bb0cb0 cdev_ioctl+0x39(1200000000, 1710002, 7ae58d04, 100003, ffffff5c53992968, ffffff01f0bb0e58)
  ffffff01f0bb0d00 spec_ioctl+0x60(ffffff431dd87200, 1710002, 7ae58d04, 100003, ffffff5c53992968, ffffff01f0bb0e58, 0)
  ffffff01f0bb0d90 fop_ioctl+0x55(ffffff431dd87200, 1710002, 7ae58d04, 100003, ffffff5c53992968, ffffff01f0bb0e58, 0)
  ffffff01f0bb0eb0 ioctl+0x9b(0, 1710002, 7ae58d04)
  ffffff01f0bb0f10 sys_syscall32+0x14a()
> ffffffffc011b270::rwlock
            ADDR      OWNER/COUNT FLAGS          WAITERS
ffffffffc011b270 ffffff4590add7e0  B111 ffffff4495e3b3a0 (W)
                                    ||| ffffffd4cd75b140 (W)
                 WRITE_LOCKED ------+|| ffffff471fda4b40 (W)
                 WRITE_WANTED -------+| ffffff4a90e78880 (W)
                  HAS_WAITERS --------+ ffffff45262dd3e0 (W)
                                        ffffff4743dcc040 (W)
> ffffff4590add7e0::findstack -v
stack pointer for thread ffffff4590add7e0: ffffff020b319450
[ ffffff020b319450 _resume_from_idle+0x112() ]
  ffffff020b319490 swtch_to+0xb6(ffffff45edb2a7a0)
  ffffff020b3194e0 shuttle_resume+0x2af(ffffff45edb2a7a0, ffffffffc0014ed0)
  ffffff020b319590 door_upcall+0x212(ffffff431de35380, ffffff020b319680, ffffff42e273ce18, ffffffffffffffff, 0)
  ffffff020b319610 door_ki_upcall_limited+0x67(ffffff431dceb998, ffffff020b319680, ffffff42e273ce18, ffffffffffffffff, 0)
  ffffff020b319660 stubs_common_code+0x59()
  ffffff020b319700 i_dls_mgmt_upcall+0xbf(ffffff020b319720, 8, ffffff020b319730, 30)
  ffffff020b3197d0 dls_mgmt_get_linkinfo+0x7c(4060b, ffffff020b319800, ffffff020b3197fc, 0, 0)
  ffffff020b319880 dls_devnet_set+0xb0(ffffff4743f3a6b8, 4060b, 0, ffffff020b3198b0)
  ffffff020b3198f0 dls_devnet_create+0x4c(ffffff4743f3a6a0, 4060b, 0)
  ffffff020b319af0 vnic_dev_create+0x6a0(4060b, 6, ffffff020b319b54, ffffff020b319b5c, ffffff020b319b60, ffffff020b319b58, ffffff420000000c, 
  ffffff420000054b, ffffff0200000000, ffffffff00000000, ffffff4646ae1044, ffffff4200000000, ffffff020b319b50, ffffffbabb924bc0)
  ffffff020b319bd0 vnic_ioc_create+0x10d(ffffff4646ae1000, 80401b0, 100003, ffffffbabb924bc0, ffffff020b319e58)
  ffffff020b319c70 drv_ioctl+0x1e4(1200000000, 1710001, 80401b0, 100003, ffffffbabb924bc0, ffffff020b319e58)
  ffffff020b319cb0 cdev_ioctl+0x39(1200000000, 1710001, 80401b0, 100003, ffffffbabb924bc0, ffffff020b319e58)
  ffffff020b319d00 spec_ioctl+0x60(ffffff431dd87200, 1710001, 80401b0, 100003, ffffffbabb924bc0, ffffff020b319e58, 0)
  ffffff020b319d90 fop_ioctl+0x55(ffffff431dd87200, 1710001, 80401b0, 100003, ffffffbabb924bc0, ffffff020b319e58, 0)
  ffffff020b319eb0 ioctl+0x9b(4, 1710001, 80401b0)
  ffffff020b319f10 _sys_sysenter_post_swapgs+0x153()

Okay, so we have a thread owns the lock trying to do an upcall into
dlmgmtd. That should be fine, because the thread that's doing the
vnic_dev_delete should already have dropped the lock.

Which is true, but we're trying to fork. If we look at the individual
process, we see:

> ::walk thread | ::stacks -c fork
mdb: couldn't read frame for thread 0x713 at 8dc2fd30: no mapping for address
THREAD   STATE    SOBJ        COUNT
2dc      UNPARKED <NONE>          1
         libc.so.1`suspend_fork+0x97
         libc.so.1`forkx+0x121
         libc.so.1`fork+0x1a
         dlmgmt_zfop+0xbf
         dlmgmt_zfopen+0xb8
         dlmgmt_process_db_onereq+0x54
         dlmgmt_process_db_req+0x4a
         dlmgmt_db_init+0xcc
         dlmgmt_zone_init+0x12e
         dlmgmt_zoneboot+0x55
         dlmgmt_handler+0x96
         libc.so.1`__door_return+0x3d

So we're going through and trying to fork, uh oh. So with any luck our
thread that was holding this lock was probably suspended. So, the
question is can we figure out what the door thread is? Or rather, why is
it that we can't make forward progress. So, the real question is what
thread we're trying to suspend, bets on it being the one that's
deadlocked in the kernel.

ffffffd4cd763b00 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait_sig+0x185
                 lwp_suspend+0xa4
                 syslwp_suspend+0x48
                 sys_syscall32+0x14a

> ffffffd4cd763b00::findstack -v
stack pointer for thread ffffffd4cd763b00: ffffff0204450da0
[ ffffff0204450da0 _resume_from_idle+0x112() ]
  ffffff0204450dd0 swtch+0x141()
  ffffff0204450e40 cv_wait_sig+0x185(ffffff431de2e0de, ffffff42e28c6380)
  ffffff0204450e80 lwp_suspend+0xa4(ffffff4743dcc040)
  ffffff0204450eb0 syslwp_suspend+0x48(83c)
  ffffff0204450f10 sys_syscall32+0x14a()
> 0t21::pid2proc | ::walk thread | ::printf "%p %x\n" kthread_t . t_tid ! grep 83c
ffffffa8a32c83c0 296
ffffff4743dcc040 83c
ffffff473c1b83c0 a44
ffffff4472083c40 a96
ffffff45bbf783c0 e31
ffffff6cb983cba0 e8c
> ffffff4743dcc040::findstack -v
stack pointer for thread ffffff4743dcc040: ffffff01f0bb09e0
[ ffffff01f0bb09e0 _resume_from_idle+0x112() ]
  ffffff01f0bb0a10 swtch+0x141()
  ffffff01f0bb0ab0 turnstile_block+0x21a(ffffff4590ad33a0, 0, ffffffffc011b270, fffffffffbc08ce0, 0, 0)
  ffffff01f0bb0b20 rw_enter_sleep+0x19b(ffffffffc011b270, 0)
  ffffff01f0bb0b90 vnic_dev_delete+0x43(405cc, 0, ffffff5c53992968)
  ffffff01f0bb0bd0 vnic_ioc_delete+0x28(ffffff44ef93f958, 7ae58d04, 100003, ffffff5c53992968, ffffff01f0bb0e58)
  ffffff01f0bb0c70 drv_ioctl+0x1e4(1200000000, 1710002, 7ae58d04, 100003, ffffff5c53992968, ffffff01f0bb0e58)
  ffffff01f0bb0cb0 cdev_ioctl+0x39(1200000000, 1710002, 7ae58d04, 100003, ffffff5c53992968, ffffff01f0bb0e58)
  ffffff01f0bb0d00 spec_ioctl+0x60(ffffff431dd87200, 1710002, 7ae58d04, 100003, ffffff5c53992968, ffffff01f0bb0e58, 0)
  ffffff01f0bb0d90 fop_ioctl+0x55(ffffff431dd87200, 1710002, 7ae58d04, 100003, ffffff5c53992968, ffffff01f0bb0e58, 0)
  ffffff01f0bb0eb0 ioctl+0x9b(0, 1710002, 7ae58d04)
  ffffff01f0bb0f10 sys_syscall32+0x14a()

And there we have it, another amazing dlmgmtd deadlock.

Note that this is very similar to OS-5363. I don't have quite enough confidence to say whether or not they're the exact same or not, though likely the act of forking and suspending while threads may be blocked in the kernel is the same problem.

Comments

Comment by Former user
Created at 2016-06-02T18:10:36.000Z

Based on some further looking at the problem, I believe that this actually shares the same fundamental problem as OS-5363 and that it's in fact a duplicate.