OS-7074: deadlock with vmm_detach() and iommu_cleanup()

Details

Issue Type:Bug
Priority:4 - Normal
Status:Resolved
Created at:2018-07-12T09:54:44.057Z
Updated at:2018-07-20T18:33:02.112Z

People

Created by:John Levon
Reported by:John Levon
Assigned to:Hans Rosenfeld [X]

Resolution

Fixed: A fix for this issue is checked into the tree and tested.
(Resolution Date: 2018-07-16T12:17:10.276Z)

Fix Versions

2018-07-19 Wall Market (Release Date: 2018-07-19)

Related Links

Labels

bhyve

Description

My system hung shutting down a bhyve instance. The relevant two threads are:

[10]> fffffcc26c444c40::findstack -v
stack pointer for thread fffffcc26c444c40: fffffcc26c444740
[ fffffcc26c444740 _resume_from_idle+0x12b() ]
  fffffcc26c444770 swtch+0x198()
  fffffcc26c444810 turnstile_block+0x2ea(0, 0, ffffffffc014c030, fffffffffbc22520, 0, 0)
  fffffcc26c444880 mutex_vector_enter+0x3fd(ffffffffc014c030)
  fffffcc26c4448b0 vmm`vmm_detach+0x30(fffffe25fef48c20, 0)
  fffffcc26c444920 devi_detach+0xd7(fffffe25fef48c20, 0)
  fffffcc26c444960 detach_node+0xec(fffffe25fef48c20, 2000)
  fffffcc26c4449c0 i_ndi_unconfig_node+0xf4(fffffe25fef48c20, 4, 2000)
  fffffcc26c4449f0 i_ddi_detachchild+0x59(fffffe25fef48c20, 2000)
  fffffcc26c444a60 devi_detach_node+0xac(fffffe25fef48c20, 2000)
  fffffcc26c444af0 unconfig_immediate_children+0x1b5(fffffe25b6dde148, 0, 2000, ffffffff)
  fffffcc26c444b80 devi_unconfig_common+0x107(fffffe25b6dde148, 0, 2000, ffffffff, 0)
  fffffcc26c444c20 mt_config_thread+0x259(fffffe2ab8515c40)
  fffffcc26c444c30 thread_start+8()

[10]> ffffffffc014c030::mutex
            ADDR  TYPE             HELD MINSPL OLDSPL WAITERS
ffffffffc014c030 adapt fffffe2ab3d53820      -      -     yes


[10]> fffffe2ab3d53820::findstack -v
stack pointer for thread fffffe2ab3d53820: fffffcc26bdcf950
[ fffffcc26bdcf950 _resume_from_idle+0x12b() ]
  fffffcc26bdcf980 swtch+0x198()
  fffffcc26bdcf9b0 cv_wait+0x89(fffffe25b6dde29c, fffffe25b6dde1b0)
  fffffcc26bdcf9f0 ndi_devi_enter+0x97(fffffe25b6dde148, fffffcc26bdcfa1c)
  fffffcc26bdcfa70 walk_devs+0xfe(fffffe25b6ddde88, fffffffff85191b0, 0, 1)
  fffffcc26bdcfaf0 walk_devs+0xc6(fffffe257edd0b88, fffffffff85191b0, 0, 1)
  fffffcc26bdcfb40 ddi_walk_devs+0x74(fffffe257edd0b88, fffffffff85191b0, 0)
  fffffcc26bdcfb50 vmm`iommu_cleanup+0x3c()
  fffffcc26bdcfb60 vmm`vmm_mod_unload+9()
  fffffcc26bdcfb70 vmm`vmmdev_mod_decr+0x35()
  fffffcc26bdcfbb0 vmm`vmm_do_vm_destroy_locked+0x10a(fffffe257d19bb00, 0)
  fffffcc26bdcfbf0 vmm`vmm_do_vm_destroy+0x3d(fffffe257d19bb00, 0)
  fffffcc26bdcfc20 vmm`vmm_zsd_destroy+0x6f(3, fffffe2ac478ad80)
  fffffcc26bdcfca0 zsd_apply_destroy+0x165(0, 1, fffffe261bd8d800, 1c)
  fffffcc26bdcfcf0 zsd_apply_all_keys+0x5f(fffffffffbf21a00, fffffe261bd8d800)
  fffffcc26bdcfd40 zone_zsd_callbacks+0x10e(fffffe261bd8d800, 2)
  fffffcc26bdcfda0 zone_destroy+0x124(3)
  fffffcc26bdcfef0 zone+0x1e7(1, 3, 0, 0, 0)
  fffffcc26bdcff00 sys_syscall+0x2a5()

Lock ordering issue between ndi_devi_enter() and the vmm_mutex.

Comments

Comment by Hans Rosenfeld [X]
Created at 2018-07-13T12:54:41.596Z

When iommu_cleanup() is called, vmmdev_mtx is already held, and ddi_walk_devs() will call ndi_devi_enter() for each device before calling the walk callback. Similarly detach(9e) will always be called inside ndi_devi_enter()/ndi_devi_exit(), so vmm_detach() running will effectively prevent ddi_walk_devs() from making progress, while vmm_detach() will happily block waiting to enter vmmdev_mtx which is held by the thread running ddi_walk_devs().

We can't really change the lock ordering in iommu_cleanup(), but we can prevent vmm_detach() from blocking on vmmdev_mtx. Instead of blocking in detach with the devi locked we can just fail the detach at this point.


Comment by Hans Rosenfeld [X]
Created at 2018-07-16T08:43:18.016Z

Testing: I repeatedly started up and shut down a bhyve zone using passthru on my test system (running DEBUG bits), this being the only bhyve zone running at a time. I verified that vmm_detach() was called and checked the mutex status each time. At least once I encountered the deadlock case where the mutex was held, and as expected vmm_detach() returned failure. I did not notice any unwanted side effects.


Comment by Jira Bot
Created at 2018-07-16T11:53:34.019Z

illumos-joyent commit 7ded4aeabf16d6abbb8a3f24364a66a321136b44 (branch master, by Hans Rosenfeld)

OS-7074 deadlock with vmm_detach() and iommu_cleanup()
Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
Reviewed by: John Levon <john.levon@joyent.com>
Approved by: John Levon <john.levon@joyent.com>