OS-7064: failed elfexec leads to segnp, and worse

Details

Issue Type:Bug
Priority:4 - Normal
Status:Resolved
Created at:2018-07-05T14:52:04.484Z
Updated at:2019-08-20T13:19:55.007Z

People

Created by:Former user
Reported by:Former user
Assigned to:Former user

Resolution

Fixed: A fix for this issue is checked into the tree and tested.
(Resolution Date: 2018-07-30T08:52:56.016Z)

Fix Versions

2018-08-02 XCannon (Release Date: 2018-08-02)

Description

As described here:

https://gist.github.com/cneira/305b416c70db1c2d16061730b495d259

running that binary will #df current SmartOS, even with all previous LDT fixes in. Running the same binary in a different image doesn't cause the issue: there is something particularly wrong in that image that causes the problem.

Comments

Comment by Former user
Created at 2018-07-05T15:07:28.232Z

This also manifests as:

/lib/ld-linux.so.2: not found.
/lib/ld-linux.so.2: not found.

panic[cpu0]/thread=fffffe0fbb835080: 
BAD TRAP: type=e (#pf Page fault) rp=fffffe0015fc8f10 addr=fffffe0fbb835210


bash: 
#pf Page fault
Bad kernel fault at addr=0xfffffe0fbb835210
pid=4469, pc=0xfffffffffb8631f4, sp=0xfffffffffbc4dc70, eflags=0x10282
cr0: 80050033<pg,wp,ne,et,mp,pe>  cr4: 3626b8<smap,smep,osxsav,pcide,vmxe,xmme,fxsr,pge,pae,pse,de>
cr2: fffffe0fbb835210  
cr3: 2bffd3000  
cr8: 0

        rdi: fffffffffbc4dd70 rsi:     7fffef354b0a rdx:                0
        rcx:                0  r8: fffffffffbc8a420  r9:       9018f03a32
        rax: fffffffffbc8a420 rbx: fffffe0fbb835080 rbp: fffffffffbc4dd60
        r10: fffffe0fba34daa8 r11: fffffe0fba34dab8 r12: fffffffffbc4dd70
        r13:                0 r14: fffffffffbc4de68 r15:     7fffef354b0a
        fsb:     7fffef0c2a40 gsb: fffffffffbc4c000  ds:               4b
         es:               4b  fs:                0  gs:              1c3
        trp:                e err:                0 rip: fffffffffb8631f4
         cs:               30 rfl:            10282 rsp: fffffffffbc4dc70
         ss:                0

fffffe0015fc8e00 unix:die+89 ()
fffffe0015fc8f00 unix:trap+e60 ()
fffffe0015fc8f10 unix:_cmntrap+e6 ()
fffffffffbc4dd60 unix:trap+44 ()
fffffffffbc4dd70 unix:_cmntrap+e6 ()

With the #df case being:

/lib/ld-linux.so.2: not found.
/lib/ld-linux.so.2: not found.

panic[cpu1]/thread=fffffe0fbb70c7e0: 
BAD TRAP: type=8 (#df Double fault) rp=fffffe0fb528df10 addr=0


bash: 
#df Double fault
pid=4859, pc=0xfffffffffb802bc1, sp=0xfffffe0fb4e8fed0, eflags=0x2
cr0: 80050033<pg,wp,ne,et,mp,pe>  cr4: 3626b8<smap,smep,osxsav,pcide,vmxe,xmme,fxsr,pge,pae,pse,de>
cr2: fffffe00155a1f58  
cr3: 1e000000  
cr8: 0

        rdi:           6ec170 rsi:           6f2090 rdx: fffffffffbc07280
        rcx:               4b  r8:               a3  r9:      3d6719e912f
        rax: fffffffffbc4c000 rbx:                0 rbp: fffffe00155a2050
        r10: fffffe0fbb5deda8 r11: fffffe0fbb5dedb8 r12:           6ec170
        r13: fffffe0fb4e8fed0 r14: fffffe00155a1f60 r15:                0
        fsb:     7fffef0c2a40 gsb: fffffe0fb4e8e000  ds:               4b
         es:               4b  fs:                0  gs:              1c3
        trp:                8 err:                0 rip: fffffffffb802bc1
         cs:               30 rfl:                2 rsp: fffffe0fb4e8fed0
         ss:                0
tss.tss_rsp0:   0xfffffe0fb4e8fe90
tss.tss_rsp1:   0x0
tss.tss_rsp2:   0x0
tss.tss_ist1:   0xfffffe0fb528e000
tss.tss_ist2:   0xfffffe0fb5293000
tss.tss_ist3:   0xfffffe0fb5298000
tss.tss_ist4:   0xfffffe0fb4e8ffb0    
tss.tss_ist5:   0xfffffe0fb4e8ff20
tss.tss_ist6:   0xfffffe0fb4e8fe90
tss.tss_ist7:   0x0

fffffe0fb528de00 unix:die+89 ()
fffffe0fb528df00 unix:trap+628 ()
fffffe00155a2050 unix:_smap_enable_patch_25+627e ()
fffffe00155a2140 unix:_cmntrap+ad ()
fffffe00155a2230 unix:_cmntrap+ad ()
fffffe00155a2320 unix:_cmntrap+ad ()
fffffe00155a2410 unix:_cmntrap+ad ()
fffffe00155a2500 unix:_cmntrap+ad ()
...
fffffe0fb4e8fd70 unix:segnptrap+46 ()

The #df is due to the explicit int $8 in DBG_INTERRUPT_TRAMPOLINE_P()


Comment by Former user
Created at 2018-07-05T15:20:47.327Z

Trying to run under gdb or strace doesn't reproduce the issue, just SEGV's the process correctly (as it's 32-bit).


Comment by Former user
Created at 2018-07-20T12:36:06.280Z
Updated at 2018-07-20T12:48:24.279Z

With kpti=0 we see:

  1. ./bin/linux-x86/nwserver-linux
    /lib/ld-linux.so.2: not found.Segmentation fault (core dumped)

tudebug:

2018-07-20T12:29:25.884651+00:00 volcano unix: [ID 839527 kern.notice] bash: 
2018-07-20T12:29:25.884672+00:00 volcano unix: [ID 753105 kern.notice] #pf Page fault
2018-07-20T12:29:25.884682+00:00 volcano unix: [ID 532287 kern.notice] Bad user fault at addr=0x7fffef3584fa
2018-07-20T12:29:25.884694+00:00 volcano unix: [ID 243837 kern.notice] pid=106483, pc=0x7fffef3584fa, sp=0x8047698, eflags=0x10213
2018-07-20T12:29:25.884704+00:00 volcano unix: [ID 619397 kern.notice] cr0: 80050033<pg,wp,ne,et,mp,pe>  cr4: 3426f8<smap,smep,osxsav,vmxe,xmme,fxsr,pge,mce,pae,pse,de>
2018-07-20T12:29:25.884717+00:00 volcano unix: [ID 152204 kern.notice] cr2: 7fffef3584fa  
2018-07-20T12:29:25.884727+00:00 volcano unix: [ID 634440 kern.notice] cr3: 207ef77000  
2018-07-20T12:29:25.884734+00:00 volcano unix: [ID 625715 kern.notice] cr8: 0
2018-07-20T12:29:25.884742+00:00 volcano unix: [ID 100000 kern.notice] 
2018-07-20T12:29:25.884750+00:00 volcano unix: [ID 592667 kern.notice] #011rdi:           6e9020 rsi:           6e74c0 rdx:           6f1aa0
2018-07-20T12:29:25.884757+00:00 volcano unix: [ID 592667 kern.notice] #011rcx:     7fffef35981b  r8:                0  r9:                1
2018-07-20T12:29:25.884765+00:00 volcano unix: [ID 592667 kern.notice] #011rax:                2 rbx:           6e74c0 rbp:     7fffef0adce0
2018-07-20T12:29:25.884773+00:00 volcano unix: [ID 592667 kern.notice] #011r10:     7fffef35981b r11:     7fffef0ad830 r12:           6e9020
2018-07-20T12:29:25.884780+00:00 volcano unix: [ID 592667 kern.notice] #011r13:     7fffef0add90 r14:     7fffef4dbe70 r15:           6e7560
2018-07-20T12:29:25.884787+00:00 volcano unix: [ID 592667 kern.notice] #011fsb:     7fffef0c2a40 gsb: fffffffffbc5a000  ds:               4b
2018-07-20T12:29:25.884795+00:00 volcano unix: [ID 592667 kern.notice] #011 es:               4b  fs:                0  gs:                0
2018-07-20T12:29:25.884803+00:00 volcano unix: [ID 592667 kern.notice] #011trp:                e err:               14 rip:     7fffef3584fa
2018-07-20T12:29:25.884810+00:00 volcano unix: [ID 592667 kern.notice] #011 cs:               53 rfl:            10213 rsp:          8047698
2018-07-20T12:29:25.884820+00:00 volcano unix: [ID 266532 kern.notice] #011 ss:               4b
2018-07-20T12:29:25.884829+00:00 volcano unix: [ID 839527 kern.notice] bash: 
2018-07-20T12:29:25.884843+00:00 volcano unix: [ID 753105 kern.notice] #np Segment not present
2018-07-20T12:29:25.884855+00:00 volcano unix: [ID 243837 kern.notice] pid=106483, pc=0xfffffffffb80248a, sp=0xfffffcc26c4c5fc8, eflags=0x10097
2018-07-20T12:29:25.884865+00:00 volcano unix: [ID 619397 kern.notice] cr0: 80050033<pg,wp,ne,et,mp,pe>  cr4: 3426f8<smap,smep,osxsav,vmxe,xmme,fxsr,pge,mce,pae,pse,de>
2018-07-20T12:29:25.884875+00:00 volcano unix: [ID 152204 kern.notice] cr2: 7fffef0c2ae8  
2018-07-20T12:29:25.884884+00:00 volcano unix: [ID 634440 kern.notice] cr3: 207ef77000  
2018-07-20T12:29:25.884891+00:00 volcano unix: [ID 625715 kern.notice] cr8: 0
2018-07-20T12:29:25.884901+00:00 volcano unix: [ID 100000 kern.notice] 
2018-07-20T12:29:25.884908+00:00 volcano unix: [ID 592667 kern.notice] #011rdi:           6e9020 rsi:           6e74c0 rdx:           6f1aa0
2018-07-20T12:29:25.884917+00:00 volcano unix: [ID 592667 kern.notice] #011rcx:         ef35981b  r8:               47  r9:                3
2018-07-20T12:29:25.884925+00:00 volcano unix: [ID 592667 kern.notice] #011rax:                2 rbx:           6e74c0 rbp:         ef0adce0
2018-07-20T12:29:25.884932+00:00 volcano unix: [ID 592667 kern.notice] #011r10: fffffe261db816e8 r11: fffffe261db816f8 r12:           6e9020
2018-07-20T12:29:25.884940+00:00 volcano unix: [ID 592667 kern.notice] #011r13:     7fffef0add90 r14:     7fffef4dbe70 r15:     7fffef3584fa
2018-07-20T12:29:25.884948+00:00 volcano unix: [ID 592667 kern.notice] #011fsb:     7fffef0c2a40 gsb: fffffffffbc5a000  ds:               4b
2018-07-20T12:29:25.884957+00:00 volcano unix: [ID 592667 kern.notice] #011 es:               4b  fs:                0  gs:              1c3
2018-07-20T12:29:25.884965+00:00 volcano unix: [ID 592667 kern.notice] #011trp:                b err:               40 rip: fffffffffb80248a
2018-07-20T12:29:25.884972+00:00 volcano unix: [ID 592667 kern.notice] #011 cs:               30 rfl:            10097 rsp: fffffcc26c4c5fc8
2018-07-20T12:29:25.884979+00:00 volcano unix: [ID 266532 kern.notice] #011 ss:               38

The IP for that segnp is tr_iret_user's iret. So we seem to be taking - and fixing - a #pf but then getting an #np when we try to iret. Not sure why that's happening, but we then go on to create a core:

[44]> $C
fffffcc26be00c60 do_core()
fffffcc26be00db0 kern_gpfault+0x20e(fffffcc26be00ed0)
fffffcc26be00ec0 trap+0x66e(fffffcc26be00ed0, 7fffef3584fa, 2c)
fffffcc26be00ed0 0xfffffffffb8002ea()

This corresponds to the comment about bad iret's we see above kern_gpfault().


Comment by Former user
Created at 2018-07-20T13:29:55.064Z

The tudebug for the np isn't very useful, as it's just the context of the iret. But the instr_is_iret() handling places the regs back into lwp_regs, so if we stop at core() we can look at the supposed state we were trying to iret with:

[8]> fffffe286974d080::print kthread_t t_lwp->lwp_regs | ::print struct regs
{
    r_savfp = 0xef0eebf0
    r_savpc = 0xfffffffffb80248a
    r_rdi = 0x8ec8d0
    r_rsi = 0x8ec190
    r_rdx = 0x8f6d90
    r_rcx = 0xef37981b
    r_r8 = 0x46
    r_r9 = 0x2
    r_rax = 0x2
    r_rbx = 0x8ec190
    r_rbp = 0xef0eebf0
    r_r10 = 0xfffffe2ab8e2d5e8
    r_r11 = 0xfffffe2ab8e2d5f0
    r_r12 = 0x8ec8d0
    r_r13 = 0x7fffef0eeca0
    r_r14 = 0x7fffef4dbe70
    r_r15 = 0x7fffef3784fa
    __r_fsbase = 0x7fffef112a40
    __r_gsbase = 0xfffffe25ca7ff000
    r_ds = 0x4b
    r_es = 0x4b
    r_fs = 0
    r_gs = 0x1c3                      
    r_trapno = 0xb
    r_err = 0x40
    r_rip = 0x7fffef362be0
    r_cs = 0x43
    r_rfl = 0x202
    r_rsp = 0x8047408
    r_ss = 0x4b
}

Note that r_cs - that's U32CS_SEL. And by this point p_model is 32-bit for bash (as we've been through exec_args). We're likely in a very odd state, where we failed to find our interpreter in lx_elfexec, so we're half converted to a 32-bit proc, but not quite. So it's no surprise that things go badly wrong, and I suspect this particular scenario predates KPTI.

What's obviously worth with KPTI is that when we hit the problem with the iret, we don't even get as far as the np trap handling.


Comment by Former user
Created at 2018-07-20T15:52:40.420Z

I edited a 32-bit native proc to have a bad p_interp, then execve()d that from a 64-bit native proc. truss isn't too useful:

126012:	lwp_cond_broadcast(0xFFFFFBFFEF3401A8)		= 0
126012:	sysi86(SI86FPSTART, 0xFFFFFBFFFFDFFB0C, 0x0000133F, 0x00001F80) = 0x00000001
badls: Cannot find /usr/lhb/ld.so.1

This looks like it's because it hits "bad:" in elfexec() and is immediately killed. For LX procs, we instead do this:

 501         if ((level <= INTP_MAXDEPTH) &&
 502             (*brand_action != EBA_NATIVE) && (PROC_IS_BRANDED(p))) {
 503                 error = BROP(p)->b_elfexec(vp, uap, args,
 504                     idatap, level + 1, execsz, setid, exec_file, cred,
 505                     brand_action);
 506                 goto out;
 507         }

There is no understanding of a "point of no return" here where we need to SIGKILL instead of attempting to return an error from the exec syscall. This in itself seems like a (pretty minor) bug.


Comment by Former user
Created at 2018-07-20T16:42:03.273Z
Updated at 2018-07-26T18:33:44.325Z

This looks to be the reason why the iret itself is unhappy: r_cs is U32CODE_SEL, but the GDT entry is:

    usd_lolimit = 0xffff
    usd_lobase = 0
    usd_midbase = 0
    usd_type = 0x1b
    usd_dpl = 0x3
    usd_p = 0 <!-----------------------------------
    usd_hilimit = 0xf
    usd_avl = 0
    usd_long = 0
    usd_def32 = 0x1
    usd_gran = 0x1
    usd_hibase = 0

i.e. ucs32_off. This is because when gexec() fails, we never take the exec_common()->setregs()->gdt_ucode_model() path to update the GDT entries for our new p_model (as the lx code moved us to be 32-bit).

So we at least have a clear explanation (I think) of what's going on in the non-KPTI case.


Comment by Former user
Created at 2018-07-25T10:41:19.960Z
Updated at 2018-07-25T12:07:02.878Z

Here's why we're falling over with KPTI:

We follow the same path as non-KPTI up until we get to the iret. Remember at this point we have a %cs that's pointing to an invalid GDT entry.

We then take an #NP on the iret itself. The trap %rip is the iret instructions, and we're KCS_SEL (and with %cr3 the user one). With tr_segnptrap:

        cmpw    $KCS_SEL, KPTI_CS(%rsp);        \
        je      3f;                             \
1:                                              \
        /* Change to the "kernel" %cr3 */       \
        mov     KPTI_KCR3(%rsp), %r14;          \
        cmp     $0, %r14;                       \
        je      2f;                             \
        mov     %r14, %cr3;                     \
2:                                              \
        /* Get our cpu_t in %r13 */             \
        mov     %rsp, %r13;                     \
        and     $(~(MMU_PAGESIZE - 1)), %r13;   \
        subq    $CPU_KPTI_START, %r13;          \
        /* Use top of the kthread stk */        \
        mov     CPU_THREAD(%r13), %r14;         \
        mov     T_STACK(%r14), %r14;            \
        addq    $REGSIZE+MINFRAME, %r14;        \
        jmp     4f;                             \
3:                                              \
        /* Check the %rsp in the frame. */      \
        /* Is it above kernel base? */          \
        mov     kpti_kbase, %r14;               \
        cmp     %r14, KPTI_RSP(%rsp);           \
        jb      1b;                             \
        /* Use the %rsp from the trap frame */  \
        mov     KPTI_RSP(%rsp), %r14;           \
        and     $(~0xf), %r14;                  \
4:                                              \
...

Nothing here will catch that we're running with the user %cr3. So we'll happily vector off to segnptrap, and eventually fall over trying to deference CPU_GDT(%rax) from the cpu[] array, which is obviously not mapped.

It seems that after all we never tested a bad iret under KPTI.

The simplest fix appears to be to use MK_DBG_INTR_TRAMPOLINE(segnptrap). While we don't use the _dbg IST here still, it catches this scenario sufficiently, because of this:

        /* Is it within the kpti_frame page? */ \
        /* If it is, treat as user interrupt */ \
        mov     %rsp, %r13;                     \
        and     $(~(MMU_PAGESIZE - 1)), %r13;   \
        mov     KPTI_RSP(%rsp), %r14;           \
        and     $(~(MMU_PAGESIZE - 1)), %r14;   \
        cmp     %r13, %r14;                     \
        je      2b;                             \

Since we were doing the iret with a frame held within the KPTI frame region, this is true, and we jump back and load the kernel cr3 correctly.


Comment by Former user
Created at 2018-07-26T12:22:27.193Z

This was tested along with the fix for OS-7090 (so badseg can run OK, along with the rest of os-test, including the LDT test).

I verified that the NWN binary now core-dumps under KPTI, just like it "should" (I'm not sure if it's worth filing a bug on the brand exec handlers not using the "bad:" SIGKILL path ?)

We survived LTP runs in both 32-bit and 64-bit LX zones (including the modify_ldt tests).

Also ran a bunch of compiles in bhyve and native zones with these bits, and COAL.


Comment by Former user
Created at 2018-07-26T19:30:41.445Z

Copying the gist data into the ticket...

Using latest SmartOS joyent_20180426T014228Z trying to execute a linux 32-bit binary on a lx branded zone causes a kernel panic.

dmesg:

2018-05-04T13:46:40.000634+00:00 cl-west-0 savecore: [ID 570001 auth.error] reboot after panic: BAD TRAP: type=8 (#df Double fault) rp=fffffe2ce26b8f10 addr=0
2018-05-04T13:46:39+00:00 cl-west-0 savecore: [ID 676877 auth.error] Saving compressed system crash dump in /var/crash/volatile/vmdump.3

mdb info:

debugging crash dump vmcore.3 (64-bit) from cl-west-0
operating system: 5.11 joyent_20180426T014228Z (i86pc)
image uuid: (not set)
panic message: BAD TRAP: type=8 (#df Double fault) rp=fffffe2ce26b8f10 addr=0
dump content: kernel pages only
> ::msgbuf
MESSAGE
pseudo-device: fbt0
fbt0 is /pseudo/fbt@0
pseudo-device: lockstat0
lockstat0 is /pseudo/lockstat@0
pseudo-device: lx_systrace0
lx_systrace0 is /pseudo/lx_systrace@0
pseudo-device: profile0
profile0 is /pseudo/profile@0
pseudo-device: sdt0
sdt0 is /pseudo/sdt@0
pseudo-device: systrace0
systrace0 is /pseudo/systrace@0
Creating /etc/devices/devid_cache
Creating /etc/devices/pci_unitaddr_persistent
Creating /etc/devices/devname_cache
sd0 at scsa2usb0: target 0 lun 0
sd0 is /pci@0,0/pci17aa,30c1@14/storage@5/disk@0,0
device pciclass,030000@2(display#0) keeps up device sd@0,0(disk#0), but the former is not power managed
sd2 at ahci0: target 5 lun 0
sd2 is /pci@0,0/pci17aa,30c1@17/cdrom@5,0
device pciclass,030000@2(display#0) keeps up device scsiclass,05@5,0(cdrom#2), but the former is not power managed
/lib/ld-linux.so.2: not found.
/lib/ld-linux.so.2: not found.

panic[cpu1]/thread=fffffe2cf5522100:
BAD TRAP: type=8 (#df Double fault) rp=fffffe2ce26b8f10 addr=0


bash:
#df Double fault
pid=5787, pc=0xfffffffffb802bc1, sp=0xfffffe2ce2618ed0, eflags=0x2
cr0: 8005003b<pg,wp,ne,et,ts,mp,pe>  cr4: 3606f8<smap,smep,osxsav,pcide,xmme,fxsr,pge,mce,pae,pse,de>
cr2: fffffe003f3e8f58
cr3: 1e000000
cr8: 0

        rdi:           8f9d30 rsi:           8ec0b0 rdx: fffffffffbc07280
        rcx:               4b  r8:               a3  r9:      2c5e9b4fed8
        rax: fffffe2ce2617000 rbx:                1 rbp: fffffe003f3e9050
        r10: fffffe2cfb12c6a8 r11: fffffe2cfb12c6b8 r12:           8f9d30
        r13: fffffe2ce2618ed0 r14: fffffe003f3e8f60 r15:                0
        fsb:     7fffef122a40 gsb: fffffe2ce2617000  ds:               4b
         es:               4b  fs:                0  gs:              1c3
        trp:                8 err:                0 rip: fffffffffb802bc1
         cs:               30 rfl:                2 rsp: fffffe2ce2618ed0
         ss:                0
tss.tss_rsp0:   0xfffffe2ce2618e90
tss.tss_rsp1:   0x0
tss.tss_rsp2:   0x0
tss.tss_ist1:   0xfffffe2ce26b9000
tss.tss_ist2:   0xfffffe2ce26be000
tss.tss_ist3:   0xfffffe2ce26c3000
tss.tss_ist4:   0xfffffe2ce2618fb0
tss.tss_ist5:   0xfffffe2ce2618f20
tss.tss_ist6:   0xfffffe2ce2618e90
tss.tss_ist7:   0x0

fffffe2ce26b8e00 unix:die+89 ()
fffffe2ce26b8f00 unix:trap+11fd ()
fffffe003f3e9050 unix:_patch_xrstorq_rbx+18f ()
fffffe003f3e9140 unix:_cmntrap+ad ()
fffffe003f3e9230 unix:_cmntrap+ad ()
fffffe003f3e9320 unix:_cmntrap+ad ()
fffffe003f3e9410 unix:_cmntrap+ad ()
fffffe003f3e9500 unix:_cmntrap+ad ()
fffffe003f3e95f0 unix:_cmntrap+ad ()
fffffe003f3e96e0 unix:_cmntrap+ad ()
fffffe003f3e97d0 unix:_cmntrap+ad ()
fffffe003f3e98c0 unix:_cmntrap+ad ()
fffffe003f3e99b0 unix:_cmntrap+ad ()
fffffe003f3e9aa0 unix:_cmntrap+ad ()
fffffe003f3e9b90 unix:_cmntrap+ad ()
fffffe003f3e9c80 unix:_cmntrap+ad ()
fffffe003f3e9d70 unix:_cmntrap+ad ()
fffffe003f3e9e60 unix:_cmntrap+ad ()
fffffe003f3e9f50 unix:_cmntrap+ad ()
fffffe003f3ea040 unix:_cmntrap+ad ()
fffffe003f3ea130 unix:_cmntrap+ad ()
fffffe003f3ea220 unix:_cmntrap+ad ()
fffffe003f3ea310 unix:_cmntrap+ad ()
fffffe003f3ea400 unix:_cmntrap+ad ()
fffffe003f3ea4f0 unix:_cmntrap+ad ()
fffffe003f3ea5e0 unix:_cmntrap+ad ()
fffffe003f3ea6d0 unix:_cmntrap+ad ()
fffffe003f3ea7c0 unix:_cmntrap+ad ()
fffffe003f3ea8b0 unix:_cmntrap+ad ()
fffffe003f3ea9a0 unix:_cmntrap+ad ()
fffffe003f3eaa90 unix:_cmntrap+ad ()
fffffe003f3eab80 unix:_cmntrap+ad ()
fffffe003f3eac70 unix:_cmntrap+ad ()
fffffe003f3ead60 unix:_cmntrap+ad ()
fffffe003f3eae50 unix:_cmntrap+ad ()
fffffe003f3eaf40 unix:_cmntrap+ad ()
fffffe003f3eb030 unix:_cmntrap+ad ()
fffffe003f3eb120 unix:_cmntrap+ad ()
fffffe003f3eb210 unix:_cmntrap+ad ()
fffffe003f3eb300 unix:_cmntrap+ad ()
fffffe003f3eb3f0 unix:_cmntrap+ad ()
fffffe003f3eb4e0 unix:_cmntrap+ad ()
fffffe003f3eb5d0 unix:_cmntrap+ad ()
fffffe003f3eb6c0 unix:_cmntrap+ad ()
fffffe003f3eb7b0 unix:_cmntrap+ad ()
fffffe003f3eb8a0 unix:_cmntrap+ad ()
fffffe003f3eb990 unix:_cmntrap+ad ()
fffffe003f3eba80 unix:_cmntrap+ad ()
fffffe003f3ebb70 unix:_cmntrap+ad ()
fffffe003f3ebc60 unix:_cmntrap+ad ()
fffffe003f3ebd50 unix:_cmntrap+ad ()
fffffe003f3ebe40 unix:_cmntrap+ad ()
fffffe003f3ebf30 unix:_cmntrap+ad ()
fffffe003f3ec020 unix:_cmntrap+ad ()
fffffe003f3ec110 unix:_cmntrap+ad ()
fffffe003f3ec200 unix:_cmntrap+ad ()
fffffe003f3ec2f0 unix:_cmntrap+ad ()
fffffe003f3ec3e0 unix:_cmntrap+ad ()
fffffe003f3ec4d0 unix:_cmntrap+ad ()
fffffe003f3ec5c0 unix:_cmntrap+ad ()
fffffe003f3ec6b0 unix:_cmntrap+ad ()
fffffe003f3ec7a0 unix:_cmntrap+ad ()
fffffe003f3ec890 unix:_cmntrap+ad ()
fffffe003f3ec980 unix:_cmntrap+ad ()
fffffe003f3eca70 unix:_cmntrap+ad ()
fffffe003f3ecb60 unix:_cmntrap+ad ()
fffffe003f3ecc50 unix:_cmntrap+ad ()
fffffe003f3ecd40 unix:_cmntrap+ad ()
fffffe003f3ece30 unix:_cmntrap+ad ()
fffffe003f3ecf20 unix:_cmntrap+ad ()
fffffe003f3ed010 unix:_cmntrap+ad ()
fffffe003f3ed100 unix:_cmntrap+ad ()
fffffe003f3ed1f0 unix:_cmntrap+ad ()
fffffe003f3ed2e0 unix:_cmntrap+ad ()
fffffe003f3ed3d0 unix:_cmntrap+ad ()
fffffe003f3ed4c0 unix:_cmntrap+ad ()
fffffe003f3ed5b0 unix:_cmntrap+ad ()
fffffe003f3ed6a0 unix:_cmntrap+ad ()
fffffe003f3ed790 unix:_cmntrap+ad ()
fffffe003f3ed880 unix:_cmntrap+ad ()
fffffe003f3ed970 unix:_cmntrap+ad ()
fffffe003f3eda60 unix:_cmntrap+ad ()
fffffe003f3edb50 unix:_cmntrap+ad ()
fffffe003f3edc40 unix:_cmntrap+ad ()
fffffe003f3edd30 unix:_cmntrap+ad ()
fffffe003f3ede20 unix:_cmntrap+ad ()
fffffe003f3edf10 unix:_cmntrap+ad ()
fffffe2ce2618d70 unix:segnptrap+46 ()

dumping to /dev/zvol/dsk/zones/dump, offset 65536, content: kernel
dumping to /dev/zvol/dsk/zones/dump, offset 65536, content: kernel
NOTICE: ahci0: ahci_tran_reset_dport port 0 reset port
>
> $C
fffffe003f3e9050 tr_pftrap+0x11()
fffffe003f3e9140 0xfffffffffb80019d()
fffffe003f3e9230 0xfffffffffb80019d()
fffffe003f3e9320 0xfffffffffb80019d()
fffffe003f3e9410 0xfffffffffb80019d()
fffffe003f3e9500 0xfffffffffb80019d()
fffffe003f3e95f0 0xfffffffffb80019d()
fffffe003f3e96e0 0xfffffffffb80019d()
fffffe003f3e97d0 0xfffffffffb80019d()
fffffe003f3e98c0 0xfffffffffb80019d()
fffffe003f3e99b0 0xfffffffffb80019d()
fffffe003f3e9aa0 0xfffffffffb80019d()
fffffe003f3e9b90 0xfffffffffb80019d()
fffffe003f3e9c80 0xfffffffffb80019d()
fffffe003f3e9d70 0xfffffffffb80019d()
fffffe003f3e9e60 0xfffffffffb80019d()
fffffe003f3e9f50 0xfffffffffb80019d()
fffffe003f3ea040 0xfffffffffb80019d()
fffffe003f3ea130 0xfffffffffb80019d()
fffffe003f3ea220 0xfffffffffb80019d()
fffffe003f3ea310 0xfffffffffb80019d()
fffffe003f3ea400 0xfffffffffb80019d()
fffffe003f3ea4f0 0xfffffffffb80019d()
fffffe003f3ea5e0 0xfffffffffb80019d()
fffffe003f3ea6d0 0xfffffffffb80019d()
fffffe003f3ea7c0 0xfffffffffb80019d()
fffffe003f3ea8b0 0xfffffffffb80019d()
fffffe003f3ea9a0 0xfffffffffb80019d()
fffffe003f3eaa90 0xfffffffffb80019d()
fffffe003f3eab80 0xfffffffffb80019d()
fffffe003f3eac70 0xfffffffffb80019d()
fffffe003f3ead60 0xfffffffffb80019d()
fffffe003f3eae50 0xfffffffffb80019d()
fffffe003f3eaf40 0xfffffffffb80019d()
fffffe003f3eb030 0xfffffffffb80019d()
fffffe003f3eb120 0xfffffffffb80019d()
fffffe003f3eb210 0xfffffffffb80019d()
fffffe003f3eb300 0xfffffffffb80019d()
fffffe003f3eb3f0 0xfffffffffb80019d()
fffffe003f3eb4e0 0xfffffffffb80019d()
fffffe003f3eb5d0 0xfffffffffb80019d()
fffffe003f3eb6c0 0xfffffffffb80019d()
fffffe003f3eb7b0 0xfffffffffb80019d()
fffffe003f3eb8a0 0xfffffffffb80019d()
fffffe003f3eb990 0xfffffffffb80019d()
fffffe003f3eba80 0xfffffffffb80019d()
fffffe003f3ebb70 0xfffffffffb80019d()
fffffe003f3ebc60 0xfffffffffb80019d()
fffffe003f3ebd50 0xfffffffffb80019d()
fffffe003f3ebe40 0xfffffffffb80019d()
fffffe003f3ebf30 0xfffffffffb80019d()
fffffe003f3ec020 0xfffffffffb80019d()
fffffe003f3ec110 0xfffffffffb80019d()
fffffe003f3ec200 0xfffffffffb80019d()
fffffe003f3ec2f0 0xfffffffffb80019d()
fffffe003f3ec3e0 0xfffffffffb80019d()
fffffe003f3ec4d0 0xfffffffffb80019d()
fffffe003f3ec5c0 0xfffffffffb80019d()
fffffe003f3ec6b0 0xfffffffffb80019d()
fffffe003f3ec7a0 0xfffffffffb80019d()
fffffe003f3ec890 0xfffffffffb80019d()
fffffe003f3ec980 0xfffffffffb80019d()
fffffe003f3eca70 0xfffffffffb80019d()
fffffe003f3ecb60 0xfffffffffb80019d()
fffffe003f3ecc50 0xfffffffffb80019d()
fffffe003f3ecd40 0xfffffffffb80019d()
fffffe003f3ece30 0xfffffffffb80019d()
fffffe003f3ecf20 0xfffffffffb80019d()
fffffe003f3ed010 0xfffffffffb80019d()
fffffe003f3ed100 0xfffffffffb80019d()
fffffe003f3ed1f0 0xfffffffffb80019d()
fffffe003f3ed2e0 0xfffffffffb80019d()
fffffe003f3ed3d0 0xfffffffffb80019d()
fffffe003f3ed4c0 0xfffffffffb80019d()
fffffe003f3ed5b0 0xfffffffffb80019d()
fffffe003f3ed6a0 0xfffffffffb80019d()
fffffe003f3ed790 0xfffffffffb80019d()
fffffe003f3ed880 0xfffffffffb80019d()
fffffe003f3ed970 0xfffffffffb80019d()
fffffe003f3eda60 0xfffffffffb80019d()
fffffe003f3edb50 0xfffffffffb80019d()
fffffe003f3edc40 0xfffffffffb80019d()
fffffe003f3edd30 0xfffffffffb80019d()
fffffe003f3ede20 0xfffffffffb80019d()
fffffe003f3edf10 0xfffffffffb80019d()
fffffe2ce2618d70 segnptrap+0x46()
>
> ::cpuinfo -v
 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  0 fffffffffbc4c000  1f    1    0  -1   no    no t-0    fffffe003ce05c40 (idle)
                       |    |
            RUNNING <--+    +-->  PRI THREAD           PROC
              READY                60 fffffe00408cac40 sched
           QUIESCED
             EXISTS
             ENABLE

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  1 fffffffffbc51e00  1b    1    0   1   no    no t-0    fffffe2cf5522100 bash
                       |    |
            RUNNING <--+    +-->  PRI THREAD           PROC
              READY                 1 fffffe2cf54a58a0 rsyslogd
             EXISTS
             ENABLE

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  2 fffffe2ce2613000  1f    0    0   1   no    no t-0    fffffe2ceea21c20 sshd
                       |
            RUNNING <--+
              READY
           QUIESCED
             EXISTS
             ENABLE

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  3 fffffe2ce2615000  1f    0    0  59   no    no t-0    fffffe2ce94474e0 rsyslogd
                       |
            RUNNING <--+
              READY
           QUIESCED
             EXISTS
             ENABLE
             
> ::panicinfo
             cpu                1
          thread fffffe2cf5522100
         message BAD TRAP: type=8 (#df Double fault) rp=fffffe2ce26b8f10 addr=0
             rdi           8f9d30
             rsi           8ec0b0
             rdx fffffffffbc07280
             rcx               4b
              r8               a3
              r9      2c5e9b4fed8
             rax fffffe2ce2617000
             rbx                1
             rbp fffffe003f3e9050
             r10 fffffe2cfb12c6a8
             r11 fffffe2cfb12c6b8
             r12           8f9d30
             r13 fffffe2ce2618ed0
             r14 fffffe003f3e8f60
             r15                0
          fsbase     7fffef122a40
          gsbase fffffe2ce2617000
              ds               4b
              es               4b
              fs                0
              gs              1c3
          trapno                8
             err                0
             rip fffffffffb802bc1
              cs               30
          rflags                2
             rsp fffffe2ce2618ed0
              ss                0
          gdt_hi                0
          gdt_lo         a00001ef
          idt_hi                0
          idt_lo         90000fff
             ldt                0
            task               70
             cr0         8005003b
             cr2 fffffe003f3e8f58
             cr3         1e000000
             cr4           3606f8
>

How to replicate:

1. Import the centos 6 lx image

    $ imgadm import 68a837fe-1b9b-11e7-a66d-ab7961786c42

2. Create a lx zone using this json

{
"alias": "nwnee-lx",
"brand": "lx",
"kernel_version": "4.3.0",
"max_physical_memory": 2048,
"quota": 60,
"image_uuid": "68a837fe-1b9b-11e7-a66d-ab7961786c42",
"resolvers": ["8.8.8.8","8.8.8.4"],
"nics": [
{
"nic_tag": "igb1",
"ip": "192.168.1.200",
"netmask": "255.255.255.0",
"gateway": "192.168.1.1",
"primary": true
}
]
}

3. In the newly generated lx zone do the following, ignore error about multilib:

    $ yum install libgcc.i686 libstdc++.i686

4. Install neverwinter nights 8166

    wget https://nwnx.io/nwnee-dedicated-8166.zip --no-check-certificate

5. Unzip nwnee-dedicated-8166.zip and cd to bin/linux*

6. Execute nwserver-linux

This bug is also in Omniosce SunOS krondor 5.11 omnios-master-37dded565b i86pc i386 i86pc

> ::cpuinfo
 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  0 fffffffffbc4c000  1f    0    0  -1   no    no t-0    fffffe003ce05c40 (idle)
  1 fffffffffbc51e00  1b    0    0  59   no    no t-0    fffffe2d096b53a0 bash
  2 fffffe2ce943e000  1f    0    0  59   no    no t-0    fffffe2cf2afbba0 syslogd
  3 fffffe2ce980e000  1f    1    0  59  yes    no t-0    fffffe2d1c595480 rsyslogd
> ::msgbuf
MESSAGE
hid3 is /pci@0,0/pci17aa,30c1@14/hub@1/hub@3/hub@4/device@2/keyboard@0
/pci@0,0/pci17aa,30c1@14/hub@1/hub@3/hub@4/device@2/keyboard@0 (hid3) online
USB 1.10 interface (usbif5f3,7.config1.1) operating at full speed (USB 1.x) on USB 1.10 external hub: input@1, hid4 at bus address 7
hid4 is /pci@0,0/pci17aa,30c1@14/hub@1/hub@3/hub@4/device@2/input@1
/pci@0,0/pci17aa,30c1@14/hub@1/hub@3/hub@4/device@2/input@1 (hid4) online
/pci@0,0/pci17aa,30c1@14/hub@1/hub@3/hub@4/device@2 (usb_mid1) online
WARNING: tis_init: bad intf_caps value 0x2000069F
WARNING: tis_init: bad intf_caps value 0x2000069F
/pseudo/zconsnex@1/zcons@0 (zcons0) online
WARNING: tis_init: bad intf_caps value 0x2000069F
WARNING: tis_init: bad intf_caps value 0x2000069F
/pseudo/zconsnex@1/zcons@0 (zcons0) online
WARNING: tis_init: bad intf_caps value 0x2000069F
WARNING: tis_init: bad intf_caps value 0x2000069F
/pseudo/zconsnex@1/zcons@0 (zcons0) online
WARNING: tis_init: bad intf_caps value 0x2000069F
WARNING: tis_init: bad intf_caps value 0x2000069F
/pseudo/zconsnex@1/zcons@0 (zcons0) online
WARNING: tis_init: bad intf_caps value 0x2000069F
NOTICE: vnic1019 registered
NOTICE: igb2 link up, 100 Mbps, full duplex
NOTICE: vnic1019 link up, 100 Mbps, unknown duplex
/pseudo/zconsnex@1/zcons@1 (zcons1) online
WARNING: tis_init: bad intf_caps value 0x2000069F
/lib/ld-linux.so.2: not found.
/lib/ld-linux.so.2: not found.

panic[cpu1]/thread=fffffe2d096b53a0:
BAD TRAP: type=8 (#df Double fault) rp=fffffe2ce9961f10 addr=0


bash:
#df Double fault
pid=510, pc=0xfffffffffb802bc1, sp=0xfffffe2ce943ded0, eflags=0x2
cr0: 80050033<pg,wp,ne,et,mp,pe>  cr4: 3606f8<smap,smep,osxsav,pcide,xmme,fxsr,pge,mce,pae,pse,de>
cr2: fffffe003e5f9f58
cr3: c400000
cr8: 0

        rdi:           6f6f90 rsi:           6e94a0 rdx: fffffffffbc07280
        rcx:               4b  r8:               a3  r9:    26c50e1491e00
        rax: fffffe2ce943c000 rbx:                1 rbp: fffffe003e5fa050
        r10: fffffe2d50f9a6a8 r11: fffffe2d50f9a6b8 r12:           6f6f90
        r13: fffffe2ce943ded0 r14: fffffe003e5f9f60 r15:                0
        fsb:     7fffef0c2a40 gsb: fffffe2ce943c000  ds:               4b
         es:               4b  fs:                0  gs:              1c3
        trp:                8 err:                0 rip: fffffffffb802bc1
         cs:               30 rfl:                2 rsp: fffffe2ce943ded0
         ss:                0
tss.tss_rsp0:   0xfffffe2ce943de90
tss.tss_rsp1:   0x0
tss.tss_rsp2:   0x0
tss.tss_ist1:   0xfffffe2ce9962000
tss.tss_ist2:   0xfffffe2ce9967000
tss.tss_ist3:   0xfffffe2ce996c000
tss.tss_ist4:   0xfffffe2ce943dfb0
tss.tss_ist5:   0xfffffe2ce943df20
tss.tss_ist6:   0xfffffe2ce943de90
tss.tss_ist7:   0x0

fffffe2ce9961e00 unix:die+89 ()
fffffe2ce9961f00 unix:trap+628 ()
fffffe003e5fa050 unix:_smap_enable_patch_25+629e ()
fffffe003e5fa140 unix:cmntrap+ad ()
fffffe003e5fa230 unix:cmntrap+ad ()
fffffe003e5fa320 unix:cmntrap+ad ()
fffffe003e5fa410 unix:cmntrap+ad ()
fffffe003e5fa500 unix:cmntrap+ad ()
fffffe003e5fa5f0 unix:cmntrap+ad ()
fffffe003e5fa6e0 unix:cmntrap+ad ()
fffffe003e5fa7d0 unix:cmntrap+ad ()
fffffe003e5fa8c0 unix:cmntrap+ad ()
fffffe003e5fa9b0 unix:cmntrap+ad ()
fffffe003e5faaa0 unix:cmntrap+ad ()
fffffe003e5fab90 unix:cmntrap+ad ()
fffffe003e5fac80 unix:cmntrap+ad ()
fffffe003e5fad70 unix:cmntrap+ad ()
fffffe003e5fae60 unix:cmntrap+ad ()
fffffe003e5faf50 unix:cmntrap+ad ()
fffffe003e5fb040 unix:cmntrap+ad ()
fffffe003e5fb130 unix:cmntrap+ad ()
fffffe003e5fb220 unix:cmntrap+ad ()
fffffe003e5fb310 unix:cmntrap+ad ()
fffffe003e5fb400 unix:cmntrap+ad ()
fffffe003e5fb4f0 unix:cmntrap+ad ()
fffffe003e5fb5e0 unix:cmntrap+ad ()
fffffe003e5fb6d0 unix:cmntrap+ad ()
fffffe003e5fb7c0 unix:cmntrap+ad ()
fffffe003e5fb8b0 unix:cmntrap+ad ()
fffffe003e5fb9a0 unix:cmntrap+ad ()
fffffe003e5fba90 unix:cmntrap+ad ()
fffffe003e5fbb80 unix:cmntrap+ad ()
fffffe003e5fbc70 unix:cmntrap+ad ()
fffffe003e5fbd60 unix:cmntrap+ad ()
fffffe003e5fbe50 unix:cmntrap+ad ()
fffffe003e5fbf40 unix:cmntrap+ad ()
fffffe003e5fc030 unix:cmntrap+ad ()
fffffe003e5fc120 unix:cmntrap+ad ()
fffffe003e5fc210 unix:cmntrap+ad ()
fffffe003e5fc300 unix:cmntrap+ad ()
fffffe003e5fc3f0 unix:cmntrap+ad ()
fffffe003e5fc4e0 unix:cmntrap+ad ()
fffffe003e5fc5d0 unix:cmntrap+ad ()
fffffe003e5fc6c0 unix:cmntrap+ad ()
fffffe003e5fc7b0 unix:cmntrap+ad ()
fffffe003e5fc8a0 unix:cmntrap+ad ()
fffffe003e5fc990 unix:cmntrap+ad ()
fffffe003e5fca80 unix:cmntrap+ad ()
fffffe003e5fcb70 unix:cmntrap+ad ()
fffffe003e5fcc60 unix:cmntrap+ad ()
fffffe003e5fcd50 unix:cmntrap+ad ()
fffffe003e5fce40 unix:cmntrap+ad ()
fffffe003e5fcf30 unix:cmntrap+ad ()
fffffe003e5fd020 unix:cmntrap+ad ()
fffffe003e5fd110 unix:cmntrap+ad ()
fffffe003e5fd200 unix:cmntrap+ad ()
fffffe003e5fd2f0 unix:cmntrap+ad ()
fffffe003e5fd3e0 unix:cmntrap+ad ()
fffffe003e5fd4d0 unix:cmntrap+ad ()
fffffe003e5fd5c0 unix:cmntrap+ad ()
fffffe003e5fd6b0 unix:cmntrap+ad ()
fffffe003e5fd7a0 unix:cmntrap+ad ()
fffffe003e5fd890 unix:cmntrap+ad ()
fffffe003e5fd980 unix:cmntrap+ad ()
fffffe003e5fda70 unix:cmntrap+ad ()
fffffe003e5fdb60 unix:cmntrap+ad ()
fffffe003e5fdc50 unix:cmntrap+ad ()
fffffe003e5fdd40 unix:cmntrap+ad ()
fffffe003e5fde30 unix:cmntrap+ad ()
fffffe003e5fdf20 unix:cmntrap+ad ()
fffffe003e5fe010 unix:cmntrap+ad ()
fffffe003e5fe100 unix:cmntrap+ad ()
fffffe003e5fe1f0 unix:cmntrap+ad ()
fffffe003e5fe2e0 unix:cmntrap+ad ()
fffffe003e5fe3d0 unix:cmntrap+ad ()
fffffe003e5fe4c0 unix:cmntrap+ad ()
fffffe003e5fe5b0 unix:cmntrap+ad ()
fffffe003e5fe6a0 unix:cmntrap+ad ()
fffffe003e5fe790 unix:cmntrap+ad ()
fffffe003e5fe880 unix:cmntrap+ad ()
fffffe003e5fe970 unix:cmntrap+ad ()
fffffe003e5fea60 unix:cmntrap+ad ()
fffffe003e5feb50 unix:cmntrap+ad ()
fffffe003e5fec40 unix:cmntrap+ad ()
fffffe003e5fed30 unix:cmntrap+ad ()
fffffe003e5fee20 unix:cmntrap+ad ()
fffffe003e5fef10 unix:cmntrap+ad ()
fffffe2ce943dd70 unix:segnptrap+46 ()

dumping to /dev/zvol/dsk/rpool/dump, offset 65536, content: kernel + curproc
dumping to /dev/zvol/dsk/rpool/dump, offset 65536, content: kernel + curproc
NOTICE: ahci0: ahci_tran_reset_dport port 0 reset port

>  ::cpuinfo -v
 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  0 fffffffffbc4c000  1f    0    0  -1   no    no t-0    fffffe003ce05c40 (idle)
                       |
            RUNNING <--+
              READY
           QUIESCED
             EXISTS
             ENABLE

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  1 fffffffffbc51e00  1b    0    0  59   no    no t-0    fffffe2d096b53a0 bash
                       |
            RUNNING <--+
              READY
             EXISTS
             ENABLE

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  2 fffffe2ce943e000  1f    0    0  59   no    no t-0    fffffe2cf2afbba0 syslogd
                       |
            RUNNING <--+
              READY
           QUIESCED
             EXISTS
             ENABLE

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  3 fffffe2ce980e000  1f    1    0  59  yes    no t-0    fffffe2d1c595480 rsyslogd
                       |    |
            RUNNING <--+    +-->  PRI THREAD           PROC
              READY                60 fffffe003cf2ec40 sched
           QUIESCED
             EXISTS
             ENABLE

> ::panicinfo
             cpu                1
          thread fffffe2d096b53a0
         message BAD TRAP: type=8 (#df Double fault) rp=fffffe2ce9961f10 addr=0
             rdi           6f6f90
             rsi           6e94a0
             rdx fffffffffbc07280
             rcx               4b
              r8               a3
              r9    26c50e1491e00
             rax fffffe2ce943c000
             rbx                1
             rbp fffffe003e5fa050
             r10 fffffe2d50f9a6a8
             r11 fffffe2d50f9a6b8
             r12           6f6f90
             r13 fffffe2ce943ded0
             r14 fffffe003e5f9f60
             r15                0
          fsbase     7fffef0c2a40
          gsbase fffffe2ce943c000
              ds               4b
              es               4b
              fs                0
              gs              1c3
          trapno                8
             err                0
             rip fffffffffb802bc1
              cs               30
          rflags                2
             rsp fffffe2ce943ded0
              ss                0
          gdt_hi                0
          gdt_lo         a00001ef
          idt_hi                0
          idt_lo         50000fff
             ldt                0
            task               70
             cr0         80050033
             cr2 fffffe003e5f9f58
             cr3          c400000
             cr4           3606f8
> $C
fffffe003e5fa050 tr_pftrap+0x11()
fffffe003e5fa140 0xfffffffffb80019d()
fffffe003e5fa230 0xfffffffffb80019d()
fffffe003e5fa320 0xfffffffffb80019d()
fffffe003e5fa410 0xfffffffffb80019d()
fffffe003e5fa500 0xfffffffffb80019d()
fffffe003e5fa5f0 0xfffffffffb80019d()
fffffe003e5fa6e0 0xfffffffffb80019d()
fffffe003e5fa7d0 0xfffffffffb80019d()
fffffe003e5fa8c0 0xfffffffffb80019d()
fffffe003e5fa9b0 0xfffffffffb80019d()
fffffe003e5faaa0 0xfffffffffb80019d()
fffffe003e5fab90 0xfffffffffb80019d()
fffffe003e5fac80 0xfffffffffb80019d()
fffffe003e5fad70 0xfffffffffb80019d()
fffffe003e5fae60 0xfffffffffb80019d()
fffffe003e5faf50 0xfffffffffb80019d()
fffffe003e5fb040 0xfffffffffb80019d()
fffffe003e5fb130 0xfffffffffb80019d()
fffffe003e5fb220 0xfffffffffb80019d()
fffffe003e5fb310 0xfffffffffb80019d()
fffffe003e5fb400 0xfffffffffb80019d()
fffffe003e5fb4f0 0xfffffffffb80019d()
fffffe003e5fb5e0 0xfffffffffb80019d()
fffffe003e5fb6d0 0xfffffffffb80019d()
fffffe003e5fb7c0 0xfffffffffb80019d()
fffffe003e5fb8b0 0xfffffffffb80019d()
fffffe003e5fb9a0 0xfffffffffb80019d()
fffffe003e5fba90 0xfffffffffb80019d()
fffffe003e5fbb80 0xfffffffffb80019d()
fffffe003e5fbc70 0xfffffffffb80019d()
fffffe003e5fbd60 0xfffffffffb80019d()
fffffe003e5fbe50 0xfffffffffb80019d()
fffffe003e5fbf40 0xfffffffffb80019d()
fffffe003e5fc030 0xfffffffffb80019d()
fffffe003e5fc120 0xfffffffffb80019d()
fffffe003e5fc210 0xfffffffffb80019d()
fffffe003e5fc300 0xfffffffffb80019d()
fffffe003e5fc3f0 0xfffffffffb80019d()
fffffe003e5fc4e0 0xfffffffffb80019d()
fffffe003e5fc5d0 0xfffffffffb80019d()
fffffe003e5fc6c0 0xfffffffffb80019d()
fffffe003e5fc7b0 0xfffffffffb80019d()
fffffe003e5fc8a0 0xfffffffffb80019d()
fffffe003e5fc990 0xfffffffffb80019d()
fffffe003e5fca80 0xfffffffffb80019d()
fffffe003e5fcb70 0xfffffffffb80019d()
fffffe003e5fcc60 0xfffffffffb80019d()
fffffe003e5fcd50 0xfffffffffb80019d()
fffffe003e5fce40 0xfffffffffb80019d()
fffffe003e5fcf30 0xfffffffffb80019d()
fffffe003e5fd020 0xfffffffffb80019d()
fffffe003e5fd110 0xfffffffffb80019d()
fffffe003e5fd200 0xfffffffffb80019d()
fffffe003e5fd2f0 0xfffffffffb80019d()
fffffe003e5fd3e0 0xfffffffffb80019d()
fffffe003e5fd4d0 0xfffffffffb80019d()
fffffe003e5fd5c0 0xfffffffffb80019d()
fffffe003e5fd6b0 0xfffffffffb80019d()
fffffe003e5fd7a0 0xfffffffffb80019d()
fffffe003e5fd890 0xfffffffffb80019d()
fffffe003e5fd980 0xfffffffffb80019d()
fffffe003e5fda70 0xfffffffffb80019d()
fffffe003e5fdb60 0xfffffffffb80019d()
fffffe003e5fdc50 0xfffffffffb80019d()
fffffe003e5fdd40 0xfffffffffb80019d()
fffffe003e5fde30 0xfffffffffb80019d()
fffffe003e5fdf20 0xfffffffffb80019d()
fffffe003e5fe010 0xfffffffffb80019d()
fffffe003e5fe100 0xfffffffffb80019d()
fffffe003e5fe1f0 0xfffffffffb80019d()
fffffe003e5fe2e0 0xfffffffffb80019d()
fffffe003e5fe3d0 0xfffffffffb80019d()
fffffe003e5fe4c0 0xfffffffffb80019d()
fffffe003e5fe5b0 0xfffffffffb80019d()
fffffe003e5fe6a0 0xfffffffffb80019d()
fffffe003e5fe790 0xfffffffffb80019d()
fffffe003e5fe880 0xfffffffffb80019d()
fffffe003e5fe970 0xfffffffffb80019d()
fffffe003e5fea60 0xfffffffffb80019d()
fffffe003e5feb50 0xfffffffffb80019d()
fffffe003e5fec40 0xfffffffffb80019d()
fffffe003e5fed30 0xfffffffffb80019d()
fffffe003e5fee20 0xfffffffffb80019d()
fffffe003e5fef10 0xfffffffffb80019d()
fffffe2ce943dd70 segnptrap+0x46()

Latest SmartOS

/lib/ld-linux.so.2: not found.
/lib/ld-linux.so.2: not found.

panic[cpu1]/thread=fffffe0bde758080:
BAD TRAP: type=8 (#df Double fault) rp=fffffe0bcb409f10 addr=0


bash:
#df Double fault
pid=4579, pc=0xfffffffffb802bc1, sp=0xfffffe0bcb0a4ed0, eflags=0x2
cr0: 80050033<pg,wp,ne,et,mp,pe>  cr4: 606b8<osxsav,pcide,xmme,fxsr,pge,pae,pse,de>
cr2: fffffe000faf3f58
cr3: 1e000000
cr8: c

        rdi:           6f75f0 rsi:           6f7a10 rdx: fffffffffbc07280
        rcx:               4b  r8:               a3  r9:      16ff5034aab
        rax: fffffffffbc4c000 rbx:                0 rbp: fffffe000faf4050
        r10: fffffe0bcd39de68 r11: fffffe0bcd39de78 r12:           6f75f0
        r13: fffffe0bcb0a4ed0 r14: fffffe000faf3f60 r15:                0
        fsb:     7fffef0c2a40 gsb: fffffe0bcb0a3000  ds:               4b
         es:               4b  fs:                0  gs:              1c3
        trp:                8 err:                0 rip: fffffffffb802bc1
         cs:               30 rfl:                2 rsp: fffffe0bcb0a4ed0
         ss:               38
tss.tss_rsp0:   0xfffffe0bcb0a4e90
tss.tss_rsp1:   0x0
tss.tss_rsp2:   0x0
tss.tss_ist1:   0xfffffe0bcb40a000
tss.tss_ist2:   0xfffffe0bcb40f000
tss.tss_ist3:   0xfffffe0bcb414000
tss.tss_ist4:   0xfffffe0bcb0a4fb0
tss.tss_ist5:   0xfffffe0bcb0a4f20
tss.tss_ist6:   0xfffffe0bcb0a4e90
tss.tss_ist7:   0x0

fffffe0bcb409e00 unix:die+89 ()
fffffe0bcb409f00 unix:trap+628 ()
fffffe000faf4050 unix:_smap_enable_patch_25+627e ()
fffffe000faf4140 unix:_cmntrap+ad ()
fffffe000faf4230 unix:_cmntrap+ad ()
fffffe000faf4320 unix:_cmntrap+ad ()
fffffe000faf4410 unix:_cmntrap+ad ()
fffffe000faf4500 unix:_cmntrap+ad ()
fffffe000faf45f0 unix:_cmntrap+ad ()
fffffe000faf46e0 unix:_cmntrap+ad ()
fffffe000faf47d0 unix:_cmntrap+ad ()
fffffe000faf48c0 unix:_cmntrap+ad ()
fffffe000faf49b0 unix:_cmntrap+ad ()
fffffe000faf4aa0 unix:_cmntrap+ad ()
fffffe000faf4b90 unix:_cmntrap+ad ()
fffffe000faf4c80 unix:_cmntrap+ad ()
fffffe000faf4d70 unix:_cmntrap+ad ()
fffffe000faf4e60 unix:_cmntrap+ad ()
fffffe000faf4f50 unix:_cmntrap+ad ()
fffffe000faf5040 unix:_cmntrap+ad ()
fffffe000faf5130 unix:_cmntrap+ad ()
fffffe000faf5220 unix:_cmntrap+ad ()
fffffe000faf5310 unix:_cmntrap+ad ()
fffffe000faf5400 unix:_cmntrap+ad ()
fffffe000faf54f0 unix:_cmntrap+ad ()
fffffe000faf55e0 unix:_cmntrap+ad ()
fffffe000faf56d0 unix:_cmntrap+ad ()
fffffe000faf57c0 unix:_cmntrap+ad ()
fffffe000faf58b0 unix:_cmntrap+ad ()
fffffe000faf59a0 unix:_cmntrap+ad ()
fffffe000faf5a90 unix:_cmntrap+ad ()
fffffe000faf5b80 unix:_cmntrap+ad ()
fffffe000faf5c70 unix:_cmntrap+ad ()
fffffe000faf5d60 unix:_cmntrap+ad ()
fffffe000faf5e50 unix:_cmntrap+ad ()
fffffe000faf5f40 unix:_cmntrap+ad ()
fffffe000faf6030 unix:_cmntrap+ad ()
fffffe000faf6120 unix:_cmntrap+ad ()
fffffe000faf6210 unix:_cmntrap+ad ()
fffffe000faf6300 unix:_cmntrap+ad ()
fffffe000faf63f0 unix:_cmntrap+ad ()
fffffe000faf64e0 unix:_cmntrap+ad ()
fffffe000faf65d0 unix:_cmntrap+ad ()
fffffe000faf66c0 unix:_cmntrap+ad ()
fffffe000faf67b0 unix:_cmntrap+ad ()
fffffe000faf68a0 unix:_cmntrap+ad ()
fffffe000faf6990 unix:_cmntrap+ad ()
fffffe000faf6a80 unix:_cmntrap+ad ()
fffffe000faf6b70 unix:_cmntrap+ad ()
fffffe000faf6c60 unix:_cmntrap+ad ()
fffffe000faf6d50 unix:_cmntrap+ad ()
fffffe000faf6e40 unix:_cmntrap+ad ()
fffffe000faf6f30 unix:_cmntrap+ad ()
fffffe000faf7020 unix:_cmntrap+ad ()
fffffe000faf7110 unix:_cmntrap+ad ()
fffffe000faf7200 unix:_cmntrap+ad ()
fffffe000faf72f0 unix:_cmntrap+ad ()
fffffe000faf73e0 unix:_cmntrap+ad ()
fffffe000faf74d0 unix:_cmntrap+ad ()
fffffe000faf75c0 unix:_cmntrap+ad ()
fffffe000faf76b0 unix:_cmntrap+ad ()
fffffe000faf77a0 unix:_cmntrap+ad ()
fffffe000faf7890 unix:_cmntrap+ad ()
fffffe000faf7980 unix:_cmntrap+ad ()
fffffe000faf7a70 unix:_cmntrap+ad ()
fffffe000faf7b60 unix:_cmntrap+ad ()
fffffe000faf7c50 unix:_cmntrap+ad ()
fffffe000faf7d40 unix:_cmntrap+ad ()
fffffe000faf7e30 unix:_cmntrap+ad ()
fffffe000faf7f20 unix:_cmntrap+ad ()
fffffe000faf8010 unix:_cmntrap+ad ()
fffffe000faf8100 unix:_cmntrap+ad ()
fffffe000faf81f0 unix:_cmntrap+ad ()
fffffe000faf82e0 unix:_cmntrap+ad ()
fffffe000faf83d0 unix:_cmntrap+ad ()
fffffe000faf84c0 unix:_cmntrap+ad ()
fffffe000faf85b0 unix:_cmntrap+ad ()
fffffe000faf86a0 unix:_cmntrap+ad ()
fffffe000faf8790 unix:_cmntrap+ad ()
fffffe000faf8880 unix:_cmntrap+ad ()
fffffe000faf8970 unix:_cmntrap+ad ()
fffffe000faf8a60 unix:_cmntrap+ad ()
fffffe000faf8b50 unix:_cmntrap+ad ()
fffffe000faf8c40 unix:_cmntrap+ad ()
fffffe000faf8d30 unix:_cmntrap+ad ()
fffffe000faf8e20 unix:_cmntrap+ad ()
fffffe000faf8f10 unix:_cmntrap+ad ()
fffffe0bcb0a4d70 unix:segnptrap+46 ()

Comment by Jira Bot
Created at 2018-07-30T08:45:41.458Z

illumos-joyent commit 5a38f11740e61db1a5560cc3bb03bebdbd684845 (branch master, by John Levon)

OS-7064 failed elfexec leads to segnp, and worse
Reviewed by: Robert Mustacchi <rm@joyent.com>
Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>


Comment by Former user
Created at 2019-06-21T12:26:31.243Z

This also occurred on joyent_20180427T064036Z - see INC-1566.