OS-6257: proc_t changes broke genunix dmods and walker

Details

Issue Type:Bug
Priority:4 - Normal
Status:Resolved
Created at:2017-07-31T04:39:57.000Z
Updated at:2017-08-17T01:14:04.000Z

People

Created by:Ryan Zezeski [X]
Reported by:Ryan Zezeski [X]
Assigned to:Ryan Zezeski [X]

Resolution

Fixed: A fix for this issue is checked into the tree and tested.
(Resolution Date: 2017-08-05T01:47:12.000Z)

Fix Versions

2017-08-17 YEREVAN OPTION (Release Date: 2017-08-17)

Related Links

Description

Changes made to proc_t in the last couple of years have broken many of
the proc-related dmods and the proc walker when debugging older crash
dumps on newer platforms. Below is the output of running various
proc-related genunix commands on a crash from early 2016 on a platform
from early 2017.

$ thoth debug 8109ce59fb22301cf6d76158ae080f29
thoth: using database at 165.225.169.53:28015 (configured from Manta)
thoth: debugging 8109ce59fb22301cf6d76158ae080f29
 * created interactive job -- db116b3b-fb41-e2be-ecd7-99b6edca4d3e
 * waiting for session... - established
thoth: dump info is in $THOTH_INFO
mdb: warning: dump is from SunOS 5.11 joyent_20160303T211841Z; dcmds and macros may not match kernel implementation
Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc apix scsi_vhci ufs ip hook neti sockfs arp usba uhci stmf_sbd stmf zfs mm lofs idm mpt_sas sata random cpc logindmux ptm sppp nfs sd ipc ]
> ::status
debugging crash dump /manta/thoth/stor/thoth/8109ce59fb22301cf6d76158ae080f29/vmcore.0 (64-bit) from dal-mappblx001-17
operating system: 5.11 joyent_20160303T211841Z (i86pc)
image uuid: (not set)
panic message: NMI received

dump content: kernel pages only
> !uname -v
joyent_20170202T033902Z
> ::ps
S    PID   PPID   PGID    SID    UID      FLAGS             ADDR NAME
R      0      0      0      0      0 0x00000001 fffffffffbc304c0
R     90      0      0      0      0 0x00020001 ffffff4320466008
R      3      0      0      0      0 0x00020001 ffffff431d409020
R      2      0      0      0      0 0x00020001 ffffff431d40d018
R      1      0      0      0      0 0x4a004000 ffffff431d009010
R  95020      1      8      8      0 0x4a004000 ffffff4bf98ae010 /smartlogin-agent
R  43138      1      8      8      0 0x4a004000 ffffff79d0840020  -o noorphan /opt/smartdc/agents/lib/node_modules/cn-age
R  43139  43138      8      8      0 0x4a014000 ffffffaafa3f4048 /node_modules/cn-agent/node/bin/node --abort_on_uncaught
R  43158  43139      8      8      0 0x4a004000 ffffff7adf68d020
R  42863      1  42588  42588      0 0x4a014000 ffffff68dea0f058 /node_modules/config-agent/build/node/bin/node /opt/smar
R  42573      1      8      8      0 0x4a004000 ffffff79d0922028  -o noorphan /opt/smartdc/agents/lib/node_modules/vm-age
R  42574  42573      8      8      0 0x4a014000 ffffff444ab23020 /node_modules/vm-agent/node/bin/node --abort_on_uncaught
R  42606  42574      8      8      0 0x4a004000 ffffff433266a000
R  41316      1      8      8      0 0x4a004000 ffffffaafac7d008  -o noorphan /opt/smartdc/agents/lib/node_modules/net-ag
R  41317  41316      8      8      0 0x4a014000 ffffff4444147050 /node_modules/net-agent/node/bin/node --abort_on_uncaugh
R  27788  41317      8      8      0 0x4a014000 ffffff6d138d3030 ort_on_uncaught_exception /usr/sbin/vmadm lookup -j -o u
R  27797  27788      8      8      0 0x4a004000 ffffff5e1c1b2028 -p -c
R  41553  41317      8      8      0 0x4a004000 ffffffefd78a1040
R  40094      1      8      8      0 0x4a004000 ffffff431dc61038  -o noorphan /opt/smartdc/agents/lib/node_modules/hagfis
R  40098  40094      8      8      0 0x4a014000 ffffff7cbada3020 /node_modules/hagfish-watcher/build/node/bin/node /opt/s
R  39501      1  39482  39482      0 0x4a004000 ffffff7cc447c040  -o noorphan /opt/smartdc/agents/lib/node_modules/firewa
R  39502  39501  39482  39482      0 0x4a014000 ffffff7c5675c008 /node_modules/firewaller/node/bin/node --abort_on_uncaug
R  37550      1  37508  37508      0 0x4a014000 ffffff431e166020 /node_modules/cabase/cmd/node cmd/cainstsvc.js /opt/smar
>
> ::ps -f
S    PID   PPID   PGID    SID    UID      FLAGS             ADDR NAME
R      0      0      0      0      0 0x00000001 fffffffffbc304c0
R     90      0      0      0      0 0x00020001 ffffff4320466008
R      3      0      0      0      0 0x00020001 ffffff431d409020
R      2      0      0      0      0 0x00020001 ffffff431d40d018
R      1      0      0      0      0 0x4a004000 ffffff431d009010
R  95020      1      8      8      0 0x4a004000 ffffff4bf98ae010
R  43138      1      8      8      0 0x4a004000 ffffff79d0840020 /smartdc/agents/lib/node_modules/cn-age
R  43139  43138      8      8      0 0x4a014000 ffffffaafa3f4048 agent/node/bin/node --abort_on_uncaught
R  43158  43139      8      8      0 0x4a004000 ffffff7adf68d020
R  42863      1  42588  42588      0 0x4a014000 ffffff68dea0f058 fig-agent/build/node/bin/node /opt/smar
R  42573      1      8      8      0 0x4a004000 ffffff79d0922028 /smartdc/agents/lib/node_modules/vm-age
R  42574  42573      8      8      0 0x4a014000 ffffff444ab23020 agent/node/bin/node --abort_on_uncaught
R  42606  42574      8      8      0 0x4a004000 ffffff433266a000
R  41316      1      8      8      0 0x4a004000 ffffffaafac7d008 /smartdc/agents/lib/node_modules/net-ag
R  41317  41316      8      8      0 0x4a014000 ffffff4444147050 -agent/node/bin/node --abort_on_uncaugh
R  27788  41317      8      8      0 0x4a014000 ffffff6d138d3030 xception /usr/sbin/vmadm lookup -j -o u
R  27797  27788      8      8      0 0x4a004000 ffffff5e1c1b2028
R  41553  41317      8      8      0 0x4a004000 ffffffefd78a1040
R  40094      1      8      8      0 0x4a004000 ffffff431dc61038 /smartdc/agents/lib/node_modules/hagfis
R  40098  40094      8      8      0 0x4a014000 ffffff7cbada3020 fish-watcher/build/node/bin/node /opt/s
R  39501      1  39482  39482      0 0x4a004000 ffffff7cc447c040 /smartdc/agents/lib/node_modules/firewa
R  39502  39501  39482  39482      0 0x4a014000 ffffff7c5675c008 ewaller/node/bin/node --abort_on_uncaug
R  37550      1  37508  37508      0 0x4a014000 ffffff431e166020 ase/cmd/node cmd/cainstsvc.js /opt/smar
>
> ::pgrep node
S    PID   PPID   PGID    SID    UID      FLAGS             ADDR NAME
R  43138      1      8      8      0 0x4a004000 ffffff79d0840020  -o noorphan /opt/smartdc/agents/lib/node_modules/cn-age
R  43139  43138      8      8      0 0x4a014000 ffffffaafa3f4048 /node_modules/cn-agent/node/bin/node --abort_on_uncaught
R  42863      1  42588  42588      0 0x4a014000 ffffff68dea0f058 /node_modules/config-agent/build/node/bin/node /opt/smar
R  42573      1      8      8      0 0x4a004000 ffffff79d0922028  -o noorphan /opt/smartdc/agents/lib/node_modules/vm-age
R  42574  42573      8      8      0 0x4a014000 ffffff444ab23020 /node_modules/vm-agent/node/bin/node --abort_on_uncaught
R  41316      1      8      8      0 0x4a004000 ffffffaafac7d008  -o noorphan /opt/smartdc/agents/lib/node_modules/net-ag
R  41317  41316      8      8      0 0x4a014000 ffffff4444147050 /node_modules/net-agent/node/bin/node --abort_on_uncaugh
R  40094      1      8      8      0 0x4a004000 ffffff431dc61038  -o noorphan /opt/smartdc/agents/lib/node_modules/hagfis
R  40098  40094      8      8      0 0x4a014000 ffffff7cbada3020 /node_modules/hagfish-watcher/build/node/bin/node /opt/s
R  39501      1  39482  39482      0 0x4a004000 ffffff7cc447c040  -o noorphan /opt/smartdc/agents/lib/node_modules/firewa
R  39502  39501  39482  39482      0 0x4a014000 ffffff7c5675c008 /node_modules/firewaller/node/bin/node --abort_on_uncaug
R  37550      1  37508  37508      0 0x4a014000 ffffff431e166020 /node_modules/cabase/cmd/node cmd/cainstsvc.js /opt/smar
R  36198      1      8      8      0 0x4a004000 ffffff9c75aff040  -o noorphan /opt/smartdc/agents/lib/node_modules/amon-r
R  36206  36198      8      8      0 0x4a014000 ffffff444d1a7058 agents/lib/node_modules/amon-relay/bin/amon-relay -v -Z
R  36214  36206      8      8      0 0x4a014000 ffffff4331a5e010 /node_modules/amon-relay/build/node/bin/node /opt/smartd
R  35520      1      8      8      0 0x4a004000 ffffff7c56a46050  -o noorphan /opt/smartdc/agents/lib/node_modules/amon-a
R  35521  35520      8      8      0 0x4a014000 ffffff68dea09018 agents/lib/node_modules/amon-agent/bin/amon-agent -v
R  35525  35521      8      8      0 0x4a014000 ffffff8be3b32018 /node_modules/amon-agent/build/node/bin/node /opt/smartd
R  22212  24025  22212  22212      0 0x4a014000 ffffff6d6b74c018 nfig/node_modules/.bin/view-config-service -c /opt/local
R  18127  24045  24045  24045   9416 0x4a014000 ffffff5bc7724040 nfig/node_modules/@walmart/view-config/node_modules/.bin
R  18122  24045  24045  24045   9416 0x4a014000 ffffffb55c9d0048 nfig/node_modules/@walmart/view-config/node_modules/.bin
R  18116  24045  24045  24045   9416 0x4a014000 ffffff5bb5a22040 nfig/node_modules/@walmart/view-config/node_modules/.bin
R  18111  24045  24045  24045   9416 0x4a014000 ffffff5bb85bb038 nfig/node_modules/@walmart/view-config/node_modules/.bin
>
> ::ptree
fffffffffbc304c0
     ffffff4320466008
     ffffff431d409020
     ffffff431d40d018
     ffffff431d009010
          ffffff4bf98ae010  /smartlogin-agent
          ffffff79d0840020   -o noorphan /opt/smartdc/agents/lib/node_modules/cn-age
               ffffffaafa3f4048  /node_modules/cn-agent/node/bin/node --abort_on_uncaught
                    ffffff7adf68d020
          ffffff68dea0f058  /node_modules/config-agent/build/node/bin/node /opt/smar
          ffffff79d0922028   -o noorphan /opt/smartdc/agents/lib/node_modules/vm-age
               ffffff444ab23020  /node_modules/vm-agent/node/bin/node --abort_on_uncaught
                    ffffff433266a000
          ffffffaafac7d008   -o noorphan /opt/smartdc/agents/lib/node_modules/net-ag
               ffffff4444147050  /node_modules/net-agent/node/bin/node --abort_on_uncaugh
                    ffffff6d138d3030  ort_on_uncaught_exception /usr/sbin/vmadm lookup -j -o u
                         ffffff5e1c1b2028  -p -c
                    ffffffefd78a1040
          ffffff431dc61038   -o noorphan /opt/smartdc/agents/lib/node_modules/hagfis
               ffffff7cbada3020  /node_modules/hagfish-watcher/build/node/bin/node /opt/s
          ffffff7cc447c040   -o noorphan /opt/smartdc/agents/lib/node_modules/firewa
               ffffff7c5675c008  /node_modules/firewaller/node/bin/node --abort_on_uncaug
          ffffff431e166020  /node_modules/cabase/cmd/node cmd/cainstsvc.js /opt/smar
          ffffff9c75aff040   -o noorphan /opt/smartdc/agents/lib/node_modules/amon-r
>
> ::walk proc | ::print proc_t p_dtrace_count
mdb: proc ffffff5e4417b040 has invalid p_sibling ffffff5e43fd5048; skipping
mdb: proc ffffff6d37b1b030 has invalid p_sibling ffffff5e53720030; skipping
>
> ::cpuinfo
 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  0 fffffffffbc326c0  1f    2    0  60   no    no t-2    ffffffaaec8480c0 t/local/penny/node_modules/@walmart/penny/lib/worker.js
  1 ffffff42ec7bc040  1f    1    0 169   no    no t-8    ffffff01e8443c40
  2 fffffffffbc3cfc0  1b    1    0  60   no    no t-3    ffffff4445ed2c40 t/local/penny/node_modules/@walmart/penny/lib/worker.js
  3 ffffff42ec7a5500  1f    1    0 169   no    no t-6    ffffff01e85a0c40
  4 ffffff42ec963540  1f    1    0 169   no    no t-3    ffffff01e86e6c40
  5 ffffff42ec962040  1f    0    0 161   no    no t-7    ffffff01e876fc40
  6 ffffff42ec960b00  1f    1    0 169   no    no t-3    ffffff01e87efc40
  7 ffffff42ec95f500  1f    2    0 169   no    no t-3    ffffff01e886fc40
  8 ffffff42ec95e000  1f    3    0 169   no    no t-3    ffffff01e88fec40
  9 ffffff42ec958ac0  1f    1    0 169   no    no t-3    ffffff01e899fc40
 10 ffffff42ec954b00  1f   11    0 169   no    no t-3    ffffff01e8a2ec40
 11 ffffff42ec94ea80  1f    3    0 168   no    no t-3    ffffff01e8bf7c40
 12 ffffff42eca63540  1f   14    0 168   no    no t-3    ffffff01e8cd9c40
 13 ffffff42eca62040  1f    7    0 168   no    no t-3    ffffff01e8d50c40
 14 ffffff42eca60080  1f    4    0 169   no    no t-6    ffffff01e8df1c40
 15 ffffff42eca5a000  1f    2    0 169   no    no t-3    ffffff01e8fc7c40
 16 ffffff42ecb2c080  1f    3    0 161   no    no t-8    ffffff01e92d4c40
 17 ffffff42ecb2aa80  1f    2    0 169   no    no t-3    ffffff01e934bc40
 18 ffffff42ecb23540  1f    1    0 165   no    no t-3    ffffff01e93d4c40
 19 ffffff42ecb22040  1f    3    0 165   no    no t-3    ffffff01e946fc40
 20 ffffff42ecb1eb00  1f    0    0 165   no    no t-3    ffffff01e94f8c40
 21 ffffff430ce71500  1f    0    0 165   no    no t-0    ffffff01e9566c40
 22 ffffff430ce69540  1f    0    0 165   no    no t-7    ffffff01e95fec40
>
> ffffff431e166020::pfiles
FD   TYPE            VNODE INFO
   0  CHR ffffff431d5a6b80 /devices/pseudo/mm@0:null
   1  REG ffffff4333461e00 /var/svc/log/smartdc-agent-ca-cainstsvc:default.log
   2  REG ffffff4333461e00 /var/svc/log/smartdc-agent-ca-cainstsvc:default.log
   3 PORT ffffff7c4d970780 [event port (port=ffffff96eea45a40)]
   4 FIFO ffffffffbafb1e40
   5 FIFO ffffff444ad1f300
   6  REG ffffffaae317c380 /opt/smartdc/agents/log/ca/cainstsvc/amqp-2016-10-19.18:03:53.37508
   7 SOCK ffffffaae1584800 socket: AF_INET 192.168.2.167 63975 remote: AF_INET 192.168.2.23 5672
> ffffff431e166020::fd 0
mdb: process ffffff431e166020 only has -1159869698 files open.
>
> ::lminfo
ADDR             TP FLAG    PID COMM             VNODE            PATH
ffffff44424f1340 WR 0001  56600 x_agentd         ffffffaaf976c000 /zones/16c59be6-8e1
ffffff68de0c6180 WR 0021  51450                  ffffffabf9611040 /zones/4a2d66d6-091
ffffffb116282980 WR 0021  24543                  ffffffac0477c080 /zones/01fdb772-fea
ffffff4445f2cb00 WR 0021   8084                  ffffff444c21c100 /zones/1415a57e-e3b
ffffff4e4cc6d240 WR 0021  92154                  ffffffaad99a4100 /zones/d50bbf1d-6d3
ffffffa4b3e57580 WR 0021  23085                  ffffffac00f56140 /zones/68fadf2d-fb8
ffffff444a863840 WR 0001   8046 x_agentd         ffffff4444491180 /zones/1415a57e-e3b
ffffff4449bc6440 WR 0021   9025                  ffffff444c93e280 /zones/c389d77f-ef7
ffffff444389cc40 WR 0021   7769                  ffffff4441105300 /zones/16c59be6-8e1
ffffff444930f080 WR 0021   8903                  ffffff4449e37340 /zones/2270bbde-8ae
ffffff6d35f7fd40 WR 0001  25464 x_agentd         ffffff4461190340 /zones/68fadf2d-fb8
ffffff4333031a80 WR 0001   3059 sendmail -Ac -q15m ffffff4442c61380 /var/spool/clientmq
ffffff443f580540 WR 0021  91301                  ffffffaad9e01380 /zones/ee2cddf7-291
ffffff4443547980 WR 0001   3116 sendmail -bl -q15m ffffff444420f400 /var/run/sendmail.p
ffffff443f315340 WR 0021   7881                  ffffff4444bb9440 /zones/7167d6f1-554
ffffffffbefbc740 WR 0021  61870                  ffffffb84b39a400 /zones/8d230657-048
ffffffb11624ad80 WR 0021    154                  ffffffaae892a400 /zones/c1ada27d-2d4
ffffff431dc7cd80 WR 0001    218                  ffffff43313ae540 /etc/dev/.devfsadm_
ffffff4447676600 WR 0021   8883                  ffffff44443c7580 /zones/2270bbde-8ae
ffffff4446604680 WR 0021   9036                  ffffff444ba2e580 /zones/c389d77f-ef7
ffffffa4a9f9b280 WR 0001  64199 x_agentd         ffffffffb3c60580 /zones/f65d52b1-137
ffffff44462e5440 WR 0021  92150                  ffffff4f1a42f680 /zones/d50bbf1d-6d3
ffffff4449bbe400 WR 0021   7782                  ffffff4445bbe700 /zones/16c59be6-8e1
>
> ::walk task_cache ! head -1
0xffffff4480d113c8
> 0xffffff4480d113c8::walk task
0xffffff6d6b74c018
mdb: failed to read proc at 0: no mapping for address
>
> ::walk as_cache ! head -1
0xffffffaad9b732e0
> 0xffffffaad9b732e0::as2proc
mdb: proc ffffff5e4417b040 has invalid p_sibling ffffff5e43fd5048; skipping
mdb: proc ffffff6d37b1b030 has invalid p_sibling ffffff5e53720030; skipping
mdb: proc ffffff5e23aeb010 has invalid p_sibling ffffff5e1be54048; skipping

Comments

Comment by Ryan Zezeski [X]
Created at 2017-07-31T04:42:09.000Z

https://cr.joyent.us/#/c/2302


Comment by Ryan Zezeski [X]
Created at 2017-08-04T21:05:32.000Z

Here is my test output with a patched genunix:

$ thoth debug 8109ce59fb22301cf6d76158ae080f29
thoth: using database at 165.225.169.53:28015 (configured from Manta)
thoth: debugging 8109ce59fb22301cf6d76158ae080f29
 * created interactive job -- 501eba91-06c6-eeec-d527-971664d9103f
 * waiting for session... \ established
thoth: dump info is in $THOTH_INFO
mdb: warning: dump is from SunOS 5.11 joyent_20160303T211841Z; dcmds and macros may not match kernel implementation
Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc apix scsi_vhci ufs ip hook neti sockfs arp usba uhci stmf_sbd stmf zfs mm lofs idm mpt_sas sata random cpc logindmux ptm sppp nfs sd ipc ]
> ::status
debugging crash dump /manta/thoth/stor/thoth/8109ce59fb22301cf6d76158ae080f29/vmcore.0 (64-bit) from dal-mappblx001-17
operating system: 5.11 joyent_20160303T211841Z (i86pc)
image uuid: (not set)
panic message: NMI received

dump content: kernel pages only
>
> !uname -v
joyent_20170202T033902Z
>
> !mget -o ~/genunix.so ~~/stor/dmod-genunix28.so
~~/stor/dmod-genunix28.so              [==============================================================================>] 100% 924.27KB
> !mount -F lofs ~/genunix.so /usr/lib/mdb/kvm/amd64/genunix.so
> ::unload genunix
> ::load genunix
>
> ::ps
S    PID   PPID   PGID    SID    UID      FLAGS             ADDR NAME
R      0      0      0      0      0 0x00000001 fffffffffbc304c0 sched
R     90      0      0      0      0 0x00020001 ffffff4320466008 zpool-zones
R      3      0      0      0      0 0x00020001 ffffff431d409020 fsflush
R      2      0      0      0      0 0x00020001 ffffff431d40d018 pageout
R      1      0      0      0      0 0x4a004000 ffffff431d009010 init
R  95020      1      8      8      0 0x4a004000 ffffff4bf98ae010 smartlogin
R  43138      1      8      8      0 0x4a004000 ffffff79d0840020 ctrun
R  43139  43138      8      8      0 0x4a014000 ffffffaafa3f4048 node
R  43158  43139      8      8      0 0x4a004000 ffffff7adf68d020 zoneevent
R  42863      1  42588  42588      0 0x4a014000 ffffff68dea0f058 node
R  42573      1      8      8      0 0x4a004000 ffffff79d0922028 ctrun
R  42574  42573      8      8      0 0x4a014000 ffffff444ab23020 node
R  42606  42574      8      8      0 0x4a004000 ffffff433266a000 zoneevent
R  41316      1      8      8      0 0x4a004000 ffffffaafac7d008 ctrun
R  41317  41316      8      8      0 0x4a014000 ffffff4444147050 node
R  27788  41317      8      8      0 0x4a014000 ffffff6d138d3030 vmadm
R  27797  27788      8      8      0 0x4a004000 ffffff5e1c1b2028 zoneadm
R  41553  41317      8      8      0 0x4a004000 ffffffefd78a1040 zoneevent
R  40094      1      8      8      0 0x4a004000 ffffff431dc61038 ctrun
R  40098  40094      8      8      0 0x4a014000 ffffff7cbada3020 node
R  39501      1  39482  39482      0 0x4a004000 ffffff7cc447c040 ctrun
R  39502  39501  39482  39482      0 0x4a014000 ffffff7c5675c008 node
R  37550      1  37508  37508      0 0x4a014000 ffffff431e166020 node
>
> ::ps -f
S    PID   PPID   PGID    SID    UID      FLAGS             ADDR NAME
R      0      0      0      0      0 0x00000001 fffffffffbc304c0 sched
R     90      0      0      0      0 0x00020001 ffffff4320466008 zpool-zones
R      3      0      0      0      0 0x00020001 ffffff431d409020 fsflush
R      2      0      0      0      0 0x00020001 ffffff431d40d018 pageout
R      1      0      0      0      0 0x4a004000 ffffff431d009010 /sbin/init
R  95020      1      8      8      0 0x4a004000 ffffff4bf98ae010 /opt/smartdc/agents/bin/smartlogin-agent
R  43138      1      8      8      0 0x4a004000 ffffff79d0840020 /usr/bin/ctrun -l child -o noorphan /opt/smartdc/agents/lib/node_modules/cn-age
R  43139  43138      8      8      0 0x4a014000 ffffffaafa3f4048 /opt/smartdc/agents/lib/node_modules/cn-agent/node/bin/node --abort_on_uncaught
R  43158  43139      8      8      0 0x4a004000 ffffff7adf68d020 /usr/vm/sbin/zoneevent
R  42863      1  42588  42588      0 0x4a014000 ffffff68dea0f058 /opt/smartdc/agents/lib/node_modules/config-agent/build/node/bin/node /opt/smar
R  42573      1      8      8      0 0x4a004000 ffffff79d0922028 /usr/bin/ctrun -l child -o noorphan /opt/smartdc/agents/lib/node_modules/vm-age
R  42574  42573      8      8      0 0x4a014000 ffffff444ab23020 /opt/smartdc/agents/lib/node_modules/vm-agent/node/bin/node --abort_on_uncaught
R  42606  42574      8      8      0 0x4a004000 ffffff433266a000 /usr/vm/sbin/zoneevent
R  41316      1      8      8      0 0x4a004000 ffffffaafac7d008 /usr/bin/ctrun -l child -o noorphan /opt/smartdc/agents/lib/node_modules/net-ag
R  41317  41316      8      8      0 0x4a014000 ffffff4444147050 /opt/smartdc/agents/lib/node_modules/net-agent/node/bin/node --abort_on_uncaugh
R  27788  41317      8      8      0 0x4a014000 ffffff6d138d3030 /usr/node/bin/node --abort_on_uncaught_exception /usr/sbin/vmadm lookup -j -o u
R  27797  27788      8      8      0 0x4a004000 ffffff5e1c1b2028 /usr/sbin/zoneadm list -p -c
R  41553  41317      8      8      0 0x4a004000 ffffffefd78a1040 /usr/vm/sbin/zoneevent
R  40094      1      8      8      0 0x4a004000 ffffff431dc61038 /usr/bin/ctrun -l child -o noorphan /opt/smartdc/agents/lib/node_modules/hagfis
R  40098  40094      8      8      0 0x4a014000 ffffff7cbada3020 /opt/smartdc/agents/lib/node_modules/hagfish-watcher/build/node/bin/node /opt/s
R  39501      1  39482  39482      0 0x4a004000 ffffff7cc447c040 /usr/bin/ctrun -l child -o noorphan /opt/smartdc/agents/lib/node_modules/firewa
R  39502  39501  39482  39482      0 0x4a014000 ffffff7c5675c008 /opt/smartdc/agents/lib/node_modules/firewaller/node/bin/node --abort_on_uncaug
R  37550      1  37508  37508      0 0x4a014000 ffffff431e166020 /opt/smartdc/agents/lib/node_modules/cabase/cmd/node cmd/cainstsvc.js /opt/smar
>
> ::pgrep node
S    PID   PPID   PGID    SID    UID      FLAGS             ADDR NAME
R  43139  43138      8      8      0 0x4a014000 ffffffaafa3f4048 node
R  42863      1  42588  42588      0 0x4a014000 ffffff68dea0f058 node
R  42574  42573      8      8      0 0x4a014000 ffffff444ab23020 node
R  41317  41316      8      8      0 0x4a014000 ffffff4444147050 node
R  40098  40094      8      8      0 0x4a014000 ffffff7cbada3020 node
R  39502  39501  39482  39482      0 0x4a014000 ffffff7c5675c008 node
R  37550      1  37508  37508      0 0x4a014000 ffffff431e166020 node
R  36214  36206      8      8      0 0x4a014000 ffffff4331a5e010 node
R  35525  35521      8      8      0 0x4a014000 ffffff8be3b32018 node
R  22212  24025  22212  22212      0 0x4a014000 ffffff6d6b74c018 node
R  18127  24045  24045  24045   9416 0x4a014000 ffffff5bc7724040 node
R  18122  24045  24045  24045   9416 0x4a014000 ffffffb55c9d0048 node
R  18116  24045  24045  24045   9416 0x4a014000 ffffff5bb5a22040 node
R  18111  24045  24045  24045   9416 0x4a014000 ffffff5bb85bb038 node
R  18102  24045  24045  24045   9416 0x4a014000 ffffff5bb9a8e000 node
R  18101  24045  24045  24045   9416 0x4a014000 ffffffb588a16020 node
R  18085  24045  24045  24045   9416 0x4a014000 ffffff5b934bf038 node
R  18069  24045  24045  24045   9416 0x4a014000 ffffff5bcd892020 node
R  18040  24045  24045  24045   9416 0x4a014000 ffffff5bdeaee048 node
R  80388  50936  50936  50936   9415 0x4a014000 ffffff4454636040 node
R  60532  80388  50936  50936   9415 0x4a014000 ffffff68dea0e000 node
R  60524  80388  50936  50936   9415 0x4a014000 ffffff433191c040 node
R  60519  80388  50936  50936   9415 0x4a014000 ffffffffa688c038 node
>
> ::ptree
fffffffffbc304c0  sched
     ffffff4320466008  zpool-zones
     ffffff431d409020  fsflush
     ffffff431d40d018  pageout
     ffffff431d009010  init
          ffffff4bf98ae010  smartlogin
          ffffff79d0840020  ctrun
               ffffffaafa3f4048  node
                    ffffff7adf68d020  zoneevent
          ffffff68dea0f058  node
          ffffff79d0922028  ctrun
               ffffff444ab23020  node
                    ffffff433266a000  zoneevent
          ffffffaafac7d008  ctrun
               ffffff4444147050  node
                    ffffff6d138d3030  vmadm
                         ffffff5e1c1b2028  zoneadm
                    ffffffefd78a1040  zoneevent
          ffffff431dc61038  ctrun
               ffffff7cbada3020  node
          ffffff7cc447c040  ctrun
               ffffff7c5675c008  node
          ffffff431e166020  node
          ffffff9c75aff040  ctrun
>
> ::walk proc | ::print proc_t p_dtrace_count
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
>
> ::cpuinfo
 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  0 fffffffffbc326c0  1f    2    0  60   no    no t-2    ffffffaaec8480c0 node
  1 ffffff42ec7bc040  1f    1    0 169   no    no t-8    ffffff01e8443c40 sched
  2 fffffffffbc3cfc0  1b    1    0  60   no    no t-3    ffffff4445ed2c40 node
  3 ffffff42ec7a5500  1f    1    0 169   no    no t-6    ffffff01e85a0c40 sched
  4 ffffff42ec963540  1f    1    0 169   no    no t-3    ffffff01e86e6c40 sched
  5 ffffff42ec962040  1f    0    0 161   no    no t-7    ffffff01e876fc40 sched
  6 ffffff42ec960b00  1f    1    0 169   no    no t-3    ffffff01e87efc40 sched
  7 ffffff42ec95f500  1f    2    0 169   no    no t-3    ffffff01e886fc40 sched
  8 ffffff42ec95e000  1f    3    0 169   no    no t-3    ffffff01e88fec40 sched
  9 ffffff42ec958ac0  1f    1    0 169   no    no t-3    ffffff01e899fc40 sched
 10 ffffff42ec954b00  1f   11    0 169   no    no t-3    ffffff01e8a2ec40 sched
 11 ffffff42ec94ea80  1f    3    0 168   no    no t-3    ffffff01e8bf7c40 sched
 12 ffffff42eca63540  1f   14    0 168   no    no t-3    ffffff01e8cd9c40 sched
 13 ffffff42eca62040  1f    7    0 168   no    no t-3    ffffff01e8d50c40 sched
 14 ffffff42eca60080  1f    4    0 169   no    no t-6    ffffff01e8df1c40 sched
 15 ffffff42eca5a000  1f    2    0 169   no    no t-3    ffffff01e8fc7c40 sched
 16 ffffff42ecb2c080  1f    3    0 161   no    no t-8    ffffff01e92d4c40 sched
 17 ffffff42ecb2aa80  1f    2    0 169   no    no t-3    ffffff01e934bc40 sched
 18 ffffff42ecb23540  1f    1    0 165   no    no t-3    ffffff01e93d4c40 sched
 19 ffffff42ecb22040  1f    3    0 165   no    no t-3    ffffff01e946fc40 sched
 20 ffffff42ecb1eb00  1f    0    0 165   no    no t-3    ffffff01e94f8c40 sched
 21 ffffff430ce71500  1f    0    0 165   no    no t-0    ffffff01e9566c40 sched
 22 ffffff430ce69540  1f    0    0 165   no    no t-7    ffffff01e95fec40 sched
>
> ffffff431e166020::pfiles
FD   TYPE            VNODE INFO
   0  CHR ffffff431d5a6b80 /devices/pseudo/mm@0:null
   1  REG ffffff4333461e00 /var/svc/log/smartdc-agent-ca-cainstsvc:default.log
   2  REG ffffff4333461e00 /var/svc/log/smartdc-agent-ca-cainstsvc:default.log
   3 PORT ffffff7c4d970780 [event port (port=ffffff96eea45a40)]
   4 FIFO ffffffffbafb1e40
   5 FIFO ffffff444ad1f300
   6  REG ffffffaae317c380 /opt/smartdc/agents/log/ca/cainstsvc/amqp-2016-10-19.18:03:53.37508
   7 SOCK ffffffaae1584800 socket: AF_INET 192.168.2.167 63975 remote: AF_INET 192.168.2.23 5672
>
> ffffff431e166020::fd 0
ffffff4331bcb508
>
> ::lminfo
ADDR             TP FLAG    PID COMM             VNODE            PATH
ffffff44424f1340 WR 0001  56600 zabbix_agentd    ffffffaaf976c000 /zones/16c59be6-8e1
ffffff68de0c6180 WR 0021  51450 sac              ffffffabf9611040 /zones/4a2d66d6-091
ffffffb116282980 WR 0021  24543 sac              ffffffac0477c080 /zones/01fdb772-fea
ffffff4445f2cb00 WR 0021   8084 sac              ffffff444c21c100 /zones/1415a57e-e3b
ffffff4e4cc6d240 WR 0021  92154 ttymon           ffffffaad99a4100 /zones/d50bbf1d-6d3
ffffffa4b3e57580 WR 0021  23085 sac              ffffffac00f56140 /zones/68fadf2d-fb8
ffffff444a863840 WR 0001   8046 zabbix_agentd    ffffff4444491180 /zones/1415a57e-e3b
ffffff4449bc6440 WR 0021   9025 sac              ffffff444c93e280 /zones/c389d77f-ef7
ffffff444389cc40 WR 0021   7769 sac              ffffff4441105300 /zones/16c59be6-8e1
ffffff444930f080 WR 0021   8903 ttymon           ffffff4449e37340 /zones/2270bbde-8ae
ffffff6d35f7fd40 WR 0001  25464 zabbix_agentd    ffffff4461190340 /zones/68fadf2d-fb8
ffffff4333031a80 WR 0001   3059 sendmail         ffffff4442c61380 /var/spool/clientmq
ffffff443f580540 WR 0021  91301 sac              ffffffaad9e01380 /zones/ee2cddf7-291
ffffff4443547980 WR 0001   3116 sendmail         ffffff444420f400 /var/run/sendmail.p
ffffff443f315340 WR 0021   7881 sac              ffffff4444bb9440 /zones/7167d6f1-554
ffffffffbefbc740 WR 0021  61870 sac              ffffffb84b39a400 /zones/8d230657-048
ffffffb11624ad80 WR 0021    154 sac              ffffffaae892a400 /zones/c1ada27d-2d4
ffffff431dc7cd80 WR 0001    218 devfsadm         ffffff43313ae540 /etc/dev/.devfsadm_
ffffff4447676600 WR 0021   8883 sac              ffffff44443c7580 /zones/2270bbde-8ae
ffffff4446604680 WR 0021   9036 ttymon           ffffff444ba2e580 /zones/c389d77f-ef7
ffffffa4a9f9b280 WR 0001  64199 zabbix_agentd    ffffffffb3c60580 /zones/f65d52b1-137
ffffff44462e5440 WR 0021  92150 sac              ffffff4f1a42f680 /zones/d50bbf1d-6d3
ffffff4449bbe400 WR 0021   7782 ttymon           ffffff4445bbe700 /zones/16c59be6-8e1
>
> ::walk task_cache ! head -1
0xffffff4480d113c8
> 0xffffff4480d113c8::walk task
0xffffff6d6b74c018
>
> ::walk as_cache ! head -1
0xffffffaad9b732e0
> 0xffffffaad9b732e0::as2proc
ffffff444b845058

Comment by Bot Bot [X]
Created at 2017-08-05T01:31:38.000Z

illumos-joyent commit a9f065e (branch master, by Ryan Zezeski)

OS-6257 proc_t changes broke genunix dmods and walker
OS-6271 mdb_pid2proc() and consumers should honor return type
Reviewed by: Robert Mustacchi <rm@joyent.com>
Reviewed by: Jason King <jason.king@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>