[osol-help] fmd service not starting

[Rob Johnston]

Moving to fm-discuss.

Short answer is we'd need to see the fmd process core to be able to tell you
what's going on. You mention you're running build 101b, which is fairly old, so
there's a reasonable chance that you're hitting a bug that's been fixed in a
later build.

Can you send us the output of the following command?:

% pstack <path to fmd core file>

(by default fmd drops its cores in /var/fm/fmd/)

rob

Hey ya'll,
Was hoping someone could shed some light on this... I have an installation of OpenSolaris (snv_101b), and I keep noticing that my / partition is full. I discovered that /var/fm/fmd was filling it up with ~100GB of core.fmd.* files. I can delete them easily enough, but sure enough, they come back after a few days.
# svcs -xv
svc:/system/intrd:default (interrupt balancer)
State: maintenance since Sun Sep 13 19:14:36 2009
Reason: Restarting too quickly.
See: http://sun.com/msg/SMF-8000-L5
See: man -M /usr/share/man -s 1M intrd
See: /var/svc/log/system-intrd:default.log
Impact: This service is not running.
svc:/system/fmd:default (Solaris Fault Manager)
State: maintenance since Mon Sep 14 04:15:03 2009
Reason: Restarting too quickly.
See: http://sun.com/msg/SMF-8000-L5
See: man -M /usr/share/man -s 1M fmd
See: /var/svc/log/system-fmd:default.log
Impact: This service is not running.
[ Sep 14 04:14:35 Executing start method ("/usr/lib/fm/fmd/fmd"). ]
[ Sep 14 04:14:35 Method "start" exited with status 0. ]
[ Sep 14 04:14:41 Stopping because process dumped core. ]
[ Sep 14 04:14:41 Executing stop method (:kill). ]
[ Sep 14 04:14:41 Executing start method ("/usr/lib/fm/fmd/fmd"). ]
[ Sep 14 04:14:42 Method "start" exited with status 0. ]
[ Sep 14 04:14:43 Stopping because process dumped core. ]
[ Sep 14 04:14:43 Executing stop method (:kill). ]
[ Sep 14 04:14:43 Executing start method ("/usr/lib/fm/fmd/fmd"). ]
[ Sep 14 04:14:44 Method "start" exited with status 0. ]
[ Sep 14 04:14:49 Stopping because process dumped core. ]
[ Sep 14 04:14:50 Executing stop method (:kill). ]
[ Sep 14 04:14:50 Executing start method ("/usr/lib/fm/fmd/fmd"). ]
[ Sep 14 04:14:50 Method "start" exited with status 0. ]
[ Sep 14 04:14:56 Method "start" exited with status 0. ]
[ Sep 14 04:14:57 Stopping because process dumped core. ]
[ Sep 14 04:14:57 Executing stop method (:kill). ]
[ Sep 14 04:14:57 Executing start method ("/usr/lib/fm/fmd/fmd"). ]
[ Sep 14 04:14:58 Method "start" exited with status 0. ]
[ Sep 14 04:15:02 Method "start" exited with status 0. ]
Just for fun I also looked at the intrd service because that seemed to be having problems too.
# view /var/svc/log/system-intrd:default.log
...
Config.pm did not return a true value at /usr/perl5/5.8.4/lib/i86pc-solaris-64int/DynaLoader.pm line 25.
BEGIN failed--compilation aborted at /usr/perl5/5.8.4/lib/i86pc-solaris-64int/DynaLoader.pm line 25.
Compilation failed in require at /usr/perl5/5.8.4/lib/Sun/Solaris/Kstat.pm line 33.
BEGIN failed--compilation aborted at /usr/perl5/5.8.4/lib/Sun/Solaris/Kstat.pm line 33.
Compilation failed in require at /usr/lib/intrd line 69.
[ Sep 13 19:14:36 Stopping because all processes in service exited. ]
[ Sep 13 19:14:36 Executing stop method (:kill). ]
[ Sep 13 19:14:36 Executing start method ("/lib/svc/method/svc-intrd"). ]
[ Sep 13 19:14:36 Method "start" exited with status 0. ]
Config.pm did not return a true value at /usr/perl5/5.8.4/lib/i86pc-solaris-64int/DynaLoader.pm line 25.
BEGIN failed--compilation aborted at /usr/perl5/5.8.4/lib/i86pc-solaris-64int/DynaLoader.pm line 25.
Compilation failed in require at /usr/perl5/5.8.4/lib/Sun/Solaris/Kstat.pm line 33.
BEGIN failed--compilation aborted at /usr/perl5/5.8.4/lib/Sun/Solaris/Kstat.pm line 33.
Compilation failed in require at /usr/lib/intrd line 69.
[ Sep 13 19:14:36 Stopping because all processes in service exited. ]
[ Sep 13 19:14:36 Executing stop method (:kill). ]
[ Sep 13 19:14:36 Restarting too quickly, changing state to maintenance. ]
...
Sep 13 19:14:36 myserver svc.startd[7]: [ID 748625 daemon.error] system/intrd:default failed repeatedly: transitioned to maintenance (see 'svcs -xv' for details)
Does anyone have an idea of where I should look next? intrd and fmd services are not online, and I can't seem to bring them up.

[Rob Johnston]

Hi Erich,

At first glance, this looks like the following bug:

6666107 fmd core dump building disk topology

which as fixed in build 102.

I would recommend that you update you box to a newer build.

rob

Post by Rob Johnston
% pstack <path to fmd core file>

core '/var/fm/fmd/core.fmd.11000' of 11000: /usr/lib/fm/fmd/fmd
----------------- lwp# 1 / thread# 1 --------------------
feebc55b __lwp_park (86f2aa0, 86f2a88) + b
feeb5d2d cond_wait_queue (86f2aa0, 86f2a88, 0, feeb61e6) + 60
feeb623e __cond_wait (86f2aa0, 86f2a88, 8046f98, feeb627e) + 66
feeb628c cond_wait (86f2aa0, 86f2a88, 0, feeb62bc) + 24
feeb62d1 pthread_cond_wait (86f2aa0, 86f2a88, 86f2a80, 807334e) + 21
080733a3 fmd_module_create (8047860, 8081328, 400, 8074b06) + 40f
08074ad6 fmd_modhash_load (80b1328, 8047860, 8081328, feeb4db1) + ea
0807472e fmd_modhash_loaddir (80b1328, 80b6f20, 8081328, 8081dc8) + f6
0807479f fmd_modhash_loadall (80b1328, 80bff70, 8081328) + 3b
080607cf fmd_run (80996d8, 4, 8047e38, 8072be3) + 57f
08072c86 main (1, 8047e6c, 8047e74, feffb7e4) + 2fa
0805f28d _start (1, 8047f04, 0, 8047f18, 8047f2e, 8047f3f) + 7d
----------------- lwp# 2 / thread# 2 --------------------
feebc55b __lwp_park (82b47e0, 82b47c8) + b
feeb5d2d cond_wait_queue (82b47e0, 82b47c8, 0, feeb61e6) + 60
feeb623e __cond_wait (82b47e0, 82b47c8, fdf1ef2c, feeb627e) + 66
feeb628c cond_wait (82b47e0, 82b47c8, 82b47f0, feeb62bc) + 24
feeb62d1 pthread_cond_wait (82b47e0, 82b47c8, fdf1efac, 807bef1) + 21
0807be28 fmd_timerq_exec (82b47c0, 0, 10, feeba1e8) + 84
0807b2af fmd_thread_start (80c0df0, fef4f000, fdf1efec, feebc25e) + 57
feebc296 _thrp_setup (fec80a00) + 7e
feebc520 _lwp_start (fec80a00, 0, 0, feebc25e, 0, 0)
----------------- lwp# 3 / thread# 3 --------------------
feebc55b __lwp_park (80e0250, 80e0238) + b
feeb5d2d cond_wait_queue (80e0250, 80e0238, 0, feeb61e6) + 60
feeb623e __cond_wait (80e0250, 80e0238, fde1fecc, feeb627e) + 66
feeb628c cond_wait (80e0250, 80e0238, 0, feeb62bc) + 24
feeb62d1 pthread_cond_wait (80e0250, 80e0238, fde1ff2c, 80b1390) + 21
0806d800 fmd_ctl_rele (80e0238, c, fde1ff4c, 80b1390) + 28
0806ea2f fmd_event_rele (80b13c8) + 1b
0806f0fc fmd_eventq_delete (80b1378, 80b13c8, 0, feea7bd7) + b4
08072e77 fmd_module_start (80b4cc0, 0, 10, feeba1e8) + e7
0807b2af fmd_thread_start (8347618, fef4f000, fde1ffec, feebc25e) + 57
feebc296 _thrp_setup (fec81200) + 7e
feebc520 _lwp_start (fec81200, 0, 0, feebc25e, 0, 0)
----------------- lwp# 4 / thread# 4 --------------------
feebc55b __lwp_park (80e0250, 80e0238) + b
feeb5d2d cond_wait_queue (80e0250, 80e0238, 0, feeb61e6) + 60
feeb623e __cond_wait (80e0250, 80e0238, fdd20ecc, feeb627e) + 66
feeb628c cond_wait (80e0250, 80e0238, 0, feeb62bc) + 24
feeb62d1 pthread_cond_wait (80e0250, 80e0238, 0, 80e1230) + 21
0806d800 fmd_ctl_rele (80e0238, c, fdd20f4c, 80e1230) + 28
0806ea2f fmd_event_rele (80b13c8) + 1b
0806f0fc fmd_eventq_delete (80e1218, 20, 0, feea7bd7) + b4
08072e77 fmd_module_start (80b4b80, 0, 10, feeba1e8) + e7
0807b2af fmd_thread_start (80c0dc8, fef4f000, fdd20fec, feebc25e) + 57
feebc296 _thrp_setup (fec81a00) + 7e
feebc520 _lwp_start (fec81a00, 0, 0, feebc25e, 0, 0)
----------------- lwp# 5 / thread# 5 --------------------
feebc55b __lwp_park (8098b60, 8098b70) + b
feeb5d2d cond_wait_queue (8098b60, 8098b70, 0, feeb61e6) + 60
feeb623e __cond_wait (8098b60, 8098b70, fdc21488, feeb627e) + 66
feeb628c cond_wait (8098b60, 8098b70, 83c88e8, feeb62bc) + 24
feeb62d1 pthread_cond_wait (8098b60, 8098b70, 809973c, 8347708) + 21
0807a691 sysev_recv (fdc21540, 828a178, 80b4e00, fe15668c) + 55
fe1566b4 door_upcall (83c88e8, fdc21540, 8c0, 0, 0, fe156680) + 34
feec1522 __door_return () + 52
----------------- lwp# 6 / thread# 6 --------------------
feec14f1 __door_return (0, 0, 0, 0, fe000, fef4f000) + 21
08060205 fmd_door_server (0, 0, 10, feeba1e8) + 31
0807b2af fmd_thread_start (8347708, fef4f000, fdb22fec, feebc25e) + 57
feebc296 _thrp_setup (fec82a00) + 7e
feebc520 _lwp_start (fec82a00, 0, 0, feebc25e, 0, 0)
----------------- lwp# 7 / thread# 7 --------------------
feebc55b __lwp_park (80e1270, 80e1280) + b
feeb5d2d cond_wait_queue (80e1270, 80e1280, 0, feeb61e6) + 60
feeb623e __cond_wait (80e1270, 80e1280, fd9fef8c, feeb627e) + 66
feeb628c cond_wait (80e1270, 80e1280, 200, fe153772) + 24
fe1537b2 subscriber_event_handler (83eec88, fef4f000, fd9fefec,
feebc25e) + 4e
feebc296 _thrp_setup (fec83200) + 7e
feebc520 _lwp_start (fec83200, 0, 0, feebc25e, 0, 0)
----------------- lwp# 8 / thread# 8 --------------------
feec0bb7 __pollsys (8139948, 4, 0, 0, 0, fed5c000) + 7
fee66cd4 poll (8139948, 4, ffffffff, fed0baab) + 4c
fed0bc1d _svc_run_mt (fed5da80, fed5da90, fed5daa0, fed5daa0, 807b2af,
fdf2020c) + 1bd
fed0b6bf svc_run (0, 0, 10, feeba1e8) + 77
0807b2af fmd_thread_start (8347690, fef4f000, fd8befec, feebc25e) + 57
feebc296 _thrp_setup (fec83a00) + 7e
feebc520 _lwp_start (fec83a00, 0, 0, feebc25e, 0, 0)
----------------- lwp# 9 / thread# 9 --------------------
feebc55b __lwp_park (80e0250, 80e0238) + b
feeb5d2d cond_wait_queue (80e0250, 80e0238, 0, feeb61e6) + 60
feeb623e __cond_wait (80e0250, 80e0238, fd7bfecc, feeb627e) + 66
feeb628c cond_wait (80e0250, 80e0238, 0, feeb62bc) + 24
feeb62d1 pthread_cond_wait (80e0250, 80e0238, fd7bff2c, 80e1870) + 21
0806d800 fmd_ctl_rele (80e0238, c, fd7bff4c, 80e1870) + 28
0806ea2f fmd_event_rele (80b13c8) + 1b
0806f0fc fmd_eventq_delete (80e1858, 80b13c8, 0, feea7bd7) + b4
08072e77 fmd_module_start (8293580, 0, 10, feeba1e8) + e7
0807b2af fmd_thread_start (83475c8, fef4f000, fd7bffec, feebc25e) + 57
feebc296 _thrp_setup (fec84200) + 7e
feebc520 _lwp_start (fec84200, 0, 0, feebc25e, 0, 0)
----------------- lwp# 10 / thread# 10 --------------------
feebc55b __lwp_park (80e0250, 80e0238) + b
feeb5d2d cond_wait_queue (80e0250, 80e0238, 0, feeb61e6) + 60
feeb623e __cond_wait (80e0250, 80e0238, fd69eecc, feeb627e) + 66
feeb628c cond_wait (80e0250, 80e0238, 0, feeb62bc) + 24
feeb62d1 pthread_cond_wait (80e0250, 80e0238, fd69ef2c, 80e1be0) + 21
0806d800 fmd_ctl_rele (80e0238, c, fd69ef4c, 80e1be0) + 28
0806ea2f fmd_event_rele (80b13c8) + 1b
0806f0fc fmd_eventq_delete (80e1bc8, 80b13c8, 0, feea7bd7) + b4
08072e77 fmd_module_start (8293440, 0, 10, feeba1e8) + e7
0807b2af fmd_thread_start (8347500, fef4f000, fd69efec, feebc25e) + 57
feebc296 _thrp_setup (fec84a00) + 7e
feebc520 _lwp_start (fec84a00, 0, 0, feebc25e, 0, 0)
----------------- lwp# 11 / thread# 11 --------------------
feebc55b __lwp_park (80e0250, 80e0238) + b
feeb5d2d cond_wait_queue (80e0250, 80e0238, 0, feeb61e6) + 60
feeb623e __cond_wait (80e0250, 80e0238, fd19eecc, feeb627e) + 66
feeb628c cond_wait (80e0250, 80e0238, 0, feeb62bc) + 24
feeb62d1 pthread_cond_wait (80e0250, 80e0238, 0, 80e1b90) + 21
0806d800 fmd_ctl_rele (80e0238, c, fd19ef4c, 80e1b90) + 28
0806ea2f fmd_event_rele (80b13c8) + 1b
0806f0fc fmd_eventq_delete (80e1b78, 20, 0, feea7bd7) + b4
08072e77 fmd_module_start (86f2440, 0, 10, feeba1e8) + e7
0807b2af fmd_thread_start (83474b0, fef4f000, fd19efec, feebc25e) + 57
feebc296 _thrp_setup (fec85200) + 7e
feebc520 _lwp_start (fec85200, 0, 0, feebc25e, 0, 0)
----------------- lwp# 12 / thread# 12 --------------------
feebc55b __lwp_park (80e0250, 80e0238) + b
feeb5d2d cond_wait_queue (80e0250, 80e0238, 0, feeb61e6) + 60
feeb623e __cond_wait (80e0250, 80e0238, fcddeecc, feeb627e) + 66
feeb628c cond_wait (80e0250, 80e0238, 0, feeb62bc) + 24
feeb62d1 pthread_cond_wait (80e0250, 80e0238, fcddef2c, 80e1b40) + 21
0806d800 fmd_ctl_rele (80e0238, c, fcddef4c, 80e1b40) + 28
0806ea2f fmd_event_rele (80b13c8) + 1b
0806f0fc fmd_eventq_delete (80e1b28, 80b13c8, 0, feea7bd7) + b4
08072e77 fmd_module_start (86f2300, 0, 10, feeba1e8) + e7
0807b2af fmd_thread_start (8347460, fef4f000, fcddefec, feebc25e) + 57
feebc296 _thrp_setup (fec85a00) + 7e
feebc520 _lwp_start (fec85a00, 0, 0, feebc25e, 0, 0)
----------------- lwp# 13 / thread# 13 --------------------
feebc55b __lwp_park (80e0250, 80e0238) + b
feeb5d2d cond_wait_queue (80e0250, 80e0238, 0, feeb61e6) + 60
feeb623e __cond_wait (80e0250, 80e0238, fccaeecc, feeb627e) + 66
feeb628c cond_wait (80e0250, 80e0238, 0, feeb62bc) + 24
feeb62d1 pthread_cond_wait (80e0250, 80e0238, fccaef2c, 80e17d0) + 21
0806d800 fmd_ctl_rele (80e0238, c, fccaef4c, 80e17d0) + 28
0806ea2f fmd_event_rele (80b13c8) + 1b
0806f0fc fmd_eventq_delete (80e17b8, 80b13c8, 0, feea7bd7) + b4
08072e77 fmd_module_start (86f21c0, 0, 10, feeba1e8) + e7
0807b2af fmd_thread_start (8347410, fef4f000, fccaefec, feebc25e) + 57
feebc296 _thrp_setup (fec86200) + 7e
feebc520 _lwp_start (fec86200, 0, 0, feebc25e, 0, 0)
----------------- lwp# 14 / thread# 14 --------------------
feebc55b __lwp_park (80e0250, 80e0238) + b
feeb5d2d cond_wait_queue (80e0250, 80e0238, 0, feeb61e6) + 60
feeb623e __cond_wait (80e0250, 80e0238, fcb7eecc, feeb627e) + 66
feeb628c cond_wait (80e0250, 80e0238, 0, feeb62bc) + 24
feeb62d1 pthread_cond_wait (80e0250, 80e0238, fcb7ef2c, 80e1960) + 21
0806d800 fmd_ctl_rele (80e0238, c, fcb7ef4c, 80e1960) + 28
0806ea2f fmd_event_rele (80b13c8) + 1b
0806f0fc fmd_eventq_delete (80e1948, 80b13c8, 0, feea7bd7) + b4
08072e77 fmd_module_start (86f2800, 0, 10, feeba1e8) + e7
0807b2af fmd_thread_start (83473e8, fef4f000, fcb7efec, feebc25e) + 57
feebc296 _thrp_setup (fec86a00) + 7e
feebc520 _lwp_start (fec86a00, 0, 0, feebc25e, 0, 0)
----------------- lwp# 15 / thread# 15 --------------------
feebc55b __lwp_park (80e0250, 80e0238) + b
feeb5d2d cond_wait_queue (80e0250, 80e0238, 0, feeb61e6) + 60
feeb623e __cond_wait (80e0250, 80e0238, fc9ceecc, feeb627e) + 66
feeb628c cond_wait (80e0250, 80e0238, 0, feeb62bc) + 24
feeb62d1 pthread_cond_wait (80e0250, 80e0238, fc9cef2c, 80e1c30) + 21
0806d800 fmd_ctl_rele (80e0238, c, fc9cef4c, 80e1c30) + 28
0806ea2f fmd_event_rele (80b13c8) + 1b
0806f0fc fmd_eventq_delete (80e1c18, 80b13c8, 0, feea7bd7) + b4
08072e77 fmd_module_start (86f2940, 0, 10, feeba1e8) + e7
0807b2af fmd_thread_start (83473c0, fef4f000, fc9cefec, feebc25e) + 57
feebc296 _thrp_setup (fec87200) + 7e
feebc520 _lwp_start (fec87200, 0, 0, feebc25e, 0, 0)
----------------- lwp# 17 / thread# 17 --------------------
fee33d80 strlen (88551e0, 88898e0, 8889880, 0, e0db6000, e8) + 30
fc776cf1 topo_add_disk (80cef58, 85a2d30, fc89edfc, fc777bd6) + 2ed
fc777c50 gather_topo_cfg (80cef58, 85a2d30, fc89edfc, fec4e80e) + 84
fec4e868 topo_walk_step (8898ed8, 1, 0, fec4e612) + 68
fec4e677 step_sibling (84ebdf8, 8898ed8, 1, 0) + 73
fec4e8c8 topo_walk_step (8898ed8, 1, 0, fec4e612) + c8
fec4e677 step_sibling (84ebeb8, 8898ed8, 1, 0) + 73
fec4e8c8 topo_walk_step (8898ed8, 1, 0, fec4e612) + c8
fec4e677 step_sibling (84ebf18, 8898ed8, 1, 0) + 73
fec4e8c8 topo_walk_step (8898ed8, 1, 0, fec4e612) + c8
fec4e677 step_sibling (8193798, 8898ed8, 1, 0) + 73
fec4e8c8 topo_walk_step (8898ed8, 1, 0, fec4e55e) + c8
fec4e5bf step_child (829e2a8, 8898ed8, 1, 0) + 6f
fec4e8b1 topo_walk_step (8898ed8, 1, 0, fec4e612) + b1
fec4e677 step_sibling (829e188, 8898ed8, 1, 0) + 73
fec4e8c8 topo_walk_step (8898ed8, 1, 0, fec4e612) + c8
fec4e677 step_sibling (827eb60, 8898ed8, 1, 0) + 73
fec4e8c8 topo_walk_step (8898ed8, 1, 0, fec4e612) + c8
fec4e677 step_sibling (829e0c8, 8898ed8, 1, 0) + 73
fec4e8c8 topo_walk_step (8898ed8, 1, 0, fec4e612) + c8
fec4e677 step_sibling (81d0ea0, 8898ed8, 1, 0) + 73
fec4e8c8 topo_walk_step (8898ed8, 1, 0, fec4e612) + c8
fec4e677 step_sibling (829e068, 8898ed8, 1, 0) + 73
fec4e8c8 topo_walk_step (8898ed8, 1, 0, fec4e612) + c8
fec4e677 step_sibling (81d0de0, 8898ed8, 1, 0) + 73
fec4e8c8 topo_walk_step (8898ed8, 1, 0, fec4e55e) + c8
fec4e5bf step_child (81d08a0, 8898ed8, 1, 0) + 6f
fec4e8b1 topo_walk_step (8898ed8, 1, 0, fec4e612) + b1
fec4e677 step_sibling (80bba28, 8898ed8, 1, 0) + 73
fec4e8c8 topo_walk_step (8898ed8, 1, fc89ee1c, fc777c7a) + c8
fc777cd9 update_configuration_from_topo (86f2a80, 0, 8392328, fc7763a1)
+ 6d
fc7763d2 config_get (86f2a80, fc78d588, fc89ef1c, fc774b3a) + 42
fc774b54 _fmd_init (86f2a80, 0, 1, 0) + 5c
080740cb fmd_module_enter (86f2a80, fc774af8, fc89efcc, feeb4f52) + 6f
080778c1 rtld_init (86f2a80, 0, fc89efac, feea7bd7) + 8d
08072dbc fmd_module_start (86f2a80, 0, 10, feeba1e8) + 2c
0807b2af fmd_thread_start (8842528, fef4f000, fc89efec, feebc25e) + 57
feebc296 _thrp_setup (fec87a00) + 7e
feebc520 _lwp_start (fec87a00, 0, 0, feebc25e, 0, 0)
Thanks for the help!
-erich

[Gavin Maltby]

Was hoping someone could shed some light on this... I have an
installation of OpenSolaris (snv_101b), and I keep noticing that my /
partition is full. I discovered that /var/fm/fmd was filling it up
with ~100GB of core.fmd.* files. I can delete them easily enough, but
sure enough, they come back after a few days.

How far apart in time are the core dumps? The current SMF criteria
for "restarting too quickly, will place into maintenance state"
are too-easily defeated by a process that takes a few seconds to
start up (as fmd does). So if the core happens during startup
we can be restarted indefinitely and keep dumping core, and fill
up the filesystem while we're at it.

There's a fix in the works to change the "restarting too quickly"
criteria - might have helped here.

Cheers

Gavin

[Gavin Maltby]

Hi,

Got it figured out, I had to tweak the grub configuration a bit to
exclude all the splashimage and foreground/background stuff. It
appears to have upgraded OK. I'll see if my fmd core dump problem
returns....

Cool - that seems to ahve solved the fmd service from 'restarting too

Excellent!

quickly' and no more core dumps are happening. I'm still seeing this

Not so excellent.

Config.pm did not return a true value at
/usr/perl5/5.8.4/lib/i86pc-solaris-64int/DynaLoader.pm line 25.
BEGIN failed--compilation aborted at
/usr/perl5/5.8.4/lib/i86pc-solaris-64int/DynaLoader.pm line 25.
Compilation failed in require at
/usr/perl5/5.8.4/lib/Sun/Solaris/Kstat.pm line 33.
BEGIN failed--compilation aborted at
/usr/perl5/5.8.4/lib/Sun/Solaris/Kstat.pm line 33.
Compilation failed in require at /usr/lib/intrd line 67.
[ Sep 16 10:59:16 Stopping because all processes in service exited. ]
[ Sep 16 10:59:16 Executing stop method (:kill). ]
[ Sep 16 10:59:16 Executing start method ("/lib/svc/method/svc-intrd"). ]
[ Sep 16 10:59:16 Method "start" exited with status 0. ]

There almost certainly is a better list for this, but I can't suggest what.

Try running /usr/bin/kstat standalone; it uses Sun::Solaris::Kstat -
intrd has 'require Sun::Solaris::Kstat;' but I suspect you'll
see the same failure.

I'd guess this could be a package integrity issue. You could try

- 'pkg verify SUNWperl584core'; if it shows problems then
'pkg fix SUNWperl584core'. Similarly for SUNWperl584usr

- I see the repos have perl 5.8.4 and 5.10.0 available. From the
main opensolaris repo I see SUNWperl{584,510}{core,usr,man}.
Do you 5.10.0 in /usr/perl5? *Maybe* that upsets Kstat.pm?
Kstat.pm is delivered as a 5.8.4 module only. What
version does /usr/perl5/bin/perl -v show?

Cheers

Gavin