Paul Fisher
2009-11-19 11:23:55 UTC
Hopefully someone can help get to the bottom of what is going on with
this machine. Overall it appears that fmd is getting in the way of work
getting done.
I have a new 2x Intel 5520 supermicro box (prtdiag/prtconf/scanpci
below) that installed extremely slowly (4+ hours) and then once booted
is overall acting sluggishly. Load seems to stay at 0.50 all of the
time doing nothing, and of that most is system time.
Normal vmstat show not much going on:
***@dev-storage-01:/test# vmstat 5
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr s0 s1 s2 s3 in sy cs
us sy id
1 0 0 65936540 47229788 68 86 17 0 0 0 5 -0 15 21 21 496 1690 949
1 8 91
0 0 0 65222612 46351108 2 14 0 0 0 0 0 0 8 0 0 338 675 313
1 4 96
0 0 0 65222452 46351000 0 3 0 0 0 0 0 0 17 0 0 416 301 288
0 2 98
0 0 0 65222452 46351004 0 0 0 0 0 0 0 0 13 0 0 477 424 336
0 0 99
When I create a 8x mirrored vdev pool (1T samsung enterise drives) and
do a simple dd test it maxes out at 100M/s, where I'd normally expect
500M/s+ at least.
***@dev-storage-01:/storage# dd if=/dev/zero of=test.zeros bs=1M
count=64000
64000+0 records in
64000+0 records out
67108864000 bytes (67 GB) copied, 630.583 s, 106 MB/s
Interestingly, during the dd, the load goes to 7+ and vmstat looks like
the box is working really hard to get the bits out to disk:
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
2073 root 8024K 2140K cpu1 0 0 0:07:09 5.1% dd/1
1008 root 55M 43M cpu12 0 0 0:29:19 4.6% fmd/20
1261 root 13M 6288K sleep 59 0 0:01:39 0.1% intrd/1
2107 local 9556K 2832K cpu5 0 0 0:00:00 0.1% prstat/1
1924 local 9128K 2652K sleep 49 0 0:00:08 0.1% bash/1
1919 local 21M 5600K sleep 59 0 0:00:10 0.0% sshd/1
...
Total: 48 processes, 199 lwps, load averages: 8.34, 7.31, 5.23
***@dev-storage-01:~$ vmstat 5
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr s0 s1 s2 s3 in sy cs
us sy id
1 0 0 65914512 47202672 66 84 17 0 0 0 5 -0 14 21 21 493 1654 929
1 8 91
9 0 0 65221928 46350452 7 92 0 0 0 0 0 0 4 0 0 493 459 541
2 47 51
3 0 0 65221600 46350148 0 1 0 0 0 0 0 0 22 0 0 614 383 1171
4 52 44
3 0 0 65221600 46350148 0 0 0 0 0 0 0 0 13 0 0 1015 680 2022
3 28 68
4 0 0 65221600 46350148 0 0 0 0 0 0 0 0 15 0 0 585 360 916
5 52 43
9 0 0 65221580 46350128 0 0 0 0 0 0 0 0 3 0 0 530 391 758
4 60 36
***@dev-storage-01:~$ mpstat 15
...
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 0 0 1 90 16 70 3 18 106 0 70 0 63
0 37
1 0 0 0 249 204 99 2 20 177 0 1 0 52
0 48
2 0 0 0 189 140 84 1 16 150 0 44 0 60
0 40
3 0 0 1 61 2 91 2 20 147 0 33 0 46
0 54
4 2 0 1 36 1 48 1 12 56 0 25 6 45
0 49
5 0 0 0 40 0 51 1 12 66 0 3 7 48
0 44
6 0 0 1 37 1 59 1 16 67 0 9 2 48
0 49
7 0 0 0 52 3 51 3 14 64 0 9 20 36
0 44
8 0 0 0 66 3 40 2 11 116 0 33 0 50
0 50
9 0 0 1 47 2 83 0 19 169 0 0 0 38
0 62
10 0 0 1 43 2 68 0 16 128 0 1 0 34
0 66
11 0 0 552 13 4 59 0 12 92 0 18 0 49
0 51
12 0 0 0 58 0 42 0 11 45 0 2 1 69
0 30
13 0 0 1 46 1 43 1 14 60 0 14 14 52
0 34
14 1 0 0 53 1 49 1 13 60 0 26 9 44
0 47
15 0 0 1 39 1 57 0 15 75 0 4 0 47
0 53
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 0 0 0 51 15 113 0 27 208 0 1 0 43
0 56
1 0 0 0 300 282 85 2 21 163 0 51 0 54
0 46
2 0 0 0 201 188 102 3 20 147 0 39 2 63
0 35
3 0 0 0 26 4 122 1 27 252 0 2 1 35
0 64
4 1 0 0 14 1 60 1 15 68 0 22 6 42
0 52
5 0 0 0 8 1 57 1 15 68 0 27 9 43
0 48
6 0 0 0 11 1 66 1 16 83 0 44 5 45
0 50
7 0 0 0 12 3 62 3 17 71 0 26 12 43
0 45
8 0 0 0 27 3 73 2 18 168 0 24 1 45
0 53
9 0 0 1 30 2 100 4 25 142 1 44 0 45
0 55
10 0 0 0 25 3 82 3 21 153 0 6 1 36
0 63
11 0 0 0 16 3 73 2 16 201 0 53 1 50
0 49
12 0 0 0 11 1 51 1 15 71 0 21 6 47
0 47
13 0 0 49 7 1 63 1 15 72 0 19 6 47
0 46
14 0 0 0 6 1 57 1 14 82 0 1 1 46
0 52
15 0 0 0 12 1 64 2 15 78 0 35 14 45
0 41
I did finally see that fmd was running a lot and took a look at what it
thought was going on:
***@dev-storage-01:~$ pfexec fmadm faulty
--------------- ------------------------------------ --------------
---------
TIME EVENT-ID MSG-ID
SEVERITY
--------------- ------------------------------------ --------------
---------
Nov 18 07:00:37 0ddd2e6a-06c4-6936-fb63-fbe93899370d SUNOS-8000-FU
Major
Host : dev-storage-01
Platform : X8DT3 Chassis_id : 1234567890
Product_sn :
Fault class : defect.sunos.eft.undiag.fme
FRU : None
faulty
Description : The diagnosis engine encountered telemetry for which it was
unable to perform a diagnosis. Refer to
http://sun.com/msg/SUNOS-8000-FU for more information.
Response : Error reports have been logged for examination by Sun.
Impact : Automated diagnosis and response for these events will not
occur.
Action : Ensure that the latest Solaris Kernel and Predictive
Self-Healing
(PSH) patches are installed.
At this point I've reached the end of what I know to do in order to
diagnose what is going on with this box. I would really appreciate some
additional guidance on what I can to look at.
Here is the configuration information on the box:
***@dev-storage-01:~$ uname -a
SunOS dev-storage-01 5.11 snv_127 i86pc i386 i86pc Solaris
***@dev-storage-01:~$ cat /etc/release
OpenSolaris Development snv_127 X86
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 06 November 2009
***@dev-storage-01:~$ prtdiag
System Configuration: Supermicro X8DT3
BIOS Configuration: American Megatrends Inc. 080015 09/24/2009
BMC Configuration: IPMI 1.5 (KCS: Keyboard Controller Style)
==== Processor Sockets ====================================
Version Location Tag
-------------------------------- --------------------------
Intel(R) Xeon(R) CPU E5520 @ 2.27GHz CPU 2
Intel(R) Xeon(R) CPU E5520 @ 2.27GHz CPU 1
==== Memory Device Sockets ================================
Type Status Set Device Locator Bank Locator
----------- ------ --- ------------------- ----------------
other in use 0 P1-DIMM1A BANK0
other in use 0 P1-DIMM1B BANK1
other in use 0 P1-DIMM2A BANK2
other in use 0 P1-DIMM2B BANK3
other in use 0 P1-DIMM3A BANK4
other in use 0 P1-DIMM3B BANK5
other in use 0 P2-DIMM1A BANK6
other in use 0 P2-DIMM1B BANK7
other in use 0 P2-DIMM2A BANK8
other in use 0 P2-DIMM2B BANK9
other in use 0 P2-DIMM3A BANK10
other in use 0 P2-DIMM3B BANK11
==== On-Board Devices =====================================
==== Upgradeable Slots ====================================
ID Status Type Description
--- --------- ---------------- ----------------------------
1 available PCI PCI#1
2 in use PCI Express PCI-E#2
3 available PCI PCI#3
4 in use PCI Express PCI#4
5 in use PCI Express PCI-E#5
6 available PCI Express PCI-E#6
***@dev-storage-01:~$ prtconf
System Configuration: Sun Microsystems i86pc
Memory size: 49143 Megabytes
System Peripherals (Software Nodes):
i86pc
scsi_vhci, instance #0
fw, instance #0
cpu, instance #0
cpu, instance #1
cpu, instance #2
cpu, instance #3
cpu, instance #4
cpu, instance #5
cpu, instance #6
cpu, instance #7
cpu, instance #8
cpu, instance #9
cpu, instance #10
cpu, instance #11
cpu, instance #12
cpu, instance #13
cpu, instance #14
cpu, instance #15
sb, instance #1
pci, instance #0
pci15d9,1 (driver not attached)
pci8086,3408, instance #0
pci15d9,10c9, instance #0
pci15d9,10c9, instance #1
pci8086,340a (driver not attached)
pci8086,340c, instance #2
pci1000,3140, instance #1
sd, instance #5
sd, instance #6
sd, instance #7
sd, instance #8
sd, instance #9
sd, instance #10
sd, instance #11
sd, instance #12
pci8086,340e (driver not attached)
pci8086,343a (driver not attached)
pci8086,343b (driver not attached)
pci8086,343c (driver not attached)
pci8086,343d (driver not attached)
pci8086,3418 (driver not attached)
pci8086,3419 (driver not attached)
pci8086,341a (driver not attached)
pci8086,341c (driver not attached)
pci8086,341d (driver not attached)
pci8086,341e (driver not attached)
pci8086,3439 (driver not attached)
pci8086,342e (driver not attached)
pci8086,3422 (driver not attached)
pci8086,3423, instance #0
pci8086,3438 (driver not attached)
pci15d9,1 (driver not attached)
pci15d9,1 (driver not attached)
pci15d9,1 (driver not attached)
pci15d9,1 (driver not attached)
pci15d9,1 (driver not attached)
pci15d9,1 (driver not attached)
pci15d9,1 (driver not attached)
pci15d9,1 (driver not attached)
pci15d9,1, instance #0
pci15d9,1, instance #1
pci15d9,1, instance #2
device, instance #0
keyboard, instance #0
mouse, instance #1
pci15d9,1, instance #0
pci8086,3a40, instance #4
pci1000,3140, instance #0
sd, instance #13
sd, instance #14
sd, instance #15
sd, instance #16
sd, instance #17
sd, instance #18
sd, instance #19
sd, instance #20
pci15d9,1, instance #3
pci15d9,1, instance #4
pci15d9,1, instance #5
pci15d9,1, instance #1
pci8086,244e, instance #0
display, instance #0
isa, instance #0
motherboard (driver not attached)
asy, instance #0
asy, instance #1
asy, instance #2
i8042, instance #0
keyboard, instance #0
mouse, instance #0
motherboard (driver not attached)
pit_beep, instance #0
pci15d9,1, instance #0
cdrom, instance #0
disk, instance #1
disk, instance #2
disk, instance #3
disk, instance #4
pci15d9,1 (driver not attached)
used-resources (driver not attached)
pseudo, instance #0
options, instance #0
xsvc, instance #0
agpgart, instance #0
***@dev-storage-01:~$ pfexec scanpci
pci bus 0x0000 cardnum 0x00 function 0x00: vendor 0x8086 device 0x3406
Intel Corporation QuickPath Architecture I/O Hub to ESI Port
pci bus 0x0000 cardnum 0x01 function 0x00: vendor 0x8086 device 0x3408
Intel Corporation QuickPath Architecture I/O Hub PCI Express Root Port 1
pci bus 0x0000 cardnum 0x03 function 0x00: vendor 0x8086 device 0x340a
Intel Corporation QuickPath Architecture I/O Hub PCI Express Root Port 3
pci bus 0x0000 cardnum 0x05 function 0x00: vendor 0x8086 device 0x340c
Intel Corporation QuickPath Architecture I/O Hub PCI Express Root Port 5
pci bus 0x0000 cardnum 0x07 function 0x00: vendor 0x8086 device 0x340e
Intel Corporation QuickPath Architecture I/O Hub PCI Express Root Port 7
pci bus 0x0000 cardnum 0x0d function 0x00: vendor 0x8086 device 0x343a
Intel Corporation Device unknown
pci bus 0x0000 cardnum 0x0d function 0x01: vendor 0x8086 device 0x343b
Intel Corporation Device unknown
pci bus 0x0000 cardnum 0x0d function 0x02: vendor 0x8086 device 0x343c
Intel Corporation Device unknown
pci bus 0x0000 cardnum 0x0d function 0x03: vendor 0x8086 device 0x343d
Intel Corporation Device unknown
pci bus 0x0000 cardnum 0x0d function 0x04: vendor 0x8086 device 0x3418
Intel Corporation Quickpath Interconnect Physical Layer Port 0
pci bus 0x0000 cardnum 0x0d function 0x05: vendor 0x8086 device 0x3419
Intel Corporation Quickpath Interconnect Physical Layer Port 1
pci bus 0x0000 cardnum 0x0d function 0x06: vendor 0x8086 device 0x341a
Intel Corporation Device unknown
pci bus 0x0000 cardnum 0x0e function 0x00: vendor 0x8086 device 0x341c
Intel Corporation Device unknown
pci bus 0x0000 cardnum 0x0e function 0x01: vendor 0x8086 device 0x341d
Intel Corporation Device unknown
pci bus 0x0000 cardnum 0x0e function 0x02: vendor 0x8086 device 0x341e
Intel Corporation Device unknown
pci bus 0x0000 cardnum 0x0e function 0x04: vendor 0x8086 device 0x3439
Intel Corporation Device unknown
pci bus 0x0000 cardnum 0x14 function 0x00: vendor 0x8086 device 0x342e
Intel Corporation QuickPath Architecture I/O Hub System Management
Registers
pci bus 0x0000 cardnum 0x14 function 0x01: vendor 0x8086 device 0x3422
Intel Corporation QuickPath Architecture I/O Hub GPIO and Scratch Pad
Registers
pci bus 0x0000 cardnum 0x14 function 0x02: vendor 0x8086 device 0x3423
Intel Corporation QuickPath Architecture I/O Hub Control Status and
RAS Registers
pci bus 0x0000 cardnum 0x14 function 0x03: vendor 0x8086 device 0x3438
Intel Corporation QuickPath Architecture I/O Hub Throttle Registers
pci bus 0x0000 cardnum 0x16 function 0x00: vendor 0x8086 device 0x3430
Intel Corporation DMA Engine
pci bus 0x0000 cardnum 0x16 function 0x01: vendor 0x8086 device 0x3431
Intel Corporation DMA Engine
pci bus 0x0000 cardnum 0x16 function 0x02: vendor 0x8086 device 0x3432
Intel Corporation DMA Engine
pci bus 0x0000 cardnum 0x16 function 0x03: vendor 0x8086 device 0x3433
Intel Corporation DMA Engine
pci bus 0x0000 cardnum 0x16 function 0x04: vendor 0x8086 device 0x3429
Intel Corporation DMA Engine
pci bus 0x0000 cardnum 0x16 function 0x05: vendor 0x8086 device 0x342a
Intel Corporation DMA Engine
pci bus 0x0000 cardnum 0x16 function 0x06: vendor 0x8086 device 0x342b
Intel Corporation DMA Engine
pci bus 0x0000 cardnum 0x16 function 0x07: vendor 0x8086 device 0x342c
Intel Corporation DMA Engine
pci bus 0x0000 cardnum 0x1a function 0x00: vendor 0x8086 device 0x3a37
Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
pci bus 0x0000 cardnum 0x1a function 0x01: vendor 0x8086 device 0x3a38
Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
pci bus 0x0000 cardnum 0x1a function 0x02: vendor 0x8086 device 0x3a39
Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
pci bus 0x0000 cardnum 0x1a function 0x07: vendor 0x8086 device 0x3a3c
Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
pci bus 0x0000 cardnum 0x1c function 0x00: vendor 0x8086 device 0x3a40
Intel Corporation 82801JI (ICH10 Family) PCI Express Port 1
pci bus 0x0000 cardnum 0x1d function 0x00: vendor 0x8086 device 0x3a34
Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
pci bus 0x0000 cardnum 0x1d function 0x01: vendor 0x8086 device 0x3a35
Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
pci bus 0x0000 cardnum 0x1d function 0x02: vendor 0x8086 device 0x3a36
Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
pci bus 0x0000 cardnum 0x1d function 0x07: vendor 0x8086 device 0x3a3a
Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
pci bus 0x0000 cardnum 0x1e function 0x00: vendor 0x8086 device 0x244e
Intel Corporation 82801 PCI Bridge
pci bus 0x0000 cardnum 0x1f function 0x00: vendor 0x8086 device 0x3a16
Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
pci bus 0x0000 cardnum 0x1f function 0x02: vendor 0x8086 device 0x3a22
Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller
pci bus 0x0000 cardnum 0x1f function 0x03: vendor 0x8086 device 0x3a30
Intel Corporation 82801JI (ICH10 Family) SMBus Controller
pci bus 0x0001 cardnum 0x03 function 0x00: vendor 0x102b device 0x0532
Matrox Graphics, Inc. MGA G200eW WPCM450
pci bus 0x0002 cardnum 0x00 function 0x00: vendor 0x1000 device 0x0058
LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS
pci bus 0x0004 cardnum 0x00 function 0x00: vendor 0x1000 device 0x0058
LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS
pci bus 0x0006 cardnum 0x00 function 0x00: vendor 0x8086 device 0x10c9
Intel Corporation 82576 Gigabit ET Dual Port Server Adapter
pci bus 0x0006 cardnum 0x00 function 0x01: vendor 0x8086 device 0x10c9
Intel Corporation 82576 Gigabit ET Dual Port Server Adapter
pci bus 0x00fe cardnum 0x00 function 0x00: vendor 0x8086 device 0x2c40
Intel Corporation QuickPath Architecture Generic Non-Core Registers
pci bus 0x00fe cardnum 0x00 function 0x01: vendor 0x8086 device 0x2c01
Intel Corporation QuickPath Architecture System Address Decoder
pci bus 0x00fe cardnum 0x02 function 0x00: vendor 0x8086 device 0x2c10
Intel Corporation QPI Link 0
pci bus 0x00fe cardnum 0x02 function 0x01: vendor 0x8086 device 0x2c11
Intel Corporation QPI Physical 0
pci bus 0x00fe cardnum 0x02 function 0x04: vendor 0x8086 device 0x2c14
Intel Corporation QPI Link 1
pci bus 0x00fe cardnum 0x02 function 0x05: vendor 0x8086 device 0x2c15
Intel Corporation QPI Physical 1
pci bus 0x00fe cardnum 0x03 function 0x00: vendor 0x8086 device 0x2c18
Intel Corporation QuickPath Memory Controller
pci bus 0x00fe cardnum 0x03 function 0x01: vendor 0x8086 device 0x2c19
Intel Corporation QuickPath Memory Controller Target Address Decoder
pci bus 0x00fe cardnum 0x03 function 0x02: vendor 0x8086 device 0x2c1a
Intel Corporation QuickPath Memory Controller RAS Registers
pci bus 0x00fe cardnum 0x03 function 0x04: vendor 0x8086 device 0x2c1c
Intel Corporation QuickPath Memory Controller Test Registers
pci bus 0x00fe cardnum 0x04 function 0x00: vendor 0x8086 device 0x2c20
Intel Corporation QuickPath Memory Controller Channel 0 Control Registers
pci bus 0x00fe cardnum 0x04 function 0x01: vendor 0x8086 device 0x2c21
Intel Corporation QuickPath Memory Controller Channel 0 Address Registers
pci bus 0x00fe cardnum 0x04 function 0x02: vendor 0x8086 device 0x2c22
Intel Corporation QuickPath Memory Controller Channel 0 Rank Registers
pci bus 0x00fe cardnum 0x04 function 0x03: vendor 0x8086 device 0x2c23
Intel Corporation QuickPath Memory Controller Channel 0 Thermal
Control Registers
pci bus 0x00fe cardnum 0x05 function 0x00: vendor 0x8086 device 0x2c28
Intel Corporation QuickPath Memory Controller Channel 1 Control Registers
pci bus 0x00fe cardnum 0x05 function 0x01: vendor 0x8086 device 0x2c29
Intel Corporation QuickPath Memory Controller Channel 1 Address Registers
pci bus 0x00fe cardnum 0x05 function 0x02: vendor 0x8086 device 0x2c2a
Intel Corporation QuickPath Memory Controller Channel 1 Rank Registers
pci bus 0x00fe cardnum 0x05 function 0x03: vendor 0x8086 device 0x2c2b
Intel Corporation QuickPath Memory Controller Channel 1 Thermal
Control Registers
pci bus 0x00fe cardnum 0x06 function 0x00: vendor 0x8086 device 0x2c30
Intel Corporation QuickPath Memory Controller Channel 2 Control Registers
pci bus 0x00fe cardnum 0x06 function 0x01: vendor 0x8086 device 0x2c31
Intel Corporation QuickPath Memory Controller Channel 2 Address Registers
pci bus 0x00fe cardnum 0x06 function 0x02: vendor 0x8086 device 0x2c32
Intel Corporation QuickPath Memory Controller Channel 2 Rank Registers
pci bus 0x00fe cardnum 0x06 function 0x03: vendor 0x8086 device 0x2c33
Intel Corporation QuickPath Memory Controller Channel 2 Thermal
Control Registers
pci bus 0x00ff cardnum 0x00 function 0x00: vendor 0x8086 device 0x2c40
Intel Corporation QuickPath Architecture Generic Non-Core Registers
pci bus 0x00ff cardnum 0x00 function 0x01: vendor 0x8086 device 0x2c01
Intel Corporation QuickPath Architecture System Address Decoder
pci bus 0x00ff cardnum 0x02 function 0x00: vendor 0x8086 device 0x2c10
Intel Corporation QPI Link 0
pci bus 0x00ff cardnum 0x02 function 0x01: vendor 0x8086 device 0x2c11
Intel Corporation QPI Physical 0
pci bus 0x00ff cardnum 0x02 function 0x04: vendor 0x8086 device 0x2c14
Intel Corporation QPI Link 1
pci bus 0x00ff cardnum 0x02 function 0x05: vendor 0x8086 device 0x2c15
Intel Corporation QPI Physical 1
pci bus 0x00ff cardnum 0x03 function 0x00: vendor 0x8086 device 0x2c18
Intel Corporation QuickPath Memory Controller
pci bus 0x00ff cardnum 0x03 function 0x01: vendor 0x8086 device 0x2c19
Intel Corporation QuickPath Memory Controller Target Address Decoder
pci bus 0x00ff cardnum 0x03 function 0x02: vendor 0x8086 device 0x2c1a
Intel Corporation QuickPath Memory Controller RAS Registers
pci bus 0x00ff cardnum 0x03 function 0x04: vendor 0x8086 device 0x2c1c
Intel Corporation QuickPath Memory Controller Test Registers
pci bus 0x00ff cardnum 0x04 function 0x00: vendor 0x8086 device 0x2c20
Intel Corporation QuickPath Memory Controller Channel 0 Control Registers
pci bus 0x00ff cardnum 0x04 function 0x01: vendor 0x8086 device 0x2c21
Intel Corporation QuickPath Memory Controller Channel 0 Address Registers
pci bus 0x00ff cardnum 0x04 function 0x02: vendor 0x8086 device 0x2c22
Intel Corporation QuickPath Memory Controller Channel 0 Rank Registers
pci bus 0x00ff cardnum 0x04 function 0x03: vendor 0x8086 device 0x2c23
Intel Corporation QuickPath Memory Controller Channel 0 Thermal
Control Registers
pci bus 0x00ff cardnum 0x05 function 0x00: vendor 0x8086 device 0x2c28
Intel Corporation QuickPath Memory Controller Channel 1 Control Registers
pci bus 0x00ff cardnum 0x05 function 0x01: vendor 0x8086 device 0x2c29
Intel Corporation QuickPath Memory Controller Channel 1 Address Registers
pci bus 0x00ff cardnum 0x05 function 0x02: vendor 0x8086 device 0x2c2a
Intel Corporation QuickPath Memory Controller Channel 1 Rank Registers
pci bus 0x00ff cardnum 0x05 function 0x03: vendor 0x8086 device 0x2c2b
Intel Corporation QuickPath Memory Controller Channel 1 Thermal
Control Registers
pci bus 0x00ff cardnum 0x06 function 0x00: vendor 0x8086 device 0x2c30
Intel Corporation QuickPath Memory Controller Channel 2 Control Registers
pci bus 0x00ff cardnum 0x06 function 0x01: vendor 0x8086 device 0x2c31
Intel Corporation QuickPath Memory Controller Channel 2 Address Registers
pci bus 0x00ff cardnum 0x06 function 0x02: vendor 0x8086 device 0x2c32
Intel Corporation QuickPath Memory Controller Channel 2 Rank Registers
pci bus 0x00ff cardnum 0x06 function 0x03: vendor 0x8086 device 0x2c33
Intel Corporation QuickPath Memory Controller Channel 2 Thermal
Control Registers
--
paul
this machine. Overall it appears that fmd is getting in the way of work
getting done.
I have a new 2x Intel 5520 supermicro box (prtdiag/prtconf/scanpci
below) that installed extremely slowly (4+ hours) and then once booted
is overall acting sluggishly. Load seems to stay at 0.50 all of the
time doing nothing, and of that most is system time.
Normal vmstat show not much going on:
***@dev-storage-01:/test# vmstat 5
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr s0 s1 s2 s3 in sy cs
us sy id
1 0 0 65936540 47229788 68 86 17 0 0 0 5 -0 15 21 21 496 1690 949
1 8 91
0 0 0 65222612 46351108 2 14 0 0 0 0 0 0 8 0 0 338 675 313
1 4 96
0 0 0 65222452 46351000 0 3 0 0 0 0 0 0 17 0 0 416 301 288
0 2 98
0 0 0 65222452 46351004 0 0 0 0 0 0 0 0 13 0 0 477 424 336
0 0 99
When I create a 8x mirrored vdev pool (1T samsung enterise drives) and
do a simple dd test it maxes out at 100M/s, where I'd normally expect
500M/s+ at least.
***@dev-storage-01:/storage# dd if=/dev/zero of=test.zeros bs=1M
count=64000
64000+0 records in
64000+0 records out
67108864000 bytes (67 GB) copied, 630.583 s, 106 MB/s
Interestingly, during the dd, the load goes to 7+ and vmstat looks like
the box is working really hard to get the bits out to disk:
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
2073 root 8024K 2140K cpu1 0 0 0:07:09 5.1% dd/1
1008 root 55M 43M cpu12 0 0 0:29:19 4.6% fmd/20
1261 root 13M 6288K sleep 59 0 0:01:39 0.1% intrd/1
2107 local 9556K 2832K cpu5 0 0 0:00:00 0.1% prstat/1
1924 local 9128K 2652K sleep 49 0 0:00:08 0.1% bash/1
1919 local 21M 5600K sleep 59 0 0:00:10 0.0% sshd/1
...
Total: 48 processes, 199 lwps, load averages: 8.34, 7.31, 5.23
***@dev-storage-01:~$ vmstat 5
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr s0 s1 s2 s3 in sy cs
us sy id
1 0 0 65914512 47202672 66 84 17 0 0 0 5 -0 14 21 21 493 1654 929
1 8 91
9 0 0 65221928 46350452 7 92 0 0 0 0 0 0 4 0 0 493 459 541
2 47 51
3 0 0 65221600 46350148 0 1 0 0 0 0 0 0 22 0 0 614 383 1171
4 52 44
3 0 0 65221600 46350148 0 0 0 0 0 0 0 0 13 0 0 1015 680 2022
3 28 68
4 0 0 65221600 46350148 0 0 0 0 0 0 0 0 15 0 0 585 360 916
5 52 43
9 0 0 65221580 46350128 0 0 0 0 0 0 0 0 3 0 0 530 391 758
4 60 36
***@dev-storage-01:~$ mpstat 15
...
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 0 0 1 90 16 70 3 18 106 0 70 0 63
0 37
1 0 0 0 249 204 99 2 20 177 0 1 0 52
0 48
2 0 0 0 189 140 84 1 16 150 0 44 0 60
0 40
3 0 0 1 61 2 91 2 20 147 0 33 0 46
0 54
4 2 0 1 36 1 48 1 12 56 0 25 6 45
0 49
5 0 0 0 40 0 51 1 12 66 0 3 7 48
0 44
6 0 0 1 37 1 59 1 16 67 0 9 2 48
0 49
7 0 0 0 52 3 51 3 14 64 0 9 20 36
0 44
8 0 0 0 66 3 40 2 11 116 0 33 0 50
0 50
9 0 0 1 47 2 83 0 19 169 0 0 0 38
0 62
10 0 0 1 43 2 68 0 16 128 0 1 0 34
0 66
11 0 0 552 13 4 59 0 12 92 0 18 0 49
0 51
12 0 0 0 58 0 42 0 11 45 0 2 1 69
0 30
13 0 0 1 46 1 43 1 14 60 0 14 14 52
0 34
14 1 0 0 53 1 49 1 13 60 0 26 9 44
0 47
15 0 0 1 39 1 57 0 15 75 0 4 0 47
0 53
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 0 0 0 51 15 113 0 27 208 0 1 0 43
0 56
1 0 0 0 300 282 85 2 21 163 0 51 0 54
0 46
2 0 0 0 201 188 102 3 20 147 0 39 2 63
0 35
3 0 0 0 26 4 122 1 27 252 0 2 1 35
0 64
4 1 0 0 14 1 60 1 15 68 0 22 6 42
0 52
5 0 0 0 8 1 57 1 15 68 0 27 9 43
0 48
6 0 0 0 11 1 66 1 16 83 0 44 5 45
0 50
7 0 0 0 12 3 62 3 17 71 0 26 12 43
0 45
8 0 0 0 27 3 73 2 18 168 0 24 1 45
0 53
9 0 0 1 30 2 100 4 25 142 1 44 0 45
0 55
10 0 0 0 25 3 82 3 21 153 0 6 1 36
0 63
11 0 0 0 16 3 73 2 16 201 0 53 1 50
0 49
12 0 0 0 11 1 51 1 15 71 0 21 6 47
0 47
13 0 0 49 7 1 63 1 15 72 0 19 6 47
0 46
14 0 0 0 6 1 57 1 14 82 0 1 1 46
0 52
15 0 0 0 12 1 64 2 15 78 0 35 14 45
0 41
I did finally see that fmd was running a lot and took a look at what it
thought was going on:
***@dev-storage-01:~$ pfexec fmadm faulty
--------------- ------------------------------------ --------------
---------
TIME EVENT-ID MSG-ID
SEVERITY
--------------- ------------------------------------ --------------
---------
Nov 18 07:00:37 0ddd2e6a-06c4-6936-fb63-fbe93899370d SUNOS-8000-FU
Major
Host : dev-storage-01
Platform : X8DT3 Chassis_id : 1234567890
Product_sn :
Fault class : defect.sunos.eft.undiag.fme
FRU : None
faulty
Description : The diagnosis engine encountered telemetry for which it was
unable to perform a diagnosis. Refer to
http://sun.com/msg/SUNOS-8000-FU for more information.
Response : Error reports have been logged for examination by Sun.
Impact : Automated diagnosis and response for these events will not
occur.
Action : Ensure that the latest Solaris Kernel and Predictive
Self-Healing
(PSH) patches are installed.
At this point I've reached the end of what I know to do in order to
diagnose what is going on with this box. I would really appreciate some
additional guidance on what I can to look at.
Here is the configuration information on the box:
***@dev-storage-01:~$ uname -a
SunOS dev-storage-01 5.11 snv_127 i86pc i386 i86pc Solaris
***@dev-storage-01:~$ cat /etc/release
OpenSolaris Development snv_127 X86
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 06 November 2009
***@dev-storage-01:~$ prtdiag
System Configuration: Supermicro X8DT3
BIOS Configuration: American Megatrends Inc. 080015 09/24/2009
BMC Configuration: IPMI 1.5 (KCS: Keyboard Controller Style)
==== Processor Sockets ====================================
Version Location Tag
-------------------------------- --------------------------
Intel(R) Xeon(R) CPU E5520 @ 2.27GHz CPU 2
Intel(R) Xeon(R) CPU E5520 @ 2.27GHz CPU 1
==== Memory Device Sockets ================================
Type Status Set Device Locator Bank Locator
----------- ------ --- ------------------- ----------------
other in use 0 P1-DIMM1A BANK0
other in use 0 P1-DIMM1B BANK1
other in use 0 P1-DIMM2A BANK2
other in use 0 P1-DIMM2B BANK3
other in use 0 P1-DIMM3A BANK4
other in use 0 P1-DIMM3B BANK5
other in use 0 P2-DIMM1A BANK6
other in use 0 P2-DIMM1B BANK7
other in use 0 P2-DIMM2A BANK8
other in use 0 P2-DIMM2B BANK9
other in use 0 P2-DIMM3A BANK10
other in use 0 P2-DIMM3B BANK11
==== On-Board Devices =====================================
==== Upgradeable Slots ====================================
ID Status Type Description
--- --------- ---------------- ----------------------------
1 available PCI PCI#1
2 in use PCI Express PCI-E#2
3 available PCI PCI#3
4 in use PCI Express PCI#4
5 in use PCI Express PCI-E#5
6 available PCI Express PCI-E#6
***@dev-storage-01:~$ prtconf
System Configuration: Sun Microsystems i86pc
Memory size: 49143 Megabytes
System Peripherals (Software Nodes):
i86pc
scsi_vhci, instance #0
fw, instance #0
cpu, instance #0
cpu, instance #1
cpu, instance #2
cpu, instance #3
cpu, instance #4
cpu, instance #5
cpu, instance #6
cpu, instance #7
cpu, instance #8
cpu, instance #9
cpu, instance #10
cpu, instance #11
cpu, instance #12
cpu, instance #13
cpu, instance #14
cpu, instance #15
sb, instance #1
pci, instance #0
pci15d9,1 (driver not attached)
pci8086,3408, instance #0
pci15d9,10c9, instance #0
pci15d9,10c9, instance #1
pci8086,340a (driver not attached)
pci8086,340c, instance #2
pci1000,3140, instance #1
sd, instance #5
sd, instance #6
sd, instance #7
sd, instance #8
sd, instance #9
sd, instance #10
sd, instance #11
sd, instance #12
pci8086,340e (driver not attached)
pci8086,343a (driver not attached)
pci8086,343b (driver not attached)
pci8086,343c (driver not attached)
pci8086,343d (driver not attached)
pci8086,3418 (driver not attached)
pci8086,3419 (driver not attached)
pci8086,341a (driver not attached)
pci8086,341c (driver not attached)
pci8086,341d (driver not attached)
pci8086,341e (driver not attached)
pci8086,3439 (driver not attached)
pci8086,342e (driver not attached)
pci8086,3422 (driver not attached)
pci8086,3423, instance #0
pci8086,3438 (driver not attached)
pci15d9,1 (driver not attached)
pci15d9,1 (driver not attached)
pci15d9,1 (driver not attached)
pci15d9,1 (driver not attached)
pci15d9,1 (driver not attached)
pci15d9,1 (driver not attached)
pci15d9,1 (driver not attached)
pci15d9,1 (driver not attached)
pci15d9,1, instance #0
pci15d9,1, instance #1
pci15d9,1, instance #2
device, instance #0
keyboard, instance #0
mouse, instance #1
pci15d9,1, instance #0
pci8086,3a40, instance #4
pci1000,3140, instance #0
sd, instance #13
sd, instance #14
sd, instance #15
sd, instance #16
sd, instance #17
sd, instance #18
sd, instance #19
sd, instance #20
pci15d9,1, instance #3
pci15d9,1, instance #4
pci15d9,1, instance #5
pci15d9,1, instance #1
pci8086,244e, instance #0
display, instance #0
isa, instance #0
motherboard (driver not attached)
asy, instance #0
asy, instance #1
asy, instance #2
i8042, instance #0
keyboard, instance #0
mouse, instance #0
motherboard (driver not attached)
pit_beep, instance #0
pci15d9,1, instance #0
cdrom, instance #0
disk, instance #1
disk, instance #2
disk, instance #3
disk, instance #4
pci15d9,1 (driver not attached)
used-resources (driver not attached)
pseudo, instance #0
options, instance #0
xsvc, instance #0
agpgart, instance #0
***@dev-storage-01:~$ pfexec scanpci
pci bus 0x0000 cardnum 0x00 function 0x00: vendor 0x8086 device 0x3406
Intel Corporation QuickPath Architecture I/O Hub to ESI Port
pci bus 0x0000 cardnum 0x01 function 0x00: vendor 0x8086 device 0x3408
Intel Corporation QuickPath Architecture I/O Hub PCI Express Root Port 1
pci bus 0x0000 cardnum 0x03 function 0x00: vendor 0x8086 device 0x340a
Intel Corporation QuickPath Architecture I/O Hub PCI Express Root Port 3
pci bus 0x0000 cardnum 0x05 function 0x00: vendor 0x8086 device 0x340c
Intel Corporation QuickPath Architecture I/O Hub PCI Express Root Port 5
pci bus 0x0000 cardnum 0x07 function 0x00: vendor 0x8086 device 0x340e
Intel Corporation QuickPath Architecture I/O Hub PCI Express Root Port 7
pci bus 0x0000 cardnum 0x0d function 0x00: vendor 0x8086 device 0x343a
Intel Corporation Device unknown
pci bus 0x0000 cardnum 0x0d function 0x01: vendor 0x8086 device 0x343b
Intel Corporation Device unknown
pci bus 0x0000 cardnum 0x0d function 0x02: vendor 0x8086 device 0x343c
Intel Corporation Device unknown
pci bus 0x0000 cardnum 0x0d function 0x03: vendor 0x8086 device 0x343d
Intel Corporation Device unknown
pci bus 0x0000 cardnum 0x0d function 0x04: vendor 0x8086 device 0x3418
Intel Corporation Quickpath Interconnect Physical Layer Port 0
pci bus 0x0000 cardnum 0x0d function 0x05: vendor 0x8086 device 0x3419
Intel Corporation Quickpath Interconnect Physical Layer Port 1
pci bus 0x0000 cardnum 0x0d function 0x06: vendor 0x8086 device 0x341a
Intel Corporation Device unknown
pci bus 0x0000 cardnum 0x0e function 0x00: vendor 0x8086 device 0x341c
Intel Corporation Device unknown
pci bus 0x0000 cardnum 0x0e function 0x01: vendor 0x8086 device 0x341d
Intel Corporation Device unknown
pci bus 0x0000 cardnum 0x0e function 0x02: vendor 0x8086 device 0x341e
Intel Corporation Device unknown
pci bus 0x0000 cardnum 0x0e function 0x04: vendor 0x8086 device 0x3439
Intel Corporation Device unknown
pci bus 0x0000 cardnum 0x14 function 0x00: vendor 0x8086 device 0x342e
Intel Corporation QuickPath Architecture I/O Hub System Management
Registers
pci bus 0x0000 cardnum 0x14 function 0x01: vendor 0x8086 device 0x3422
Intel Corporation QuickPath Architecture I/O Hub GPIO and Scratch Pad
Registers
pci bus 0x0000 cardnum 0x14 function 0x02: vendor 0x8086 device 0x3423
Intel Corporation QuickPath Architecture I/O Hub Control Status and
RAS Registers
pci bus 0x0000 cardnum 0x14 function 0x03: vendor 0x8086 device 0x3438
Intel Corporation QuickPath Architecture I/O Hub Throttle Registers
pci bus 0x0000 cardnum 0x16 function 0x00: vendor 0x8086 device 0x3430
Intel Corporation DMA Engine
pci bus 0x0000 cardnum 0x16 function 0x01: vendor 0x8086 device 0x3431
Intel Corporation DMA Engine
pci bus 0x0000 cardnum 0x16 function 0x02: vendor 0x8086 device 0x3432
Intel Corporation DMA Engine
pci bus 0x0000 cardnum 0x16 function 0x03: vendor 0x8086 device 0x3433
Intel Corporation DMA Engine
pci bus 0x0000 cardnum 0x16 function 0x04: vendor 0x8086 device 0x3429
Intel Corporation DMA Engine
pci bus 0x0000 cardnum 0x16 function 0x05: vendor 0x8086 device 0x342a
Intel Corporation DMA Engine
pci bus 0x0000 cardnum 0x16 function 0x06: vendor 0x8086 device 0x342b
Intel Corporation DMA Engine
pci bus 0x0000 cardnum 0x16 function 0x07: vendor 0x8086 device 0x342c
Intel Corporation DMA Engine
pci bus 0x0000 cardnum 0x1a function 0x00: vendor 0x8086 device 0x3a37
Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
pci bus 0x0000 cardnum 0x1a function 0x01: vendor 0x8086 device 0x3a38
Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
pci bus 0x0000 cardnum 0x1a function 0x02: vendor 0x8086 device 0x3a39
Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
pci bus 0x0000 cardnum 0x1a function 0x07: vendor 0x8086 device 0x3a3c
Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
pci bus 0x0000 cardnum 0x1c function 0x00: vendor 0x8086 device 0x3a40
Intel Corporation 82801JI (ICH10 Family) PCI Express Port 1
pci bus 0x0000 cardnum 0x1d function 0x00: vendor 0x8086 device 0x3a34
Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
pci bus 0x0000 cardnum 0x1d function 0x01: vendor 0x8086 device 0x3a35
Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
pci bus 0x0000 cardnum 0x1d function 0x02: vendor 0x8086 device 0x3a36
Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
pci bus 0x0000 cardnum 0x1d function 0x07: vendor 0x8086 device 0x3a3a
Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
pci bus 0x0000 cardnum 0x1e function 0x00: vendor 0x8086 device 0x244e
Intel Corporation 82801 PCI Bridge
pci bus 0x0000 cardnum 0x1f function 0x00: vendor 0x8086 device 0x3a16
Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
pci bus 0x0000 cardnum 0x1f function 0x02: vendor 0x8086 device 0x3a22
Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller
pci bus 0x0000 cardnum 0x1f function 0x03: vendor 0x8086 device 0x3a30
Intel Corporation 82801JI (ICH10 Family) SMBus Controller
pci bus 0x0001 cardnum 0x03 function 0x00: vendor 0x102b device 0x0532
Matrox Graphics, Inc. MGA G200eW WPCM450
pci bus 0x0002 cardnum 0x00 function 0x00: vendor 0x1000 device 0x0058
LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS
pci bus 0x0004 cardnum 0x00 function 0x00: vendor 0x1000 device 0x0058
LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS
pci bus 0x0006 cardnum 0x00 function 0x00: vendor 0x8086 device 0x10c9
Intel Corporation 82576 Gigabit ET Dual Port Server Adapter
pci bus 0x0006 cardnum 0x00 function 0x01: vendor 0x8086 device 0x10c9
Intel Corporation 82576 Gigabit ET Dual Port Server Adapter
pci bus 0x00fe cardnum 0x00 function 0x00: vendor 0x8086 device 0x2c40
Intel Corporation QuickPath Architecture Generic Non-Core Registers
pci bus 0x00fe cardnum 0x00 function 0x01: vendor 0x8086 device 0x2c01
Intel Corporation QuickPath Architecture System Address Decoder
pci bus 0x00fe cardnum 0x02 function 0x00: vendor 0x8086 device 0x2c10
Intel Corporation QPI Link 0
pci bus 0x00fe cardnum 0x02 function 0x01: vendor 0x8086 device 0x2c11
Intel Corporation QPI Physical 0
pci bus 0x00fe cardnum 0x02 function 0x04: vendor 0x8086 device 0x2c14
Intel Corporation QPI Link 1
pci bus 0x00fe cardnum 0x02 function 0x05: vendor 0x8086 device 0x2c15
Intel Corporation QPI Physical 1
pci bus 0x00fe cardnum 0x03 function 0x00: vendor 0x8086 device 0x2c18
Intel Corporation QuickPath Memory Controller
pci bus 0x00fe cardnum 0x03 function 0x01: vendor 0x8086 device 0x2c19
Intel Corporation QuickPath Memory Controller Target Address Decoder
pci bus 0x00fe cardnum 0x03 function 0x02: vendor 0x8086 device 0x2c1a
Intel Corporation QuickPath Memory Controller RAS Registers
pci bus 0x00fe cardnum 0x03 function 0x04: vendor 0x8086 device 0x2c1c
Intel Corporation QuickPath Memory Controller Test Registers
pci bus 0x00fe cardnum 0x04 function 0x00: vendor 0x8086 device 0x2c20
Intel Corporation QuickPath Memory Controller Channel 0 Control Registers
pci bus 0x00fe cardnum 0x04 function 0x01: vendor 0x8086 device 0x2c21
Intel Corporation QuickPath Memory Controller Channel 0 Address Registers
pci bus 0x00fe cardnum 0x04 function 0x02: vendor 0x8086 device 0x2c22
Intel Corporation QuickPath Memory Controller Channel 0 Rank Registers
pci bus 0x00fe cardnum 0x04 function 0x03: vendor 0x8086 device 0x2c23
Intel Corporation QuickPath Memory Controller Channel 0 Thermal
Control Registers
pci bus 0x00fe cardnum 0x05 function 0x00: vendor 0x8086 device 0x2c28
Intel Corporation QuickPath Memory Controller Channel 1 Control Registers
pci bus 0x00fe cardnum 0x05 function 0x01: vendor 0x8086 device 0x2c29
Intel Corporation QuickPath Memory Controller Channel 1 Address Registers
pci bus 0x00fe cardnum 0x05 function 0x02: vendor 0x8086 device 0x2c2a
Intel Corporation QuickPath Memory Controller Channel 1 Rank Registers
pci bus 0x00fe cardnum 0x05 function 0x03: vendor 0x8086 device 0x2c2b
Intel Corporation QuickPath Memory Controller Channel 1 Thermal
Control Registers
pci bus 0x00fe cardnum 0x06 function 0x00: vendor 0x8086 device 0x2c30
Intel Corporation QuickPath Memory Controller Channel 2 Control Registers
pci bus 0x00fe cardnum 0x06 function 0x01: vendor 0x8086 device 0x2c31
Intel Corporation QuickPath Memory Controller Channel 2 Address Registers
pci bus 0x00fe cardnum 0x06 function 0x02: vendor 0x8086 device 0x2c32
Intel Corporation QuickPath Memory Controller Channel 2 Rank Registers
pci bus 0x00fe cardnum 0x06 function 0x03: vendor 0x8086 device 0x2c33
Intel Corporation QuickPath Memory Controller Channel 2 Thermal
Control Registers
pci bus 0x00ff cardnum 0x00 function 0x00: vendor 0x8086 device 0x2c40
Intel Corporation QuickPath Architecture Generic Non-Core Registers
pci bus 0x00ff cardnum 0x00 function 0x01: vendor 0x8086 device 0x2c01
Intel Corporation QuickPath Architecture System Address Decoder
pci bus 0x00ff cardnum 0x02 function 0x00: vendor 0x8086 device 0x2c10
Intel Corporation QPI Link 0
pci bus 0x00ff cardnum 0x02 function 0x01: vendor 0x8086 device 0x2c11
Intel Corporation QPI Physical 0
pci bus 0x00ff cardnum 0x02 function 0x04: vendor 0x8086 device 0x2c14
Intel Corporation QPI Link 1
pci bus 0x00ff cardnum 0x02 function 0x05: vendor 0x8086 device 0x2c15
Intel Corporation QPI Physical 1
pci bus 0x00ff cardnum 0x03 function 0x00: vendor 0x8086 device 0x2c18
Intel Corporation QuickPath Memory Controller
pci bus 0x00ff cardnum 0x03 function 0x01: vendor 0x8086 device 0x2c19
Intel Corporation QuickPath Memory Controller Target Address Decoder
pci bus 0x00ff cardnum 0x03 function 0x02: vendor 0x8086 device 0x2c1a
Intel Corporation QuickPath Memory Controller RAS Registers
pci bus 0x00ff cardnum 0x03 function 0x04: vendor 0x8086 device 0x2c1c
Intel Corporation QuickPath Memory Controller Test Registers
pci bus 0x00ff cardnum 0x04 function 0x00: vendor 0x8086 device 0x2c20
Intel Corporation QuickPath Memory Controller Channel 0 Control Registers
pci bus 0x00ff cardnum 0x04 function 0x01: vendor 0x8086 device 0x2c21
Intel Corporation QuickPath Memory Controller Channel 0 Address Registers
pci bus 0x00ff cardnum 0x04 function 0x02: vendor 0x8086 device 0x2c22
Intel Corporation QuickPath Memory Controller Channel 0 Rank Registers
pci bus 0x00ff cardnum 0x04 function 0x03: vendor 0x8086 device 0x2c23
Intel Corporation QuickPath Memory Controller Channel 0 Thermal
Control Registers
pci bus 0x00ff cardnum 0x05 function 0x00: vendor 0x8086 device 0x2c28
Intel Corporation QuickPath Memory Controller Channel 1 Control Registers
pci bus 0x00ff cardnum 0x05 function 0x01: vendor 0x8086 device 0x2c29
Intel Corporation QuickPath Memory Controller Channel 1 Address Registers
pci bus 0x00ff cardnum 0x05 function 0x02: vendor 0x8086 device 0x2c2a
Intel Corporation QuickPath Memory Controller Channel 1 Rank Registers
pci bus 0x00ff cardnum 0x05 function 0x03: vendor 0x8086 device 0x2c2b
Intel Corporation QuickPath Memory Controller Channel 1 Thermal
Control Registers
pci bus 0x00ff cardnum 0x06 function 0x00: vendor 0x8086 device 0x2c30
Intel Corporation QuickPath Memory Controller Channel 2 Control Registers
pci bus 0x00ff cardnum 0x06 function 0x01: vendor 0x8086 device 0x2c31
Intel Corporation QuickPath Memory Controller Channel 2 Address Registers
pci bus 0x00ff cardnum 0x06 function 0x02: vendor 0x8086 device 0x2c32
Intel Corporation QuickPath Memory Controller Channel 2 Rank Registers
pci bus 0x00ff cardnum 0x06 function 0x03: vendor 0x8086 device 0x2c33
Intel Corporation QuickPath Memory Controller Channel 2 Thermal
Control Registers
--
paul