After running some MPI jobs, the job will exit with error:
[69] MPI startup(): ofa fabric is not available and fallback fabric is not enabled
The dmesg show:
mlx4_core 0000:81:00.0: vpd r/w failed. This is likely a firmware bug on this device. Contact the card vendor for a firmware update.
mlx4_core 0000:81:00.0: vpd r/w failed. This is likely a firmware bug on this device. Contact the card vendor for a firmware update.
LustreError: 28769:0:(file.c:186:ll_close_inode_openhandle()) tc4600fs-clilmv-ffff8810745d2c00: inode [0x2000015fb:0x1f8f0:0x0] mdc close failed: rc = -13
LustreError: 28752:0:(file.c:186:ll_close_inode_openhandle()) tc4600fs-clilmv-ffff8810745d2c00: inode [0x2000015fb:0x1f8f2:0x0] mdc close failed: rc = -13
LustreError: 28752:0:(file.c:186:ll_close_inode_openhandle()) Skipped 1 previous similar message
OS: CentOS release 6.5 2.6.32-431.29.2.el6.x86_64
mstflint -d 0000:81:00.0 query:
Image type: FS2
FW Version: 2.30.8000
Device ID: 4099
Description: Node Port1 Port2 Sys image
GUIDs: 0000e8611f104684 0000e8611f104685 0000e8611f104686 0000e8611f104687
MACs: e8611f104684 e8611f104685
VSD:
PSID: MT_1100110019
ibv_devinfo:
hca_id: mlx4_0
transport: InfiniBand (0)
fw_ver: 2.30.8000
node_guid: 0000:e861:1f10:4684
sys_image_guid: 0000:e861:1f10:4687
vendor_id: 0x02c9
vendor_part_id: 4099
hw_ver: 0x1
board_id: MT_1100110019
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 80
port_lid: 149
port_lmc: 0x00
link_layer: InfiniBand
ibqueryerrors
Errors for "node11 HCA-1"
GUID 0x6c025792bf00374d port 1: [PortXmitWait == 428]
Errors for "node12 HCA-1"
GUID 0x6c025792bf0052f9 port 1: [PortXmitWait == 4034525]
Errors for "node13 HCA-1"
GUID 0x6c025792bf005029 port 1: [PortXmitWait == 1332591]
Errors for "node14 HCA-1"
GUID 0x6c025792bf00378d port 1: [PortXmitWait == 497142]
Errors for "node15 HCA-1"
GUID 0x6c025792bf003769 port 1: [PortXmitWait == 13]
Errors for "node16 HCA-1"
GUID 0x6c025792bf003795 port 1: [PortXmitWait == 467501]
Errors for "node17 HCA-1"
GUID 0x6c025792bf005349 port 1: [PortXmitWait == 24680]
Errors for "node18 HCA-1"
GUID 0x6c025792bf005365 port 1: [PortXmitWait == 293558]
Errors for "node19 HCA-1"
GUID 0x6c025792bf003799 port 1: [PortXmitWait == 3864438]
Errors for "node20 HCA-1"
GUID 0x6c025792bf003751 port 1: [PortXmitWait == 8791370]
Errors for "node36 HCA-1"
GUID 0x46d2c92000002371 port 1: [PortXmitWait == 505251]
Errors for "node25 HCA-1"
GUID 0x2c903001f7b01 port 1: [PortXmitWait == 22433658]
Errors for "node24 HCA-1"
GUID 0x2c903001f7011 port 1: [PortXmitWait == 231280]
Errors for "node37 HCA-1"
GUID 0x46d2c92000004ad1 port 1: [PortXmitWait == 35467]
Errors for "node33 HCA-1"
GUID 0x46d2c920000052b1 port 1: [PortXmitWait == 7150432]
Errors for "node21 HCA-1"
GUID 0x2c903001f8641 port 1: [PortXmitWait == 42722907]
Errors for "node31 HCA-1"
GUID 0x46d2c92000012201 port 1: [PortXmitWait == 21578]
Errors for "node39 HCA-1"
GUID 0x46d2c920000050c1 port 1: [PortXmitWait == 8356]
Errors for "node34 HCA-1"
GUID 0x46d2c920000058a1 port 1: [PortXmitWait == 58677393]
Errors for "node35 HCA-1"
GUID 0x46d2c920000049d1 port 1: [PortXmitWait == 45396994]
Errors for "node38 HCA-1"
GUID 0x46d2c920000052a1 port 1: [PortXmitWait == 7981989]
Errors for "node40 HCA-1"
GUID 0x46d2c92000005401 port 1: [PortXmitWait == 92006320]
Errors for "node41 HCA-1"
GUID 0x46d2c92000005121 port 1: [PortXmitWait == 10843583]
Errors for "node42 HCA-1"
GUID 0x46d2c92000005851 port 1: [PortXmitWait == 6418548]
Errors for "node43 HCA-1"
GUID 0x46d2c92000005441 port 1: [PortXmitWait == 2123159]
Errors for "node44 HCA-1"
GUID 0x46d2c920000053f1 port 1: [PortXmitWait == 15341643]
Errors for "node32 HCA-1"
GUID 0x46d2c9200000c971 port 1: [PortXmitWait == 102666977]
Errors for "node29 HCA-1"
GUID 0x2c903001f8601 port 1: [PortXmitWait == 6830999]
Errors for "node28 HCA-1"
GUID 0x2c903001f64b1 port 1: [PortXmitWait == 79086076]
Errors for "node22 HCA-1"
GUID 0x2c903001f8661 port 1: [PortXmitWait == 5017]
Errors for "node23 HCA-1"
GUID 0x2c903001f6f01 port 1: [PortXmitWait == 37801]
Errors for "node30 HCA-1"
GUID 0x2c903001f8651 port 1: [PortXmitWait == 97280452]
Errors for "node27 HCA-1"
GUID 0x2c903001f5641 port 1: [PortXmitWait == 126472513]
Errors for 0x6c025792bf003668 "Infiniscale-IV Mellanox Technologies"
GUID 0x6c025792bf003668 port ALL: [SymbolErrorCounter == 52034] [LinkErrorRecoveryCounter == 18] [PortRcvErrors == 52089] [PortRcvSwitchRelayErrors == 6233] [PortXmitDiscards == 13] [PortXmitWait == 100747894]
GUID 0x6c025792bf003668 port 1: [PortRcvErrors == 1] [PortXmitDiscards == 1] [PortXmitWait == 4294967295]
GUID 0x6c025792bf003668 port 2: [PortXmitWait == 4294967295]
GUID 0x6c025792bf003668 port 3: [PortXmitWait == 4294967295]
GUID 0x6c025792bf003668 port 4: [PortXmitWait == 4294967295]
GUID 0x6c025792bf003668 port 5: [PortXmitWait == 12550054]
GUID 0x6c025792bf003668 port 6: [PortXmitDiscards == 1] [PortXmitWait == 4294967295]
GUID 0x6c025792bf003668 port 7: [PortXmitWait == 3662292368]
GUID 0x6c025792bf003668 port 8: [PortXmitWait == 4294967295]
GUID 0x6c025792bf003668 port 9: [PortXmitWait == 4294967295]
GUID 0x6c025792bf003668 port 10: [PortXmitWait == 4294967295]
GUID 0x6c025792bf003668 port 17: [LinkErrorRecoveryCounter == 2] [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 2] [PortXmitWait == 4294967295]
GUID 0x6c025792bf003668 port 19: [LinkErrorRecoveryCounter == 2] [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 1] [PortXmitWait == 4294967295]
GUID 0x6c025792bf003668 port 21: [SymbolErrorCounter == 52034] [LinkErrorRecoveryCounter == 2] [PortRcvErrors == 51880] [PortRcvSwitchRelayErrors == 2051] [PortXmitDiscards == 1] [PortXmitWait == 4294967295]
GUID 0x6c025792bf003668 port 23: [LinkErrorRecoveryCounter == 2] [PortRcvSwitchRelayErrors == 169] [PortXmitDiscards == 1] [PortXmitWait == 4294967295]
GUID 0x6c025792bf003668 port 25: [PortRcvSwitchRelayErrors == 29] [PortXmitDiscards == 1] [PortXmitWait == 4294967295]
GUID 0x6c025792bf003668 port 27: [LinkErrorRecoveryCounter == 2] [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 1] [PortXmitWait == 4294967295]
GUID 0x6c025792bf003668 port 29: [LinkErrorRecoveryCounter == 2] [PortRcvErrors == 208] [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 2] [PortXmitWait == 4294967295]
GUID 0x6c025792bf003668 port 31: [LinkErrorRecoveryCounter == 2] [PortRcvSwitchRelayErrors == 3292] [PortXmitDiscards == 2] [PortXmitWait == 4294967295]
GUID 0x6c025792bf003668 port 33: [LinkErrorRecoveryCounter == 2] [PortRcvSwitchRelayErrors == 687] [PortXmitWait == 720872785]
GUID 0x6c025792bf003668 port 35: [LinkErrorRecoveryCounter == 2] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
Errors for "node103 HCA-1"
GUID 0xe8611f104a35 port 1: [PortXmitWait == 961]
Errors for "node110 HCA-1"
GUID 0xe8611f104af5 port 1: [PortXmitWait == 1183983]
Errors for "node102 HCA-1"
GUID 0xe8611f104865 port 1: [PortXmitWait == 1714217]
Errors for "node109 HCA-1"
GUID 0xe8611f1047b5 port 1: [PortXmitWait == 2812689]
Errors for "io1 HCA-1"
GUID 0x2c90300418f61 port 1: [PortXmitWait == 39237]
Errors for "node2 HCA-1"
GUID 0x2c903001ba811 port 1: [PortXmitWait == 9660175]
Errors for "node1 HCA-1"
GUID 0x2c903001ba381 port 1: [PortXmitWait == 51238]
Errors for "node9 HCA-1"
GUID 0x2c90300419061 port 1: [PortXmitWait == 25329]
Errors for "node8 HCA-1"
GUID 0x2c90300418f71 port 1: [PortXmitWait == 121]
Errors for "node108 HCA-1"
GUID 0xe8611f104795 port 1: [PortXmitWait == 1111446]
Errors for "node107 HCA-1"
GUID 0xe8611f104a15 port 1: [PortXmitWait == 1889]
Errors for "node4 HCA-1"
GUID 0x2c903001ba7f1 port 1: [PortXmitWait == 26625247]
Errors for 0x2c903008ccf00 "SwitchX - Mellanox Technologies"
GUID 0x2c903008ccf00 port ALL: [LinkDownedCounter == 88] [PortRcvRemotePhysicalErrors == 16231] [PortRcvSwitchRelayErrors == 157] [PortXmitWait == 4294967295]
GUID 0x2c903008ccf00 port 1: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008ccf00 port 2: [PortRcvSwitchRelayErrors == 129] [PortXmitWait == 4294967295]
GUID 0x2c903008ccf00 port 6: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008ccf00 port 7: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008ccf00 port 8: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008ccf00 port 9: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008ccf00 port 10: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008ccf00 port 12: [LinkDownedCounter == 9] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008ccf00 port 13: [LinkDownedCounter == 8] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008ccf00 port 14: [LinkDownedCounter == 5] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008ccf00 port 15: [LinkDownedCounter == 5] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008ccf00 port 16: [LinkDownedCounter == 6] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008ccf00 port 17: [LinkDownedCounter == 6] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008ccf00 port 19: [LinkDownedCounter == 4] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008ccf00 port 22: [LinkDownedCounter == 4] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008ccf00 port 23: [LinkDownedCounter == 5] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008ccf00 port 24: [LinkDownedCounter == 5] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008ccf00 port 26: [LinkDownedCounter == 10] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008ccf00 port 27: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008ccf00 port 28: [LinkDownedCounter == 5] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008ccf00 port 29: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 2] [PortXmitWait == 1]
GUID 0x2c903008ccf00 port 30: [PortRcvSwitchRelayErrors == 2]
GUID 0x2c903008ccf00 port 31: [LinkDownedCounter == 4] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008ccf00 port 32: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008ccf00 port 34: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008ccf00 port 35: [LinkDownedCounter == 9] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008ccf00 port 36: [PortRcvRemotePhysicalErrors == 16231] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
Errors for "node106 HCA-1"
GUID 0xe8611f104a85 port 1: [PortXmitWait == 370282]
Errors for "node105 HCA-1"
GUID 0xe8611f104bc5 port 1: [PortXmitWait == 1415449]
Errors for "node10 HCA-1"
GUID 0x2c90300419071 port 1: [PortXmitWait == 1682321]
Errors for "node104 HCA-1"
GUID 0xe8611f104995 port 1: [PortXmitWait == 2769787]
Errors for "node7 HCA-1"
GUID 0x2c903001bb811 port 1: [PortXmitWait == 6544359]
Errors for "node101 HCA-1"
GUID 0xe8611f104b35 port 1: [PortXmitWait == 599737]
Errors for "node6 HCA-1"
GUID 0x2c903001ba801 port 1: [PortXmitWait == 56043]
Errors for "node5 HCA-1"
GUID 0x2c90300357551 port 1: [PortXmitWait == 52033813]
Errors for "node3 HCA-1"
GUID 0x2c903001ba4b1 port 1: [PortXmitWait == 25888335]
Errors for "io4 HCA-1"
GUID 0xf452140300338911 port 1: [PortXmitWait == 20]
Errors for "io3 HCA-1"
GUID 0xf452140300339751 port 1: [PortXmitWait == 427]
Errors for "io2 HCA-1"
GUID 0xf452140300339731 port 1: [PortXmitWait == 2658548]
Errors for "io1 HCA-1"
GUID 0xf452140300339471 port 1: [PortXmitWait == 263]
Errors for "tc4600 HCA-1"
GUID 0xf45214030082cab1 port 1: [PortXmitWait == 1735150]
Errors for 0x2c903008d5600 "SwitchX - Mellanox Technologies"
GUID 0x2c903008d5600 port ALL: [LinkDownedCounter == 255] [PortRcvErrors == 2] [PortRcvRemotePhysicalErrors == 45963] [PortRcvSwitchRelayErrors == 27939] [PortXmitDiscards == 1927] [PortXmitWait == 4294967295]
GUID 0x2c903008d5600 port 1: [PortXmitWait == 98295522]
GUID 0x2c903008d5600 port 2: [PortXmitWait == 51845652]
GUID 0x2c903008d5600 port 3: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008d5600 port 4: [PortRcvSwitchRelayErrors == 3] [PortXmitDiscards == 1079] [PortXmitWait == 4294967295]
GUID 0x2c903008d5600 port 5: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008d5600 port 6: [LinkDownedCounter == 2] [PortRcvSwitchRelayErrors == 123] [PortXmitWait == 4294967295]
GUID 0x2c903008d5600 port 7: [PortRcvSwitchRelayErrors == 27159] [PortXmitWait == 4294967295]
GUID 0x2c903008d5600 port 8: [PortRcvSwitchRelayErrors == 13] [PortXmitDiscards == 161] [PortXmitWait == 4294967295]
GUID 0x2c903008d5600 port 9: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 116] [PortXmitWait == 4294967295]
GUID 0x2c903008d5600 port 10: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008d5600 port 11: [PortRcvRemotePhysicalErrors == 128] [PortXmitWait == 4294967295]
GUID 0x2c903008d5600 port 12: [PortRcvRemotePhysicalErrors == 38462] [PortXmitWait == 4294967295]
GUID 0x2c903008d5600 port 13: [PortXmitWait == 4294967295]
GUID 0x2c903008d5600 port 14: [PortXmitWait == 4294967295]
GUID 0x2c903008d5600 port 15: [PortRcvRemotePhysicalErrors == 2016] [PortXmitWait == 4294967295]
GUID 0x2c903008d5600 port 16: [PortXmitWait == 4294967295]
GUID 0x2c903008d5600 port 17: [PortRcvRemotePhysicalErrors == 219] [PortXmitWait == 4294967295]
GUID 0x2c903008d5600 port 18: [PortXmitWait == 4294967295]
GUID 0x2c903008d5600 port 19: [PortRcvRemotePhysicalErrors == 74] [PortXmitWait == 674164601]
GUID 0x2c903008d5600 port 20: [PortRcvRemotePhysicalErrors == 4983] [PortXmitWait == 4294967295]
GUID 0x2c903008d5600 port 21: [PortRcvSwitchRelayErrors == 2] [PortXmitWait == 281438]
GUID 0x2c903008d5600 port 22: [PortRcvRemotePhysicalErrors == 81] [PortXmitWait == 4294967295]
GUID 0x2c903008d5600 port 23: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 220212]
GUID 0x2c903008d5600 port 24: [LinkDownedCounter == 66] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008d5600 port 25: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 2198114]
GUID 0x2c903008d5600 port 26: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008d5600 port 27: [PortRcvErrors == 2] [PortRcvSwitchRelayErrors == 3] [PortXmitWait == 4294967295]
GUID 0x2c903008d5600 port 29: [LinkDownedCounter == 202] [PortRcvSwitchRelayErrors == 241] [PortXmitWait == 4294967295]
GUID 0x2c903008d5600 port 30: [LinkDownedCounter == 2] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0x2c903008d5600 port 31: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4257910]
GUID 0x2c903008d5600 port 33: [PortRcvSwitchRelayErrors == 381] [PortXmitWait == 4294967295]
GUID 0x2c903008d5600 port 34: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 323] [PortXmitWait == 4294967295]
GUID 0x2c903008d5600 port 35: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 484796]
GUID 0x2c903008d5600 port 36: [PortRcvSwitchRelayErrors == 2] [PortXmitDiscards == 248] [PortXmitWait == 4294967295]
Errors for "node99 HCA-1"
GUID 0xe8611f104a75 port 1: [PortXmitWait == 9345108]
Errors for "node100 HCA-1"
GUID 0xe8611f104a55 port 1: [PortXmitWait == 22436]
Errors for "node97 HCA-1"
GUID 0xe8611f104a95 port 1: [PortXmitWait == 3856160]
Errors for "node98 HCA-1"
GUID 0xe8611f104ae5 port 1: [PortXmitWait == 28418]
Errors for "node96 HCA-1"
GUID 0xe8611f104675 port 1: [PortXmitWait == 3173368]
Errors for "node95 HCA-1"
GUID 0xe8611f104665 port 1: [PortXmitWait == 4704227]
Errors for "node94 HCA-1"
GUID 0xe8611f104735 port 1: [PortXmitWait == 5456975]
Errors for "node93 HCA-1"
GUID 0xe8611f104725 port 1: [PortXmitWait == 2931296]
Errors for "node92 HCA-1"
GUID 0xe8611f1046c5 port 1: [PortXmitWait == 19874]
Errors for "node91 HCA-1"
GUID 0xe8611f104ab5 port 1: [PortXmitWait == 550147]
Errors for "node72 HCA-1"
GUID 0xe8611f104945 port 1: [PortXmitWait == 215794]
Errors for "node71 HCA-1"
GUID 0xe8611f104935 port 1: [PortXmitWait == 29770]
Errors for "node70 HCA-1"
GUID 0xe8611f1047a5 port 1: [PortXmitWait == 26135]
Errors for "node69 HCA-1"
GUID 0xe8611f1046e5 port 1: [PortXmitWait == 8863391]
Errors for "node68 HCA-1"
GUID 0xe8611f104975 port 1: [PortXmitWait == 1319]
Errors for "node67 HCA-1"
GUID 0xe8611f1045f5 port 1: [PortXmitWait == 3451879]
Errors for "node66 HCA-1"
GUID 0xe8611f1048d5 port 1: [PortXmitWait == 1255020]
Errors for "node65 HCA-1"
GUID 0xe8611f1048f5 port 1: [PortXmitWait == 98193]
Errors for "node64 HCA-1"
GUID 0xe8611f1048e5 port 1: [PortXmitWait == 2364]
Errors for "node63 HCA-1"
GUID 0xe8611f1047c5 port 1: [PortXmitWait == 37412]
Errors for "node62 HCA-1"
GUID 0xe8611f1049b5 port 1: [PortXmitWait == 8185167]
Errors for "node61 HCA-1"
GUID 0xe8611f104785 port 1: [PortXmitWait == 1464886]
Errors for "node60 HCA-1"
GUID 0xe8611f1048a5 port 1: [PortXmitWait == 2090245]
Errors for "node59 HCA-1"
GUID 0xe8611f104855 port 1: [PortXmitWait == 5425108]
Errors for "node58 HCA-1"
GUID 0xe8611f104895 port 1: [PortXmitWait == 10235]
Errors for "node57 HCA-1"
GUID 0xe8611f104625 port 1: [PortXmitWait == 11568]
Errors for "node56 HCA-1"
GUID 0xe8611f104ad5 port 1: [PortXmitWait == 10900]
Errors for "node55 HCA-1"
GUID 0xe8611f104ac5 port 1: [PortXmitWait == 8138]
Errors for "node54 HCA-1"
GUID 0xe8611f104b25 port 1: [PortXmitWait == 19559]
Errors for "node53 HCA-1"
GUID 0xe8611f104b15 port 1: [PortXmitWait == 3337]
Errors for "node52 HCA-1"
GUID 0xe8611f104a65 port 1: [PortXmitWait == 2305000]
Errors for "node51 HCA-1"
GUID 0xe8611f1048b5 port 1: [PortXmitWait == 2859967]
Errors for "node50 HCA-1"
GUID 0xe8611f104875 port 1: [PortXmitWait == 142054]
Errors for "node47 HCA-1"
GUID 0xe8611f1046b5 port 1: [PortXmitWait == 384681]
Errors for "node46 HCA-1"
GUID 0xe8611f102bf9 port 1: [PortXmitWait == 113900]
Errors for "node45 HCA-1"
GUID 0xe8611f102c49 port 1: [PortXmitWait == 1]
Errors for "node44 HCA-1"
GUID 0xe8611f102dd9 port 1: [PortXmitWait == 2109061]
Errors for "node43 HCA-1"
GUID 0xe8611f102be9 port 1: [PortXmitWait == 1492859]
Errors for "node41 HCA-1"
GUID 0xe8611f102c29 port 1: [PortXmitWait == 1319431]
Errors for "node42 HCA-1"
GUID 0xe8611f102db9 port 1: [PortXmitWait == 1767831]
Errors for "node40 HCA-1"
GUID 0xe8611f102c79 port 1: [PortXmitWait == 8022]
Errors for "node39 HCA-1"
GUID 0xe8611f102d39 port 1: [PortXmitWait == 1275902]
Errors for "node38 HCA-1"
GUID 0xe8611f102bb9 port 1: [PortXmitWait == 681445]
Errors for "node37 HCA-1"
GUID 0xe8611f102ca9 port 1: [PortXmitWait == 149394]
Errors for "node36 HCA-1"
GUID 0xe8611f102c99 port 1: [PortXmitWait == 2817]
Errors for "node35 HCA-1"
GUID 0xe8611f102b99 port 1: [PortXmitWait == 4314308]
Errors for "node34 HCA-1"
GUID 0xe8611f102e49 port 1: [PortXmitWait == 4357]
Errors for "node33 HCA-1"
GUID 0xe8611f102d69 port 1: [PortXmitWait == 2630012]
Errors for "node32 HCA-1"
GUID 0xe8611f102b79 port 1: [PortXmitWait == 6621266]
Errors for "node31 HCA-1"
GUID 0xe8611f102b39 port 1: [PortXmitWait == 2057]
Errors for "node30 HCA-1"
GUID 0xe8611f102b29 port 1: [PortXmitWait == 13528]
Errors for "node29 HCA-1"
GUID 0xe8611f102cd9 port 1: [PortXmitWait == 674]
Errors for "node28 HCA-1"
GUID 0xe8611f102da9 port 1: [PortXmitWait == 1597]
Errors for "node27 HCA-1"
GUID 0xe8611f102d09 port 1: [PortXmitWait == 189324]
Errors for "node26 HCA-1"
GUID 0xe8611f102c09 port 1: [PortXmitWait == 443]
Errors for "node25 HCA-1"
GUID 0xe8611f102d59 port 1: [PortXmitWait == 1691]
Errors for "node24 HCA-1"
GUID 0xe8611f102b89 port 1: [PortXmitWait == 18]
Errors for "node23 HCA-1"
GUID 0xe8611f102cb9 port 1: [PortXmitWait == 69911]
Errors for "node22 HCA-1"
GUID 0xe8611f102d49 port 1: [PortXmitWait == 23050]
Errors for "node21 HCA-1"
GUID 0xe8611f102d29 port 1: [PortXmitWait == 5232157]
Errors for "node20 HCA-1"
GUID 0xe8611f102bd9 port 1: [PortXmitWait == 200427]
Errors for "node19 HCA-1"
GUID 0xe8611f102b49 port 1: [PortXmitWait == 2110]
Errors for "node18 HCA-1"
GUID 0xe8611f102c19 port 1: [PortXmitWait == 5403]
Errors for "node17 HCA-1"
GUID 0xe8611f102ce9 port 1: [PortXmitWait == 261964]
Errors for "node16 HCA-1"
GUID 0xe8611f102b69 port 1: [PortXmitWait == 2790552]
Errors for "node15 HCA-1"
GUID 0xe8611f102ba9 port 1: [PortXmitWait == 5652]
Errors for "node14 HCA-1"
GUID 0xe8611f102c69 port 1: [PortXmitWait == 3789]
Errors for "node13 HCA-1"
GUID 0xe8611f102e59 port 1: [PortXmitWait == 2459]
Errors for "node12 HCA-1"
GUID 0xe8611f102b59 port 1: [PortXmitWait == 1023]
Errors for "node11 HCA-1"
GUID 0xe8611f102d19 port 1: [PortXmitWait == 136582]
Errors for "node10 HCA-1"
GUID 0xe8611f102e29 port 1: [PortXmitWait == 381880]
Errors for "node9 HCA-1"
GUID 0xe8611f102cf9 port 1: [PortXmitWait == 7573536]
Errors for "node8 HCA-1"
GUID 0xe8611f102e39 port 1: [PortXmitWait == 924132]
Errors for "node7 HCA-1"
GUID 0xe8611f102e19 port 1: [PortXmitWait == 21617]
Errors for "node6 HCA-1"
GUID 0xe8611f102e69 port 1: [PortXmitWait == 2949430]
Errors for "node5 HCA-1"
GUID 0xe8611f102c39 port 1: [PortXmitWait == 2317637]
Errors for "node4 HCA-1"
GUID 0xe8611f102d79 port 1: [PortXmitWait == 6303]
Errors for "node3 HCA-1"
GUID 0xe8611f102bc9 port 1: [PortXmitWait == 4568803]
Errors for "node2 HCA-1"
GUID 0xe8611f102c59 port 1: [PortXmitWait == 94382]
Errors for "node1 HCA-1"
GUID 0xe8611f102dc9 port 1: [PortXmitWait == 841]
Errors for 0xf4521403008b76c0 "MF0;switch-993d34:SX6506/L06/U1"
GUID 0xf4521403008b76c0 port ALL: [LinkDownedCounter == 11] [PortRcvSwitchRelayErrors == 22802] [PortXmitDiscards == 8057] [PortXmitWait == 4294967295]
GUID 0xf4521403008b76c0 port 1: [PortRcvSwitchRelayErrors == 3] [PortXmitDiscards == 1079] [PortXmitWait == 4294967295]
GUID 0xf4521403008b76c0 port 2: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 336] [PortXmitWait == 4294967295]
GUID 0xf4521403008b76c0 port 3: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 3678] [PortXmitWait == 4294967295]
GUID 0xf4521403008b76c0 port 4: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 129] [PortXmitWait == 4294967295]
GUID 0xf4521403008b76c0 port 5: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 522] [PortXmitWait == 4294967295]
GUID 0xf4521403008b76c0 port 6: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 198] [PortXmitWait == 4294967295]
GUID 0xf4521403008b76c0 port 7: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 1870] [PortXmitWait == 4294967295]
GUID 0xf4521403008b76c0 port 8: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 204] [PortXmitWait == 4294967295]
GUID 0xf4521403008b76c0 port 9: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 41] [PortXmitWait == 4294967295]
GUID 0xf4521403008b76c0 port 10: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b76c0 port 11: [LinkDownedCounter == 2] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 1771]
GUID 0xf4521403008b76c0 port 12: [PortXmitWait == 571701]
GUID 0xf4521403008b76c0 port 13: [LinkDownedCounter == 3] [PortRcvSwitchRelayErrors == 2] [PortXmitWait == 4294967295]
GUID 0xf4521403008b76c0 port 14: [PortXmitWait == 9250848]
GUID 0xf4521403008b76c0 port 15: [LinkDownedCounter == 2] [PortRcvSwitchRelayErrors == 5817] [PortXmitWait == 4294967295]
GUID 0xf4521403008b76c0 port 16: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 6027] [PortXmitWait == 4294967295]
GUID 0xf4521403008b76c0 port 17: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 5569] [PortXmitWait == 4294967295]
GUID 0xf4521403008b76c0 port 18: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 5374] [PortXmitWait == 4294967295]
GUID 0xf4521403008b76c0 port 19: [PortXmitWait == 3297636]
GUID 0xf4521403008b76c0 port 20: [PortXmitWait == 65313094]
GUID 0xf4521403008b76c0 port 21: [PortXmitWait == 1165031]
GUID 0xf4521403008b76c0 port 22: [PortXmitWait == 5774741]
GUID 0xf4521403008b76c0 port 23: [PortXmitWait == 2620130]
GUID 0xf4521403008b76c0 port 24: [PortXmitWait == 4708293]
GUID 0xf4521403008b76c0 port 25: [PortXmitWait == 8904962]
GUID 0xf4521403008b76c0 port 26: [PortXmitWait == 27577749]
GUID 0xf4521403008b76c0 port 27: [PortXmitWait == 29928363]
GUID 0xf4521403008b76c0 port 28: [PortXmitWait == 12385189]
GUID 0xf4521403008b76c0 port 29: [PortXmitWait == 16828511]
GUID 0xf4521403008b76c0 port 30: [PortXmitWait == 39950914]
GUID 0xf4521403008b76c0 port 31: [PortXmitWait == 7779423]
GUID 0xf4521403008b76c0 port 32: [PortXmitWait == 972493]
GUID 0xf4521403008b76c0 port 33: [PortXmitWait == 24165400]
GUID 0xf4521403008b76c0 port 34: [PortXmitWait == 9421146]
GUID 0xf4521403008b76c0 port 35: [PortXmitWait == 4778723]
GUID 0xf4521403008b76c0 port 36: [PortXmitWait == 1660990943]
Errors for 0xf4521403008b6fa0 "MF0;switch-993d34:SX6506/L04/U1"
GUID 0xf4521403008b6fa0 port ALL: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 270] [PortXmitDiscards == 105] [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 1: [PortRcvSwitchRelayErrors == 3] [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 2: [PortRcvSwitchRelayErrors == 2] [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 3: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 4: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 5: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 6: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 7: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 8: [PortRcvSwitchRelayErrors == 74] [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 9: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 10: [PortRcvSwitchRelayErrors == 8] [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 11: [PortRcvSwitchRelayErrors == 2] [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 12: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 13: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 14: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 15: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 16: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 17: [PortRcvSwitchRelayErrors == 169] [PortXmitDiscards == 79] [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 18: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 20] [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 19: [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 20: [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 21: [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 22: [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 23: [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 24: [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 25: [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 26: [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 27: [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 28: [PortXmitDiscards == 6] [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 29: [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 30: [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 31: [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 32: [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 33: [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 34: [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 35: [PortXmitWait == 4294967295]
GUID 0xf4521403008b6fa0 port 36: [PortXmitWait == 4294967295]
Errors for 0xf4521403008b7060 "MF0;switch-993d34:SX6506/L03/U1"
GUID 0xf4521403008b7060 port ALL: [LinkDownedCounter == 12] [PortRcvSwitchRelayErrors == 22] [PortXmitDiscards == 134] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 1: [PortRcvSwitchRelayErrors == 5] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 2: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 3: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 4: [LinkDownedCounter == 2] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 5: [LinkDownedCounter == 2] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 6: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 10747]
GUID 0xf4521403008b7060 port 7: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 36858]
GUID 0xf4521403008b7060 port 8: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 84156103]
GUID 0xf4521403008b7060 port 9: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 10: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 11: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 12: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 13: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 14: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 15: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 16: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 17: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 18: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 19: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 20: [PortXmitDiscards == 59] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 21: [PortXmitDiscards == 68] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 22: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 23: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 24: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 25: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 26: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 27: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 28: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 29: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 30: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 31: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 32: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 33: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 34: [PortXmitDiscards == 7] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 35: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7060 port 36: [PortXmitWait == 4294967295]
Errors for 0xf4521403008b68d0 "MF0;switch-993d34:SX6506/L02/U1"
GUID 0xf4521403008b68d0 port ALL: [LinkDownedCounter == 255] [PortRcvSwitchRelayErrors == 19435] [PortXmitDiscards == 18511] [LocalLinkIntegrityErrors == 15] [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 1: [PortRcvSwitchRelayErrors == 4] [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 2: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 3: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 4: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 5: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 6: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 7: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 8: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 9: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 10: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 11: [PortRcvSwitchRelayErrors == 581] [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 12: [PortRcvSwitchRelayErrors == 1444] [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 13: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 14: [PortRcvSwitchRelayErrors == 17391] [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 15: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 16: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 17: [LinkDownedCounter == 3] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 18: [LinkDownedCounter == 255] [LocalLinkIntegrityErrors == 15] [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 19: [PortRcvSwitchRelayErrors == 2] [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 20: [PortXmitDiscards == 4596] [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 21: [PortXmitDiscards == 6650] [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 22: [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 23: [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 24: [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 25: [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 26: [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 27: [PortXmitDiscards == 1155] [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 28: [PortXmitDiscards == 5403] [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 29: [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 30: [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 31: [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 32: [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 33: [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 34: [PortXmitDiscards == 707] [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 35: [PortXmitWait == 4294967295]
GUID 0xf4521403008b68d0 port 36: [PortXmitWait == 4294967295]
Errors for 0xf4521403008b7300 "MF0;switch-993d34:SX6506/L01/U1"
GUID 0xf4521403008b7300 port ALL: [LinkDownedCounter == 9] [PortRcvSwitchRelayErrors == 349] [PortXmitDiscards == 2898] [VL15Dropped == 22] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 1: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 2: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 3: [PortRcvSwitchRelayErrors == 15] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 4: [PortRcvSwitchRelayErrors == 4] [VL15Dropped == 22] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 5: [LinkDownedCounter == 3] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 6: [PortRcvSwitchRelayErrors == 4] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 7: [PortRcvSwitchRelayErrors == 10] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 8: [PortRcvSwitchRelayErrors == 2] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 9: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 10: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 11: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 12: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 13: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 14: [PortRcvSwitchRelayErrors == 247] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 15: [LinkDownedCounter == 5] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 16: [PortRcvSwitchRelayErrors == 56] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 17: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 18: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 19: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 20: [PortXmitDiscards == 884] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 21: [PortXmitDiscards == 1118] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 22: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 23: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 24: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 25: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 26: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 27: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 28: [PortXmitDiscards == 648] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 29: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 30: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 31: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 32: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 33: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 34: [PortXmitDiscards == 248] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 35: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7300 port 36: [PortXmitWait == 4294967295]
Errors for 0xf452140300476130 "MF0;switch-993d34:SX6506/S03/U1"
GUID 0xf452140300476130 port ALL: [PortXmitDiscards == 291] [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 1: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 2: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 3: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 4: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 5: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 6: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 7: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 8: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 9: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 10: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 11: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 12: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 13: [PortXmitWait == 601]
GUID 0xf452140300476130 port 14: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 15: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 16: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 17: [PortXmitWait == 3499372837]
GUID 0xf452140300476130 port 18: [PortXmitWait == 2966702]
GUID 0xf452140300476130 port 19: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 20: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 21: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 22: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 23: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 24: [PortXmitDiscards == 291] [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 25: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 26: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 27: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 28: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 29: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 30: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 34: [PortXmitWait == 4294967295]
GUID 0xf452140300476130 port 35: [PortXmitWait == 2967821492]
GUID 0xf452140300476130 port 36: [PortXmitWait == 4280804153]
Errors for 0xf452140300476100 "MF0;switch-993d34:SX6506/S02/U1"
GUID 0xf452140300476100 port ALL: [PortXmitDiscards == 343] [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 0: [PortXmitWait == 15675680]
GUID 0xf452140300476100 port 1: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 2: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 3: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 4: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 5: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 6: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 7: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 8: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 9: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 10: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 11: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 12: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 13: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 14: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 15: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 16: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 17: [PortXmitDiscards == 323] [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 18: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 19: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 20: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 21: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 22: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 23: [PortXmitDiscards == 20] [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 24: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 25: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 26: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 27: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 28: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 29: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 30: [PortXmitWait == 4294967295]
GUID 0xf452140300476100 port 31: [PortXmitWait == 498646609]
GUID 0xf452140300476100 port 32: [PortXmitWait == 796440845]
GUID 0xf452140300476100 port 33: [PortXmitWait == 293780019]
Errors for "node89 HCA-1"
GUID 0xe8611f104775 port 1: [PortXmitWait == 519674]
Errors for 0xf452140300476080 "MF0;switch-993d34:SX6506/S01/U1"
GUID 0xf452140300476080 port ALL: [PortXmitDiscards == 1603] [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 0: [PortXmitWait == 3773520]
GUID 0xf452140300476080 port 1: [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 2: [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 3: [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 4: [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 5: [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 6: [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 7: [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 8: [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 9: [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 10: [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 11: [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 12: [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 13: [PortXmitWait == 44639]
GUID 0xf452140300476080 port 14: [PortXmitWait == 174723]
GUID 0xf452140300476080 port 15: [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 16: [PortXmitWait == 33152813]
GUID 0xf452140300476080 port 17: [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 18: [PortXmitWait == 137950884]
GUID 0xf452140300476080 port 19: [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 20: [PortXmitDiscards == 2] [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 21: [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 22: [PortXmitDiscards == 1601] [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 23: [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 24: [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 25: [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 26: [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 27: [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 28: [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 29: [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 30: [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 31: [PortXmitWait == 4294967295]
GUID 0xf452140300476080 port 32: [PortXmitWait == 145892]
GUID 0xf452140300476080 port 33: [PortXmitWait == 5768275]
GUID 0xf452140300476080 port 34: [PortXmitWait == 15651998]
GUID 0xf452140300476080 port 35: [PortXmitWait == 301315601]
GUID 0xf452140300476080 port 36: [PortXmitWait == 2095531879]
Errors for "node88 HCA-1"
GUID 0xe8611f1046f5 port 1: [PortXmitWait == 36631]
Errors for "node87 HCA-1"
GUID 0xe8611f104755 port 1: [PortXmitWait == 178718]
Errors for "node85 HCA-1"
GUID 0xe8611f1046d5 port 1: [PortXmitWait == 692609]
Errors for "node86 HCA-1"
GUID 0xe8611f104645 port 1: [PortXmitWait == 205382]
Errors for "node83 HCA-1"
GUID 0xe8611f104695 port 1: [PortXmitWait == 1159110]
Errors for "node82 HCA-1"
GUID 0xe8611f104715 port 1: [PortXmitWait == 531365]
Errors for "node80 HCA-1"
GUID 0xe8611f104765 port 1: [PortXmitWait == 44858]
Errors for "node81 HCA-1"
GUID 0xe8611f1049d5 port 1: [PortXmitWait == 337661]
Errors for "node79 HCA-1"
GUID 0xe8611f104915 port 1: [PortXmitWait == 26707]
Errors for "node78 HCA-1"
GUID 0xe8611f104965 port 1: [PortXmitWait == 79891]
Errors for "node77 HCA-1"
GUID 0xe8611f1046a5 port 1: [PortXmitWait == 1177117]
Errors for "node76 HCA-1"
GUID 0xe8611f104a45 port 1: [PortXmitWait == 2876827]
Errors for "node74 HCA-1"
GUID 0xe8611f104605 port 1: [PortXmitWait == 39148]
Errors for "node73 HCA-1"
GUID 0xe8611f1049c5 port 1: [PortXmitWait == 2293292]
Errors for 0xf4521403008b7720 "MF0;switch-993d34:SX6506/L05/U1"
GUID 0xf4521403008b7720 port ALL: [LinkDownedCounter == 8] [PortRcvSwitchRelayErrors == 176] [PortXmitDiscards == 784] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7720 port 1: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 102] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7720 port 2: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7720 port 3: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7720 port 4: [LinkDownedCounter == 6] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7720 port 5: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7720 port 6: [PortRcvSwitchRelayErrors == 3] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7720 port 7: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7720 port 8: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 39] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7720 port 9: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7720 port 10: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 46] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7720 port 11: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7720 port 12: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 383] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7720 port 13: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7720 port 14: [PortRcvSwitchRelayErrors == 157] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7720 port 15: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7720 port 16: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7720 port 17: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7720 port 18: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7720 port 19: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7720 port 20: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7720 port 21: [PortXmitWait == 372817446]
GUID 0xf4521403008b7720 port 22: [PortXmitWait == 5065172]
GUID 0xf4521403008b7720 port 23: [PortXmitWait == 42903661]
GUID 0xf4521403008b7720 port 24: [PortXmitWait == 2743754029]
GUID 0xf4521403008b7720 port 25: [PortXmitWait == 978145227]
GUID 0xf4521403008b7720 port 26: [PortXmitWait == 1575017077]
GUID 0xf4521403008b7720 port 27: [PortXmitDiscards == 214] [PortXmitWait == 4294967295]
GUID 0xf4521403008b7720 port 28: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7720 port 29: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7720 port 34: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7720 port 35: [PortXmitWait == 4294967295]
GUID 0xf4521403008b7720 port 36: [PortXmitWait == 4294967295]
Errors for "node84 HCA-1"
GUID 0xe8611f104685 port 1: [PortXmitWait == 8651819]
## Summary: 175 nodes checked, 167 bad nodes found
## 597 ports checked, 552 ports have errors beyond threshold
## Thresholds:
## Suppressed:
Can you help me? Thank you very much.