AIX5.,2 gigbit problem

[umjw01]

Hi,

The problem - Just started looking into it and wonder if anybody else come across this -

When I reboot the system or configure the network gigabit adapters it produces an error in the OErrs and takes about 30-40 seconds to start sending pings down it. This is causing me major issues with HACMP as when I start the cluster the system swaps on the service address but becuase of the time its taking to initiate a ping it thinks the adapter fails so swps it to its standby and so on.

Details below -

Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
en0 1500 link#2 0.2.55.33.5c.e0 10478 0 54 1 0
en0 1500 10.200.19 10.200.19.224 10478 0 54 1 0
en1 1500 link#3 0.2.55.af.f2.be 21082 0 5491 0 0
en1 1500 10.200.10 10.200.10.184 21082 0 5491 0 0
en2 1500 link#4 0.2.55.6f.af.4d 239508 0 2071685 0 0
en2 1500 10.200.254 10.200.254.83 239508 0 2071685 0 0
en3 1500 link#5 0.2.55.33.6c.92 10701 0 450 1 0
en3 1500 10.200.200 10.200.200.223 10701 0 450 1 0
en4 1500 link#6 0.2.55.ef.1c.56 1166406 0 3838 0 0
en4 1500 10.200.18 10.200.18.185 1166406 0 3838 0 0
lo0 16896 link#1 156266 0 247099 0 0
lo0 16896 127 127.0.0.1 156266 0 247099 0 0
lo0 16896 ::1 156266 0 247099 0 0

Look at en0 and en3. Details of the adapters are below -

alt_addr 0x42acdcbedbed Alternate ethernet address True
busintr 37 Bus interrupt level False
busmem 0xe8080000 Bus memory address False
chksum_offload no Enable hardware transmit and receive checksum True
copy_bytes 2048 Copy packet if this many or less bytes True
flow_ctrl yes Enable Transmit and Receive Flow Control True
intr_priority 3 Interrupt priority False
intr_rate 10000 Interrupt events processed per interrupt True
jumbo_frames no Transmit jumbo frames True
large_send no Enable hardware TX TCP resegmentation True
media_speed Auto_Negotiation Media speed True
rom_mem 0xe8040000 ROM memory address False
rx_hog 1000 RX buffers processed per RX interrupt True
rxbuf_pool_sz 1024 RX descriptor queue size True
rxdesc_que_sz 1024 RX descriptor queue size True
slih_hog 10 Interrupt events processed per interrupt True
tx_que_sz 8192 Software transmit queue size True
txdesc_que_sz 512 TX descriptor queue size True
use_alt_addr no Enable alternate ethernet address True


alt_addr 0x42acdcbedbed Alternate ethernet address True
busintr 549 Bus interrupt level False
busmem 0xe8080000 Bus memory address False
chksum_offload no Enable hardware transmit and receive checksum True
copy_bytes 2048 Copy packet if this many or less bytes True
flow_ctrl yes Enable Transmit and Receive Flow Control True
intr_priority 3 Interrupt priority False
intr_rate 10000 Interrupt events processed per interrupt True
jumbo_frames no Transmit jumbo frames True
large_send no Enable hardware TX TCP resegmentation True
media_speed Auto_Negotiation Media speed True
rom_mem 0xe8040000 ROM memory address False
rx_hog 1000 RX buffers processed per RX interrupt True
rxbuf_pool_sz 1024 RX descriptor queue size True
rxdesc_que_sz 1024 RX descriptor queue size True
slih_hog 10 Interrupt events processed per interrupt True
tx_que_sz 8192 Software transmit queue size True
txdesc_que_sz 512 TX descriptor queue size True
use_alt_addr no Enable alternate ethernet address True

Network Routes -


Route tree for Protocol Family 2 (Internet):
default 10.200.10.1 UG 0 646 en1 - -
10.200.10.0 127.0.0.1 UHb 0 0 lo0 - - =>
10.200.10.0 10.200.10.184 UHb 0 0 en1 - - =>
10.200.10/24 10.200.10.184 U 3 164 en1 - -
10.200.10.184 127.0.0.1 UGHS 0 11 lo0 - -
10.200.10.255 127.0.0.1 UHb 0 550 lo0 - - =>
10.200.10.255 10.200.10.184 UHb 0 4 en1 - -
10.200.18.0 10.200.18.185 UHb 0 0 en4 - - =>
10.200.18/24 10.200.18.185 U 0 669 en4 - -
10.200.18.185 127.0.0.1 UGHS 0 3223 lo0 - -
10.200.18.255 10.200.18.185 UHb 0 1296 en4 - -
10.200.19.0 127.0.0.1 UHb 0 0 lo0 - - =>
10.200.19.0 10.200.19.224 UHb 0 0 en0 - - =>
10.200.19/24 10.200.19.224 U 0 0 en0 - -
10.200.19.224 127.0.0.1 UGHS 0 30 lo0 - -
10.200.19.255 127.0.0.1 UHb 0 423 lo0 - - =>
10.200.19.255 10.200.19.224 UHb 0 9 en0 - - =>
10.200.19.255/32 10.200.200.224 U 0 0 en0 - - =>
10.200.19.255/32 10.200.200.224 U 0 0 en3 - -
10.200.200.0 127.0.0.1 UHb 0 0 lo0 - - =>
10.200.200.0 10.200.200.223 UHb 0 0 en3 - - =>
10.200.200/24 10.200.200.223 U 0 411 en3 - -
10.200.200.223 127.0.0.1 UGHS 0 0 lo0 - -
10.200.200.255 127.0.0.1 UHb 0 621 lo0 - - =>
10.200.200.255 10.200.200.223 UHb 0 6 en3 - -
10.200.254.0 10.200.254.83 UHb 0 0 en2 - - =>
10.200.254/24 10.200.254.83 U 2 54425 en2 - -
10.200.254.83 127.0.0.1 UGHS 0 1545 lo0 - -
10.200.254.255 10.200.254.83 UHb 0 2 en2 - -
10.201.10.1/32 10.200.10.1 UG 0 0 en1 - -

127/8 127.0.0.1 U 13 133019 lo0 - -

Route tree for Protocol Family 24 (Internet v6):
::1 ::1 UH 0 0 lo0 16896 -


Running on AIX5.2 ML2 - Filesets all okay. Errors produced are -

LABEL: GOENT_RCVRY_EXIT
IDENTIFIER: 4507DE58

Date/Time: Fri Mar 19 15:49:13 2004
Sequence Number: 504
Machine Id: 003764BF4C00
Node Id: sapaix18
Class: H
Type: INFO
Resource Name: ent3
Resource Class: adapter
Resource Type: 14106802
Location: U1.5-P2-I2/E1
VPD:
Product Specific.( ).......Gigabit Ethernet-SX PCI-X Adapter
Part Number.................00P3055
FRU Number..................00P3055
EC Level....................H11634A
Manufacture ID..............YL1021
Network Address.............000255336C9E
ROM Level (alterable).......GOL001

Description
ETHERNET NETWORK RECOVERY MODE

Probable Causes
CSMA/CD ADAPTER

Failure Causes
CSMA/CD ADAPTER

Recommended Actions
NONE
Detail Data
FILE NAME
line: 223 file: goent_intr.c
PCI ETHERNET STATISTICS
0000 0000 0023 081B 0000 0002 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0004 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000
DEVICE DRIVER INTERNAL STATE
2222 2222 0000 0000 0000 0000
SOURCE ADDRESS
0002 5533 6C9E
---------------------------------------------------------------------------
LABEL: GOENT_LINK_DOWN
IDENTIFIER: DED8E752

Date/Time: Fri Mar 19 15:49:13 2004
Sequence Number: 503
Machine Id: 003764BF4C00
Node Id: localhost
Class: H
Type: TEMP
Resource Name: ent3
Resource Class: adapter
Resource Type: 14106802
Location: U1.5-P2-I2/E1
VPD:
Product Specific.( ).......Gigabit Ethernet-SX PCI-X Adapter
Part Number.................00P3055
FRU Number..................00P3055
EC Level....................H11634A
Manufacture ID..............YL1021
Manufacture ID..............YL1021
Network Address.............000255336C9E
ROM Level (alterable).......GOL001

Description
ETHERNET DOWN

Probable Causes
CABLE
CSMA/CD ADAPTER

Failure Causes
LINK TIMEOUT

Recommended Actions
CHECK CABLE AND ITS CONNECTIONS

Detail Data
FILE NAME
line: 180 file: goent_intr.c
PCI ETHERNET STATISTICS
0000 0000 0023 0853 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000
DEVICE DRIVER INTERNAL STATE
5555 5555 0000 0000 0000 0000
SOURCE ADDRESS
0002 5533 6C9E
---------------------------------------------------------------------------

Same for both adapters......

I have mucked about with jumbo frames, MTU sizes etc, but to no avail. On our 4.3.3 ML11 systems I dont have these issues.They ping straight away. Also the loopback address appears to be routing with any address that configures on the system...

Any info would be good......cheers

 

[ademeter]

Hi,
check the ether switch port for STP (Spanning Tree Protocol). If the full STP is enabled, it may take about 40 sec for the ether switch port to become online.

 

[ogniemi]

Maybe your problem is related to this (got from IBM):


Required update for Gigabit Ethernet Adapter on AIX 5.1 and 5.2

USERS AFFECTED:
Systems using one of the following Gigabit Ethernet PCI-X
adapters with the large_send option enabled, which is the
default setting, and with the devices.pci.14106902.rte
fileset below the level of 5.1.0.57 for AIX 5.1 or 5.2.0.18
for AIX 5.2.

Feature Adapter
------- -------
5700 IBM Gigabit Ethernet-SX PCI-X Adapter
5701 IBM 10/100/1000 Base-TX Ethernet PCI-X Adapter
5706 IBM 2-Port 10/100/1000 Base-TX Ethernet PCI-X
Adapter
5707 IBM 2-Port Gigabit Ethernet-SX PCI-X Adapter

PROBLEM DESCRIPTION:
TX_ERR errors may be logged in the system error log
associated with the Gigabit Ethernet PCI-X adapters.
Corruption of transmitted packets is also possible in
conjunction with the errors. This problem has only been
observed running at 10Mbit and 100Mbit speeds, although a
reduced possibility of failure does exist when running at
the 1000Mbit speed.

RECOMMENDATION:
Disable the large_send option until the fix is applied. To
disable this option:

ifconfig en# detach
chdev -l ent# -a large_send=no

Alternatively, you may add the -P option to the chdev
command as follows to change the option for the next reboot.

chdev -P -l ent# -a large_send=no

Install one of the following APARs depending upon the level
of your system:

AIX 5.1: IY54323
AIX 5.2: IY54068

APARs are available from Fix Central at: