UDP packet receive errors on interconnect rac cluster

[Scunningham99]

Hi

We are receiving packet receive errors on interconnect.

We have a 3 node Oracle Rac Cluster running on Red Hat 5.

Linux dc1-atl-pdba01 2.6.18-92.1.22.el5 #1 SMP Fri Dec 5 09:28:22 EST 2008 x86_64 x86_64 x86_64 GNU/Linux

Some of the statistics in oracle were stating that the globa cluster interconnect was not performing optimally.

Whilst investigating we noticed udp dropped packets etc.

Is this normal to have a little packet loss or should this be 0. As we have another rac system running Financials release 12 which has 0 packet loss.... the only difference is that this system is no where near the size or as busy!


Udp:
936188957 packets received
277 packets to unknown port received.
260040 packet receive errors
654115869 packets sent


[root@dc1-atl-pdba01 ~]# ifconfig -a
bond0 Link encap:Ethernet HWaddr 00:22:64:0F:40:74
inet addr:10.127.103.30 Bcast:10.127.103.255 Mask:255.255.255.0
inet6 addr: fe80::222:64ff:fe0f:4074/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:464993773 errors:0 dropped:0 overruns:0 frame:0
TX packets:1395828922 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:56156260552 (52.2 GiB) TX bytes:1829497124152 (1.6 TiB)

bond0:1 Link encap:Ethernet HWaddr 00:22:64:0F:40:74
inet addr:10.127.103.40 Bcast:10.127.103.255 Mask:255.255.255.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1

eth0 Link encap:Ethernet HWaddr 00:22:64:0F:40:70
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Interrupt:169 Memory:f8000000-f8012100

eth1 Link encap:Ethernet HWaddr 00:22:64:0F:40:74
inet6 addr: fe80::222:64ff:fe0f:4074/64 Scope:Link
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:429010357 errors:0 dropped:0 overruns:0 frame:0
TX packets:967241405 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:49155376047 (45.7 GiB) TX bytes:1312810825637 (1.1 TiB)
Interrupt:177 Memory:fa000000-fa012100

eth2 Link encap:Ethernet HWaddr 00:22:64:0E:AF:18
inet6 addr: fe80::222:64ff:fe0e:af18/64 Scope:Link
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:35983416 errors:0 dropped:0 overruns:0 frame:0
TX packets:428587517 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:7000884505 (6.5 GiB) TX bytes:516686298515 (481.2 GiB)
Interrupt:177

eth3 Link encap:Ethernet HWaddr 00:22:64:0E:AF:19
inet addr:192.168.100.30 Bcast:192.168.100.255 Mask:255.255.255.0
inet6 addr: fe80::222:64ff:fe0e:af19/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1253526626 errors:0 dropped:1455708 overruns:0 frame:49924
TX packets:877589490 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1018462188978 (948.5 GiB) TX bytes:600545931400 (559.3 GiB)
Interrupt:98

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:6340746 errors:0 dropped:0 overruns:0 frame:0
TX packets:6340746 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:9069891539 (8.4 GiB) TX bytes:9069891539 (8.4 GiB)

sit0 Link encap:IPv6-in-IPv4
NOARP MTU:1480 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)



Sy UK

 

[Annihilannic]

What speed and duplex are the NICs in question running at (check with ethtool)? Have you checked whether the switch is accumulating errors too, and whether the speed and duplex settings match the NICs? Are all three nodes accumulating errors?

Annihilannic.

 

[Scunningham99]

HI

Thanks for reply I have asked system engineer above questions and will get back to you.

looking at netstat -su and ifconfig -a

what is the difference between "dropped" in ifconfig -a and
"packet receive errors" in netstat -su as netstat shows 0 for this and ifconfig shows dropped packets?

Do We have a problem of not?


Link encap:Ethernet HWaddr 00:22:64:0E:CC:41
inet addr:192.168.100.31 Bcast:192.168.100.255 Mask:255.255.255.0
inet6 addr: fe80::222:64ff:fe0e:cc41/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:181121182 errors:0 dropped:730915 overruns:0 frame:20466
TX packets:205945934 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:145542170394 (135.5 GiB) TX bytes:170942495960 (159.2 GiB)
Interrupt:98



netstat -su

Udp:
130816334 packets received
6383 packets to unknown port received.
0 packet receive errors
124596978 packets sent

thanks in advance




Sy UK

 

[Annihilannic]

I think that just means the packets were rejected at a different layer (i.e. TCP not Ethernet)... but I'm not sure.

Annihilannic.

 

[Annihilannic]

It just occurred to me, I think packets can be dropped by an iptables rule; not sure whether they would appear in the ifconfig dropped output in that scenario, but might be worth checking your iptables stats to see if they match.

Annihilannic.