PE NIC Teaming - How should the network see them? 1

[PoliMalaka]

NIC Teaming - How should the network see them?

We are in the process of teaming the NICs on all of our PowerEdge 17xx, 26xx, 28xx and 29xx series servers and have noticed an interesting situation when scanning the network.

All servers are running Server 2003.
All NIC are imbedded.
All virtual NICs or teams are statically configured and operational (i.e. we are using them without any problems).

Server 1: METAB1
PE1750
2 Broadcom NetXtreme Gigabit NICs teamed with BACS

Server 2: FS7
PE 2850
2 Intel Pro/1000 MT NICs teamed with Intel PROset

Server 3: RS3
PE 2650
2 Broadcom NetXtreme Gigabit NICs teamed with BACS

Server 4: RS4
PE 2850
2 Intel Pro/1000 MT NICs teamed with Intel PROset


We scan our network with the Fluke Etherscope Series II. From these scans using SNMP we can see into the HP switches to the port and node level.

Servers 2, 3, and 4 show up twice on the scan (1 IP per connected NIC) and are reported as problems (duplicate IP addresses). Traffic levels are elevated on these teams.

Server 1 shows up only once. Both NICs are connected to the network and the switch can see each port as operational (traffic patterns are normal).

Not sure if there is a problem or if this is a normal situation.

Anyone out there see this before?
TIA

 

[ADB100]

Unless you are purely doing fault tolerant teams (active/standby) then configuration is required on the network switches for teaming (load balancing) to work. From what you have posted server 1 is configured for fault tolerance and servers 2, 3 & 4 are load-balanced. If the network is purely layer-2 then it may work, however if there are any routers (or layer-3 switches) involved then there will be confusion and address flapping.

I see this quite a lot with people deploying servers without involving the network people and just assuming the default NIC teaming options will work. I suggest you speak to your networking people and explain what you have configured, they will probably give you some grief but you should be able to work it out....

Andy

 

[PoliMalaka]

Thanks for the tips Andy. I am the network team as well at our location!
We do have a consultant with whom I will discuss these issues as well.

Should the network see 2 NICs or only 1 when load balancing?
What configuration should be used on the switches (HP ProCurve 26xx, 28xx, and 41xx?

 

[ADB100]

When load-balancing generally you will see two, however there are different algorithms - Cisco EtherChannel, 802.3ad Link Aggregation as well as the adaptive algorithms that are purely for Layer-2 environments (clients on the same broadcast domain as the server). With EtherChannel & 802.3ad you set up a 'Channel' between the server & the switch and packets are load-balanced based on an algorithm, the load-balancing happens at each end (i.e. the switch & the server).
With adaptive load-balancing the behaviour is either receive always on the same NIC but transmit on multiple or the server selectively replies with a different MAC address to ARP requests. The idea of this is that the clients will talk to the server on the same IP address but will use different Layer-2 (MAC) addresses - i.e. PC 1 will see IP address X with MAC address A, whilst PC 2 will see IP address X with MAC address B. When the clients are on the same broadcast domain as the server this can work OK, however put a router in the mix and it gets weird as the router will have an ARP entry for the server but any broadcasts the server does will likely upset this and cause flapping.

My personal preference is fault-tolerance as it simplifies things...

HTH

Andy

 

[PoliMalaka]

Still researching and gathering information.
Both the Intel PROset and BACS have the ability to test the network switch's configuration. This configuration test passes on all 3 switches from all 4 servers.

I am reading the some switch configuration may be necessary.
However, I can't seem to find out what the might be!

As of this morning, Server 2 is now behaving like Server 1.
Only seen once on the network.

I wonder if the switches are able to adjust depending on the type.

The two servers that only show up once are on HP ProCurve 2810 and 4104.

The others are on a HP 2524.

 

[PoliMalaka]

ADB100

My personal preference is fault-tolerance as it simplifies things..

We are hoping to increase the bandwidth each server can use. From what I understand, the fault tolerance approach will not accomplish this. True or am I misinformed?

 

[PoliMalaka]

Well well well.

It seems I have configured the teams incorrectly.

Doing a LOT of research has lead me to the conclusion that the teams should be configured as
802.3ad Link Aggregation

Thank you for all your help!

 

[ADB100]

It seems I have configured the teams incorrectly.

Doing a LOT of research has lead me to the conclusion that the teams should be configured as
802.3ad Link Aggregation

You need to enable this on the switches also.... Depending on the switch types then it might be referred to as different things - Cisco refer to them as Channels (port-channels), 3Com use the term 'trunks' which is more confusing as Cisco use the term 'trunks' to refer to VLAN trunks. Read the documentation to understand what it is you want to configure.

Do your servers really generate enough traffic to warrant teaming for increasing bandwidth? The disk subsystems are usually the bottlenecks. I have yet to see any 'real' performance improvements with load-balanced NIC teams....

I am a big fan of keeping things as simple as they need to be (KISS).....

Andy

 

[PoliMalaka]

How busy are your servers?

Our main file servers gets hit big time all day with very large 30-40MB. I noticed quicker response time on the large files after making the change.

 

[PoliMalaka]

By the way, the HP ProCurve switches have 802.3ad LACP is enabled by default.
No configuration needed when the NICs are teamed using this protocol.

Thanks again for all your help!

By the way:

After a longer review of the servers, we have noticed a marked increase in response times specifically in directories that hold large numbers of pictures. These directories are viewed as thumbnails. The thumbs appear in only 2-3 seconds as opposed to taking 7-10 before the teams were enabled.

GO TEAM! /funny