ip multipathing- all cards fail when link is down

[gameover]

Our 4800 is running solaris 8 with 2 nics. We had some consultants build the server for us so I am ignorant is some areas of the server like multipathing. One nic is production and one is a failover. In our environment, our switch goes down quite often. When our switch goes down, both network cards fail. Is this normal? The only way we know how to get the cards back is to reboot. Reboot take about 20 minutes and we loose quite a bit of production time.

If this is normal behavior, is there any way to get the cards back without rebooting?

Thanks for the help,
Mitch

 

[jmiturbe]

Hello,

Can you post the output of ifconfig -a? Does the second NIC have the FAILOVER flag?

I'm not sure because I only know the theory but I think that this flags means that the address does not fail when the interface fails, so if using an active-passive configuration with the failover flag on the second interface, it should work as you expect.

Anyway, you should find the way to take the interfaces up again without rebooting: plumbing and getting them up again, restarting the proper daemons, etc...

Bye,

jmiturbe

 

[gameover]

Thanks again for the help.


# ifconfig -a
lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
ge0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 192.168.5.206 netmask ffff0000 broadcast 192.168.255.255
groupname db
ether 8:0:20:fd:1a:12
ge0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>
mtu 1500 index 2
inet 192.168.5.208 netmask ffff0000 broadcast 192.168.255.255
ge1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 192.168.5.207 netmask ffff0000 broadcast 192.168.255.255
groupname db
ether 8:0:20:fd:24:6f
ge1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>
mtu 1500 index 3
inet 192.168.5.209 netmask ffff0000 broadcast 192.168.255.255
hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
inet 192.168.10.100 netmask ffffff00 broadcast 192.168.10.255
ether 8:0:20:bf:af:bf
#

 

[jmiturbe]

Hello,

Seems like you have an active-active configuration (both are working at the same time), because you have virtual interfaces with the flags deprecated and nofailover on both cards.

Is normal that both interfaces fail and jump to the other card as a virtual interface. Anyway, when the switch is up again both virtual interfaces (ge0:1 and ge1:1) should check that the link is up again and switch the configurations to the initial state.

If you put an active-passive configuration, the second card configuration wouldn't change when the switch is down (but logically it wouldn't work because the switch doesn't), only the active card should &quot;jump&quot; to a virtual interface on the second card.

However, in both cases when the switch is back, the configurations should switch back to the initial state. Is the initial configuration restored when the switch works again?

Bye,

jmiturbe

 

[TobyR]

The question I would ask right off the bat is whether or not the switch is a single point of failure. If so, then the cards will fail to do anything useful regardless of whether they have failed in terms of IPMP. Does each cable (from the cards in question) lead to the dodgy switch? If so, then IPMP is a problem rather than a help.

Also, if one card is intended as purely as a backup, why are there 4 addresses (and in the same net)? At the moment you have one address per card that will try to switch to the other in the group, and one that will not. To my eyes that looks strange. Typically with IPMP you have three addresses shared over two cards, two of which do not failover, and one that does. The IP that does failover should be the active address, and, for the sake of intuitive simplicity, be a &quot;virtual&quot; address, ie a :1. If the card on which that :1 address sits fails, then the address leaps over to the other card in the group, and everything is good.

If the problem is still there, perhaps you could explain exactly how IPMP is supposed to be heling you in this situtation, which address is the &quot;active&quot; one (that should switch), and perhaps also whether or not load balancing is desired.

Cheers

Toby

 

[jmiturbe]

Hello,

TobyR, I think that both cards should be working again with the initial configurations when the switch is back, even if they are connected to the same buggy device. Do you agree?

Anyway with an active-passive configuration, the failover card wouldn't switch it's configuration to the active card, so maybe this kind of IPMP could work better when the switch crashes. But I'm not sure.

You are right when you say that it is strange to have four interfaces, but only because gameover told that one card is active and the other failover. It's clear that this is an active-active configuration and not active-passive configuration.

But it's not a strange thing to have active-active and not because of load balancing, because IPMP doen't give load-balancing (maybe a bit with the output traffic). The most usefull situation that I can think of to use IPMP with two active cards, is to have a server that responds to very different services through different IPs. It's not a traffic load balancing, just a distribution of inbound traffic. For a full load balancing Sun Trunking should be used.

jmiturbe

 

[TobyR]

Hi jmiturbe,

the point about having both interfaces go through the same switch is that, in the case where the switch fails, there is no longer any path to the client/customer, so ipmp is moot; the configuration of it becomes irrelevant. In the case where a network card fails, then it's failover-able IP will switch over to the other card in the group. In the setup listed in the first post, either NIC can fail and have its failover address failover. The question is, why should this be necessary? The only reason I can think of (I agree with you) is that the IPs are serving different services, with perhaps different DNS names etc. If that is the case then the setup is OK, but nothing anyone does to ipmp is going to make any difference in the case where the sole switch fails. In that case the infrastructure is the problem, not the software.

As to load balancing, yes, ipmp only manages that outbound, and is perhaps not relevant to this situation. At best it perhaps is a nice yet not mind-blowingly useful advantage.

Toby

 

[TobyR]

Hello again,

just thought it might be helpful to explain (in case it's at all necessary) how ipmp decides a card has failed. It pings (typically) a router on its subnet and waits ten seconds (this value is adjustable) for a reply. If it fails to get a reply five times in a row, it considers the card failed. So if the router in the above setup is on the other side of the switch, that switch is causing ipmp big problems when it falls over.

Cheers

Toby

 

[jmiturbe]

Hello,

I agree with all you posted, but the question is: What happens once the switch is working again?

I think that the test interface (in this case one per NIC) will detect with pings that the link is working again, and this way return the interfaces to their initial configuration.

So gameover's system should be working normally when the switch works, and it shouldn't hung up for ever. He didn't told if the switch is working again when they try to get the cards back without success, but maybe he thinks that it's not working properly because he tries to put the interfaces working while the switch is down.

IPMP is designed against router failures too. I don't exactly know the process, but it pings the router. If it doesn't get any answer, it tries with local machines (maybe looking the arp tables) and finally it tries with a broadcast to check if there's any hosts there. If no one answers, it determines that the link is down (the card is broken, the cable is disconnected or broken,....) and it switches it's configuration to a virtual interface on the other card.

Gameover, I supose that your environment is quite critical because you talk about a SF4800 with HA on the network, so repairing the switch or getting a new one would be a good idea, maybe another switch in HA with a card connected on each switch device. This way you would have a &quot;complete&quot; network HA.

Bye,

jmiturbe

 

[gameover]

Thanks again for all the responses. While reading my initial post, I realized I was very unclear. Sorry.

We have 2 NICs. Each with 2 IPs. 192.168.5.206 192.168.5.207 are floating IPs. That is, in event of a failed card, they float over to the no-failed card. Jmiturbe's assumption was correct that we distribute network services across the 2 NICs.

OK, our network department is suspect. They have crap go down all the time.

We have 600 PCs/servers on our network that do not need a reboot when a switch goes down. Then we have the 1/3 million dollar server that needs a reboot when a switch goes down. I am sure it is a configuration problem so that is why I am asking the experts.

The following is mpathd logs of a failover that did not required a reboot to get the NICs up again:

Apr 1 12:49:03 sunfire1 in.mpathd[33]: [ID 594170 daemon.error] NIC failure det
ected on ge0 of group db
Apr 1 12:49:03 sunfire1 in.mpathd[33]: [ID 832587 daemon.error] Successfully fa
iled over from NIC ge0 to NIC ge1
Apr 1 12:58:43 sunfire1 in.mpathd[33]: [ID 237757 daemon.error] At least 1 inte
rface (ge1) of group db has repaired
Apr 1 12:58:43 sunfire1 in.mpathd[33]: [ID 299542 daemon.error] NIC repair dete
cted on ge1 of group db
Apr 1 12:58:43 sunfire1 in.mpathd[33]: [ID 620804 daemon.error] Successfully fa
iled back to NIC ge1
Apr 1 12:58:47 sunfire1 in.mpathd[33]: [ID 299542 daemon.error] NIC repair dete
cted on ge0 of group db
Apr 1 12:58:47 sunfire1 in.mpathd[33]: [ID 620804 daemon.error] Successfully fa
iled back to NIC ge0

The following is mpathd logs of a failover that required a reboot to get the NICs up again:

Apr 3 16:57:01 sunfire1 in.mpathd[33]: [ID 168056 daemon.error] All Interfaces
in group db have failed
Apr 3 18:29:39 sunfire1 in.mpathd[32]: [ID 255185 daemon.error] Failures cannot
be detected on ge1 as no IFF_NOFAILOVER address is available
Apr 3 18:30:19 sunfire1 in.mpathd[32]: [ID 490503 daemon.error] Failure detecti
on restored on ge1 as an IFF_NOFAILOVER address is available

Anyone know how mpathd starts? Can't find it in startup scripts.


Thanks,
Gameover

 

[jmiturbe]

Hello,

The first group of logs (when it works) indicates that ge0's and ge1's configurations are switched correctly to virtual interfaces on the other NIC, probably ge0 to ge1:2 and ge1 to ge0:2. This tells that the configuration is working like expected on an active-active configuration.

I think that you should have a log concerning to the failover from ge1 to ge0 at the same time, right?

I don't know what is forcing the system to work wrong sometimes. The output of ifconfig -a when multipathing doesn't failback could help in the resolution of the problem.

Anyway, like the manual indicates:

&quot;The starting of in.mpathd daemon is controled by the TRACK_INTERFACES_ONLY_WITH_GROUPS parameter in the /etc/default/mpathd file.

If the TRACK_INTERFACES_ONLY_WITH_GROUPS variable is set to yes, the ifconfig utility's group opton starts the in.mpath process automatically. That is, as soon as you use the ifconfig utility with the group option in the command, the in.mpathd process starts. If the TRACK_INTERFACES_ONLY_WITH_GROUPS variable is set to no, the /etc/rcS.d/S30network.sh run control script starts the in.mpathd process at boot time.

.... (a section of this script is printed)


If necessary, start the in.mpathd process from the command line by performing the command as root:

sys11# /sbin/in.mpathd
sys11# &quot;

I think that this indicates that if TRACK_INTERFACES_ONLY_WITH_GROUPS is set to yes, ipmp is started when configuring the interfaces with the group option, so maybe reconfiguring the interfaces at the command line with group option could cause your system to work properly without rebooting. Try to test this (for example turning off the switch) when the system has low load.

Doy you need the steps to configure IPMP from the command line with ifconfig command?

Anyway you should discover the source of the problem. If nobody helps you on the forums, try to ask Sun's people, because with a system like yours, you sure have a warranty upgrade with Solaris support. And if not, try to ask anyway because you have a &quot;big&quot; system!!!!

Bye,

jmiturbe