Nortel 8610 ( 8600 ) Port Connectivity Issue

[leedsit]

Have a weird one. We have 8x8600 switches in our company, however on one particular 8600 we seem to have connectivity issue with Random/Blocks of Ports. If we try and connect a new device ( laptop , server etc ) the port seems to be up by looking at the connectivity lights on the switch, however there is NO connectivity. Device manager see`s the port as down even though its enabled etc. We have tried combinations of Auto NEG, 100 Full, 10 Half 10 FUll, 100 Half, however we dont get connectivity.

Anyone else experienced this before?

Thanks, Lee.


LEEroy
MCNE6,CCNP,CWNA,CCSA,Project+

 

[Kwazzy]

When you are looking at the port level, what does the spanning tree tab state?

Even though you have a single link to the switch, it could be a rare chance that STP is blocking on the port. Just a thought.

 

[anthonyanderberg]

Could be a hardware problem that is affecting FPGA/ASIC chips but not the physical layer hardware. What are the affected blades and port numbers?

 

[leedsit]

Hi,

Thanks Kwazzy, STP is working fine, the port is not being shutdown etc because of STP as checked this, from what I can see it does not even get a chance to pass BPDU`s as the port does not come up and stats are all at Zero...

Antonyanderberg, it seems random. For example existing ports are working fine, however if we unplugg the server, there is a good chance it will not come back up. Card 4, some ports will work, some wont.... some are groupped other are not!!!! same for all the other cards!!!

We are planning a reboot, and this issue has been around for 4-5 months, Im having real issues getting this though change.......

LEEroy
MCNE6,CCNP,CWNA,CCSA,Project+

 

[Kwazzy]

I noticed you mentioned slot 4, if that is the only blade affected, and if you cannot risk the reboot of the switch, how about a reset of the card?

Another note, has nortel been contacted? If so, they maybe able to determine the if you are having a hardware issue thru shell commands. (I am sure you contacted them but depending on your contract, I do not know how far up you were able to get with their tech support. And what have they suggested thus far?)

 

[leedsit]

Hi Kwazzy,

Our support contract is with *Cough* BT and they are scratching their heads.... their current plan is for a chassis swap if the cold reboot does not fix it.

Its not just slot 4. We have replaced cards ( BT ). Slot 9/10 is empty, if we move a card or install a new card in those slots.... same issue!!!!!

This issue is sooo strange!

Thanks, Lee.

LEEroy
MCNE6,CCNP,CWNA,CCSA,Project+

 

[anthonyanderberg]

What microcode/hardware is involved?

 

[leedsit]

Hi,

I believe its running 3.5 ( will doubly confirm tomorrow ) along with another 8600 that has an IST MLT into in ( both running same code )

Lee.

LEEroy
MCNE6,CCNP,CWNA,CCSA,Project+

 

[Kwazzy]

Sorry Lee, I am at a loss on this.

I was warned on several codes in the 3.5 rev. but I do not recall which ones. I do know that the 3.5.3.0 code is a solid and stable revision. We have ran that for a while on the 8691 SF blades.

I have been around these boxes for a long time. The company I work for was one of the first ten companies to roll out the 8600s. And I have been involved with them since the ol' Accellar days, and this truely is a first from my experiences. We currently have over 60+ 8000 chassis's deployed and never seen this on any platform. Please keep this thread updated if you have any resolution to this problem. I am highly interested.

-Kwazzy

 

[leedsit]

Hi,

No Worries, thanks for the effort. Well as mentioned our plan is a Hard reset and then a chassis change if that does not work. I`ll let you know how it goes.

Thanks, Lee.

P.S Code is 3.5.1.0.


LEEroy
MCNE6,CCNP,CWNA,CCSA,Project+

 

[anthonyanderberg]

Does the affected server show up in any tables (ARP, layer 2 forwarding database, ect)? That would provide a valuable clue.

I ran various 3.5 releases for many years with only minor issues, and your description doesn't 'feel' like a software problem anyway... really weird problems always make me think hardware. By your description I'd guess your running 8648TXE modules, which have always worked really well for us.

The only thing I've seen that's similar to your problem was an Octapid ASIC failure on an 8616 module that caused two ports to act flaky - but the problem always followed those ports. There isn't much to the 8000 backplanes, just traces and an EPROM that sets the base MAC address... but I suppose there could be something going on.

Good luck man.

 

[MaCinTof]

have you checked the multicast/broadcast limitation on your port ?
You should have this kind of entry in your log

"WARNING Task=tMainTask Shutdown port xxxx due to excessive control frames multicast 166, broadcast 10770 packet per second

 

[leedsit]

Hi MaCinTof,

Yup, however the ports are not coming up period! As soon as you plug the physical cable in, Nothing, Nada no matter what physical speed that is set, so the ports do not even get the chance to receive excessive frames.

Thanks, Lee.


LEEroy
MCNE6,CCNP,CWNA,CCSA,Project+

 

[MaCinTof]

May be your port is administratively down, you can't make it work with device manager. You have to go on CLI and type the following command:

config eth <your port (ex: 2/6)> stat enable

 

[leedsit]

Hi MacinTof,

Nope all the ports are enabled. Physically active, with the correct speed, and all different combinations of speeds, with Spanning tree, no spanning tree, port fast, no port fast, they just will not come up at all!

Lee.


LEEroy
MCNE6,CCNP,CWNA,CCSA,Project+

 

[leedsit]

Hi,

Chassis swap with 200+ Servers this weekend.... wish me luck and fingers x it fixes this $%£$£" issue......

Lee

LEEroy
MCNE6,CCNP,CWNA,CCSA,Project+

 

[leedsit]

Chassis swap seems to have fixed the issue, wether its a result of the hard reset or the change of the chassis swap, I dont know.... but all looks good so far!!! strange one!

Lee

LEEroy
MCNE6,CCNP,CWNA,CCSA,Project+

 

[HanKa]

Hi

I have experience with the same problem, in two diffrent system.
I use code 3.5.1 and it has worked fine in more then 3 years now.

I have got this message before and usually someone have put a unmanaged switch in the network and connect 2 ports in the switch together witch give a brodacast storm, and as a resulat of that the uplinks port get blocked on the 8600 (blocked to protect the CPU from the broadcast). After having removed the cables from the unmanaged switch, and use the cli command

/config ethernet 2/1 state disable
and
/config ethernet 2/1 state enable

I am up and running again. ( I am using SMLT)

If you get a broadcast storm on the port it will not operate before using the command dis and ena or if you restart the 8600 (boot).


But for maybe 2 month ago I have got excessive control frames in two diffrent system at several isolated events.

It has now happend more then 4 times and i can´t find any loop so fare, and after disable and enable the port i the 8600 everything works fine.

There are diffrent VLAN connection between the system on a Nortel 5510.

I don´t think this is any problem.

I am now trying to put a sniffer on one of the segment so i can be able to see the mac address from the
broadcast storm (excessive control frames).

One theory i have is that some networkcards is sending out this broadcast storms, maybe we have the same problem with some new software problem on our network, malfunction network card(intermittent)or some new equiment, I'm aware of two new 5510 24 ports.

I also have some 5510 48 ports and they have been there a long time before this trouble start.
But i think i have differt firmware on the switches. I don´t think the switches are the problem but who knows.

As you can see of the example it is broadcast and sometimes multicast also.
On the port 2/1 i have two Nortel 5510 (24ports) and on port 1/3 two Nortel 5510 (48 ports)
I have also Nortel 5510 (48 ports) on other ports on the 8600.(witch never have give me problem)


This is from two differt 8600 and on diffrent network.

CPU3 [12/31/06 12:17:54] SNMP INFO Smlt Link Down Trap(SmltId=5)
CPU3 [12/31/06 12:17:54] SNMP INFO Smlt Link Down Trap(SmltId=4)
CPU3 [12/31/06 12:17:54] SNMP INFO Smlt Link Down Trap(SmltId=3)
CPU3 [12/31/06 12:17:54] SNMP INFO Smlt Link Down Trap(SmltId=2)
CPU3 [12/31/06 12:17:54] SNMP INFO Smlt Ist Link Down trap received
CPU3 [12/31/06 12:17:54] SNMP INFO Link Down 2/8.Port is a
trunk port
CPU3 [12/31/06 12:17:54] SNMP INFO Link Down(2/8) due to excessive control frames
CPU3 [12/31/06 12:17:54] SNMP INFO Link Down 2/7.Port is a trunk port
CPU3 [12/31/06 12:17:54] SNMP INFO Link Down(2/7)
CPU3 [12/31/06 12:17:54] SNMP INFO Smlt Link Down Trap(SmltId=2)
CPU3 [12/31/06 12:17:54] SNMP INFO Link Down 2/1.Port is a
trunk port
CPU3 [12/31/06 12:17:54] SNMP INFO Link Down(2/1) due to excessive control frames
CPU3 [12/31/06 12:17:54] CPU WARNING Shutdown port 2/8 due to excessive control frames multicast 9945, broadcast 14719 packet per second
CPU3 [12/31/06 12:17:54] MLT INFO All the SMLTs are down
CPU3 [12/31/06 12:17:54] MLT INFO smltIstSessionDown
CPU3 [12/31/06 12:17:53] CPU WARNING Shutdown port 2/1 due to excessive control frames multicast 12810, broadcast 17478 packet per second



CPU5 [12/30/06 05:52:37] SNMP INFO Smlt Link Down Trap(SmltId=2)
CPU5 [12/30/06 05:52:37] CPU WARNING Shutdown port 1/3 due to excessive control frames multicast 0, broadcast 33604 packet per second
CPU5 [12/30/06 05:52:37] MLT INFO SMLT 2 DOWN
CPU5 [12/30/06 05:52:37] SNMP INFO Port 1/3 is a trunk port
CPU5 [12/30/06 05:52:37] SNMP INFO Link Down(1/3) due to excessive control frames