Why do IPSIs go bad so often, and Y if duplicated does it cause outage

[LeFoneChique]

This is the third or fourth time an IPSI board problem has caused a major production outage for us in about a year. These boards are duplicated in the system, and yet we continue to experience outages.

Does Avaya or anyone have an explanation for this situation?

Why when my IPSI’s are duplicated, do problems seem to look past the duplicated board, and affect the entire system?

Examples:

1) New S8710 system installed for 2 months. Entire system (12 port networks/ 3K stations) would periodically BOUNCE. All phones down - all calls in progress lost. Took avaya MONTHS to figure out that the problem was multiple bad IPSIs. Replaced 10.

2) Last month. One bad IPSI took DOWN the entire division - (S8710 CM 3.1.) Took Avaya 8-9 hrs to determine it was the 1 bad IPSI. replaced the board.

3) Yesterday - took all circuits down in the port network with the bad ipsi. replaced board.

Yes, In all cases all boards are duplicated.

 

[rjblanch]

Very good post - how did you fight with Avaya to prove that it was Bad IPSI boards?

We are currently going through the same situation where 2 duplicated port networks have died on a couple of occassions and everytime Avaya have come back with "bad network" or "network drop out".

I am interested how you got about proving that it was IPSI issues? Please let us know how you actually proved it so that we can put the same to Avaya here!!!

Thanks.

 

[ewaaijenberg]

Hi,

if you have HW 11 it is a known issue. With other boards never had problems.

 

[Toni269]

I was baffled by your post. I have HW12 and HW14 and they are rock-solid.

 

[amarc5300]

HW12 here. Never had a problem.

 

[mikeydidit]

What is shown when you do a "list isp" ? All should be A active AA which is All-A control.

Also the errors for these are listed at the back of this command. They should be 0 0 0 0 , if they are all working correctly.

"Wise men speak because they have something to say; Fools because they have to say something."
(Plato)

 

[GMinTX]

Have HW2, HW13 and HW20, never a problem.

 

[RFWatts]

This is actually a very common problem...You a likely having IPSI sanity issues. IPSIs are EXTREMELY sensitive to network blips...even blips lasting less than one second! I have been working the same problem for months now. Avaya has told me that the problem is in my network, but I am still unconviced that this is not a product development issue. Here's another forum discussion on this topic:

http://www.avayausers.com/showthread.php?s=&threadid=2865&highlight=ipsi+reset

 

[JediBMC]

I would verify that all of your ISPI are HW12 or greater and have the latest FW. I have had no problems with my IPSI since they were installed 6 months ago. Of course, now you have me worried!

-Brian-
Semper Paratus

 

[RFWatts]

Is there any interest in possible group discussions on IPSI technical info, Control Network routing, and issues and possible fixes? There is much buzz about various IPSI sanity issues, and it might be good to get some live discussion of other folks' woes and ails. Would any one be interested in such an event? Post you thoughts to this post.

tnx,

RFWatts

 

[Toni269]

Yes, definately. I'd also like to learn more about how to administer ESS servers as well as the new features like IGAR and to understand better how the whole IPSI-port network works in comparison to the C-Lan-Gateway/LSP's.

 

[LeFoneChique]

Replying to RJ Blanch: We didn't have to prove it, they admited it. I don't know why they concluded it was the IPSIs, but when they replaced them the problem went away, so I guess they were correct.

 

[foned00d]

I would be very interested in a forum specific to IPSI's, ESS, and all Avaya IP related topics.

 

[langl3y]

We have just been fighting with Avaya for over a month with IPSI issues. We have 4 ESS pairs and 42 G700 gateways and IPSI problems were sending our Avaya network into ESS. We ended up proving the IPSI problems by logging in to the IPSI boards and checking "show internet stats". This would show TIME_WAIT when trying to connect to our main S8710 pair. We were advised yesterday - Nov 7 - that there is a new IPSI firmware 32 that addresses IPSI socket connection issues. We are waiting to see if it will resolve our issues.
Avaya have lost a lot of credibilty in my company, due to this issue.

 

[row1030]

We have a duplicate reliability system, 2 port networks, One port network (#2) has dropped out twice in less than a year. Each time the actual alarm has been a minute. The active media server is on the same (Avaya) network switch as the IPSI. We are replacing the IPSI (HW12)this weekend with a new HW12. Are we wasting our time? Yes we are talking to them about adding redundant IPSIs but, even that seems to not be functioning as designed?

P.S. I'm not seeing firmware 32 on the download site yet.

Richard

 

[TreyS]

We ended up proving the IPSI problems by logging in to the IPSI boards and checking "show internet stats". This would show TIME_WAIT when trying to connect to our main S8710 pair.

How do you log into the IPSI board? I want to check mine out.

Thanks,

Trey

 

[FicBen]

Would also like to know.

 

[mikeydidit]

http://support.avaya.com/elmodocs2/comm_mgr/r3_1/pdfs/maintenance/03_300431_2_1.pdf

Above is the link to the commands and methods for accessing the IPSI's. Connecting from a laptop was the only way I knew of, but it looks like you can access these from the web brouser too. I will let you know If I can get in that way.

"You don't stop playing because you get old. You get old because you stopped playing."

 

[Spookytooth]

you can try ipsisession -p xxx.xxx.xxx.xxx to telnet to the ipsi from bash

 

[jimbojimbo]

FYI, IPSI version 36 has been released and should be available from the download center.

James Middleton
ACSCI/ACSCD/MCSE
Xeta Technologies
jim.middleton@xeta.com