IP Office Crashing

[Rappy]

good afternoon, everyone!

could you please look at the error messages below? i understand that the IP office really has no real reason for rebooting, cept for all the jokes such as picking up the phones, dialing a phone number, getting a dial tone.
From what i have been able to gather, the issue can be caused by spaces in your config. how is this possible? are they talking about spaces in the users' names and extensions? could someone please, please, please give me some direction? this has happened one too many times at our corporate branch, and it's making it seem as if the system is unstable. (yeah yeah! i know! Avaya is as stable as the british pound!

=H=

2948mS RES: Thu 10/1/2008 15:55:30 FreeMem=13950112(25) CMMsg=8 (9) Buff=200 850 1000 1271 4 Links=6489 Elements=166
3000mS PRN: +++ START OF ALARM LOG DUMP +++
3000mS PRN: ALARM: 27/05/2006 11:03:14 IP 412 3.1(56) <DSI Exception> CRIT RAISED addr=11bca5b9 d=7 pc=008e3c28 aaaaaaaa 008e1c34 008dea44 008deae4 00a6f070
3000mS PRN: ALARM: 21/02/2007 09:49:34 IP 412 3.1(65) <TLB Data Store Error> CRIT RAISED addr=b056cab3 d=7 pc=008e3cd0 aaaaaaaa 008e1cdc 008deaec 008deb8c 00a7005c
3001mS PRN: ALARM: 21/02/2007 11:44:27 IP 412 3.1(65) <TLB Data Load Error> CRIT RAISED addr=4445b774 d=15 pc=008dcc40 008dcf30 008dcf30 008e58fc 00c93eac 00c92490
3001mS PRN: ALARM: 23/03/2007 14:20:09 IP 412 3.1(65) <TLB Data Store Error> CRIT RAISED addr=22fee9b7 d=7 pc=008e3cd0 aaaaaaaa 008e1cdc 008deaec 008deb8c 00a7005c
3001mS PRN: ALARM: 11/12/2007 12:58:36 IP 412 3.1(65) <TLB Data Store Error> CRIT RAISED addr=54504b54 d=7 pc=008e3cd0 aaaaaaaa 008e1cdc 008deaec 008deb8c 00a7005c
3001mS PRN: ALARM: 08/01/2008 12:24:01 IP 412 3.1(65) <TLB Data Store Error> CRIT RAISED addr=54504b54 d=7 pc=008e3cd0 aaaaaaaa 008e1cdc 008deaec 008deb8c 00a7005c

Beets? Yes! Ask me how I feel about Beets! I like Beets!

 

[ProviderReborn]

The TLB's are usually issues caused from a currupt config.

Those are the recorded errors. Do you have an actual output of the trace when the system is crashing?

ACA - IP Office Implement
ACA - IP Telephony
ACS - IP Office Implement
CCNA - Working towards.

 

[PrivateBuild]

The last reboot in the error list shows you are running 3.1.65. The next build which was 3.2.17 addressed a couple of TLB reboot errors and build 3.2.55 addressed even more. Suggest you upgrade to at least the latest of 3.2 (67) or higher still to 4.0.14. Once upgraded check you config for errors.
TheProvider is right,an actual output when the issue occurs is best.

 

[Rappy]

Thank you both for your time.

1. What can be done about a corrupted configuration? Can I run the configuration through the IPO 4.0 wizard to correct some of the errors?

2. Where can i find the output trace? I have searched the box for all . mon and TXT files and found nothing pointing to 01/08/08 at 12.24 PM

3. Upgrading to 3.2.67 would probably be the best course of action, as 4.0 requires that we upgrade some of our units in the other branches, as they would not support 4.0- Avaya obviously didnt build a scalable solution with the 400 series, lest you got the top of the line stuff.

Again, thank you!

Beets? Yes! Ask me how I feel about Beets! I like Beets!

 

[IPGuru]

1 escalate the issue to your maintainer!

2 upgrade firmware

3 ensure the config has no obvious errors (Manager V3.2 + will list items t considers suspect).

Observations on your 2 points

Spaces in usere names should noty be an issue (this advise dates back to early V1.X releases)

British pound? have you seen the state of the US dollars $2=£1 & incresing, great or us british tourists

 

[sizbut]

i understand that the IP office really has no real reason for rebooting"

- actually it has good reason for rebooting and you as a customer would rather it did!

Whilst it would be nice to have a system that never had a problem we know that's never going to happen. Now when it meets a fatal error it could do a Windows blue screen of death equivalent and sit there waiting for you to manually pull the plug. Or a really really low level bit of firmware could say to itself "something has gone very wrong" and reboot the whole lot so that you do at least get back to a working (hopefully) system in a few minutes (hopefully) rather than waiting for the guy with the key to the comms room to turn up. Yes there's still been a problem and yes you probably need to get it sorted the system is still going so you're still doing business.

As it is you alarm logs show only 6 reboots in 20'ish months. Not good but far from the worst. 4 of those are the same fault so it would suggest one particular but infrequent activity that causes the crash rather than any config curruption. Knowing what the TLB's mean is hard as they change for each software build so only the top level Avaya developers have a chance of interpreting them if you can get it escalated that far. The fact that its the same fault several time is good (in a bad way) as means it might be a know problem, but whichever way in the course of escalation the most likely words will be "upgrade to the lastest maintenance release" and see if it still happens.

 

[ronromano]

TLB=BUG

 

[gknight1]

it wouldnt hurt to check the obvious things like hunt group names no longer than 12 characters and no spaces. i also dont think you are supposed to have spaces the the name box, you can have spaces in the Full Name, but not the name. just a little tip you might trying before upgrading

 

[tnlcom]

Anyone thought about the fact that this may just be a brown out?

 

[Rappy]

Thank you all so very much!
While i understand your logic, Sizbut, i would rather have a piece of hardware/software which can handle things such as spaces, and special characters in the configuration. I believe such issues were resolved in windows 3.11, and I'm sure the software/firmware used in these devices are more advanced than windows 95.
I am hessitant to upgrade to a newer version of IPO due to the fact that we need to upgrade some hardware in order to do that. i may go to something such as 3.2.17 or 3.2.55 as suggested by Privatebuild.
i will go through the config and see if any of your suggestions may be found- a special thanks to you, GK- i will probably need to remove all the spaces in the name box.

The British pound joke is a reference to an old 80s flick- no longer applicable, certainly.

=H=

Beets? Yes! Ask me how I feel about Beets! I like Beets!

 

[ProviderReborn]

3.2 61 has solved many outstanding issues for me, I would personally always upgrade to the latest release of a build.

ACA - IP Office Implement
ACA - IP Telephony
ACS - IP Office Implement
CCNA - Working towards.

 

[sizbut]

Its not my logic, you'll find its what most phone systems do, its called a watchdog timer (

http://en.wikipedia.org/wiki/Watchdog_timer).

We would all rather have equipment that could cope with everything but when the worst happens you certainly don't want your phone system to just lock up and wait.

 

[Rappy]

thank you all for all the valuable information. i will put these suggestions to work Monday next, and report my progress.
enjoy the humor!

=H=

http://www.jokes-o-matic.com/jokes/...ke-a-walk-to-the-pier-near-his-church-He.html

Beets? Yes! Ask me how I feel about Beets! I like Beets!

 

[aarenot]

Rappy,

" While i understand your logic, Sizbut, i would rather have a piece of hardware/software which can handle things such as spaces, and special characters in the configuration. I believe such issues were resolved in windows 3.11, and I'm sure the software/firmware used in these devices are more advanced than windows 95. "

While I understand your not knowing these things because I did not know them before I became competent with the IPO system(if I am even competent yet), there is no issue with the IPO accepting characters. The issue is with the technician inputing data that according to proper training, and documentation is not valid input. I started out blaming the IPO for every issue which happened, and after I became somewhat more competent I realized that learning how to properly program the IPO was more productive than blaming it for my incompetent input in programming.

As with probably most IPO techs, except those who unlike me, never started off ignorant, untrained, and unexperienced with the IPO we learned that 99% of the issues you run into are your own fault due to a lack of knowledge, experience, or training. From my limited experience of walking into, or helping remote on other peoples systems, rebooting issues are most often caused by programming errors as well.
My experience with upgrades fixing issues is that sometimes bug fixes take care of the issue by upgrading. Also, sometimes the IPO software is rewritten to accept the garbage input by the programmers who do not know, not to do certain things from training, or RTFM.

While I am not saying that your situation is not going to be a bug fix which resolves itself by upgrading to a newer release which fixes an issue with the software itself. My experience tells me as competence goes up, the number of issues that lead to reboots goes down. My advice therefore would be to have soemone who knows the details of how to program the IPO from implementation experience, servicing experience, training, and reading the manuals take a look at it before just thinking an upgrade is your solution.
I suggest IPOFFICEINFO.COM as the best on the web source of info, and documentation on the IPO. Better than Avayas in my opinion.

 

[gknight1]

well said aarenot... i needed that this morning.

 

[aarenot]

gknight,
I am not one, as so many have harrassed me about, to go about claiming the IPO is perfect. Part of competence in my opinion is to know that not all new features work as advertised right away.

Another part of competence is to know that most errors are human not the IPO, and that blaming the IPO is really not something a less than highly experienced tech should even be doing. In fact, unless you find a specific CQ to support your conclusion, I would always lean toward programmer error, unless I know the programmer, and know they are a better programmer than I am.

 

[intrigrant]

IP Office is not "fool" proof but it is getting better.
Partly it is Avaya to blaim that a lot of companies think they can handle the IP Office as they probably can with a lot of other systems and start straight away with the most exotic configs you can imagine and then they fail...
The installer shoud have invested more in training and start at a low level but Avaya said : as easy as LEGO, put the bricks together and build something beautiful.
Who is to blaim here?


NIHIL NOVI SUB SOLE

 

[aarenot]

Intrigant,
If the BP has two product authorised techs as Avaya specifies in order to implement the IPO, and they do not cheat on getting the PA's they should have a good start toward your basic implements. Any BP who has techs get their PA's, and does not know the IPO is not as easy as that, are not listening, or do not care what their techs say.
Also, sales guys who have their PA in sales know it is a bit more complicated if they are diligent.

So, I do not give any BP any slack on not being prepared for the IPO when they have resources available. I had no help from my BP of any kind, no training of any kind, and I got my PA's for implement, and CCC, etc. before I ever even touched an IPO simply by downloading, and reading the PDF's. My first implement also was a referal for other implements, as well as them buying another one. So it can be done if you actualy care about the customer. IPO is not that hard if you RTFM. I do realize I am good at learning things quickly, but I am no "guru", or "god" I am pretty much an average guy who works hard.

 

[tlpeter]

You are not that average, you read the manuals


ACA - Implement IP Office
ACA - Voice Services Management
______________
Women and cats can do as they please and men and dogs should relax and get used to the idea!

 

[aarenot]

tlpeter,
Since we are exchanging compliments, you are not that average either, LOL.

To be honest with the read out loud option for PDF's, I kinda let it read itself to me in the car more than reading it myself. You should try that feature if you have the time at the wheel or when you are doing something that lets you listen to audio.