having issues with my system where I'm running various status, list and other commands and getting an error: error encountered, can't complete request, check errors before retrying. system has no alarms, but unable to perform various administration and may have a down trunk.
Tek-Tips is the largest IT community on the Internet today!
Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!
-
Congratulations Mike Lewis on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!
SOS: in CM getting an error running commands saying: Error Encountered, can't complete request;
- Thread starter wpetilli
- Start date
AvayaTier3
Technical User
Hardware Platform?
CM Software Version?
You have software corruption and most likely cannot save translations in addition to the other signatures.
I can help you with a solution
A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"
bsh
45 years Bell, AT&T, Lucent, Avaya
Tier 3 for 35 years and counting
[URL unfurl="true"]http://bshtele.com[/url]
CM Software Version?
You have software corruption and most likely cannot save translations in addition to the other signatures.
I can help you with a solution
A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"
bsh
45 years Bell, AT&T, Lucent, Avaya
Tier 3 for 35 years and counting
[URL unfurl="true"]http://bshtele.com[/url]
- Thread starter
- #3
Hey- thanks for the response and offer. I had opened an Avaya ticket and you’re spot on. There was something corrupt. I had to run and send the corruption team a backup (xln). I guess they identified the issue and went in and fixed it. They didn’t tell me exactly what but it seems fine now.
First got some calls from our help desk who operates out of a cc. They were having some strange behaviors. One of them was unable to log into their 9630 IP set. When I looked it was set as a 9630 but had a port from one of the ds1’s. Super weird. Was then getting errors running some commands and missing data on others. Service otherwise was fine. Never seen this before. We gave quite a bit of help desk guys access to CM so I thought something happened due to the volume of people in there. Not sure.
What would you have done if you don’t mind me asking?
It is a CM 6.3. The original tech did a save trans and it did work.
First got some calls from our help desk who operates out of a cc. They were having some strange behaviors. One of them was unable to log into their 9630 IP set. When I looked it was set as a 9630 but had a port from one of the ds1’s. Super weird. Was then getting errors running some commands and missing data on others. Service otherwise was fine. Never seen this before. We gave quite a bit of help desk guys access to CM so I thought something happened due to the volume of people in there. Not sure.
What would you have done if you don’t mind me asking?
It is a CM 6.3. The original tech did a save trans and it did work.
AvayaTier3
Technical User
Glad you were able to get this resolved.
Command failures generate entries in log files to assist with diagnosis of root cause for corruption.
Generally speaking, all software corruption is caused by software malfunctions. Resolution for these issues are many. Side effects
and signatures are many. These would include the correction of software tables, installing patches for correction of software,
and even possible reboot of CM or restore and reboot of CM from a previous backup.
A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"
bsh
45 years Bell, AT&T, Lucent, Avaya
Tier 3 for 35 years and counting
[URL unfurl="true"]http://bshtele.com[/url]
Command failures generate entries in log files to assist with diagnosis of root cause for corruption.
Generally speaking, all software corruption is caused by software malfunctions. Resolution for these issues are many. Side effects
and signatures are many. These would include the correction of software tables, installing patches for correction of software,
and even possible reboot of CM or restore and reboot of CM from a previous backup.
A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"
bsh
45 years Bell, AT&T, Lucent, Avaya
Tier 3 for 35 years and counting
[URL unfurl="true"]http://bshtele.com[/url]
- Thread starter
- #5
Seems like things are ok now. All commands and data is there. Had a user unable to register to CM though. Getting various errors on the phone: discover, user dat being restored and request error. Had to blow away and recreate. Not sure if this is a one off or there’s other things I’m not going to find out about till volumes pickup. Considering doing a reset sys 4 or an interchange but don’t want to jump the gun.
- Thread starter
- #6
Another thing I saw was I status'd a station and it was in-service, but showed another extension number in the extension field. Super weird. So far these issues are with our call center users. I wiped this guys station out and re-created from scratch and it's fine now. Any thoughts on what I should do here? To catch all these little things isn't possible, but nothing is showing wrong on the system.
AvayaTier3
Technical User
This sounds like corruption with agent tables caused by software bug.
A controlled reboot will reload translations and rebuild agent tables correctly as agents login.
This bug can block agent login and logout.
You could ask Avaya if they have a fix for your 6.3 in a service pack.
A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"
bsh
45 years Bell, AT&T, Lucent, Avaya
Tier 3 for 35 years and counting
[URL unfurl="true"]http://bshtele.com[/url]
A controlled reboot will reload translations and rebuild agent tables correctly as agents login.
This bug can block agent login and logout.
You could ask Avaya if they have a fix for your 6.3 in a service pack.
A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"
bsh
45 years Bell, AT&T, Lucent, Avaya
Tier 3 for 35 years and counting
[URL unfurl="true"]http://bshtele.com[/url]
- Thread starter
- #8
AvayaTier3
Technical User
reset system 4 from CM SAT
to reload from known good CM translations (before corruption existed). Use restore xln from known good backup on active CM server,
then do reset system 4 from CM SAT. After CM comes up, do save translation all to update standby, LSP/ESS servers.
A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"
bsh
45 years Bell, AT&T, Lucent, Avaya
Tier 3 for 35 years and counting
[URL unfurl="true"]http://bshtele.com[/url]
to reload from known good CM translations (before corruption existed). Use restore xln from known good backup on active CM server,
then do reset system 4 from CM SAT. After CM comes up, do save translation all to update standby, LSP/ESS servers.
A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"
bsh
45 years Bell, AT&T, Lucent, Avaya
Tier 3 for 35 years and counting
[URL unfurl="true"]http://bshtele.com[/url]
- Thread starter
- #10
Sitting on the edge of the seat all morning and things have been normal. Does that.. actually, can that mean what Avaya fixed with the corruption was resolved or it's a bug that can just show up anytime.. now or a year from now? Trying to make final decisions. What you proposed about restoring a known good backup and then doing a reset sounds like the most sound thing to do. However, the longer we wait to do that the good backup becomes older and older. In a short conversation with my direct report, since Avaya resolved the corruption and things were fine all weekend there wouldn't be enough basis to do something drastic like the restore and reset.
Generally you'd get Avaya to fix it and have backups every day.
Patch. Release notes include stuff. Usually when they add features is when things can go sideways. When they added the Media Server as a DSP farm, announcements could go corrupt. When they upped trunk group members to 10K, trunk groups could go corrupt, etc.
Was anyone banging in a lot of stuff with ProVision lately?
Patch. Release notes include stuff. Usually when they add features is when things can go sideways. When they added the Media Server as a DSP farm, announcements could go corrupt. When they upped trunk group members to 10K, trunk groups could go corrupt, etc.
Was anyone banging in a lot of stuff with ProVision lately?
- Thread starter
- #12
All I can really say is we have a lot of service desk team members with limited access accounts that do the low-level MAC work. They didn't break anything by making a change, but I thought the volume of people in there could have caused something. Or maybe not. Nobody with Provision, just out of the blue this issue occurred. We were told to do a save trans, run a backup of just the xln and send to Avaya's corruption group. We did that and in a short time, they must have gone in and made whatever change they saw from the backup that was sideways. Whatever they did wasn't service affecting and things have been fine since. I just don't know if that's the end of it or something on need to be on the edge of my seat for.
I remember giving a gang a crash course on SMGR 6.2.
I told them "go add a user, name it for yourself" and the first guy was 1001, the second 2002 and so on.
When they all hit save, the guy with comm profile 1001@domain.com got CM station 2002. Basically the order in which they all clicked new user, whoever was first filled out their user. Whoever clicked commit first put their station in that guy's profile.
So yeah, iptcm debug logs on SMGR would be where you'd look. In log config, you can make it have more megs per file and more files retained.
I told them "go add a user, name it for yourself" and the first guy was 1001, the second 2002 and so on.
When they all hit save, the guy with comm profile 1001@domain.com got CM station 2002. Basically the order in which they all clicked new user, whoever was first filled out their user. Whoever clicked commit first put their station in that guy's profile.
So yeah, iptcm debug logs on SMGR would be where you'd look. In log config, you can make it have more megs per file and more files retained.
- Thread starter
- #14
While the CM is a managed element the masses choose to administer using ASA still. We are in the planning phases of moving/upgrading our whole stack into CM into AWS, but the end state for that will be a few months. In the interim, I'm trying not to have to invest time into the current unless I absolutely have to. From dealing with Avaya it sounds like this corruption thing is something they come across quite a bit.
Well, there's a corruption team. That's all they deal with. Yeah, they see that all day.
I've never heard of corruption the way you mentioned - seeing a B channel port where a digital station port should be.
If you've been doing stuff the same way you always have, I'd just chalk it up to bad luck. FWIW, we run into it about once a year across all our customers for a decent sized BP with some really large systems and the last time was around using the >255 trk members per trk group and cluster SMs in sig groups at 8.1.
I've never heard of corruption the way you mentioned - seeing a B channel port where a digital station port should be.
If you've been doing stuff the same way you always have, I'd just chalk it up to bad luck. FWIW, we run into it about once a year across all our customers for a decent sized BP with some really large systems and the last time was around using the >255 trk members per trk group and cluster SMs in sig groups at 8.1.
- Thread starter
- #16
Yea, a really weird block of time. The call center guys were getting the same call delivered to multiple agents at the same time. One reported hearing the auto attendant for voicemail in the background of a call. Then 2 guys were getting an error logging in. While tshooting that is when I saw a set type 9630 with the port field showing a B channel of the PRI. Once I busied out that port and blew the station away, is sort of when the majority of the commands were erroring out. While statusing another station it showed in service, but the extension field showed someone else in the call centers extension. I fixed the 2 guys by completed wiping and re-adding. The symptoms with same call being offered to multiple agents and the vm background stuff went away.
My options are to:
1. Do nothing since things are fine now
2. I can do a reset sys 4 to reload everything
3. Restore a backup from before this all occurred and do the reset system 4, which AvayaTier3 mentions above. The longer I wait to do that, the more changes that would have to be redone.
4. I know 6.3 is pretty much EOL/EOS, but we are on .15 of that release and have the option to patch to .18
Not sure-
My options are to:
1. Do nothing since things are fine now
2. I can do a reset sys 4 to reload everything
3. Restore a backup from before this all occurred and do the reset system 4, which AvayaTier3 mentions above. The longer I wait to do that, the more changes that would have to be redone.
4. I know 6.3 is pretty much EOL/EOS, but we are on .15 of that release and have the option to patch to .18
Not sure-
- Thread starter
- #18
This is what I got back from support on what was fixed- Doesn't make sense though with the symptoms:
The specific issue was that for trunk group 6 it was missing an entry for the PR_MOPORT PREC for one of the member ports and had to be readded to the system via TCM. Without this PREC the CM was unable to determine the MO (maintenance object) so when messages were passed to the HMM process it could not determine how the maintenance should be done on the missing trunk member port and produced an error.
In short “it was missing a needed call processing record and we added it back”.
The specific issue was that for trunk group 6 it was missing an entry for the PR_MOPORT PREC for one of the member ports and had to be readded to the system via TCM. Without this PREC the CM was unable to determine the MO (maintenance object) so when messages were passed to the HMM process it could not determine how the maintenance should be done on the missing trunk member port and produced an error.
In short “it was missing a needed call processing record and we added it back”.
AvayaTier3
Technical User
The description from Avaya corruption support would explain issues with one member of trunk-group 6 as well as administration and maintenance commands related to the trunk-group and said trunk and port. Without knowing what software errors were recorded, it would be hard to tell what else was affected.
With "dadmin" login you can use SAT Command: display software high-resolution. Use the front page to filter for the window of time
to match when you experienced problems.
A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"
bsh
45 years Bell, AT&T, Lucent, Avaya
Tier 3 for 35 years and counting
[URL unfurl="true"]http://bshtele.com[/url]
With "dadmin" login you can use SAT Command: display software high-resolution. Use the front page to filter for the window of time
to match when you experienced problems.
A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"
bsh
45 years Bell, AT&T, Lucent, Avaya
Tier 3 for 35 years and counting
[URL unfurl="true"]http://bshtele.com[/url]
- Thread starter
- #20
- Status
Similar threads
- Question