New (to me) Avaya CM Troubles

[horuden]

So I hope it is ok to post this here, and inb4 "Call your BP" this is just one of my (admittedly pretty weird) hobbies. I do work for a small MSP and we manage a couple of IP Offices which I have become fairly proficient at, I know absolutely nothing compared to my boss or any of you guys, but I love telephony so I am just happy to be able bang my way around in the dark.

Long story short we decommed an older Avaya CM (I think it is running CM 6-ish) at a client location about a year ago. I never really got the chance to learn much about it other than some very, very basics (changing a couple little things for a station), but it grabbed my attention my first week on the job. We finally got around to removing it from the closet and I was lucky enough to be able to take it with me. This is a dream come true for me, a dream that turned into a nightmare (much like my AS/400).

I have a G450 cabinet with an S8300C processor, a PRI card, and a couple analog cards. I also have a G250 with an S8300C processor that (I am fairly certain) acts as an SRG, not sure if I have the terminology correct, but it will pick up call processing if the main unit ends up MIA.

I figured since I knew the IP subnet and had at least the dadmin password it would be pretty easy to get it running again, but you know what they say about assuming. So I get it home and get it makeshift set up in my living room and things seem to be going ok, until I try to connect to the S8300. The chassis boots up no problems, I can ping the chassis controller, and after some fiddling with an old windows xp laptop, and an ancient version of Java, I can even get to the Avaya Device Management console for the G450.

That, however, is where the fun times end. Once connected to the ADM I can see all of the cards, except for the S8300. The PRI card has the alarm light lit up, which is to be expected as I am not wealthy enough to have a whole PRI coming into my house. The S8300 also has the alarm light lit, on both of the chassis.

Some of the things I have tried so far are:
- Re-seating the S8300 card
- Connecting directly to the Services port on the S8300 (I don't even get a link light)
- Continually pinging the Services port as I reboot the chassis (Still not even a link light)
- Resetting the chassis from ADM (I get a "Failed to reset the device! SNMP Error: Timeout" error.

I feel like I have stopped just short of sacrificing a chicken to the Avaya Gods, but I am sure I have missed quite a few things to try. After the first boot failed I pulled the drive out of the S8300 and got a dd image of it at the advice of a friend that works with PBXs for his day job (admittedly not CM hardware, he does have the VM variety of Aura running though). I am willing to provide any further information that would help in diagnosing my issues, but I may need pointed to the appropriate Avaya manuals. If there is anyone that would be willing to throw some suggestions at me I can't tell you how much I would appreciate it.

This is my telephony holy grail, and I am hoping that I don't have a very pretty (and giant) paperweight. But I feel like it probably takes more than unplugging and moving locations to kill a CM. I am dying to try getting my Merlin Legend set up in Behind Switch mode just to see how that all works, and I am sure my IPO V8 wouldn't mind having a newer Avaya buddy to talk to either.

Sorry for the wall of text, but thank you in advance for any pointer you may be able to provide.

--
Holden

 

[avayaguy23]

What are you ultimately trying to accomplish with the s8300c? That model hasn’t been supported by Avaya in a very long time. You are better off using VMware if you have access to the CM ova.

 

[horuden]

Well to be fair the age of it doesn't bother me all that much, except for the fact that it is a little too modern for my tastes. I would really love an older Definity, or better yet a Dimension, but beggars can't be choosers right? But I guess ultimately I just want the CM to boot and process a few calls a week, I also want some of those beautiful 74XX series phones and at least one 302 attendant console. It would also be neat to create a little private network and put my Merlin Legend in behind switch mode and have it run off the CM.

I don't have access to the CM ova, and I am unwilling (and unable) to pay $texas to license a new system. Since this system was free to me, and already has licenses, it is perfect for my use case of a) learning a little bit about the more enterprise level Avaya stuff b) having a bunch of neat older phones and c) amazing the three people that call me with music on hold at my house.

Most of the stuff I own and use every day is no longer supported by their manufacturer, which can lead to a bit of heartache, but the sense of accomplishment of getting something working, and keeping some useful old tech out of the landfill is worth it to me. For now I can live with the IP Office and the Legend, but eventually I would really like to see this sweet little piece of telephony history doing what it was intended to do.

Thanks for the reply!

--
Holden

 

[NewNic]

It doing just what its supposed to do....DIE after 3 years. (joking)

If you cant ping the services port, your pretty much dead and need a new/different 8300

 

[kyle555]

If you can't ping 192.11.13.6 with your end 192.11.13.5 on the S8300C, like NewNic says, it's toast.

S8300C supported up to CM 5.1 I believe, definitely not release 6.

Licenses on release 5 are specific to the medaia gateway serial number. So, you could replace a S8300C in the same gateway and restore a backup if that included the license file. But if you don't have/can't get that, you're SOL.


What's on the dd of the S8300C from the G450?

If you post a link to something that can mount, I can peek.

Otherwise, services on the G450 should do the same thing as S8300 - ping on 192.11.13.6. You should be able to login with root and ggdaseuaimhrke on the services or console port of the G450. They changed that in later firmware, but if you're running R5 on a 8300C, you're definitely still able to use the backdoor password.

if you get into the G450, does the prompt say G450-001 or G450-???. if there's a number, that means it's registered to a CM and it's gateway number 001. ??? means it's not registered.
If you have a number there, try a session icc or something.

 

[horuden]

> It doing just what its supposed to do....DIE after 3 years. (joking)

I mean, it probably lasted a good couple decades, give the poor old girl some credit! I don't think built in planned obsolescence was a thing everywhere until around 2015 haha.

> If you can't ping 192.11.13.6 with your end 192.11.13.5 on the S8300C, like NewNic says, it's toast.

That is kind of what I was worried about/expecting, also probably why it was replaced with an IPO. I did find out today it is running CM 5.something, I was only one number off and I was even in the right software family so I am going to count that as mostly a win haha. I am very happy to hear that the licenses are tied to the media gateway chassis and not the processor card. Out of curiosity is the serial number tied to the actual chassis or is it tied to the card that controls it (the one with the services and console ports)

> What's on the dd of the S8300C from the G450?

Good question, I was lazy and dd'd the entire drive and not the individual partitions, and after a bit of googling it is apparently possible to mount them in linux, but that involves some crazy math to tell the mount command which partition you want to mount out of the image. What I will probably do when I have a minute is write it out to an extra HD and snoop through the files that way. I also grabbed an image of the CF card that is on the board. I am interested to find out what that is used for, the schemes on both of the drive look a little crazy.

Since the thing was used in production I'm not sure it would be wise to just post a link to them on the internet, nothing personal, I'm just not sure what is stored on there and I don't want to be the cause of any troubles. Especially if it kills my chances of getting more neat enterprise gear to shove in my house and role play GiantMegaCorp, llc. haha

> Otherwise, services on the G450 should do the same thing as S8300

Good to know, luckily I got a serial cable that should work with the console port today. Honestly I feel like that is the easiest route to go at this point, the services ports did me dirty and I now am holding a grudge.

Life is kinda hectic at the moment, I am hoping I will have some time this evening to at least try to get connected to the console port, but worst case scenario it will just have to wait for the weekend. I assume session icc is a command that starts a session with the CM if it is available? I will definitely report back here with my findings though, I really appreciate the pointers!

Another question, since this board is just a linux box could I purchase a replacement and just pop the hard drive and SD card from my current board into the replacement? I know that regular world linux doesn't throw a fit if you take out a drive and put it in a different computer with the same hardware, but if there is one thing I have learned on my trip down the Ma Bell/Lucent/Avaya rabbit hole it is that they play by their own rules.

Thanks again guys, I know we are all busy, but I really appreciate you taking the time to reply to me!

--
Holden

 

[horuden]

Welp, should have know with my previous serial experiences that wasn't the way to go, but the services port worked! There are no numbers in the prompt, so it isn't registering with the 8300 unfortunately, but at least the cabinet isn't dead. I ran the show faults command, it wasn't as verbose as I was hoping.

[pre]
CURRENTLY ACTIVE FAULTS
--------------------------------------------------------------------------
-- Hardware Faults --
+ Clock synchronization signal lost, 06/03-01:36:45.00

-- MGP Faults --
+ H248 registration failure, 06/03-01:36:38.00

-- Media Module Faults --
+ Insertion failure, mmid = v1, 06/03-01:38:59.00

Current Alarm Indications, ALM LED is on
--------------------------------------------------------------------------
+ H248 Link
+ Clock SYNC Primary Fault

Done!
[/pre]

I tried session icc just for giggles, and it just yells at me that "The Media Module inserted in slot 1 is not an ICC". I also plugged in the SRG just to see if the services port worked on that 8300, and it looks like they are both dead, just my luck. But at least I know the one cabinet is good to go, maybe I will try the services port on the g250 tomorrow, maybe that one's processor has a different error.

 

[kyle555]

Yeah, Avaya serial cable is a little different.

Everything's VMware now, but in those days, the CM install CD only boots on Avaya specific hardware even though it's just a plain old computer. I've heard from Cisco guys that Cisco was even worse. It would check the hardware part # of the server and you couldn't install Unity voicemail on a server meant for Call Manager even if it was the same model/spec because you didn't pay the voicemail team for the server!

The serial number of the gateway is bound to the chassis - no way to change that.

Maybe you can just create a new VM in Linux, mount the disks there, boot off a CentOS DVD into rescue mode and be able to chroot to the file system it found. That's what I do when I need to root something/someone forgot a password for something. Boot the VM off the full CentOS DVD, troubleshooting, rescue mode, try to find the partitions, and it'll get you to a CLI where you can chroot /mnt/sysimage to change that to the live partition. It could be enough for you to grab the license file off of because you'd need access to Avaya's remote feature activation website (rfa.avaya.com) with an account of the business partner that owns that customer account to be able to generate licenses.

Might also check /var/log/messages and see if you can diagnose a boot problem you might be able to fix yourself.

So, if you can get the licenses, great.

Otherwise, a s8300C can be had for peanuts. Thing is you need an Avaya CD/DVD drive. The S8300 doesn't work with just any, but Avaya's isn't particularly special, just that some won't work.

 

[AvayaTier3]

s8300B and s8300C can be imaged with CM software from MV11 (MultiVantage) and CM2 to CM5 (R012 to R015 Communication manager) and will work with any g700, g350, g430, g450 and can be used for lab system with no license or expired license.
Phones, etc. would be restricted to the media-modules inserted in the media-gateway.

A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"

bsh

45 years Bell, AT&T, Lucent, Avaya
Tier 3 for 35 years and counting
[URL unfurl="true"]http://bshtele.com[/url]

 

[kyle555]

By "imaged" you mean the normal way with a CD drive in the USB port?

As in, any old S8300 can run a single gateway CM?

 

[AvayaTier3]

yes
& yes

A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"

bsh

45 years Bell, AT&T, Lucent, Avaya
Tier 3 for 35 years and counting
[URL unfurl="true"]http://bshtele.com[/url]

 

[kyle555]

Learn somethin' new everyday

 

[horuden]

kyle555,

Sorry for the delay in replying it was definitely one of those weeks last week. So I have a new, supposedly working, S8300 on the way, should be here in the next day or two. I am hoping that the issues with the current one are hardware not software, and that a hard disk swap will solve my current set of issues. Fingers crossed!

Yeah I don't think that almost anyone could beat out Cisco at the "our shit is crazy proprietary and we are going to make you jump through all of our hoops to use any of it" game, probably IBM but there is always the exception that makes the rule.

I am happy the serial number is associated with the cabinet, I didn't think to create a VM and chroot into the CM images, but that is a great idea! I did write the images to an extra hard drive and poked through them. Didn't really find anything helpful in /var/log, BUT I found 4 license files and a password file. So worst case scenario I will have to get my hands on some CM5 install disks, which is probably less impossible than I assume it is. Good to know about the CD rom drive, if reinstalling is the way to go then hopefully one of my USB cd drives will work, I'm not sure I have ever even seen an Avaya branded drive. Hopefully they are not too hard to find if you go looking.

Thanks for all the pointers, I actually feel like eventually I will have a running CM now!

--
Holden

 

[horuden]

AvayaTier3,

That is fantastic! Thanks for sharing your knowledge, it is good to know that at the very least I will eventually be able to have a CM to play around with, even if it is a crippled one!

--
Holden

 

[kyle555]

HAHAHA. Finding a CM5 disc ain't going to be easy. Buy one on ebay maybe.

There's 3rd party code in everything and Avaya quit licensing a particular piece of what's in CM 6.x and older. They could either rework their old releases to not contain that code, or just stop distributing those releases. They picked number 2.

Either way, without passwords or a friend that can give you a craft ASG challenge response, doing a chroot /mnt/sysimage and resetting the root password would help, but you'll still need a valid account that has access to the CM web page and terminal. cat /etc/passwd would show you a list of logins and there's probably one near the bottom, probably profile 18 that you'd want to set a password for.

 

[horuden]

Yeah I figured I probably couldn’t call Avaya and be like “Can I have an installation cd for a 10 year old product, also I have no customer number”. I’m also not surprised they didn’t want to bother reworking the older releases, just makes it easier to say you need to upgrade.

Luckily I do have at least the dadmin password, and I am pretty sure I may know what they would have used for the root password. So I’ve lucked out there.

My replacement S8300 came today, I’m going to see if I can give it a try this evening... fingers crossed!

 

[AvayaTier3]

s8300 CM software version?
modem connected to s8300C usb port?
backup CD/DVD version?
g450 serial number?

G350-003(super)# show system
System Name        :
System Location    :
System Contact     :
Uptime (d,h:m:s)   : 1,03:08:49
MV Time            : 07:41:52 27 DEC 2019
MAC Address        : 00:04:0d:f2:1e:71
WAN MAC address    : 00:04:0d:f2:1e:70
Serial No          : 07IS12106xxx
Model No           : G350
HW Vintage         : 3
HW Suffix          : B
FW Vintage         : 26.31.0
HW Ready for FIPS  : No
G350-003(super)# show run
G350-003(super)#
! version 26.31.0
Config info release 26.31.0 time "07:42:02 27 DEC 2019 " serial_number 07IS12106
xxx
 !
encrypted-username hx8whv6bbb69X//GCJKHVg== password 3KPvfQ7UqZVs3pBv1x41NJXITIN
jr6YncrnfbfCyqnk= access-type 2YSgXzw9yz3w/DbmPRPMQA==
set logging file enable
set logging file condition all Error
set logging file condition BOOT Debug
!
no ip telnet
no snmp-server community
!
ds-mode t1
!
interface Vlan 1
 icc-vlan
 ip address 10.157.10.11    255.255.254.0
 pmi
 exit
!
interface FastEthernet 10/2
 exit
!
interface Console
 exit
!
interface USB-Modem
 shutdown
 exit
!
ip default-gateway 10.157.10.1     1 low
!
set mgc list 11.9.0.1
set mediaserver 11.9.0.1 11.9.0.1 23 telnet
set mediaserver 11.9.0.1 11.9.0.1 5023 sat
set sync interface primary v5
set sync source primary
rtp-stat qos-trap
no rtp-stat fault
!#
!# End of configuration file. Press Enter to continue.

G350-003(super)# show mg list
SLOT  TYPE     CODE        SUFFIX  HW VINTAGE  FW VINTAGE
----  -------- ----------  ------  ----------  -----------
v0    G350                 D       1           26.31.0(B)
v1    S8300    ICC         A       0           1
v2    -- Not Installed --
v3    -- Not Installed --
v4    Analog   MM711       A       31          88
v5    E1T1     MM710       A       5           18
v6    -- Not Installed --
v7    Analog   (on-board)  A       6           88
G350-003(super)# show mgc

CALL CONTROLLER STATUS
-------------------------------------------
Registered          : YES
Active Controller   : 11.9.0.1
H248 Link Status    : UP
H248 Link Error Code: 0x0

CONFIGURED MGC HOST
---------------------
11.9.0.1
-- Not Available --
-- Not Available --
-- Not Available --
survivable-call-engine      disabled

Done!

A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"

bsh

45 years Bell, AT&T, Lucent, Avaya
Tier 3 for 35 years and counting
[URL unfurl="true"]http://bshtele.com[/url]

 

[Wanebo]

Even if you had a Sold To number and all the requisite info Avaya would not be able to supply you with CM 5 software. They removed all of that a few years ago and no longer offer any, repeat ANY, support for such an old version.

Kyle wasn't kidding when he said find one on Ebay maybe. It actually would be your best bet for getting one.

 

[AvayaTier3]

Horuden

There is still support out there for your s8300c / g450 platform. It does not have to be Avaya.
Hopefully you have ftp backups or usb flash card backups which will allow replacement server to be built and restored to the original (before crash) state.


A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"

bsh

45 years Bell, AT&T, Lucent, Avaya
Tier 3 for 35 years and counting
[URL unfurl="true"]http://bshtele.com[/url]

 

[NewNic]

if it were me building a lab, get hold of a new blade, install 7 or 8 on it and have at it. once installed licenses are wide open for a grace period like 30 days, reboot and it starts over