Completely lost weekend trying to upgrade to .67 – bad firmware – bad retrieval

[mpisano]

I’m home, remote desktoped into my computer with the Manager doing a “quick” weekend upgrade, I cant even test functionality of the digital handsets or the SCN, I only know my remote two IP units are dead – the following I wrote over the past 24 hours, I started with….

I have (had) a working SCN until I deiced to upgrade one site from 8.1.43 to 8.1.67. After the upgrade my two remote 9621G failed to get I dial tone and the contacts and features buttons just beep. I can see the digits being dialed in system status, and lots of logs when they go off hook., They can even call my cell but no audio or directory? I know everybody is gonna say “IP Routing”…. Keep reading….

 2736884mS H323Evt:    Recv GRQ from 0a60c08e
   2736884mS H323Evt:    e_H225_AliasAddress_dialedDigits alias
   2736885mS H323Evt:    found number <410>
   2736931mS H323Evt:    Recv: RegistrationRequest   10.96.192.142; Endpoints registered: 2; Endpoints in registration: 0
   2736931mS H323Evt:    e_H225_AliasAddress_dialedDigits alias
   2736931mS H323Evt:    found number <410>
   2736932mS H323Evt:    RRQ --- CallSigProtocol is H323AnnexL_P. Go for Avaya 4600IP phone
   2736932mS H323Evt:    RRQ --- Treat this as a forced login, the phone was probably rebooted (same_cs_addr 0)
   2736932mS H323Evt:    GK: Unregister endpoint EXO_516a9cdd6f5d0d0d for extension 410 reason 4
   2736932mS CMExtnEvt:    EXO Mp Home Dn:410 ExtnFault now 1
   2736933mS CMExtnTx: v=410, p1=8002
            CMReleaseComp
            Line: type=IPLine 250 Call: lid=253 id=7 in=1
            Cause=16, Normal call clearing
   2736934mS H323Evt:    GK: Send URQ
   2736935mS H323Evt:    RRQ --- Register extn 410 using product IP_Phone, version 6.1000
   2736938mS H323Evt: v=0 stacknum=250  State, new=ReleaseReq, old=Active id=7
   2736938mS H323Evt: v=0 stacknum=250  State, new=NullState, old=ReleaseReq id=7
   2737273mS RES: Sun 14/4/2013 12:12:32 FreeMem=57514060 57055844(1) CachedMem=458216 CMMsg=5(9) Buff=5200 790 999 12463 4 Links=4446 BTree=0 CPU=1/4/6874/13640/14963/1
   2737273mS RES2: IP 500 V2 8.1(67) Tasks=56 RTEngine=0 CMRTEngine=0 ExRTEngine=0 Timer=70 Poll=0 Ready=0 CMReady=0 CMQueue=0 VPNNQueue=0 Monitor=2 SSA=1 TCP=25 TAPI=2 ASC=1 SYS=MNTD OPT=UMNT SDSPD=2034
   2737274mS RES4: XML MemObjs=45 PoolMem=2097152(1) FreeMem=2082064(1)
   2737291mS H323Evt: v=0 stacknum=250  State, new=NullState, old=NullState id=-1
   2737291mS H323Evt: v=0 stacknum=250  State, new=Present, old=NullState id=8

I upgraded one to 9621G’s the shipping firmware with .67 (6.2) they both were on (6.1) and I also found an even newer version (6.2313) as a separate download (which might be CM specific) But loaded. None seemed to help

I then:
a) I rolled back to .43 and one with the original (6.1) firmware worked.
b) Rolled up to .67 again (I'm persistent) both failed again.
c) Rolled back now to .63 which worked "Sometimes" - Seems if I reboot the Phone it fails, but if I reboot the 500V2 it works until the phone power cycles? - Bizarre
d) Rolled again back to .43 and again one worked
e) Rolled up to .57 still one
(now thinking 6.2313 maybe I should of stayed with 6.2... I Rolled the phone firmware back to S9621_41HALBR6_2_2_09U) - Still spastic
f) Still determined – Rolled back up .67 Now two problems surface
1) I have 3 1416’s that wont take firmware they get to 100% on 1400Boot25.bin and bomb with engine(f5b4a538:0) Abort download: Reason=Status NOK: 0x2 status=252 Retry=1 One of them continues to 1400R32.bin and then bombs
Very similar to

http://tek-tips.com/viewthread.cfm?qid=1700537

2) I decide to create a new 323 port and user and log off one of the ip phones and log it in as the newer extension and I get a dialtone, contacts and features ?!?!

Convinced now that this has something to do with the user profile I compared the new user to my original. After many reboots while syncing the two, I have found that my “private line” which is on button 4 as “Mike” “Line Appearance” 712:10 is causing the problem?


Now that I have determined that all these newer versions stink, I’m again rolling back to .43 and hop I can get my three 1416’s alive again

I’ve rolled back now and the 1 9521g with the older firmware works with the user with the Line appearance. The other phone with the newer firmware doesn’t… guess I’m rolling the other phone back to 6.1

The Three 1416’s are still giving me issues, I guess when I’m at the office on Monday I will try them in a DS8 or will RMA them – BTW – these 3 are all on a combo card #2 (with 2 phones that are upgrading\downgrading ok)

It’s now Sunday and I’ve had it - don’t know if anybody see anything similar after .57? Would welcome any suggestions, if not I leave the above as notes for anyone who ends up in the same dilemma

Thanks,
Mike

 

[Westi]

I upgraded 3 systems within the last week (2 of them today) to the new version and so far no problem. I have 1600, 9620, 9630 and 9641 IP sets on them as well as digital sets (not too many 1400 series but 9508 sets) and older version of digital sets 6408 and 5400 and 2400 phones and so far all good (I hope i didn't jinx it now)

I have however learned the hard way that the downloads from Avaya need to be checked with the md5 checksum

Try that and see if there was maybe a fault on the downloaded file which could be causing all kinds of problems

If I see it correctly then the 9621 phone should have new firmware on releases above the GA release (S9621_41HALBR6_2_2_09U_V452) this is the same release for all 8.1 except the 8.1.43 (GA) so there should be no issue with upgrades within these releases. Anything above that 6.2.2 release will be for CM as you correctly stated.

You said that you did that remotely? Sometimes a hard reboot helps the system over some of the issues.

Thanks for taking the time to post all this in detail and congratulations for being persistent and trying as you are right someone might get good use out of this post.

Joe W.

FHandw, ACSS (SME), ACIS (SME)

http://convergednetworks.ca

Give a tech a solution and he will be back tomorrow to ask you the next question, teach a tech how to read the manual and he will be able to solve the problems for a life time.

 

[mpisano]

Thanks Joe!

I did re-download from avaya and pointed manager to the bins at some point which did't help. Also manually deleted the 14xx files and transferred them using card level tools too.

After the upgrades\downgrades i normally do a timed reset for 5 minutes via System Status. I know it's not a complete power cycle but it's better then driving the hour to the office.

My saga continued today....

The three 1416's are still dead - i'll swap them tomorrow when i'm on site - presume Avaya will release a fix one day or I will RMA them.

My 2 remote 9521G still were acting weird so I have to bit the bullet ans i've upgraded again to .67 - I'm risking it and hoping that there are no SCN or surprises at the office tomorrow

I have also updated both of my 9521G's to 6.2. and cleared the users buttons except for the basic ABC and 1 Cover button. I've used a few version of the Non Color touchscreen 95xx phones and will try them next week, I've only used these 9521G's for two weeks and so far i'm not impressed with the functionality or stability of them.

Will know more in the morning, Thanks again

Regards,
Mike

 

[tlpeter]

Mike, i have had no issues at all with upgrading.
All files are in the manager folder so no need to find and replace them.
It clearly sounds like you have no file server setup.

The digital phones use tftp and the IP phones use http.
The IP500v2 has an SD card which you can use as fileserver location.
Set the tftp and http IP address the same as the Lan 1 port on your IPO and reboot the IPOffice.
The 9621's have big firmware files so this can take a while and can affect the upload to other phones as well.
Open monitor and turn on tftp and http in it to see what actually is happening.


BAZINGA!

I'm not insane, my mother had me tested!

 

[mpisano]

First typo in post - Two IP phones failing to get dial tone \ proper retrieval are 9621G not 9521G. 9521G is not even a proper model, but may of added confusion given 95xx us a DS not IP series - Sorry. We do have 9508 DS phones at this site which have upgraded perfectly, the DS phones that failed to upgrade were 1416's

Thanks for the info peter,

Partially Correct - I manually (craft) set the IP phones to point to the managers IP (with IIS) to download and control the version of the TAR files. My 2 IP Phones are remote across the VPN are are routed through LAN2. I believe the TFTP\HTTP settings are only for LAN1 (not currently in use) and only if you use the internal DHCP server which will propagate those parameters along the the Address\Subnet. The mucking with the SD that I did was to make sure that the upgrade wasn't corrupted for the 1400*.* digital station files and the possible reason why the three phones wont upgrade.

Question, you stated "The digital phones use tftp and the IP phones use http."? It was always my understanding that IP phones use either TFTP or HTTP depending on model. Digital phones use their own protocol and will always pull from IPO regardless of managers\IPO's setting?

My two remaining issues are:
The three Digital phones will load the 1400Boot25.bin to 99/100% and either:
Die at "Abort download: Reason=Status NOK: 0x2 status=252 "
or
Continue to load 1400R32.bin to 99/100% and
Die at "Abort download: Reason=Status NOK: 0x1 status=251 "
They never get to load the language file.
Very similar to

http://tek-tips.com/viewthread.cfm?qid=1700537

And my other issue is the lack of proper retrieval for the 9521G leaving them without dial tone or functional Contacts\History\Features... buttons; provoked by assigning a "line appearance" to a button. Believe this is still buggy firmware and will prove it buy bringing one to the office and trying it locally and bringing a 9608 home tonight and see if it properly retrieves.

Glad you both have had success with this newer version, make me feel better about upgrading to it and dealing with these last two remaining issues.

Regards,
Mike Pisano

 

[tlpeter]

The old 46xx and 56xx could do tftp too but the 96xx always use http or https.


BAZINGA!

I'm not insane, my mother had me tested!

 

[mpisano]

Back at the site today and happy to report that the 3 phones that would not upgrade their firmware have completed. Not sure if it was he Combo card that these were attached to or the internal cat 5e wiring (bad drops), but when attached directly to the 8 port card with a cat6 patch they worked first shot. They are now back in their offices and operational.

Also happy to report that the 9621G is also working on site now - this is a confusing one... I've been using a 9650 and 9630G at the home office without a problem for close to two years. I recently (2 weeks ago) picked up these 9621G and they have been working without a problem.

I havent rebooted the IPO during these two week leading up the the upgrade. It seemed like the 323 port got corrupted during the bad registration and it maybe would work depending on the user buttons; after rebooting the IPO sometimes it worked other times it failed. Only thing that was consistant was a new port and default user allways gave me dialtone and features. My normal users\port were in some half logged in state with no audio and dyfunctional buttons.

Will know more when I get home tonight and bring a 9650 with me

Regards,
Mike

 

[Westi]

good to hear Mike
was already interested if you would have gotten somewhere with the problem and look there you solved it. This is keeping our lives interesting or makes us age prematurely

Joe W.

FHandw, ACSS (SME), ACIS (SME)

http://convergednetworks.ca

Give a tech a solution and he will be back tomorrow to ask you the next question, teach a tech how to read the manual and he will be able to solve the problems for a life time.

 

[mpisano]

Confirmed - They messed up the firmware on the 9621G (Maybe others?)

I now have at the home office (far side of the VPN) the 9621G and a 9650 plugged into the same switch. I can go back and forth logging in as the same extension and 9650 works and the 9621G wont get a dialtone and locks up after loading the retierval. Although I can pick up the 9621G handset and call my cell phone - just cant talk or hear. It's logged in and dysfunctional.

Now - Follow this - I remote into my Manager machine, copied the User and assigned it to an available 323 port and can now use this copy on the 9621G! It gets a dialtone and features/contacts. At some point, this copied user will meet the same fate as the original which was what I was dealing with over the weekend. Now I fell like diving into wire shark,except this has already consumed may to much of my life!

Do I even take this to Avaya, or shelf them for a month and use the 9650?

Regards,
Mike

 

[Westi]

now we have 9621G on remote and park zones messed up. Lovely

Joe W.

FHandw, ACSS (SME), ACIS (SME)

http://convergednetworks.ca

Give a tech a solution and he will be back tomorrow to ask you the next question, teach a tech how to read the manual and he will be able to solve the problems for a life time.

 

[mpisano]

Bummer, Hope I didn't jinx you.

There is Firmware 6.2.3 - but seems to only be CM related fixes

http://downloads.avaya.com/css/P8/documents/100169016

I tried it, but it didn't help my half logged in mode so I rolled back.

I'm just glad my 323 trunks across the SCN sites are holding and this is somewhat localized.

On the bright side, your not alone, and I don't feel crazy (ok, I'm still crazy - My parents never tested me)
Chin up brother, welcome the the bleeding edge, drop back 10 Yards and send them a 9630G or 9650

Regards,
Mike