OPS manager V7.6.3.7 directory syncing problem

[slapin]

Today I've started OPS/ENT manager upgrade. Decided to build the system from scratch on Win2003 platform. Installation went through without any issues. System discovery was successfull as well. But the next step was to get Network elements directory synced with the new OPS manager.

Here is the problem. I start full collect from all elements. Seems that everything is OK, but when I open directory in the Directory Editor or Change/Move form the only extensions with Home Element assigned are the last imported. Tried to make full sync, the same result. Full collection from a single element overwrites entire directory, so external extensions for this element loose their home element assignments.

I have got various 3300 boxes with lowest version of the system software 7.0.9.5 and highest 8.0.7.16_2

Any ideas?

 

[MitelInMyBlood]

Are the "external extensions" outside the cluster or do you mean they are simply on different network elements? Also unsure if I understand you regarding full collect. Are you saying performing a full collect from one single network element overwrites the entire OPS directory?

I would not attempt to perform a full collect from all network elements at once. Do them one at a time and at least 10 minutes apart, same for full propagate. Suck it all in, one system at a time and blow it all out, again to only one box at a time.

Like the big red bumper sticker said atop the old SX2000-SG in the I&M lab at Boca Raton: "PATIENCE"

You are apt to find the Administrator's Guide is a far better (clearer, more concise) tool to walk you through a new system setup from scratch than the Tech handbook. It literally spoon-feeds it.

 

[slapin]

It overwrites the previous full collect. I've tried to recover ops manager backup from the old version. It installed it successfully, but the result was that templates were screwed up and not working now. In fact I cannot make any changes as it fails to use default change template and I cannot even edit it. I think that not the only one problem after recovery.
Right now I'm dropping the database and will try to manually add systems to enterprise manager one after one and make one full collect every time.

 

[slapin]

Ok. I have reinstalled the whole system. Manually added one system to Enterprise manager. Synced with OPS manager, and configured ther rest of the settings in the OPS manager. Collected full directory. In change/add form got extensions which belong to the first element. Created distribution list with the same name as it

Added the second element to the Enterprise manager, synced with OPS manager and configured it. Full collect for the second element came out the way so I have extensions for the first and the second elements in the directory, but home element field populated for extensions which belong to the second element only. PFM...

 

[MitelInMyBlood]

Sounds to me like time to get on the phone to Product Support. Sorry I have no idea where else to go at this point.

 

[Lundah]

When you collect from the 2nd node (and each following), don't add it to the PDG before you do the collect. Do it like this:

1. Create PDG and add 1st node; perform full collect from it.
2. Perform full collect from 2nd node; add it to the PDG AFTER the collect is complete, then go into TelDir Editor and change the dist. list for all of the new entries you just collected, since the collect will toss them into a new system-generated dist. list.
3. Rinse, repeat.

I built a cluster of 100 nodes and 15,000 entries this way, one node at a time.

 

[Lundah]

Whoops, I forgot a step:

2.5: Synch to the PDG members

 

[slapin]

It's not PDG. This elements are not in the cluster.
I got into the MySQL database and see that HOMENODE_ID changing to 0 for all "external" directory records collected from the element. All other information including MAC addresses, location IDs, device types is untouched.
Looks like I'm missing something very little.

 

[Lundah]

What if you manually corrected the Home Element field? Like with a SchedMAC import or something? Hell, you could probably just write a MySQL script to do it if you really wanted.

 

[slapin]

If I correct the field manually the record appeares in the Change/Move/Delete application. I really don't want to use workarounds when the software has to work as intended. Besides that I have noticed that Voicemail information is not being imported. I don't think that Mitel testing team will miss that major problem, so I assume that I'm doing something wrong. Already contacted my vendor, but so far there is no resolution.

Let me give you better description what I'm trying to do.

I've got Enterprise manager V.3 running on Win2000 platform.
It turned to be that the old OPS manager cannot work with ICP V8 any more and I had to make an upgrade.
Since this was the last W2K system in the entire company I have decided to upgrade the system as well.
Installed Win 2003 R2 server.
Logged into the system with account equal to windows domain admin.
Called my vendor and got AMC cleared.
Started installation of the Enterprise Manager.
Obtained license keys from AMC
Finished EM installation and restarted the system.

In fact I was going to backup and restore config from the old EM/OPS managers. According to documentation it should be able to convert the database and config files as needed. And failed. The procedure went OK but nothing worked after that.

I uninstalled everthing, saved AMC keys in a secure place. Copied the database and all files from the old EM/OPS managers to the new system. Started installation using floppy disk(offline) licensing using saved keys. Upgrade went OK, the database appeared to be migrated but I couldn't make any chages because of some problems with default change/move/delete templates. Template editing didn't work as well. And I didn't try to sync using this new database with issues.

I don't want to take chances and make an upgrade of the existing OPS manager which is still operational for all 3300 boxes with SW V.7

What could be the possible problem in this backup/restore based upgrade procedure.

 

[Lundah]

So you're trying to take backups from an older version of EM/Ops and restore them to a fresh install of a newer version, do I have that right? That WILL lead to problems, particularly with the Ops MAC templates. Install the same version on the new server, move your databases, THEN upgrade to the final release you wish to use.

 

[slapin]

Yeah, feels like I'll need to try that option. I'll drop a meaasage how it will go

 

[slapin]

It's not over yet. Observing strange activities.
So far I went through following steps.

1. Created a VMWare clone of the existing OPS manager server. Enterprise manager refused to work because it couldn't validate the licence but MySQL was alive.
2. Upgraded Win2000 to Win2003 on this VM.
3. Disconnected VM network adaptor and adjusted its MAC address so it will match the MAC address of the new enterprise manager server where I ran AMC license checkuot.
4.Started Enterprise manager installation and used offline licensing option. Selected Upgrade option. Setup converted the old EM database. Restarted the VM.
5.Started OPS manager installation and selected Upgrade option. Setup successfully converted the database. Restarted the VM. After that I got upgraded system.
6. Startred EM client on the local host. sheduled and created an EM backup. Started OPS manager client from there and created OPS manager backup. Extracted backup files from the VM disk and placed on the new EM server. Reinstalled all the stuff and restored beckups into the fresh database.


From the first look everything was OK.

The first Full Collect from one of the elements erased Home element field for all "external" records. First sync after full collect didn't recover anything but raised pending collection flag on ONE of the elements. The second sync recovered home_element ID for some of the nodes but not for all from unknown source. I have checked the database between sync's.

PFM.

P.S. Still no resolution from MITEL. Good thing that I got a clean backup.

 

[Lundah]

I'm confused as to why you're still trying to do a full collect. If the whole point of this exercise was simply to upgrade EMan/Ops, and you're able to do that, what's the full collect for? If you don't collect from ALL managed nodes, you will have issues with non-local entries.

 

[slapin]

From ALL at the same time? MitelInMyBlood suggesten not to do so. Any way Full collect has to work regardless of situation. I'll try to full collect all nodes in the same dist list and then sync. Or full propagation might be better option?

 

[Lundah]

The thing is, since you aren't clustered, Ops doesn't know what the home element is for the non-local entries that are collected, so it make sense that their Home Element data field is getting blanked out.

Check the Telephone Directory Assignment form for the element you're doing the full collect from, if the non-local entries are all showing up as "External", then Ops is doing exactly what it should with the collected entries. There's no CEID data collected from the PBX, so how would Ops know how to populate that field with anything else?

Doesn't help your situation out too much, but I don't have enough experience with your scenario to be much help for you, sorry.

 

[slapin]

Lundah,
I appreciate your input. The only thing is that the previous version didn't overwrite this field. I'm positive because I have done that many times. This one does not overwrite anything else except home element ID and cluster ID. Full collect from all nodes still in progress so there is no output yet

 

[Lundah]

how many nodes, and how many TelDir entries are we talking about here?

Keep in mind that a full collect is going to generate delta records for every entry on every node you collect from, and you're limited to a max of 150k deltas for the whole thing. So if you have 150 nodes, you'd be limited to 1000 entries per node before the delta table is full and thing get really interesting. That's why MIMB recommended doing the collection one at a time.

 

[slapin]

Not that much. In fact since I have a working backup I'm trying to figure out the procedure or what I'm doing wrong.
Just tested full collect from the old server. No problems. Nothing got damaged. So it's eather bug or new feature.

 

[slapin]

Lundah,
You were partially right. In fact I got full directory collected and synced.
1. Full Collect from all elements. Totall mess in the directory. Last collected element available for move/change. Everything else 0. Lots of deltas
2. Sync with elements. Home elements not recovered. One of PDG members raised pending collection flag.
3. Second sync recovered just 10 elements.
4 Another Full collect from all elements. Again only the last collected element left in the directory plus some records from other elements. Number of deltas changed...
5. Sync with elements. Same picture. One of PDG raised collection flag.
6. Final sync came out with all elements recovered. Didn't check records count yet but I think it would be easy to fix minor errors manualy.

I have got no explanations. There is a chance that conflicts resolution method gives unreliable results. According to engineering guide it depends on the sequence how records get in the delta table. Or there is a bug in this version and it deletes wrong records during the conflict resolution pass.

Any way it's up and running now