CMS v12 Synchronization Question

[AvayaHelp]

We have two CMS Servers v12. One Primary and one secondary. I have been told that when the Primary is down the Secondary will continue to collect the data. Once the Primary is back up there only needs to be a manual synchronization performed to ensure they both match up. Does the synch include the historical data?

S8700 & S8710 v12/CMIV/Octel 250/CMS v12/Intuity AUDIX/IP Agent/VoIP/Programming
==========================================================
Does anyone have a quarter, I need to make a phone call?

 

[RTMCKEE]

The sync would only include the historical data right? Once the real-time data has passed the current interval it becomes historical. I would assume that histrical is what the sync is reparing, since real-time does not need to sync.

RTMCKEE

CM 2.1.1
Prologix R9.05
Modular Messaging 1.1

 

[BurningChr0me]

I will let you know i'm installing 2 v12 HA-CMS on thursday all the documentation says that the manual sync does do this but we wait and see.

 

[AvayaHelp]

Not sure if this is the way to submit this document but I never check my internet email account as I use it for junk and I have no other account I can give you all if you wanted a copy of it so here it is.

I just found this, hope this helps you - note that the Admin operations synchronized by Auto-synch for Split/Skill Call Profile should have an asterix from Avaya. We found that although any changes you made to the Profile on the Primary does populate to the Secondary during an Auto-Synch, it will not be reflected in the reporting field on the Secondary. This is a known glitch by Avaya but there seems to be no resolution yet.



HA CMS Auto-Sync User’s Guide
(Provided by CMS Technical Integration Team – formerly Professional Services)


This guide explains how the Avaya CMS Technical Integration (TI) Team’s add-on software package, Auto-Sync, works and what it does.


WHAT IS HA CMS?

The term “HA CMS” is somewhat misleading. HA CMS is not truly a CMS feature. HA CMS is actually a PBX feature just as vectoring is actually a PBX feature. Purchasing HA CMS means that the HA CMS feature gets activated on the PBX. The activation of the HA CMS feature in the PBX enables the PBX to transmit identical data streams to two CMS’s simultaneously. The two identical, simultaneous data streams are how the call data is kept in synchronization within the two CMS databases. Auto-Sync keeps only the administrative data in synchronization.

There are no configuration changes done on the CMS. Each HA CMS is a totally independent, stand alone system that is unaware of the existence of the other CMS. The requirement for the CMS systems is that the two CMS’s be closely matched, i.e. similar hardware and identical CMS load. The two machines must be closely matched so that data and administrative information can be transferred back and forth. So HA CMS is simply two standard CMS machines connected to the same PBX. This discussion is continued later in this document in the section labeled “HA CMS and Add-On Interface Software”.


The Auto-Sync package adds a new CMS Main Menu Addition, HAusermenu. HAusermenu is the customer’s menu driven interface to the Auto-Sync software.


1) To run HAusermenu:
a) Login to CMS with the user ID ‘cms’
b) Enter: cms
c) Pick “HAusermenu>” from the Main Menu
d) Pick from the menu and follow the on-screen instructions


Example HAusermenu menu:

HA CMS User Menu – PRIMARY
for
Interface Software Activation/Deactivation

1) Configure this system as active (put into service)
2) Configure this system as inactive (take out of service)
3) Auto-Sync sub-menu
0) Exit
Selection?


The menu item terms active and inactive refer solely to the data transmissions from the CMS interfaces, e.g. ECH, RTA-IEX, etc. Both HA CMS’s maintain their PBX data links and collect and store call data regardless of their Auto-Sync status. See sections #5 and #6 below for further details.


Example Auto-Sync Sub-Menu:

Auto-Sync for HA CMS – Version 3.3

1) Schedule Auto-Sync
2) Un-Schedule Auto-Sync
3) Synchronize Administrative Data Now
4) Transfer & Restore Historic Call Data, (pull) FROM Secondary CMS
5) Transfer & Restore Historic Call Data, (push) TO Secondary CMS
0) Exit
===========
Choice ==>


2) Applicable administrative data (user logins, synonyms, etc.) is automatically synchronized nightly (see lists on a later page). The administrative data can also be synchronized on-demand via “HAusermenu”. Operations that allocate database space must be performed manually on both HA CMS machines, e.g. Data Storage Allocation.

NOTE: Operations that can be synchronized via tape backup and restore are automatically synchronized by Auto-Sync. Operations that cannot be synchronized via tape backup and restore are not (and can not be) synchronized via Auto-Sync (exception: UNIX passwords are synchronized).


3) CMS Supervisor created Report Designer reports and Custom Screen Painter reports will be synchronized as part of the nightly synchronization.


4) Timetables and Short Cuts will not be synchronized by Auto-Sync. This allows customers to load balance by scheduling different Timetable jobs on each HA CMS. The Shortcuts and Timetables are closely tied (coupled) together so the two can’t be separated for synchronization purposes.


5) When in the "normal" state the Primary is marked as "active" and the Secondary is not marked. When the Primary or Secondary is marked as "active" it sets a flag. If an add-on interface is duplicated on both CMS’s the flag will be examined by the interface. If the active flag is set the respective add-on interface will transmit data. If the “active” flag is not present the respective interface will not transmit data. If the interface software package is loaded on only one HA CMS the flag will not be examined.


6) The Primary is always the Primary and the Secondary is always the Secondary. What changes is their "active" status. Using "HAusermenu" you can individually mark them as "active" or "inactive". The Primary and Secondary should not both be marked as “active” at the same time (you’ll get duplicate data from ECH, etc.)

Auto-Sync notifies you if you try marking both CMS's as "active" at the same time (provided both CMS’s are up). You are prompted (Do you really want to do this?) before it lets you mark both CMS’s as “active”.


7) Auto-Sync Version 3.3 and later includes a “heartbeat” function. Approximately every 20 seconds the heartbeat function tests LAN/WAN communication between the Secondary and the Primary. If the Secondary cannot ‘talk’ to the Primary the Secondary will mark itself as “active”. When the Secondary marks itself as “active” it will call the startup programs for IEX-RTA, TCS-RTA and BP-RTA (if they are present & loaded on both HA CMS’s). The Secondary will also take over the ECH data feed (if present).

The heartbeat function applies only to total outages of the Primary. If the Primary has suffered a partial outage and can still respond to LAN/WAN communication requests from the Secondary the heartbeat function will not mark the Secondary “active”. In partial outage situations the customer must manually mark the Primary as “inactive” via HAusermenu if the customer desires that the Secondary take over the interface data feeds duties. In other words, if the customer can still log into both HA CMS’s and if they want the Secondary to be active they should mark the Primary as inactive. The heartbeat function will detect that the Primary is marked inactive and configure the Secondary as inactive.

The heartbeat function does not apply to the GeoTel interface (RTA-Geo). The details of the GeoTel interface in an HA environment are covered in a separate document.

The switch-over functions performed by Auto-Sync’s heartbeat feature do not apply to nor have any effect on user login sessions (e.g. CMS Supervisor logins). A prolonged total outage of the Primary HA CMS would necessitate that the users log into the Secondary HA CMS. A partial outage of the Primary would necessitate that only the affected users log into the Secondary. As an example, if only the PBX data link for ACD 3 goes down then only those CMS users that monitor ACD 3 would need to log into the Secondary (provided the ACD 3 data link is up on the Secondary).

When the Primary has had a total outage and is restored to operation the heartbeat function will restore the Secondary to inactive status and stop any interfaces that it started. If the customer manually marked the Primary as inactive because of a partial outage the Secondary will remain marked as active until the customer marks the Primary as active.


8) Historic call data (interval and daily tables) can be manually synchronized (one day at a time) via Auto-Sync menu items #4 and #5. This feature can be used to restore missing call data after one of the HA CMS boxes has been temporarily out of service. If one of the HA CMS boxes is out of service for an extended time period it is recommend that a standard tape backup/restore procedure be used to synchronize the two historic data bases.


9) The Historic data synchronization feature supports synchronizing ‘per ACD’ or “all” ACDs. The ‘per ACD’ feature facilitates support for HA CMS boxes that are in geographically different locations (disaster recovery model) and that have multiple ACDs.


10) R3V9 CMS (and above): The ACD Admin Log is unique to each HA CMS machine so synchronization is not applicable.
11) Configuration data (splits/skills to be monitored) for GeoTel, IEX-RTA, and TCS-RTA is synchronized nightly or on demand via “HAusermenu”.


12) Pseudo ACDs should be created/loaded only on the Primary CMS. Pseudo ACD administrative data is excluded from the Auto-Sync nightly and manual synchronization. When restoring call data from the Primary to the Secondary do not select “[all]” because it will copy the pseudo ACD data to the Secondary (needlessly using hard disk space).



Admin operations “automatically synchronized” by the switch/PBX

When CMS is used to make switch/PBX translation changes those changes will appear to be automatically synchronized between the Primary and Secondary CMS. This is because the changes are actually done to the switch/PBX translations. Those changes include:
a) changes to VDN Skill Preferences
b) VDN assignments
c) Vector contents
d) Multi-agent skill changes
e) Change Agent Skills



Admin operations requiring manual synchronization

The following operations cannot be synchronized between the two CMS servers via Auto-Sync. Instead, these operations must be performed manually on each CMS server.
a) Agent Trace Administration
b) Activate Agent Trace
c) Exception Administration
d) CMS Supervisor scripts
e) Scheduling of Timetables
f) Changing the CMS state
g) Data Storage Allocation changes
h) External Call History state
i) Load Pseudo ACD data
j) Pseudo ACD setup
k) Storage Interval changes
l) Turning data collection on and off
m) Data Summarizing
n) Call Work Code administration


Admin operations synchronized by Auto-Sync

The following CMS administration operations will be synchronized from the Primary to the Secondary HA server.
a) Custom Reports
b) CMS Supervisor Designer Reports
c) Data Dictionary operations
d) User Permissions
e) User passwords
f) User home directories
o) VDN Call Profile
g) Split/Skill Call Profile


Call Profile data is copied nightly from the Primary to the Secondary. However, changes to Call Profile monitoring do not take effect until the associated Informix dB tables are re-read by the associated process. The Call Profile parameters are read by the associated process when the system is booted, when CMS is stopped and restarted, and when a “Modify” is done to an individual Call Profile.

A Call Profile “Modify” may be done via a CMS Timetable task. If this method is utilized, a Timetable task is needed for each Call Profile (each skill & each VDN).

Avaya recommends that the Secondary HA CMS not be used for routine generation of Call Profile reports unless the Call Profiles are virtually static and any recent changes have been put into effect as discussed above.


Security

SSH (sftp) adds a non-trivial amount of overhead to data transmissions. The nightly synchronization of the CMS administrative data is both computer resource and network bandwidth intensive. Therefore, SSH is not supported by Auto-Sync. If increased security hardening is required, the recommended solution is TCP Wrappers.

Firewalls

It is expected that both CMS’s will be on the same side of the firewall in “normal” situations and configurations. This software package will work through a Screening Firewall provided the customer “opens” the relevant ports on their Screening Firewall. However, this software package will NOT work through a “Proxy Server” (sometimes referred to as a Proxy Firewall).


HA CMS and Add-On Interface Software


If you have any prior experience with disk mirroring, Sun Clustering or Solstice HA, put aside those concepts and ideas when thinking about HA CMS. The workings of HA CMS are totally different. However, as with Sun Clustering, current users of CMS are impacted during the failover period. Specifically, clients connected to the node on which the failure occurs are impacted. Those clients must reconnect to the surviving node. CMS Supervisor does not have an automatic reconnect function so the reconnection is manual (the user must login to the surviving node).

The call data on the two HA CMS machines does not have to be periodically synchronized because the two machines are receiving identical data streams from the PBX. In other words, the data is kept in sync by virtue of the fact that the two CMS’s are receiving identical data streams from the PBX.

However, changes that users (CMS administrators) make to the CMS system must be synchronized. The user could type their changes into both CMS machines. But that would be a constant source of problems because of typing errors (typos) and omissions.

The Auto-Sync package automates the transfer process of administrative changes. Auto-Sync transfers the administrative data via the customer’s LAN/WAN.


Avaya Add-On Interface Software in an HA CMS Environment

Software interface packages provided by the Avaya CMS Technical Integration Team (formerly Professional Services) may be loosely referred to as custom or semi-custom software. However, Avaya add-on interface software is actually Off-The-Shelf.

Third party vendor’s applications run on either a Windows NT/2000 Server or a UNIX server. So the term vendor’s server will be used in this document to represent a third party application such as TCS, IEX, NICE Analyzer, Blue Pumpkin, etc. (excluding Cisco/GeoTel – which is covered in a separate document).

If both HA CMS machines were to send call data to the vendor’s server the vendor’s server would receive the same data twice, i.e. double data. Therefore, the two HA CMS machines must have a mechanism to designate which CMS system is to send data to the vendor’s server. The CMS that is currently designated/configured to send data to the vendor’s server is referred to as the active CMS.

The two HA CMS machines must also be designated as a primary and a secondary. The primary vs. secondary designation applies to the physical CMS machines. The primary CMS is the one where the customer does their administration and configuration work. Since these are physical designations, the primary is always the primary and the secondary is always the secondary regardless of the scenario or operational status. The active designation is a logical designation that could apply to either physical CMS machine depending on operational status.

When Avaya add-on interface software is installed on HA CMS machines the customer receives a user interface (menu) named HAusermenu. The customer must manually run HAusermenu to designate which CMS is the active CMS (Avaya will run it the 1st time as part of the installation). The CMS that is configured as active is the CMS that sends data to the vendor’s server. If the primary CMS is in service by convention it is configured as the active CMS.

Avaya add-on software will look for a flag (special lock file) that was created by running HAusermenu. If the file is present that particular CMS system will send data to the vendor’s server. If the file is absent (not the active CMS) that particular CMS will not send data to the vendor’s server. Thus only one of the two HA CMS systems will send data to the vendor’s server at any given time.



SYNCHRONIZATION

Since the customer does their day-to-day administration (e.g. Data Dictionary entries) on the primary CMS the secondary CMS’s administrative database (dB) must be periodically synchronized with the primary’s administrative dB. The standard way to do this synchronization of administrative data is via a tape backup on the primary and a corresponding restore on the secondary. Since this procedure is very cumbersome, Avaya offers the Auto-Sync software package to automate this process. The features and usage of the Auto-Sync package are explained in the Auto-Sync User’s Guide.

If one of the HA CMS boxes is out of service it obviously does not receive call data (ACD data). After the disabled CMS is repaired it needs to receive a copy of the missing call data from the CMS that remained in service. Again, the standard way to do this synchronization of call data is a very cumbersome tape backup/restore process. The Auto-Sync package has a function (menu choice) that will transfer one day’s worth of call data from one HA CMS to the other via TCP/IP over the customer’s LAN/WAN. If three days of data are missing the customer would run the menu item three times, once for each missing day.

The call data synchronization cannot be run for the current day. As an example, if it’s Tuesday you can synchronize Monday’s call data but you can’t synchronize Tuesday’s data until at least Wednesday.

Neither the administrative data nor the call data synchronization process requires taking the CMS out of service (non service affecting).



FREQUENTLY ASKED QUESTIONS


Q: Do the HA CMS’s switch over automatically on failure?

A: Yes, if Auto-Sync is installed. If the primary CMS goes out of service, a heartbeat signal mechanism from the secondary detects the failure and runs HAusermenu on the secondary CMS to configure it as the active CMS. Thus enabling the data transfers to the various 3rd party vendor’s systems.

However, the heartbeat function only detects a total loss of service, e.g. a root hard disk failure. If a PBX data link goes down, but the CMS (Solaris) system is still running, the heartbeat function will not detect a failure.

Also,the CMS users will need to manually log into the CMS that is still in service. Therefore the user community needs to know the IP address or DNS name of both HA CMS machines.


Q: How often does Auto-Sync run?

A: Auto-Sync runs once a day around 2 or 3 AM (configurable). If the customer makes critical or extensive administrative changes and doesn’t want to wait until the automatic nightly synchronization they may manually run the administrative data synchronization via the Auto-Sync menu.


Q: If my primary CMS has a service outage and my split/skill supervisors switch over to (log into) the secondary CMS what restrictions are there?

A: To maintain control of changes, all CMS administration (with the exception of Timetables and Short Cuts) must be done on the primary CMS. However, user changes such as dynamically moving agents between skills can be performed on either CMS since those changes are stored on the PBX. In other words, configuration changes that are stored on the PBX can be done from either CMS. Configuration changes that are stored on the CMS must be done on the primary CMS only.


Q: Can users (split supervisors) be distributed across the two HA machines for load balancing?

A: The HA CMS product documentation states not to do this (it’s not condoned by Avaya Labs). However, there is no physical or logical reason to preclude doing this. If the primary CMS is heavily loaded it may even be advisable to distribute the load between the two systems. The only concern is that all administrative changes that are stored on CMS must be made on the primary CMS, e.g. Data Dictionary changes.


Q. Can Timetable jobs be distributed across the two HA machines for load balancing?

A. Again, load balancing is not officially condoned by Avaya Labs. However, there is no physical or logical reason to preclude doing this.


Q: Do I need duplicate copies of Avaya add-on interface software on the secondary HA CMS machine?

A: It depends of the software package. RTA-Geo and ECH handling software absolutely should be duplicated.

Duplicating Real Time Adherence packages such as IEX-RTA and TCS-RTA is strictly a business decision (Can you do without RTA data if the primary goes down?).

Historic data feeds such as IEX Historic and TCS Historic have a resend feature. When the primary CMS is down the secondary CMS collects and stores the historic call data. When the primary is put back into service the missed data can be transferred from the secondary to the primary. Then the missed IEX or TCS historic data can be sent to the vendor’s server via the resend function. So there is no compelling reason to duplicate IEX Historic or TCS Historic software.


Q: I understand that HA CMS means obtaining closely matched (duplicated) CMS machines. What other hardware has to be duplicated?

A: The CLAN cards in the PBX should always be duplicated, i.e. a separate CLAN card for each HA CMS. This prevents the CLAN card from being a single point of failure. Having duplicated PBX processors is also highly recommended.


Q: I have an existing SUN E3500 CMS connected to my PBX via an X.25 data link. When I add the second CMS to create HA CMS can I connect it to my PBX via the same X.25 card?

A: No. X.25 is not supported for HA CMS. All the connections must be TCP/IP over a LAN/WAN. The PBX and both CMS’s must all be able to ‘talk’ to each other via TCP/IP.

S8700 & S8710 v12/CMIV/Octel 250/CMS v12/Intuity AUDIX/IP Agent/VoIP/Programming
==========================================================
Does anyone have a quarter, I need to make a phone call?