backups fail 2

[newglide]

I have a few servers that the backups are starting to fail.
I logged into the cdom this morning and am seeing failures.
Ver 6.3 CM.

Her is what the log is showing.
2016-09-28 10:17:38,747|bdb_db_open: database "dc=vsp": unclean shutdown detected; attempting recovery.
2016-09-28 10:17:38,747|bdb_db_open: warning - no DB_CONFIG file found in directory /var/lib/ldap: (2).
2016-09-28 10:17:38,747|Expect poor performance for suffix "dc=vsp".
2016-09-28 10:17:38,747|bdb_db_open: database "dc=vsp": recovery skipped in read-only mode. Run manual recovery if errors are encountered.
2016-09-28 10:17:38,747|bdb(dc=vsp): Logging region out of memory; you may need to increase its size
2016-09-28 10:17:38,747|bdb_db_open: database "dc=vsp": db_open(/var/lib/ldap/id2entry.bdb) failed: Cannot allocate memory (12).
2016-09-28 10:17:38,747|backend_startup_one (type=bdb, suffix="dc=vsp"): bi_db_open failed! (12)
2016-09-28 10:17:38,747|slap_startup failed
2016-09-28 10:17:38,753|11369 finished
2016-09-28 10:17:38,865|kill -- -11367
2016-09-28 10:17:38,865|BACKUP ldap/backup.sh - Failed with exit code 1
2016-09-28 10:17:38,865|BACKUP ****** FAILED ******

 

[BENANDSERA]

Hey Newglide

Do you show any alarms on your server? Anything like a DLY-MTCE alarm?

 

[montyzummer]

Here is your issue

https://support.avaya.com/ext/index?page=content&id=SOLN146574&group=UG_ENTITLED_CUSTOMER

Follow this link and it shows you what to do to resolve , below is what the link has content wise in case you don't have an avaya SSO login.

DETAILS
System Platform 1.1.1.x.x. All versions before 6.0.3.10.3 are affected.Few instance fro 6.3
In cdom's /vspdata/backup/backup.log the following is seen:
2011-01-04 15:59:03,254|bdb(dc=vsp): Logging region out of memory; you may need to increase its size
2011-01-04 15:59:03,254|bdb_db_open: database "dc=vsp": db_open(/var/lib/ldap/id2entry.bdb) failed: Cannot allocate memory (12).
2011-01-04 15:59:03,254|backend_startup_one: bi_db_open failed! (12)
2011-01-04 15:59:03,254|slap_startup failed
2011-01-04 15:59:03,255|BACKUP ldap/backup.sh - Failed with exit code 1
2011-01-04 15:59:03,282|BACKUP ****** FAILED ******
PROBLEM CLARIFICATION
Scheduled and Manual System Platform backups fail.
CAUSE
System Platform uses LDAP for user authentication and this issue typically happens when the LDAP database on domain0 is corrupted.
SOLUTION
Per request solution is restricted to Entitled Partner, Internal & Internal Development.
First reported in SP 1.1.1.x.x.
Fix is delivered in 6.0.3.10.3 and is GA.
WI01020567.
RESTRICTED SOLUTIONS ELEMENTS
1) Log into System Platform dom0 as root (you will first need to log in as admin and su - root).
2) Check to make sure there is plenty of disk space available (if the disk is full this could cause further issues). This can be done using command "df -h" and checking the "Use%" column on the / partition (shown under "Mounted On" column) to make sure its not really high (like over 90%). If higher than that please open a ticket with Avaya services for advice. See sample image below:

3) *** Note if disk space is low and you follow next step you run the risk of corrupting the system. Please first determine what is taking up the disk space and that it is not LDAP or LADP log files. You will need approx 20 – 30 megs of free space for step to complete. If unsure open a Service Request via the Support.avaya.com web portal. *******
4) If there is plenty of disk space available, run the following command on dom0 to fix the LDAP Database: /usr/sbin/slapd_db_recover -v -h /var/lib/ldap
5) Then restart the LDAP using command: service ldap restart
The above commands are not service affecting so can be done without affecting template operations.
To permanently resolve this issue you should upgrade to the latest available System Platform version. There have been numerous improvements and updates in the underlying LDAP service which has reduced the chances of database corruption.
First reported in SP 1.1.1.x.x.
Fix is delivered in 6.0.3.10.3 and is GA.
This work around even worked on 6.3 version of the system platform.
WI01020567

ACSS (UC/SBCE/SM/SME)

Not that they mean a thing anymore , get a brain dump pass the test crash the system.

 

[montyzummer]

Also this link seems to have more verbatim info.

https://support.avaya.com/ext/index?page=content&id=SOLN221171&group=UG_ENTITLED_CUSTOMER

In cdom's /vspdata/backup/backup.log the following is seen:

|bdb(dc=vsp): Logging region out of memory; you may need to increase its size
|bdb_db_open: database "dc=vsp": db_open(/var/lib/ldap/id2entry.bdb) failed: Cannot allocate memory (12).
|backend_startup_one: bi_db_open failed! (12)
|slap_startup failed
|BACKUP ldap/backup.sh - Failed with exit code 1
|BACKUP ****** FAILED *****
check service ldap status, the service is down.
Tried to restart ldap service ,it prompted "cannot allocate memory"
PROBLEM CLARIFICATION
This issue has been reported on System Platform 6.0.3.0.3 and 6.0.3.6.3 and 6.3.1.08002.0 **Issue also found on System Platform 6.0.3.4.3 using HA, 6.0.1.0.5,6.3.4.08011.0 and 6.3.5.01003.0, 6.3.6.01005.0 HA and 6.3.7.0.05001
CAUSE
LDAP Database is corrupted.
First reported in SP 1.1.1.x.x.
Also can see below error after LDAP database recovered.
2016-08-02 21:01:11,554|+ runuser -G vsplog vspvm -- ./.backup -d /tmp/tmp.OiDiS10339/backup_matrixcdom_2016_08_02_21_00_00/vm-data/Midsize_Ent/smgr
2016-08-02 21:01:11,555|+ /opt/avaya/vsp/util/logger.py -f /opt/avaya/vsp/backup/scripts/log-vm.conf
2016-08-02 21:01:11,555|./backup.sh pid is 11047
2016-08-02 21:01:16,666|11047 finished
2016-08-02 21:01:16,768|kill -- -11045
2016-08-02 21:01:16,768|BACKUP vm-data/backup.sh - Failed with exit code 126
2016-08-02 21:01:18,033|BACKUP ****** FAILED ******
Which resolved once we re-scheduled the backup at different time.


SOLUTION
Work around is restricted to Entitled Partner, Internal. Entitled customers can open a ticket with Avaya to have LDAP corruption fixed.
For VSP 6.03: Fix originally delivered in 6.0.3.10.3. WI01020567
FOR VSP 6.3.X: The problem appears to be corruption and the restricted solution works. If this is repeatable, may need to rebuild the server and restore from backup. Open a case to consult an Avaya engineer
LDAP corruption has been seen in other lesser known ways also. If the above markings are present AVAYA support fix should be the same.
RESTRICTED SOLUTIONS ELEMENTS
It looks like, SMGR/dom0/cdom is running out of Memory. Suspecting either of these two below causes…

a. There is LDAP corruption in the system.
b. The disk space on the system has become full or above threshold. You can check the disk usage from command line “ df –h”

2. Check disk space: Disk space looks normal. Only 22% used

3. Restart ldap to clear ldap corruption (service ldap restart)

To resolve the issue please perform the following procedure (which is non service affecting):
1. Log into System Platform dom0 as root (you will first need to log in as admin and su - root). Determine the cause of the LDAP Status being stopped: We should not multiple accounts returned in the results:
Example of the failure:
root@server #> slapcat
/etc/openldap/slapd.conf: line 121: rootdn is always granted unlimited privileges.
bdb_db_open: database "dc=vsp": unclean shutdown detected; attempting recovery.
bdb_db_open: database "dc=vsp": recovery skipped in read-only mode. Run manual recovery if errors are encountered.
bdb_db_open: database "dc=vsp" cannot be opened, err 22. Restore from backup!
backend_startup_one (type=bdb, suffix="dc=vsp"): bi_db_open failed! (22)
slap_startup failed
Example of working slapcat results:
[root@testvsp ldap]# slapcat
/etc/openldap/slapd.conf: line 123: rootdn is always granted unlimited privileges.
dn: dc=vsp
dc: vsp
objectClass: top
objectClass: domain
structuralObjectClass: domain
entryUUID: 6cd91c6c-5ac3-43e2-9e1a-ee8809683b8a
creatorsName: cn=manager,dc=vsp
createTimestamp: 20140801132136Z
entryCSN: 20140801132136.358796Z#000000#000#000000
modifiersName: cn=manager,dc=vsp
modifyTimestamp: 20140801132136Z
... There will be many of these accounts listed.
2. Check to make sure there is plenty of disk space available (if the disk is full this could cause further issues). This can be done using command "df -h" and checking the "Use%" column on the / partition (shown under "Mounted On" column) to make sure its not really high (like over 90%). If higher than that please open a ticket with Avaya services for advice.
3. Note if disk space is low and you follow step 4, you run the risk of corrupting the system. Please first determine what is taking up the disk space and that it is not LDAP or LDAP log files. You will need approx 20 – 30 megs of free space for step 4 to complete. If unsure open a Service Request via the support.avaya.com web portal.
4. As root run the following command (after su - root): /usr/sbin/slapd_db_recover -v -h /var/lib/ldap
5. Then run the command: chown -R ldap:ldap /var/lib/ldap
6. Then execute the following command: service ldap restart
7. Execute the following command to run a backup: /opt/avaya/vsp/backup/bin/backup.sh

ACSS (UC/SBCE/SM/SME)

Not that they mean a thing anymore , get a brain dump pass the test crash the system.