NetBackup too verbose

[kidswipe]

Hi, I'm trying to see if anybody else is running into the same issue that I am having right now. Currently Netbackup sends multiple messages that are generated for a single server about the same issue/failure/completion. Granted I should be happy that it is sending me messages and notifying me but its sort of a pain having to sort through 4 or 6 of the same messages from a single server several dozen times. My goal is to try and get it down to 1 status message per server.

any and all help is appreciated, Thank You.

FYI I'm running Netbackup Enterprise Server 6.0 MP6 on a windows 2003 server.

 

[Stumpr]

DOCUMENTATION: Best Practice recommendations for enabling and gathering Veritas NetBackup (tm) 6.0 logging

http://seer.entsupport.symantec.com/docs/282865.htm

Details:
Manual: Veritas NetBackup (tm) 6.0 Troubleshooting Guide for UNIX and Windows, Chapter 3: Using Logs and Reports

Modification Type: Supplement

Modification:
For every case that comes into Symantec Enterprise Technical Support, there is a significant percentage that requires the collection of log files for troubleshooting. Starting with NetBackup 6.0 there are two styles of NetBackup logging. The previous logging method of creating a log directory for each daemon and enabling logging by changing the verbose setting is now called legacy logging. This method of logging is still used by some daemons within NetBackup.

NetBackup 6.0 also introduces a new style of logging, which is referred to as unified logging (VxUL). Unified logging is used by the Intelligent Resource Manager (IRM) and Enterprise Media Manager (EMM) services. This new logging method is enabled by default and can generate significantly larger log files, especially at higher debug levels. This logging method will keep logs based on the same setting used for legacy logging which is a default of 28 days.

Due to the changes in NetBackup 6.0 there are additional considerations for planning how to enable and gather logging. Enabling logging without the proper precautions can cause NetBackup to consume all available disk space on the system. This can cause problems for NetBackup and the operating system itself. One common problem that has been seen is logging consuming all the space and causing corruption of the EMM database.

The EMM server now uses a relational database to store media and device management information. If the system runs out of disk space this can potentially corrupt the EMM database and require that NetBackup be restored from the last good catalog backup. Depending on when this was last run, this could lead to the loss of valid backup images.

I. Setup of log directories
When setting up logging directories different approaches must be taken for legacy and unified log files. Symantec recommends that log files be moved to a dedicated file system. If this is not possible, then log files should be moved to a location other than the root file system or the file system on which NetBackup or other production data resides.

There are two considerations for selecting a partition to store log files: enough space to contain required logs, and not filling up a production file system. Since some of the logs are quite large, care should be taken to ensure that the file system is large enough to contain required logs. If the file system containing your image database or the EMM database is full, there is a possibility that you could lose data (as described in TechNote 282805, found below in the Related Documents section). It is therefore recommended that a separate file system, at least 32 GB in size be used, and it should be mounted or linked to the appropriate log directory. There are some log directories in that location, and those should be removed before mounting the new file system.

On UNIX systems:
Legacy logging is stored in: /usr/openv/netbackup/logs/<process>
Unified logging is stored in: /usr/openv/logs

On Windows systems:
Legacy logging is stored in: <install_dir>\NetBackup\logs\<process>
Unified logging is stored in: <install_dir>\NetBackup\logs

To move legacy log files on a UNIX system it will be necessary to make /usr/openv/netbackup/logs a symbolic link to another file system.
- It will be necessary to stop the NetBackup daemons to prevent any updates to the log files while they are being moved. This should be done when there are no active backups, restores, duplications, etc.
# /usr/openv/netbackup/bin/goodies/netbackup stop

- Move the logs directory to a dedicated file system.
# mv /usr/openv/netbackup/logs /disk3/netbackup/logs

- Make a symbolic link to the new log directory
# ln -s /disk3/netbackup/logs /usr/openv/netbackup/logs

- Verify the logs directory is accessible.
# cd /usr/openv/netbackup/logs
# ls -l
This should list the various log directories that have been created on the server.

- Restart the NetBackup daemons to begin using the new log directory.
# /usr/openv/netbackup/bin/goodies/netbackup start

To move legacy log files on a Windows system refer to TechNote 273593, found below in the Related Documents section.

To move unified log files to a different file system refer to TechNote 281360, found below in the Related Documents section. The NetBackup 6.0 vxlogcfg command can be used to change the default logging location for the originators that use unified logging. This procedure is the same for both UNIX and Windows systems.

II. Enable "verbose" NetBackup logging
There are different methods to enable legacy and unified logs in NetBackup 6.0.

For legacy logging the verbose level is controlled by the VERBOSE setting in the /usr/openv/netbackup/bp.conf on UNIX systems or the Verbose setting in the Windows registry. Legacy logging also requires that log directories be created for each daemon or process individually. Unless directed by Symantec Technical Support do not use a higher debug level.
To lower the debug level for legacy logging do the following:
1. Launch the NetBackup Java Administration console and go to Host Properties > Master servers
2. Select the master server and bring up the Properties window
3. Go to the Logging section and set Global logging level: to 0 (minimum)
4. Click OK to save the changes. It will be necessary to restart NetBackup for these changes to take effect.

For unified logging do not use a higher debug level, unless troubleshooting issues with Symantec Enterprise Technical Support. The following commands can be used to set the debug and diagnostic levels to three for unified logging. This should some basic troubleshooting information without filling up the disk as quickly as higher debug levels. To set the debug level for unified logging run the following commands:

For UNIX systems:
# cd /usr/openv/netbackup/bin
# ./vxlogcfg -a -p NB -o Default -s DebugLevel=3 -s DiagnosticLevel=3

For Windows systems:
% cd <install_dir>\NetBackup\bin
% vxlogcfg -a -p NB -o Default -s DebugLevel=3 -s DiagnosticLevel=3

Note: The above commands will enable debug and diagnostic unified logs. It will be necessary to restart NetBackup for these changes to take effect.

III. Configuring log file rotation
Set the Keep logs configuration setting to a lower value. The default is to keep 28 days of debug logging information. Setting this to a lower setting, such as 3 to 5 days, will result in fewer log files being kept on the server.
To change this setting via the NetBackup Java Administration Console:

- Go to Host Properties > Master servers and bring up the properties for the master server.
- Select Clean-up and change the value for Keep Logs from the default of 28 days to a value of 5 days.
- Click OK to save the changes.
- It is necessary to restart NetBackup for these changes to take effect.

To change this setting via the command line on UNIX Systems:
# /usr/openv/netbackup/bin/admincmd/bpconfig -kl 5
It is necessary to restart NetBackup for this change to take effect.

To change this setting via the command line on Windows systems:
% <install_dir>\NetBackup\bin\admincmd\bpconfig -kl 5

Note: It is necessary to restart NetBackup for this change to take effect.

IV. Saving logs for future reference
It may be necessary to have an archive of older logs in order to troubleshoot long-running problems. A good practice would be to create a policy that will back up older logs before they are removed, and retain those logs for a time that is slightly greater than, or equal to, the time between your average full backups. For example, if you run a full backup once a month, then it would be a good idea to retain at least a month's worth of logs for reference.

Refer to the System Administrator's Guide for information on creating a policy. The policy will need to be active, and have a User Backup schedule. It would also be a good idea to keep the volumes used for these backups in a separate pool, so they don't get used for regular backups. Then a shell script could be run from cron to run a user backup everyday to back up older logs. The following script is provided merely as an example, and is not officially supported:

#!/bin/sh

DATE=`date +%m%d`
FILELIST=/tmp/flist.$DATE
LOGFILE=/tmp/log_backup.$DATE

/bin/find /usr/openv/logs -atime 6 > $FILELIST

/usr/openv/netbackup/bin/bpbackup -p $policy -L $LOGFILE -f $FILELIST

[ $? == 0 ] && rm -f $FILELIST $LOGFILE

V. Summary of NetBackup logging
Newer features in the NetBackup 6.0 release require additional considerations when enabling and gathering logging. Features such as the EMM database can become corrupt if proper care isn't taken to prevent the file system from running out of disk space.

Making changes mentioned in this TechNote will make it easier to keep track of NetBackup logs, provide enough detail to troubleshoot problems, and minimize the effects of logging on the normal operation of a server.

Related Documents:

273593: How to move NetBackup log files to a mount point on a Windows 2000 and 2003 server

http://support.veritas.com/docs/273593

279295: VERITAS NetBackup (tm) 6.0 Troubleshooting Guide for UNIX and Windows

http://support.veritas.com/docs/279295

281360: DOCUMENTATION: How to configure Veritas NetBackup (tm) 6.x to write unified log files to a different directory

http://support.veritas.com/docs/281360

282805: All Veritas NetBackup (tm) 6.0 logging should reside on a dedicated file system to prevent corruption of the NetBackup Enterprise Media Manager database when a disk full condition occurs.

http://support.veritas.com/docs/282805

Bob Stump
VERITAS - "Ain't it the truth?"

 

[kidswipe]

Thanks bob... but its not exactly what i'm talking about, my fault for not being detailed enough. What I have going on, is that netbackup sends me emails when ever it tries to start a backup and when it fails. Whats happening is that, sometimes the backup will fail and it tries again, and will continue to fail and in the time slot I had given it to run it will keep trying. But each time it tries I get a message that its trying and each time it fails I get a message that it fails.

What I'm hoping to find is a way to just be alerted at the end of the time period that hey, it tried this many times and it finally was successful or that it completed failed in just one email alert and not be sent an email every time it tries and fails or succeeds.

hopefully thats a little clearer.

 

[RustyMajor]

You can't get what you're asking without setting up and running a script. You are getting an email for each failure. Each time the job retries and fails, you are going to get an email, whether that's one or 10.