Backups fail every Friday

[jfilmore]

My friday backups have been failing every friday night. I have narrowed down that there must another process conflicting with backup exec on friday night. (replaced tapes, backup to disk) Only 1 backup job, same for every night.

I have reviewed all windows event viewer logs with no ideas. Also reviewed backup exec logs and job history with nothing.

If there is anyone out there who might be able to help me with this, please let me know.

 

[LawnBoy]

The job log will give some kind of clue as to why the job stopped. What does it say?


--
The stagehand's axiom: "Never lift what you can drag, never drag what you can roll, never roll what you can leave.

 

[jfilmore]

Job Log: BEX_WCCEXCHANGE_01264.xmlJob Log for Full Backup



Completed status: Failed Expand AllCollapse All
Job Information
Job server: WCCEXCHANGE
Job name: Full Backup
Job started: Saturday, September 13, 2008 at 10:00:04 PM
Job type: Backup
Job Log: BEX_WCCEXCHANGE_01264.xml


Drive and media mount requested: 9/13/2008 10:00:05 PM
Device and Media Information
Drive and media information from media mount: 9/13/2008 10:00:06 PM
Drive Name: QUANTUM 2
Media Label: DLT000015
Media GUID: {a5116bd7-384f-4564-a569-ab8fb7bfc55c}
Overwrite Protected Until: 12/31/1899 7:00:00 PM
Appendable Until: 12/30/9999 7:00:00 PM
Targeted Media Set Name: Media Set 1


Job Operation - Backup
Backup Options
Media operation - overwrite.
Hardware compression enabled.
WARNING: The option 'Verify after backup completes' was not selected.
Performing a verify operation to make sure that media can be read after the backup has completed is recommended.



Server - WCCEXCHANGE
Set Information - C:
Backup Set Information
Family Name: "Media created 9/13/2008 10:00:05 PM"
Backup of "C: "
Backup set #1 on storage media #1
Backup set description: "Full Backup"
Backup Type: Full - Back Up Files - Reset Archive Bit



Backup started on 9/13/2008 at 10:00:26 PM.
Backup Set Detail Information
The item C:\Program Files\Common Files\Symantec Shared\EENGINE\EPERSIST.DAT in use - skipped.
The item C:\Program Files\Exchsrvr\bin\store.fcl in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\AP0.LOG in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\BF0.LOG in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000003.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000005.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000008.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000009.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db00000a.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db00000b.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db00000c.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db00000d.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db00000e.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db00000f.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000010.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000011.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000012.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000013.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000014.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000015.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000016.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000017.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000018.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000019.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db00001a.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db00001b.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db00001c.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db00001d.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db00001e.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db00001f.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000021.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000022.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\DB000027.DAT in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\DBREFS in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\EV0.LOG in use - skipped.
The item C:\Program Files\Exchsrvr\WCCEXCHANGE.log\20080914.log in use - skipped.
The item C:\Program Files\Microsoft SQL Server\MSSQL$BKUPEXEC\LOG\ERRORLOG in use - skipped.
The item C:\Program Files\VERITAS\Backup Exec\NT\Logs\adamm.log in use - skipped.
The item C:\Program Files\VERITAS\Backup Exec\NT\Logs\SGMon.log in use - skipped.
The item C:\Program Files\VERITAS\Backup Exec\NT\Logs\WCCEXCHANGE-bengine00.log in use - skipped.
The item C:\Program Files\VERITAS\Backup Exec\NT\Logs\WCCEXCHANGE-beremote00.log in use - skipped.
The item C:\WINDOWS\WindowsUpdate.log in use - skipped.
The item C:\WINDOWS\system32\h323log.txt in use - skipped.
The item C:\WINDOWS\system32\LogFiles\HTTPERR\httperr4.log in use - skipped.
The item C:\WINDOWS\system32\LogFiles\W3SVC1\ex080914.log in use - skipped.


Backup completed on 9/13/2008 at 10:09:19 PM.
Backup Set Summary
Backed up 21876 files in 3049 directories.
45 items were skipped.
Processed 4,263,653,337 bytes in 8 minutes and 53 seconds.
Throughput rate: 458 MB/min




Set Information - E: DATA
Backup Set Information
Family Name: "Media created 9/13/2008 10:00:05 PM"
Backup of "E: DATA"
Backup set #2 on storage media #1
Backup set description: "Full Backup"
Backup Type: Full - Back Up Files - Reset Archive Bit



Backup started on 9/13/2008 at 10:09:22 PM.
Backup completed on 9/13/2008 at 10:21:09 PM.
Backup Set Summary
Backed up 30053 files in 2299 directories.
Processed 32,699,588,629 bytes in 11 minutes and 47 seconds.
Throughput rate: 2647 MB/min




Set Information - \\WCCEXCHANGE\Microsoft Exchange Public
Folders
Backup Set Information
Family Name: "Media created 9/13/2008 10:00:05 PM"
Backup of "\\WCCEXCHANGE\Microsoft Exchange Public Folders "
Backup set #3 on storage media #1
Backup set description: "Full Backup"
Backup Type: Full - Back up messages - Reset archive bit
Microsoft Exchange Server Agent: Started



Backup started on 9/13/2008 at 10:21:10 PM.
Backup completed on 9/13/2008 at 10:21:23 PM.
Backup Set Summary
Backed up 926 mail messages in 6 folders in 0 mailbox(es)
Processed 5,603,502 bytes in 13 seconds.
Throughput rate: 24.7 MB/min




Set Information - \\WCCEXCHANGE\Microsoft Information
Store\First Storage Group
Backup Set Information
Family Name: "Media created 9/13/2008 10:00:05 PM"
Backup of "\\WCCEXCHANGE\Microsoft Information Store\First Storage Group "
Backup set #4 on storage media #1
Backup set description: "Full Backup"
Backup Type: Full - Database & Logs (flush committed logs)
Microsoft Exchange Server Agent: Started



Backup started on 9/13/2008 at 10:21:23 PM.
Backup Set Detail Information


Backup completed on 9/13/2008 at 10:30:49 PM.
Backup Set Summary
Backed up 2 Exchange Server stores
Backed up 3 Exchange Server logs
Processed 27,304,370,976 bytes in 9 minutes and 26 seconds.
Throughput rate: 2760 MB/min




Set Information - \\WCCEXCHANGE\Microsoft Exchange Mailboxes
Backup Set Information
Family Name: "Media created 9/13/2008 10:00:05 PM"
Backup of "\\WCCEXCHANGE\Microsoft Exchange Mailboxes "
Backup set #5 on storage media #1
Backup set description: "Full Backup"
Backup Type: Full - Back up messages - Reset archive bit
Microsoft Exchange Server Agent: Started



Backup started on 9/13/2008 at 10:30:50 PM.
Backup Set Detail Information
WARNING: "\\WCCEXCHANGE\Microsoft Exchange Mailboxes\Dawn Bove [dbove]?Top of Information Store?Inbox?Dawn????????????????????????????????????????????????????????????????????????Delivery Status Notification (Failure)" is a corrupt file.
This file cannot verify.
WARNING: "\\WCCEXCHANGE\Microsoft Exchange Mailboxes\Squash [squash]?Top of Information Store?Inbox????????????????????????????????????????????????????????????????????????Delivery Status Notification (Failure)" is a corrupt file.
This file cannot verify.
WARNING: "\\WCCEXCHANGE\Microsoft Exchange Mailboxes\Squash [squash]?Top of Information Store?Inbox????????????????????????????????????????????????????????????????????????Delivery Status Notification (Failure)" is a corrupt file.
This file cannot verify.


Backup completed on 9/14/2008 at 1:48:09 AM.
Backup Set Summary
Backed up 212951 mail messages in 2533 folders in 71 mailbox(es)
3 corrupt files were backed up
Processed 23,235,783,403 bytes in 3 hours, 17 minutes, and 19 seconds.
Throughput rate: 112 MB/min






Tape Alerts

----- TapeAlert Events -----

The device has reported the following TapeAlert diagnostic information.
Please contact your device manufacturer for more information regarding TapeAlerts.


Information - You cannot eject the cartridge because the tape drive is in use.
Wait until the operation is complete before ejecting the cartridge.
Robotic Library:
Drive: QUANTUM 2
Slot:


Warning - You have loaded a cartridge of a type that is read-only in this drive.
The cartridge will appear as write-protected.
Robotic Library:
Drive: QUANTUM 2
Slot:



Job Completion Status
Job ended: Sunday, September 14, 2008 at 1:50:38 AM
Completed status: Failed
Final error: 0xe00084ef - The media is write protected.
Final error category: Backup Media Errors

For additional information regarding this error refer to link V-79-57344-34031

Exceptions
Click an exception below to locate it in the job log
Backup- C: The item C:\Program Files\Common Files\Symantec Shared\EENGINE\EPERSIST.DAT in use - skipped.
The item C:\Program Files\Exchsrvr\bin\store.fcl in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\AP0.LOG in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\BF0.LOG in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000003.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000005.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000008.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000009.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db00000a.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db00000b.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db00000c.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db00000d.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db00000e.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db00000f.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000010.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000011.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000012.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000013.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000014.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000015.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000016.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000017.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000018.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000019.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db00001a.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db00001b.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db00001c.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db00001d.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db00001e.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db00001f.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000021.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\db000022.dat in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\DB000027.DAT in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\DBREFS in use - skipped.
The item C:\Program Files\Exchsrvr\mtadata\EV0.LOG in use - skipped.
The item C:\Program Files\Exchsrvr\WCCEXCHANGE.log\20080914.log in use - skipped.
The item C:\Program Files\Microsoft SQL Server\MSSQL$BKUPEXEC\LOG\ERRORLOG in use - skipped.
The item C:\Program Files\VERITAS\Backup Exec\NT\Logs\adamm.log in use - skipped.
The item C:\Program Files\VERITAS\Backup Exec\NT\Logs\SGMon.log in use - skipped.
The item C:\Program Files\VERITAS\Backup Exec\NT\Logs\WCCEXCHANGE-bengine00.log in use - skipped.
The item C:\Program Files\VERITAS\Backup Exec\NT\Logs\WCCEXCHANGE-beremote00.log in use - skipped.
The item C:\WINDOWS\WindowsUpdate.log in use - skipped.
The item C:\WINDOWS\system32\h323log.txt in use - skipped.
The item C:\WINDOWS\system32\LogFiles\HTTPERR\httperr4.log in use - skipped.
The item C:\WINDOWS\system32\LogFiles\W3SVC1\ex080914.log in use - skipped.
Backup- \\WCCEXCHANGE\Microsoft Exchange Mailboxes WARNING: "\\WCCEXCHANGE\Microsoft Exchange Mailboxes\Dawn Bove [dbove]?Top of Information Store?Inbox?Dawn????????????????????????????????????????????????????????????????????????Delivery Status Notification (Failure)" is a corrupt file.
This file cannot verify.
WARNING: "\\WCCEXCHANGE\Microsoft Exchange Mailboxes\Squash [squash]?Top of Information Store?Inbox????????????????????????????????????????????????????????????????????????Delivery Status Notification (Failure)" is a corrupt file.
This file cannot verify.
WARNING: "\\WCCEXCHANGE\Microsoft Exchange Mailboxes\Squash [squash]?Top of Information Store?Inbox????????????????????????????????????????????????????????????????????????Delivery Status Notification (Failure)" is a corrupt file.
This file cannot verify.

 

[LawnBoy]

OK, you've got some skipped files and some corrupted files, but it looks like each backup set is completing before the tape deck errors. Are you missing any backup sets or data, or is the problem only the tape deck errors?



--
The stagehand's axiom: "Never lift what you can drag, never drag what you can roll, never roll what you can leave.

 

[jfilmore]

The data that completes is backed up fine, only missing data that should be backed after failure.

 

[LawnBoy]

Is there any chance that NTBackup or some other backup software is running on the server? According to the log the tape deck is getting the command to eject the tape in the middle of the job.

This only happens on one day of the week? You do a full backup every night?

You could try recreating the job, sometimes they get corrupted in wierd ways.

--
The stagehand's axiom: "Never lift what you can drag, never drag what you can roll, never roll what you can leave.

 

[jfilmore]

NT backup is NOT running at the same time, I tried recreating backup and selection list last night, still fails.

 

[LawnBoy]

Can you tell if it fails at the same time? If so I'd think there must be some scheduled process on the machine that is sending an "eject" to the scsi bus. Check your scheduled tasks and disable anything that might run during the backup time.

If it doesn't happen at the same time, you might need to look at environmental factors. Does someone run a vacuum cleaner around the equipment on Fri night?

Also, you state that you run the same job every night. Does the job perform the same actions every time it runs? Do you run a full backup every night, or do you run incrementals or differentials on regular days and only do the full backup on Friday? If the error occurs only when you run a full backup, you should try changing your SCSI cables and terminator.

If none of this helps, I'm out of ideas...



--
The stagehand's axiom: "Never lift what you can drag, never drag what you can roll, never roll what you can leave.

 

[jfilmore]

I was wrong, backup-to-disk works. As a test I changed Friday's media to back-to-disk and it was successful. So it is something with TAPE backups on FRIDAY night. I have no scheduled tasks at all. If it was faulty H/W, I can't imagine it would only happen only Friday's for the past 12 weeks.

Also, it is not the tape, because this tape worked fine on another night.

There has to be some explication behind this craziness!!!