Server crashes randomly running Veritas BE 9.1

[Sverre]

I am running BE 9.1 on win 2003 server sp1, taking backup to External USB disks every night. It is backing up local drives and a total of 5 drives on two other win 2003 sp1 servers with remote agents. After a period of time with successfull backups the backupserver now crashes some nights (about every second to fifth night - no obvious pattern). It sometimes crashes during backup, but mostly during verify. There is nothing in the eventlog on the backupserver other than "The last shutdown was unexpected"
On the remote server witch is being backed up at that time the eventlog are logging events in relation to starting and stopping VSS services. I have installed most of the patches from Veritas and have reinstalled the remote agents. Even upgraded the servers to sp1. No luck.
Any Idea - anyone

---------------------------
There are only 10 types of people in the world. Those who know binary numbers and those who don't

 

[NortonES2]

Disable any anti-virus software on the backup server.

-------------------------------

If it doesn't leak oil it must be empty!!

 

[steveot]

I would right click on my computer - properties - advanced - startup and recovery - uncheck the box to automatilly reboot. If the server blue screen you will probably see the error on the screen. I suspect its the USB. If your running Norton AV there is known issue with the latest symevent file that were recently released about 3 weeks ago. You have to get the latest ones

http://service1.symantec.com/SUPPORT/ent-security.nsf/docid/2005062909055948?Open&src=w

 

[Sverre]

really quick replies! Thank you, but... Here is more facts:
No antivirus is running. Dedicated USB2.0 PCI card installed. Seems to be working allright when accessing the disk (Maxtor One touch II - 100 GB).
The server is not reeboting, shut crashing down. I have not seen any bluescreen because this is happening during the night. I searched for the file MEMORY.DMP to see if lastdate matched last crash, but did not find any such file. So the crash dos not dump the memory. I have now unchecked the box "automaticly reboot", but again the server does not reboot, only crash/hang (black screen).

I very much feel that this has something with the VSS to do?



---------------------------
There are only 10 types of people in the world. Those who know binary numbers and those who don't

 

[steveot]

http://support.microsoft.com/default.aspx?scid=kb;en-us;887827

 

[Sverre]

I contacted Microsoft. They said that this hotfix was included into sp1. I just checked one of the files listed in the hotfix to find out that my file from the sp1 upgrade was a newer version than the one from the hotfix. Considering this I dont think I will aply this hotfix now. I am recieving it from Microsoft anyway so may be if nothing else seem to work I will test the hotfix (may be the newer files has some new errors - even if I really dont think so). The backup crashed last night again. Nothing in the eventlog except from "the last shutdown was unexcpected." Log on the server being backed up had some of this errors:

--------------------

Event Type: Error
Event Source: Backup Exec
Event Category: None
Event ID: 57860
Date: 16.08.2005
Time: 00:39:52
User: N/A
Computer: EDB
Description:
An error occurred while attempting to log in to the following server: "EDB".
SQL error number: "0011".
SQL error message: "[DBNETLIB][ConnectionOpen (Connect()).]SQL Server does not exist or access denied.
".

For more information, click the following link:

http://eventlookup.veritas.com/eventlookup/EventLookup.jhtml

Data:
0000: 11 00 00 00 00 00 00 00 ........
0008: 00 00 00 00 00 00 00 00 ........

---------------------

Event Type: Error
Event Source: VSS
Event Category: None
Event ID: 6013
Date: 16.08.2005
Time: 00:59:59
User: N/A
Computer: EDB
Description:
Sqllib error: OLEDB Error encountered calling IDBInitialize::Initialize. hr = 0x80004005. SQLSTATE: 08001, Native Error: 17
Error state: 1, Severity: 16
Source: Microsoft OLE DB Provider for SQL Server
Error message: [DBNETLIB][ConnectionOpen (Connect()).]SQL Server does not exist or access denied.


For more information, see Help and Support Center at

http://go.microsoft.com/fwlink/events.asp.

Data:
0000: 2d 20 43 6f 64 65 3a 20 - Code:
0008: 53 51 4c 43 4f 4e 4e 43 SQLCONNC
0010: 30 30 30 30 30 34 39 31 00000491
0018: 2d 20 43 61 6c 6c 3a 20 - Call:
0020: 53 51 4c 43 4f 4e 4e 43 SQLCONNC
0028: 30 30 30 30 30 33 39 37 00000397
0030: 2d 20 50 49 44 3a 20 20 - PID:
0038: 30 30 30 30 31 36 34 30 00001640
0040: 2d 20 54 49 44 3a 20 20 - TID:
0048: 30 30 30 30 35 31 30 38 00005108
0050: 2d 20 43 4d 44 3a 20 20 - CMD:
0058: 43 3a 5c 57 49 4e 44 4f C:\WINDO
0060: 57 53 5c 53 79 73 74 65 WS\Syste
0068: 6d 33 32 5c 76 73 73 76 m32\vssv
0070: 63 2e 65 78 65 20 20 20 c.exe
0078: 2d 20 55 73 65 72 3a 20 - User:
0080: 4e 54 20 41 55 54 48 4f NT AUTHO
0088: 52 49 54 59 5c 53 59 53 RITY\SYS
0090: 54 45 4d 20 20 20 20 20 TEM
0098: 2d 20 53 69 64 3a 20 20 - Sid:
00a0: 53 2d 31 2d 35 2d 31 38 S-1-5-18
-----------------

The first error is mention by Veritas as a problem conected to versions of BE 9.0 not 9.1 as far as I could find out on their website. Anyway I will try to fix this by renameing the remote servers bedssql2.dll to .old as Veritas says. I am not backing up any SQL data this way anyway - only the files from MSSQLs internal backup earlier in the evening.

The second error makes me wonder? Maybe this also is gone by renaming the mentioned file.

Still not shure if this is what that makes the backupserver crash.

I am planning to change the backup options for a few nights now - running without any open file options, to see if the problem then will be the same or disappear.



---------------------------
There are only 10 types of people in the world. Those who know binary numbers and those who don't

 

[justin2000]

If your server is rebooting without a memory.dmp or blue screen I would tend to think that you have a hardware problem.

The 2 events you are getting are nothing to worry about and they wouldn't cause the server to reboot like it is.

 

[Sverre]

The server does not reboot, only crashing/hanging/not responsive when the person who is changing disk comes to work in the morning.

Since last post I was going to prepare a new backup without any shadow copy data checked. So I selected a new backup, imported the selection. Then I discovered that when I clicked on the checkbox for shadowcopy components to remove it, the hourglass came on and the BE interface would not respond for å while. Checked the log on the other server. Not much wiser. Well anyway - I will run the new selections tonight to see if anything happens.

I am also planning to recheck some of the hardware on the backupserver.

---------------------------
There are only 10 types of people in the world. Those who know binary numbers and those who don't

 

[Sverre]

Just a short update:
The new backup selection went on just fine last night. This happened with to old one some nights to so I will wait until next week before I can be shure this is one of the right steps towards isolating and solving the problem.

---------------------------
There are only 10 types of people in the world. Those who know binary numbers and those who don't