Can't get a decent backup done under BEWS 9.1 !!!! 3

[TigerDivision]

Hi!

I just don't know what else to do. Veritas Support have had me go round in circles and they cannot offer me a decent solution to my problem. I was wondering whether a kind soul out there could help me.

I am running BEWS 9.1 (upgraded from 9.0) on a Windows 2000 Server (SP4) connected to a Dell PV132T Tape Library.

I am backing up remote servers running respective BEWS 9.1 agents such as SQL, Lotus Domino.

These servers have dual nics installed (NIC teaming) and are operating at 100MB/sec Full Duplex. Now some of you might point out the NIC teaming issue. I don't think this is the case as backups were run successfully in the past.

Anyway I am having backups fail on my remote servers. The problem floats from remote server to remote server. For example last night it was our Domino Server, tonight it might be our Domain Server.

I am getting the following errors in the job log. By the way running SGMON does not record this problem!!!!

Final error: 0xa00084f8 - The network connection to the Backup Exec Remote Agent has been lost. Please check for network errors.

Further down I see also

A timeout occurred waiting for completion of media server data processing.

Veritas Support told me to uninstall and install again the Remote Agent but this is useless as it will not solve the problem and anyway this was already done last week when I upgraded BEWS to 9.1

Guys I'm really lost and don't know what to do next. I'm regretting the day we migrated to Veritas Backup Exec.

Thanks
Pierre

 

[Spinifex]

Hello Pierre,

We are a service provider and one of our biggest customers has exactly the same problem.
We run Bex 9.1 and backup 5 remote servers. In our case it's only one of the remote servers that fails to backup correctly. This is the biggest fileserver on which the company works all day. Veritas support also had us test about everything (re-installing remote agent, changing registry settings,...) but they couldn't come up with a solution.

Any help or ideas are more then welcome since we really don't know what to try next.

 

[TigerDivision]

Hi Spinifex

Glad to read that I'm not the only one experiencing such problems. I was earlier on looking at an earlier post within this forum and the same problem was experienced. Veritas is not admitting to its shortcomings and problems with this product. I firmly believe that this version, both 9.0 amd 9.1, is absolutely crap. Apparently my issue existed both under 9.0 and 9.1 Shame on them.
My problem as explained earlier floats from one remote server to another. Sometimes (very rare) you get 100% successful backups of all servers.
This has now become a serious and critical issue for us because we can no longer depend on our daily backups running to completion.
I wish we could sue them !!!

 

[justin2000]

Tigerdivision

OK so whats changed if the backup has been working in the past and now doesn't work, or works intermittently?
You are going to say nothing has changed - thats OK - so look at it this way, if this had been working for a while with no issues and now it suddenly stops - what could it be?

To answer that one..
Is it..
1. Does software decide to break after a while if nothing has changed?
or
2. Does hardware break after a while?

I would suggest you try the following to see if it resolves the issue you are having..

1. Check the switch these servers are attached to for any errors on the ports - if there are errors - reset them to 0, and watch them closely.
2. Break the NIC teaming on all servers.
3. You will now have 2 NIC cards to use, so possibly use one of them on each server for the Backup network - You can configure this in BE, Tools, Options, Network.
4. make sure all NIC cards and ports on the switches are hard set.
5. This could even be a hardware error from the tape drive and is reporting a false error.
Check the Windows Sys log for any Error events, specifically 7,9,11 and 15 - if you are getting these errors on either the tape hardware or even the NIC cards, this could be the issue and needs to be resolved.

I also suggest if this is such a critical issue to log a voice support call with Veritas and pay for support on this issue so it can be more closely looked at (I think the cost is less than $100) - email support for your sort of problem will not solve it.

 

[AndyGolf]

I am having similar problems, It may be worth you turning on the debug log - look at website for instructions. This gives a very detailed look at the dialogues going on at device and program level.

 

[TigerDivision]

I am currently restarting the BEWS Remote Agents on each server prior to running the backup. I've had backups run to 100% completion these past two days! However it is still too early to cry Victory! It might have solved the problem it might have not. Am still monitoring the situation and will keep you all posted as time goes by.

 

[Spinifex]

We will also schedule a restart of all remote agents before the backup this evening.
I'll inform you how things run.

 

[Spinifex]

We tried the restart of the remote agent just before the backup but unfortunately it didn't work.
Backup failed again with the same error.

I can only conclude that Backup Exec 9.1 sucks or has been released before it was actually tested...

Painfull to make such a conclusion for a product made by one of the biggest players on the storage market.

Spinifex.

 

[TigerDivision]

You're absolutely right! In fact it should be withdrawn from the market. Now onto the problem. Yes mine failed also after running successfully for 2 consecutive nights. Next Monday I am going to make some settings to the Network of BEWS 9.1 This was recommended to me by someone on another forum who experienced the same problem and is successfully taking backups. Will inform you about these settings next Monday

 

[justin2000]

Yeah withdraw it from the market.. I wonder how many licenses they have sold of 9.x - must be in the millions..
And I wonder how many are these installations are working without issue?

Again, if you backups are so critical and you can't find a resolution, log a voice support call and speak to someone at Veritas, moaning and groaning on support forums is not going to help your casue at all.

Tigerdivision, when you fix your network up and everything is working again, is backup Exec going to be the worst piece of backup software you have ever used?

The restarting of the remote agents would not have resolved the issue, and it does prove that Backup Exec does work as it worked without issue for 2 days backups - and then suddenly the next day it wasn't?

 

[TigerDivision]

Of course I've phoned Veritas and sent them e-mails. But they've led me nowhere but only to more frustration hence my postings in this forum! I've had a Veritas Advanced Engineer look into my problem and after 4 days still no reply!
The possible and likely reason is that BEWS 9.1 is running out of available ports when it comes to back remote servers. They have a network section where you can increase the ports pool. Will post about these next Monday.

 

[steveot]

You make no note if your servers are local or out on a DMZ. By default BE uses all ports. Have you tried backing up to disk to take the tape device out of the loop?
Also, check the event logs on the remote servers....has one of your servers rebooted or blue screened and reboot during the backup without you knowing? This would generate this error. Are there any virus scans, database maint, anything else going on during backup. Try moving the time windows for the jobs. . Im on with Justin...fix the network.

 

[TigerDivision]

There is no DMZ and no firewall either. Servers do not crash either and the event logs do not show any entries re failures, etc.
I am also making sure that the Anti Virus is disabled prior and remains disabled during the backup jobs and that there are no maintenance jobs scheduled during backup times either.
Our network is fine. What needs fine tuning is the networking component of BE. Come Monday I'll fine tune these and will report whether problem has been resolved or not on this forum. Watch this space

 

[TigerDivision]

Here are the settings recommended to me by someone from the Veritas Newsgroup.

Under Tools, Options, Network make the following settings:

Network Interface
'Select the NIC installed in the server'

Use any available network route for remote agents not bound to the above network ID
'Select it'

Enable Agents using TCP/IP Protocol
'Select it'

Enable Media Server Ports
'Set these to 10025 to 18025'

Enable Remote Agent Ports
'Set these to 18026 to 26026'

With thanks to Shawn Hartford for the above recommeded settings. According to him he was experiencing the same problem as I was and the above worked for him. He is getting daily backups done.

I've applied the settings this morning. We'll see how we'll get on this week fingers crossed Will keep you all posted.

 

[vo1ce]

Hi Tiger,

Short post to say:

I am just another of many who has been experiencing this intermittent problem (about one in four backups fail). Veritas have been giving me the "what have you changed then to break it" line. I will be trying the network settings suggestion and will also feed back.

 

[TigerDivision]

Here is the update.

The Network Interface 'Select the NIC installed in the server' setting had to be reset to Use Any Available Network Adapter since one of my remote servers running Lotus Domino just wouldn't communicate with the Media Server's Job Engine. The other servers did not complain which is strange. Later on I discovered that this is a known issue and will only be corrected at the next major release!! The only revised settings I kept were the port ranges.
Last night the backup ran to 100% completion. However this is too early to say whether the problem has gone away or not. I've also turned on debugging on the media server and this remote Lotus Domino server so that if the problem crops up again the debug log would hopefully log what went wrong!

 

[flatfour]

Hi

Having similar issues with BEWS9.1, espeically with a Domino server: how do you turn on debug on the remote server?

Ta

Bob

 

[TigerDivision]

Go to the following link:

http://seer.support.veritas.com/docs/254212.htm

 

[flatfour]

Thanks!

Bob

 

[vo1ce]

It seems that TigerDivisions suggestion that he gleaned from tha Veritas Newsgroup above (Network settings)
has worked for me....well chuffed

tx all esp TigerDivision