automatic cloning errors

[esdeedee]

Hi,

Some questions about the errors we get when automatic cloning:

Every Sunday there is a full backup from our servers to pool A. After the full backup, we start an automatic cloning of these servers to pool B. The full backup to pool A is ok:

NetWorker savegroup: (notice) Linux Servers completed, total 4 client(s), 0 Hostname(s) Unresolved, 0 Failed, 4 Succeeded.
Start time: Tue Jan 10 21:00:01 2006
End time: Tue Jan 10 21:26:57 2006

But before legato starts cloning we get this message:
"NetWorker media: (emergency) inactive ssid"

When the cloning session stops, we get this:

"NetWorker savegroup: (alert) Compagny Full-Clone completed, total 7 client(s), 0 Hostname(s) Unresolved, 1 Failed, 7 Succeeded.
Start time: Tue Jan 10 23:59:01 2006
Clone Start: Wed Jan 11 02:18:05 2006
End time: Wed Jan 11 03:18:32 2006
Automatic cloning of save sets to pool Compagny Clone succeeded."

So, you should think, that everything is ok and the automatic cloning succeeded. But when we scroll in the logfile, we get this message also:

"
--- Cloned Save Sets ---

Automatic cloning has failed for 23 clients in the savegroup Compagny Full-Clone. Please check the daemon.log for details."

In spite of the message "automatic cloning succeeded", there is also a warning that automatic cloning has failed.
When we ask what has been cloned, we get just a few server/directory's, not all (mminfo -q "pool=Full Clone,savetime> 1 day ago")

Is the reason of this failure, the error "NetWorker media: (emergency) inactive ssid"? Or something else? What should we do?

 

[605]

This seems to be a bit more complicated. But most important: Is this a NW server/Windows which has RSM enabled?

 

[esdeedee]

NW server runs on a windows 2003 server. The RSM service isn't started.

 

[605]

Could it be that multiple groups are started so that you are running out of devices for simulteanous operations?

As the error indicates, there should be more info available from daemon.log.

 

[esdeedee]

We still get the errors!

That's what we see in the daemon.log file during the clone operation:

NetWorker media: (emergency) inactive ssid
...
NetWorker savegroup: (alert) Compagnie Full-Clone completed, total 8 client(s), 0 Hostname(s) Unresolved, 1 Failed, 7 Succeeded. (kastaar Failed)
Start time: Sun Feb 26 01:30:01 2006
Clone Start: Sun Feb 26 05:50:32 2006
End time: Sun Feb 26 05:59:08 2006
Automatic cloning of save sets to pool Compagnie Clone succeeded.
...
--- Cloned Save Sets ---
Automatic cloning has failed for 45 clients in the savegroup Compagnie Full-Clone. Please check the daemon.log for details.
Dus in het begin van de logging wordt er aangegeven dat het clonen gelukt is, echter wanneer we het laatste lijntje van de logging bekijken , zien we dat het clonen helemaal niet gelukt is. In de daemon.log vinden we het volgende terug:
02/26/06 05:53:45 nsrd: media event cleared: Waiting for 1 writable volumes to backup pool 'Compagnie Clone' tape(s) on legato.Compagnie.be
02/26/06 05:55:07 nsrd: media info: can not read record 0 of file 3874 on LTO Ultrium-2 tape 000008L2
02/26/06 05:55:07 nsrmmd #31: inactive ssid
02/26/06 05:55:07 nsrd: media emergency: inactive ssid
02/26/06 05:55:07 nsrd: cloning session:1 of 45 save set(s) reading from 000008L2 2873 MB of 270 GB
02/26/06 05:55:07 nsrmmd #31: inactive ssid
02/26/06 05:55:07 nsrmmd #31: inactive ssid
...
02/26/06 05:55:10 nsrd: legato.Compagnie.be:cloning session done saving
02/26/06 05:55:10 nsrd: cloning session:save sets done reading 50 GB
02/26/06 05:55:10 ansrd: ansrd_clone FAILED: errnum is -7 and errstr is can not read record 0 of file 3874 on LTO Ultrium-2 tape 000008L2
02/26/06 05:55:10 ansrd: failed to execute MODE_CLONE
...
02/26/06 05:55:12 savegrp: command 'nsrclone -s legato.Compagnie.be -b Compagnie Clone -S -f - ' exited with return code 1.
02/26/06 05:56:39 nsrd: legato.Compagnie.be:bootstrap saving to pool 'OurServers' (000008L2)
02/26/06 05:56:44 nsrmmdbd: media db is saving its data. This may take a while.
02/26/06 05:56:44 nsrmmdbd: media db is open for business.
02/26/06 05:56:49 nsrd: legato.Compagnie.be:bootstrap done saving to pool 'OurServers' (000008L2) 4490 KB
02/26/06 05:56:57 nsrd: deactivating mmd #31
02/26/06 05:56:57 nsrd: Calling mm_deactivate for mmd 31 thats using device \\.\Tape1 with volume 000012L2 on host null
02/26/06 05:56:57 nsrd: write completion notice: Writing to volume 000012L2 complete
02/26/06 05:57:24 nsrd: write completion notice: Writing to volume 000008L2 complete
02/26/06 05:59:05 nsrd: legato.Compagnie.be:cloning session saving to pool 'Compagnie Clone' (000012L2)
02/26/06 05:59:06 nsrd: cloning session:1 of 1 save set(s) reading from 000008L2 4490 KB of 4490 KB
02/26/06 05:59:06 nsrd: legato.Compagnie.be:cloning session done saving to pool 'Compagnie Clone' (000012L2)
02/26/06 05:59:06 nsrd: cloning session:save sets done reading 4490 KB
02/26/06 05:59:08 savegrp: Automatic cloning of saveset SYSTEM STATE:\ during savegroup operation has failed!
02/26/06 05:59:08 savegrp: Automatic cloning of saveset SYSTEM FILES:\ during savegroup operation has failed!
...
02/26/06 05:59:08 savegrp: Automatic cloning of saveset /netappdata/fstore during savegroup operation has failed!02/26/06 05:59:08 savegrp: Automatic cloning of saveset E:\ during savegroup operation has failed!
02/26/06 05:59:08 savegrp: Automatic cloning of saveset index:2905b6e5-00000004-43d642ba-43d642b9-04560000-c14ac303 during savegroup operation has failed!

Anyone who can help us?

 

[605]

It looks like a problem with your device and/or media. Fact is:
- Tape 000008L2 obviously has read i/o errors after a while.
- It aborts the clone process due to this problem.
- Then it uses this tape to save the bootstrap, which most likely will fail.
However, it seems that it will succeed.
- Anyway, it NW seems to deactivate the device (stopping nsrmmd)
- But cloning of the bootstrap seems to work fine.

So obviously it points to the fact that there is auch a hard a read error on the backup media that clones can not be completed.


But something is really weird:
- at the end it reports "../netappdata/fstore during savegroup operation has failed!"
- Does this mean that
- You use automatic cloning for NDMP save sets?
- You mix OTF and NDMP save sets on one media?
This will most likely fail, depending on the NW version. Which one do you use?

 

[esdeedee]

the savegroup /netappdata/fstore is an nfsmount of the data stored on the netapp. NDMP isn't yet implemented, but we know when you backup normal data and NDMP data, that we need separate tapes to store the save sets.

 

[605]

Not necessarily ...
- You can even mix the save sets on one media
- But you can not clone the volume because
- nsrclone can not read NDMP save sets
- nsrndmp_clone can not read non-NDMP save sets
Honestly, in practice nobody would do this.


However, if you use the Data Service Agent (NW 7.2+) then the data will be converted to OTF and you can mix both on one media and treat it like 'normal' NW media. This also means that you can avoid to buy specific NDMP backup hardware.