DR plans for DR exercise 2

[MrOttoX]

Hi,
I am getting ready to create a DR plan on how to recover my NetBackup environment at an offsite DR provider. I have 1 master and 2 media servers, an LTO2 Library with 8 drives, I am using Vault and we duplicate our offsite copies, and I am at version NBU 4.5 DataCenter FP6.
Can anyone with a couple successfull DRs with NetNackup share their DR plans with me, I am looking for the technical steps mainly.
Thank you in advance.
Otto

 

[doctorski]

Otto,
How are you you have recovering the OS ?

We use HP-UX Ignite to recover the OS, which includes NBU (minus the images).

Once recovered, identify your device files, as they may have changed.

Recreate any pass-through drivers.

Use tpconfig to correct any device anomolies, including your library.

Recover your images etc using

bprecover -v -r -ev TAPEID -d density

One gotcha is the bprecover will change the device file configuration back, so any changes as above need to be redone.

Identify all images needed and make the duplicates the primary copy.

Restore your data.

We run the coverage command every two hours, and use the output from that command to build our list of file systems to be recovered.

Hope this helps.

 

[enterprisebu]

What type of hardware for the servers? Is the equipment the same offsite?

 

[ihpen]

good morning otto.
this is rough but i'll clean it up as i figure it out!!!

-rebuild server, in my case, solaris 8
-network, patched
-install netbackup 4.5_3ga
-install fp5
-from media manager device config guide, config sg/st drivers.
-restore catalog
--bprecover -r -tpath /dev/rmt/Xcbn -dhost "new_server_name"

-config netbackup
--modify vm.conf/bp.conf to reflect "new server name"
--rm files: globDB, ltidevs, robotic_def
--vmglob -set_gdbhost
--tpautoconf -upgrade
--tpconfig or use wizard to setup drives

-restore
--bpimmedia -L -client "client_name" -d xxxxxxxx -e xxxxxx
you are looking for the "backupid" at this point.
--bpduplicate -npc 2 -backupid "client_name_backupid"
--bprestore/jnbSA, run a restore. it should ask you for the copy, not the primary tape.

again, this is a rough draft as i am working out the details myself. ttfn..

 

[saiplent]

Hi,

one question regarding DR.

Does someone has a script, or something, to have the list of the tape having the last succesfull backup.

I have tape sent to the vault everyday, I have to take these tapes, but some backups are done once in a week, so how can I ask for the correct tape to restore this specific backup ? (the tapes has been sent to the vault but what is the tape ID ???)

I could send the catalog tape to the DR site , restore the catalog and check the info, but it will be too long. Someone use a script ? a vaulting option ?

Thanks !

 

[enterprisebu]

We record the info on a spreadsheet that goes offsite with the tapes.

 

[ihpen]

good morning saiplent.
i have a copy of the vault reports sent to another email account other then the one at work, which i can access through my own isp anytime/anywhere.
quick and dirty, under the directory ../netbackup/vault/session/your_vault_profile, there is a file called "eject.failed". if this file size is zero, your vault was successful. also check the file "reportout.11", same location.

in my dr scenario, i would ask for all of the tapes from the last two set of full's and the preceding week of inc's [i do fulls once a week, inc's sunday-thursday]. i actively monitor for failed backup and vaulting sessions and take corrective action.

no cat tape?
i create a catalog tape with each vault. from the offsite emails[or daily reports] i can quickly determine the mediaid of this tape. to much time/money/my_butt at stake, reload time of cat tape; 57m@41Gb.
i never know in advance what files/directories i will be asked to restore.

take care..

 

[saiplent]

Thanks,

I'm going to use a /netbackup/bin/admincmd/bpdbjobs -all_columns. With that command I have the info. I use it to generate a daily backup report (size, succes rate, failure and failure explanation), I will add the tape info.
(I'm too lazy to check the activity monitor everyday)

For the catalog tape, yes I do send one offsite everyday. Of course.
It's just a DR test that we will do soon, in an other backup environment, we won't restore the catalog but import the tapes (long, but no other choice...)

 

[enterprisebu]

Not true. We have a different environment too. For the machines I need to restore for the DR test I've added their specific db/images path to the catalog files. Then at DR test I only bprecover those specific directories so the restore window knows what images are on the tape. I create the new "environment" on the master, request the restore, select the date I want and insert the correct tape and it works like a charm.

 

[saiplent]

Ok, that is a good new.

So for example in my catalog backup config I have to had
/db/image/server1
and when I do the catalog restore I choose only this directory.

But a few questions remaining :
- Do I need to have the same master server name ?
- Does the client need to have the same name ?

Thanks

 

[enterprisebu]

That's exactly what I do and yes the names and ips are duplicated.

 

[ihpen]

neat idea!

 

[saiplent]

So, seems that we can do the restore in a different environment (like restore Prod in Dev) without using the same IP's and name:

- For the master/media server in the bp.conf we can use the FORCE_RESTORE_MEDIA_SERVER

- For the clients if we want to restore on a diffent name: we can create a host.xlate file (not sure for that one)

- To check the catalog image we need to restore:
bprecover -l -m AA0001 -d hcart -v, will liste the image and then:
bprecover –r IMAGE2 -m AA0001 -d hcart –v –dhost destination_host

Will make some test.

Bye

 

[MrOttoX]

ihpen" thank you very much for yr response outline of the steps needed for the DR, they are pretty much inline with what I have figured out so far, but I have a couple questions in the second set "config netbackup", in there you are mentioning two steps that I was not aware of:
--rm files: globDB, ltidevs, robotic_def
--vmglob -set_gdbhost

Could you please explain what are these two steps and why we need them ? do we really need them ?

Thanks again,

Otto

/Quote
good morning otto.
this is rough but i'll clean it up as i figure it out!!!

-rebuild server, in my case, solaris 8
-network, patched
-install netbackup 4.5_3ga
-install fp5
-from media manager device config guide, config sg/st drivers.
-restore catalog
--bprecover -r -tpath /dev/rmt/Xcbn -dhost "new_server_name"

-config netbackup
--modify vm.conf/bp.conf to reflect "new server name"
--rm files: globDB, ltidevs, robotic_def
--vmglob -set_gdbhost
--tpautoconf -upgrade
--tpconfig or use wizard to setup drives

-restore
--bpimmedia -L -client "client_name" -d xxxxxxxx -e xxxxxx
you are looking for the "backupid" at this point.
--bpduplicate -npc 2 -backupid "client_name_backupid"
--bprestore/jnbSA, run a restore. it should ask you for the copy, not the primary tape.

again, this is a rough draft as i am working out the details myself. ttfn.. Unquote/

 

[ihpen]

Good morning mrottox. I apologize for the delay.

In my dr test I was unnecessarily restoring the volmgr/database directory. In testing I was using an stk l180 vs. l700 with fewer tape drives. If I do not recover the volmgr/database directory I do not need to remove the globDB and the robotic_def files. Messing with the ltidevs file requires you to reset the global device database host, using vmglob –set_gdbhost. So no, you do not need these two steps.

Here is a draft of my dr test.

--- part 1
1. Procedure Description:
Installation of Veritas NetBackup and related patches.

2. Prerequisite:
OS media. Installation CD's 1 and 2, Sun Microsystems Solaris 8.
Installation of Sun E450, Tape Library, and network interface card.

3. Special Requirements:
verify SCSI connection between Sun E450 and Tape Library
Loadpoint, "/usr/openv" (or some referencing link).
Partition size, fifty gigabytes [minimum].
License Key list.
Memory configuration for Veritas NetBackup. Reboot required.
modifications made to "/etc/system" file.
---------
* modify parameters for veritas:
* so... do what veritas says...
*
set maxusers=40
set msgsys:msginfo_msgmap=512
set msgsys:msginfo_msgmax=8192
set msgsys:msginfo_msgmnb=131072
set msgsys:msginfo_msgmni=256
set msgsys:msginfo_msgssz=16
set msgsys:msginfo_msgseg=8192
set msgsys:msginfo_msgtql=512
*
set semsys:seminfo_semume=64
set semsys:seminfo_semmap=64
set semsys:seminfo_semopm=32
set semsys:seminfo_semmni=1024
set semsys:seminfo_semmns=1024
set semsys:seminfo_semmnu=1024
set semsys:seminfo_semmsl=300
*
set shmsys:shminfo_shmmin=1
set shmsys:shminfo_shmmni=220
set shmsys:shminfo_shmmax=2147483648
set shmsys:shminfo_shmseg=100
*
*


4. Detailed Instructions:
a) Login as root.
b) Create loadpoint, "/usr/openv" directory
c) Load Veritas NetBackup CD from DR-1 kit.
d) Run "install" from installation directory on CD.
e) From "installation options", Select NetBackup, option 1.
f) Prompts for Netbackup and Media Manager binaries to be installed in "/usr/openv".
g) Prompts for client binaries installation. Select Solaris. Installation begins.
h) Prompts for license key. Input the "Base NetBackup" and the "Library Based Tape Drives" license key.
i) Decline prompt for the full domainname name of the NetBackup server, ex: "hostname.domainname"
j) Input "hostname" only! for the name of the NetBackup server. Accept verification.
k) Decline prompt for "media (slave) servers".
l) Accept prompt for the "global device database" host, default [this servers host_name, not pds-nbu].
m) Accept prompt for startup of the NetBackup process for client installation.
n) Accept prompt for creatation of policy and schedule examples.
o) Accept prompt to start the Media Manager device daemon processes.
p) Accept prompt to start the NetBackup bprd process. End of Vertias NetBackup installation.
q) Installation of feature pack 5, or fp5.
r) Install CD.
s) Copy "071604_NB_45_5_F_SERVERS_260885.solaris.tar" to /tmp. Extract tar file.
t) run "Vrts_pack.install". Input pack name "NB_45_5_F".
u) Copy "071604_NB_CLT_45_5_F_260893.solaris.tar" to /tmp. Extract tar file.
v) run "Vrts_pack.install". Input pack name "NB_45_5_CLT".


--- part 2



1. Procedure Description:
Device Installation / Configuration for Veritas NetBackup.

2. Prerequisite:
OS media. Installation CD's 1 and 2, Sun Microsystems Solaris 8.
Installation of Sun E450, Tape Library, and network interface card.

3. Special Requirements:
SCSI connection between Sun E450 and Tape Library
Tape drives will initially be defined as "hcart2", redefine as "hcart".
stop netbackup; /usr/openv/netbackup/bin/goodies/bp.kill_all.
restart netbackup; /etc/init.d/netbackup start.

4. Detailed Instructions:
a) Configure tape drive. gui; "/usr/openv/netbackup/bin/jnbSA"
b) In left panel; Select "media and device management"
c) In right panel, Select "Configure Storage Devices [define robots and drives]".
d) Within the device configuration wizard, Select "next".
e) Next; within the device configuration wizard, the "hostname" should be displayed. Select "next".
f) Next; "scanning hosts", tape drive will be discovered under the "hostname". Select "next".
g) Next; backup devices output will display vendor specific devices. Select "next".
h) Next; drag and drop configuration will output display with vendor specified devices. Select "next".
i) Next; commit changes. Select "continue".
j) Next; updating device configuration. Select "next".
k) Next; display of storage devices. Select "next". Select "finish".
l) In left panel; under netbackup management, Select "storage units",
m) remove all storage units NOT associated with "hostname"; right click; "delete".
n) Right click "hostname" of storage unit. Right click "change", under devices select "standalone - hcart". Select "ok"
o) In the left panel, under "media and device management", Select "devices", then click "hostname".
p) In right panel, Select "devices shown [Drive type]", right click and Select "change".
q) Change option "drive type" to "1/2" cartridge (hcart)". Select "ok".
r) Select "yes" to update device configuration.
.


--- part 3




1. Procedure Description:
Veritas NetBackup Data Recovery process.

2. Prerequisite:
OS media. Installation CD's 1 and 2, Sun Microsystems Solaris 8.
Installation of Sun E450, Tape Library, and network interface card.
Configuration of Veritas NetBackup and associated hardware.

3. Special Requirements:
Restoration of data based upon date of duplicated tapes.

4. Detailed Instructions:
a) Performing a restore requires two peices of information; the "time frame" and the "backupid".
The "time frame" is based upon the date the files were backed up.

b) Obtaining the "backup_id" and "media_id" for specific file/directory to restoration;
/usr/openv/netbackcup/bin/admincmd/ bpimmedia -L -client -client "client_name" -d "beginning_date_of_search" -e "ending_date_of search"

c) identifing the media_id / backup_id example::

$ bpimmedia -L -client seawolf -d 04/29/2004 -e 05/02/2004
Backup-ID Policy Type RL Files C E T PC Expires
Copy Frag KB Type Density FNum Off Host DWO MPX Expires RL MediaID
---------------------------------------------------------------------------------------------------------------------------

seawolf_1083422363 umservers FULL 3 9760 N N R 1 07:39 08/02/2004

1 1 477440 RMed hcart 124 852941 pds-nbu 2 Y 07:39 08/02/2004 3 005238
1 2 1536 RMed hcart 125 860946 pds-nbu 2 005238
1 3 1536 RMed hcart 126 861180 pds-nbu 2 005238
1 4 34766 RMed hcart 127 861216 pds-nbu 2 005238
^
this column, "1" indicates primary copy

2 1 478464 RMed hcart 64 486203 pds-nbu 7 Y 07:39 08/02/2004 3 003902
2 2 512 RMed hcart 65 494208 pds-nbu 7 003902
2 3 1536 RMed hcart 66 494442 pds-nbu 7 003902
2 4 34766 RMed hcart 67 494478 pds-nbu 7 003902
^
this column, "2" indicates the secondary copy

backup_id:: seawolf_1083422363 media_id:: 005238 and 003902
^^^^^^^^^^^^^^^^^

d) NetBackup always defaults to the primary copy. NetBackup will always ask for the primary copy during a restore.
Running the "bpduplicate -npc 2" command temporaily reassigns the secondary tape as the primary tape for restore/disaster
recovery purposes. To switch the primary tape [the original] with the secondary [the copy];

/usr/openv/netbackcup/bin/admincmd/bpduplicate -npc 2 -backupid "client_name_backup_id"


e) To perform a restore through the gui run the administration console; "/usr/openv/netbackup/bin/jnbSA" .