I have attached all the steps performed below , please share your views on this.
Third Day :
There has been a bit more progress here.
It appears that the Legato media manager was not running on the DR site and this tied in with the errors we were seeing.
I have now managed to perform crosscheck and this now seems to be working (see below).
oracle,PARISLIV [/scc/oracle/admin/rman_DR]>rman target / cmdfile=ph_crosscheck.rcv
Recovery Manager: Release 11.1.0.6.0 - Production on Thu Dec 13 15:30:19 2012
Copyright (c) 1982, 2007, Oracle. All rights reserved.
connected to target database: PARISLIV (DBID=2934680327, not open)
RMAN> allocate channel for maintenance type 'SBT_TAPE' parms 'ENV=(NSR_SERVER=svr-blk-legato1,NSR_CLIENT=svr-unx-hscare,NSR_DATA_VOLUME_POOL=DataDomain)';
2> crosscheck backup;
3>
4>
5>
using target database control file instead of recovery catalog
allocated channel: ORA_MAINT_SBT_TAPE_1
channel ORA_MAINT_SBT_TAPE_1: SID=112 device type=SBT_TAPE
channel ORA_MAINT_SBT_TAPE_1: NMDA Oracle v1.1.0
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=db_PARISLIV_798691697.3140 RECID=3046 STAMP=798691736
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=arch_PARISLIV_798705963.3141 RECID=3047 STAMP=798706008
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=arch_PARISLIV_798706453.3142 RECID=3048 STAMP=798706497
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=c-2934680327-20121107-00 RECID=3049 STAMP=798706964
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=db_PARISLIV_798759076.3152 RECID=3058 STAMP=798759116
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=arch_PARISLIV_798787776.3153 RECID=3059 STAMP=798787817
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=arch_PARISLIV_798788802.3154 RECID=3060 STAMP=798788843
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=c-2934680327-20121108-00 RECID=3061 STAMP=798789731
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=db_PARISLIV_798846492.3164 RECID=3070 STAMP=798846532
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=arch_PARISLIV_798860761.3165 RECID=3071 STAMP=798860803
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=arch_PARISLIV_798861489.3166 RECID=3072 STAMP=798861529
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=c-2934680327-20121109-00 RECID=3073 STAMP=798862415
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=db_PARISLIV_798955061.3176 RECID=3082 STAMP=798955101
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=arch_PARISLIV_798967687.3177 RECID=3083 STAMP=798967726
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=arch_PARISLIV_798968141.3178 RECID=3084 STAMP=798968180
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=c-2934680327-20121110-00 RECID=3085 STAMP=798968585
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=db_PARISLIV_799205175.3204 RECID=3110 STAMP=799205216
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=arch_PARISLIV_799230449.3205 RECID=3111 STAMP=799230496
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=arch_PARISLIV_799230858.3206 RECID=3112 STAMP=799230903
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=c-2934680327-20121113-00 RECID=3113 STAMP=799231383
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=db_PARISLIV_799282774.3216 RECID=3122 STAMP=799282820
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=arch_PARISLIV_799290440.3217 RECID=3123 STAMP=799290486
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=arch_PARISLIV_799290731.3218 RECID=3124 STAMP=799290773
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=c-2934680327-20121114-00 RECID=3125 STAMP=799291182
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=db_PARISLIV_799359037.3222 RECID=3134 STAMP=799359080
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=arch_PARISLIV_799374563.3229 RECID=3135 STAMP=799374605
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=arch_PARISLIV_799375070.3230 RECID=3136 STAMP=799375112
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=c-2934680327-20121115-00 RECID=3137 STAMP=799376179
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=db_PARISLIV_799444593.3234 RECID=3146 STAMP=799444653
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=arch_PARISLIV_799461297.3241 RECID=3147 STAMP=799461341
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=arch_PARISLIV_799461587.3242 RECID=3148 STAMP=799461635
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=c-2934680327-20121116-00 RECID=3149 STAMP=799461854
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=db_PARISLIV_799560007.3252 RECID=3158 STAMP=799560049
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=arch_PARISLIV_799581509.3253 RECID=3159 STAMP=799581551
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=arch_PARISLIV_799581956.3254 RECID=3160 STAMP=799582000
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=c-2934680327-20121117-00 RECID=3161 STAMP=799582438
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=db_PARISLIV_799814613.3298 RECID=3204 STAMP=799814654
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=arch_PARISLIV_799828949.3299 RECID=3205 STAMP=799828996
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=arch_PARISLIV_799829172.3300 RECID=3206 STAMP=799829215
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=c-2934680327-20121120-00 RECID=3207 STAMP=799829485
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=db_PARISLIV_799891783.3310 RECID=3216 STAMP=799891824
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=arch_PARISLIV_799900111.3311 RECID=3217 STAMP=799900154
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=arch_PARISLIV_799900319.3312 RECID=3218 STAMP=799900358
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=c-2934680327-20121121-00 RECID=3219 STAMP=799900583
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=db_PARISLIV_799961341.3316 RECID=3228 STAMP=799961398
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=arch_PARISLIV_799987872.3323 RECID=3229 STAMP=799987913
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=arch_PARISLIV_799988829.3324 RECID=3230 STAMP=799988870
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=c-2934680327-20121122-00 RECID=3231 STAMP=799989609
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=db_PARISLIV_800048249.3328 RECID=3240 STAMP=800048301
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=arch_PARISLIV_800064130.3335 RECID=3241 STAMP=800064172
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=arch_PARISLIV_800064718.3336 RECID=3242 STAMP=800064761
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=c-2934680327-20121123-00 RECID=3243 STAMP=800065270
Crosschecked 52 objects
When I try to restore a single file I am now getting a different error. This suggests to me that it may just be the tape not being available.
RMAN> run {
2> allocate channel t1 type 'SBT_TAPE'
3> parms 'ENV=(
4> NSR_SERVER=svr-blk-legato1,
5> NSR_CLIENT=svr-unx-hscare,
6> NSR_DATA_VOLUME_POOL=DataDomain)';
7> #
8> #
9> restore spfile to '/scc/oracle/admin/rman_DR/phtest.sp' ;
10> }
11>
using target database control file instead of recovery catalog
allocated channel: t1
channel t1: SID=111 device type=SBT_TAPE
channel t1: NMDA Oracle v1.1.0
Starting restore at 13-DEC-12
List of remote backup files
============================
Handle: c-2934680327-20121123-00 Media: FIE250
released channel: t1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 12/13/2012 15:33:01
RMAN-20507: some targets are remote - aborting restore
Recovery Manager complete.
RMAN> list backup tag TAG20121123T002026;
List of Backup Sets
===================
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
3243 Full 14.00M SBT_TAPE 00:00:48 23-NOV-12
BP Key: 3243 Status: AVAILABLE Compressed: NO Tag: TAG20121123T002026
Handle: c-2934680327-20121123-00 Media:
SPFILE Included: Modification time: 22-NOV-12
SPFILE db_unique_name: PARISLIV
Control File Included: Ckp SCN: 1363133826 Ckp time: 23-NOV-12
=============================================================================================================================================
Second Day :
I have just tried few more commands.
Firstly I have tried listing backups while allocating an SBT_TAPE channel. As you can see below this works ok.
oracle,PARISLIV [/scc/oracle/admin/rman_DR]>rman target / cmdfile=ph_list_backups.rcv
Recovery Manager: Release 11.1.0.6.0 - Production on Thu Dec 13 13:38:57 2012
Copyright (c) 1982, 2007, Oracle. All rights reserved.
connected to target database: PARISLIV (DBID=2934680327, not open)
RMAN> allocate channel for maintenance type 'SBT_TAPE' parms 'ENV=(NSR_SERVER=svr-blk-legato1,NSR_CLIENT=svr-unx-hscare,NSR_DATA_VOLUME_POOL=DataDomain)';
2> list backup of database summary device type 'SBT_TAPE' completed after '20-jan-10';
3>
4>
5>
using target database control file instead of recovery catalog
allocated channel: ORA_MAINT_SBT_TAPE_1
channel ORA_MAINT_SBT_TAPE_1: SID=113 device type=SBT_TAPE
channel ORA_MAINT_SBT_TAPE_1: NMDA Oracle v1.1.0
List of Backups
===============
Key TY LV S Device Type Completion Time #Pieces #Copies Compressed Tag
------- -- -- - ----------- --------------- ------- ------- ---------- ---
3046 B F A SBT_TAPE 07-NOV-12 1 1 NO TAG20121107T024816
3058 B F A SBT_TAPE 08-NOV-12 1 1 NO TAG20121107T213115
3070 B F A SBT_TAPE 09-NOV-12 1 1 NO TAG20121108T214811
3082 B F A SBT_TAPE 10-NOV-12 1 1 NO TAG20121110T035740
3110 B F A SBT_TAPE 13-NOV-12 1 1 NO TAG20121113T012615
3122 B F A SBT_TAPE 14-NOV-12 1 1 NO TAG20121113T225933
3134 B F A SBT_TAPE 15-NOV-12 1 1 NO TAG20121114T201036
3146 B F A SBT_TAPE 16-NOV-12 1 1 NO TAG20121115T195632
3158 B F A SBT_TAPE 17-NOV-12 1 1 NO TAG20121117T040006
3204 B F A SBT_TAPE 20-NOV-12 1 1 NO TAG20121120T024332
3216 B F A SBT_TAPE 21-NOV-12 1 1 NO TAG20121121T000942
3228 B F A SBT_TAPE 22-NOV-12 1 1 NO TAG20121121T192900
3240 B F A SBT_TAPE 23-NOV-12 1 1 NO TAG20121122T193728
Recovery Manager complete.
oracle,PARISLIV [/scc/oracle/admin/rman_DR]>cat
I have then tried to run a command to crosscheck backups. As you can see below this is failing with the same error as when we try the restore.
"ph_crosscheck.rcv" 4 lines, 171 characters
oracle,PARISLIV [/scc/oracle/admin/rman_DR]>rman target / cmdfile=ph_crosscheck.rcv
Recovery Manager: Release 11.1.0.6.0 - Production on Thu Dec 13 13:45:24 2012
Copyright (c) 1982, 2007, Oracle. All rights reserved.
connected to target database: PARISLIV (DBID=2934680327, not open)
RMAN> allocate channel for maintenance type 'SBT_TAPE' parms 'ENV=(NSR_SERVER=svr-blk-legato1,NSR_CLIENT=svr-unx-hscare,NSR_DATA_VOLUME_POOL=DataDomain)';
2> crosscheck backup;
3>
4>
5>
using target database control file instead of recovery catalog
allocated channel: ORA_MAINT_SBT_TAPE_1
channel ORA_MAINT_SBT_TAPE_1: SID=113 device type=SBT_TAPE
channel ORA_MAINT_SBT_TAPE_1: NMDA Oracle v1.1.0
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of crosscheck command on ORA_MAINT_SBT_TAPE_1 channel at 12/13/2012 13:45:31
ORA-27191: sbtinfo2 returned error
Additional information: 3335
Recovery Manager complete.
My conclusion with this is that we need to check the basic functionality of the Legato system here.
===============================================================================================================================================
First Day
I have made a few attempts at restoring from Legato this morning. All of them have been unsuccessful.
In order to try and simplify the problem I thought we would start by seeing if we could restore a single file.
In order to prove the theory I tested the following command. In this example I have restored a backup of an spfile from TIVOLI on one of our other systems. The purpose of this was simply to check the syntax of the command that we were using. As you can see the command was successful.
RMAN> run {
2> allocate channel c1 device type sbt_tape parms 'ENV=(TDPO_OPTFILE=/home/oradev/TSM/tdpo.opt_SMCOGDEV)' ;
3> restore spfile to '/home/oradev/PHTMP/RMAN_TEST/ph_sp_bkup3';
4> }
allocated channel: c1
channel c1: SID=198 device type=SBT_TAPE
channel c1: Data Protection for Oracle: version 5.4.1.0
Starting restore at 13-DEC-12
channel c1: starting datafile backup set restore
channel c1: restoring SPFILE
output file name=/home/oradev/PHTMP/RMAN_TEST/ph_sp_bkup3
channel c1: reading from backup piece sp_DLY_43nsnbll_1_1
channel c1: piece handle=sp_DLY_43nsnbll_1_1 tag=TAG20121212T234804
channel c1: restored backup piece 1
channel c1: restore complete, elapsed time: 00:03:35
Finished restore at 13-DEC-12
released channel: c1
I have then used the following command to try the same thing on the Southampton system using Legato. The results from this are as follows:
oracle,PARISLIV [/scc/oracle/admin/rman_DR]>rman target / debug trace=ph_debug2.log cmdfile=ph_test_rest2.rcv
Recovery Manager: Release 11.1.0.6.0 - Production on Thu Dec 13 13:23:00 2012
Copyright (c) 1982, 2007, Oracle. All rights reserved.
RMAN-06568: connected to target database: PARISLIV (DBID=2934680327, not open)
RMAN> run {
2> allocate channel c1 type 'SBT_TAPE' parms 'ENV=(NSR_SERVER=svr-blk-legato1,NSR_CLIENT=svr-unx-hscare,NSR_DATA_VOLUME_POOL=DataDomain)';
3> restore spfile to '/scc/oracle/admin/rman_DR/phtest.sp' ;
4> }
5>
RMAN-06009: using target database control file instead of recovery catalog
RMAN-08030: allocated channel: c1
RMAN-08500: channel c1: SID=40 device type=SBT_TAPE
RMAN-08526: channel c1: NMDA Oracle v1.1.0
RMAN-03090: Starting restore at 13-DEC-12
RMAN-08031: released channel: c1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 12/13/2012 13:23:07
RMAN-03009: failure of IRESTORE command on c1 channel at 12/13/2012 13:23:07
ORA-27191: sbtinfo2 returned error
Additional information: 3335
Recovery Manager complete.
oracle,PARISLIV [/scc/oracle/admin/rman_DR]>
I have also attached the trace file from the above command.