Redefining Tivoli Storage Manager Library and Drives for UNIX OS

Question
Frequently, when hardware or firmware changes have been made, it is necessary to remove the tape library and drive definitions from the Tivoli Storage Manager Server, then re-define them.

Cause
Sometimes there are specific errors, such as:

ANR0523W Transaction failed - error on output storage device
ANR8300E I/O error on library (OP=xx, CC=xx, KEY=xx, ASC=xx, ASCQ=xx, SENSE=xx)
ANR8301E I/O Error on library
ANR8355E I/O error reading label for volume NNNNNN on drive XXXXX
ANR8359E Media fault detected on volume NNNNNN in drive XXXXX
ANR8441E Initialization failed for SCSI library
ANR8779E Unable to open drive XXXXX, error number=ZZZ
ANR8944E Hardware or media error on drive
ANR8963E Unable to find path to match the serial number defined for drive

Frequently, the Tivoli Storage Manager Server can automatically rediscover devices when using "SANDISCOVERY ON" option, or by running the "UPDATE PATH" command with "AUTODETECT=YES", to refresh the values.

However, there are times when that may not be successful. For example, if a tape drive, tape library, fibre/SCSI HBA, or SAN have experienced changes (such as hardware, firmware or device drivers), it may require rebuilding the Tivoli Storage Manager "special files" to re-establish connectivity to the library and drives. To rebuild the "special files," you must delete and re-define the hardware devices to the Tivoli Storage Manager Server (UPDATE does not rebuild).

Answer
Perform these tasks in the following order, to completely re-define the tape devices to Tivoli Storage Manager. These steps should be taken only if attempts to update the devices/paths using the autodetect feature have failed:

1) Before deleting anything, gather the output from these commands so that you can use the same naming conventions when re-defining the tape devices:
QUERY STATUS (get SERVERNAME value for "<tsm_server>")
QUERY DEVCLASS FORMAT=DETAIL
QUERY LIBRARY FORMAT=DETAIL
QUERY DRIVE FORMAT=DETAIL
QUERY PATH FORMAT=DETAIL

2) Run the appropriate OS command to produce a list of the configured HW 'special file' device names.
AIX ==> lsdev -Cc tape (-or- 'cfgmgr')
lsdev -Cc adsmtape (for TSM devices)
lsdev -Cc library
Solaris ==> ls -l /dev/rmt/*st (-or- 'sysdef')
ls -l /dev/rmt/*smc
HP-UX ==> /usr/sbin/ioscan -funC tape
(-or 'ioscan -kfn')
Linux ==> ls -l /dev/IBM*
ls -l /dev/tsmscsi/*
(-or- 'more /etc/sysconfig/hwconf')

If the tape devices are not defined to the OS, please work with your OS or SCSI/SAN hardware support to configure them. Until the OS can use the drives (can write to them, for example using 'tar' or 'dd') the tape devices cannot be defined to Tivoli Storage Manager.

3) From the '/dev' directory, write down the OS-level device definitions for the library and drives:

AIX Linux Solaris HP-UX
TSM Drives mt# tsmscsi/mt# rmt/# rmt/tsmmt#
IBM Drives rmt# IBMtape# rmt/#st rmt/#m
TSM Library lb# tsmscsi/lb# rmt/#lb tsmchgr#
358x Library smc# IBMchanger# rmt/#smc rmt/#chng
3494 Library lmcp# 3494lib libmgrc# libmgrc#

4a) First, the drives and drive paths must be deleted. From a Tivoli Storage Manager Server admin commandline, for all the drives:

DELETE PATH <tsm_server> <drive_name> SRCTYPE=SERVER DESTTYPE=DRIVE LIBRARY=<library_name>

4b) Then delete all the Tivoli Storage Manager drive definitions:
DELETE DRIVE <library_name> <drive_name>

5a) Next, delete the path for the tape library:
DELETE PATH <tsm_server> <library_name> SRCTYPE=SERVER DESTTYPE=LIBR

5b) Finally, delete the Tivoli Storage Manager library definition:
DELETE LIBRARY <library_name>

If the OS cannot access the tape drives at this point, stop. Check hardware, device drivers, update firmware, swap cables; consider power-cycling the tape library, then deleting and re-defining the devices to the OS. There is no point attempting to get Tivoli Storage Manager to write to the devices if they are not recognized by the OS; work with OS and/or hardware vendors to resolve HW issues before proceeding.

6a) Now the tape library and library path can be re-defined. Use the Tivoli Storage Manager QUERY outputs from "Step 1" as a guide for the library name and LIBTYPE; no additional parameters are necessary in the syntax below. Redefine the library:

DEFINE LIBRARY <library_name> LIBTYPE=<library_type> SERIAL=AUTODETECT

Note: If this Tivoli Storage Manager Server is hosting a tape library for other systems, for example any "Tivoli Storage Manager Server Library Clients" or "Tivoli Storage Manager Storage Agents" then you also need "SHARED=YES" on the "DEFINE LIBRARY".

6b) Redefine the path to the library. For SCSI libraries, confirm the DEVICE value matches the latest OS-level info gathered from "Step 2" (if you needed to remove and re-add the devices to the OS, repeat Step 2 to obtain the updated device names). For 3494, ACSLS, and other types of libraries using software configuration files, use the previous values from "Step 1" to redefine the DEVICE or ACSID, and so on:

DEFINE PATH <tsm_server> <library_name> SRCTYPE=SERVER DESTTYPE=LIBRARY DEVICE=</dev/lb#> (adjust device name for OS and device type; e.g. /dev/smc#, /dev/IBMChanger#, lb#.#.#.#, etc)

7a) Redefine the drives and drive paths. Redefine all the drives using names from
"Step 1" for example:

DEFINE DRIVE <library_name> <drive_name> SERIAL=AUTODETECT ELEMENT=AUTODETECT

7b) Redefine paths to all drives, using the OS-level info gathered from "Step 2" for the DEVICE values. Keep in mind the OS-level DEVICE values may have changed since the they were previously defined, in which case repeat Step 2 to obtain the updated device names..

DEFINE PATH <tsm_server> <drive_name> SRCTYPE=SERVER DESTTYPE=DRIVE LIBRARY=<library_name> DEVICE=</dev/mt#> (adjust device name for OS and device type; e.g. /dev/rmt#, /dev/IBMTape#, mt#.#.#.#, etc)

Note: If this Tivoli Storage Manager Server is hosting a tape library for other systems, for example any "Tivoli Storage Manager Server Library Clients" or "Tivoli Storage Manager Storage Agents" then in addition to the "Tivoli Storage Manager Server Library Manager" DRIVE PATH, you also need to define a new PATH for each drive for those systems, substituting the SERVERNAME (shown by "Q SERVER") for the value of "<tsm_server>" and the local DEVICE value for the drive as seen by that other system.

8) Verify the library, drives, and paths are online:
QUERY LIBRARY <library_name> FORMAT=DETAIL
QUERY DRIVE <library_name> * FORMAT=DETAIL
QUERY PATH * * FORMAT=DETAIL

9) Since the library is "new" to Tivoli Storage Manager, the volumes must be checked in again to re-create the inventory (AUDIT LIBRARY does not CHECKIN). Use the following sequence: first SCRATCH, then PRIVATE:
CHECKIN LIBVOL <library> SEARCH=Y STATUS=SCR CHECKL=BARC
CHECKIN LIBVOL <library> SEARCH=Y STATUS=PRIV CHECKL=BARC

NOTE: For ACSLS libraries, use "CHECKLABEL=NO" on the CHECKIN commands, because "CHECKLABEL=BARCODE" is not supported for an ACSLS Library.

NOTE: The reason for checking in scratch tapes first, then private tapes, is that tapes known by the Tivoli Storage Manager server to contain data cannot be checked in as SCRATCH (a warning will be issued against that tape, and the scratch checkin will continue), but scratch tapes can be checked in as PRIVATE but as long as they are checked in as PRIVATE the Tivoli Storage Manager will not attempt to use them as it would a tape checked in as SCRATCH.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If that does not resolve the issue, then the tape drive problem may be beyond the control of the Tivoli Storage Manager Server (software). Review the output from the OS-level logs for additional hardware error information:
Remove OS Install
OS Diagnostics Devices OS Devices
------- ------------ --------- ------------
AIX errpt –a rmdev cfgmgr
Linux dmesg /dev/MAKEDEV
Solaris mbin/prtdiag rem_drv drvconfig
HP-UX dmesg rmsf insf -e

If you cannot reach HW support immediately, you could take the additional action of power-cycling in this order:

1) Tape library.
2) SAN switch (if any).
3) Consider updating to latest device drivers and/or firmware.
4) Halt Tivoli Storage Manager and reboot the system with the Tivoli Storage Manager Server.
5) Re-define the tape device to the OS (see commands above).
6) If tape device definitions have changed, DELETE & re-DEFINE to Tivoli Storage Manager.

That is most of what can be done, from a software perspective, to correct potential hardware issues. If errors persist, it may point to an issue at a layer which Tivoli Storage Manager cannot repair.