Our Networker is version 7.1.2, Build 325.
We have a single Linux box with two Sony 10 tape jukeboxes (1 drive each) directly attached. After 3 years of relatively troublefree operation, we all of a sudden had both jukeboxes start to give us problems at nearly the same time. Jukebox1 had to be removed a week ago because it was having tapes hung (eating the tapes and the tape hanging out caused it to get stuck between the slots and the drive). The only correlation between time and events my coworker (the backup manager) can make is that he performed a firmware upgrade to the Sony jukeboxes around 3 months ago, close to the time the tapes started getting eaten, but not exactly the same time. We have not ruled that out though as part of the problem. Enough setting the stage, now on to the main question.
Jukebox1 was eating tapes, so we removed it. After removal, Networker sees this and Jukebox1 is disabled. All data should get sent to Jukebox2. Initially we were able to load and label tapes (bar code enabled) with no problems. We started a normal nightly backup and we began to get some strange errors. The errors are strange and incomplete. The software is reporting a blank status of the drive. I've gone down the analytical path of "is it hardware", but I'm not seeing any scsi errors, so I don't think it is. Now it cannot load nor label tapes. The backup server has been rebooted more than once attempting to fix this strange error. Jukebox2 has been power cycled as well.
This is from /nsr/logs/messages:
Feb 14 13:28:55 backup01 logger: NetWorker media: (warning) /dev/nst1 reading: read open error: drive status is
Feb 14 13:38:22 backup01 last message repeated 2 times
Feb 14 13:48:09 backup01 last message repeated 2 times
Feb 14 13:48:10 backup01 logger: NetWorker media: (warning) /dev/nst1 reading: read open error: drive status is
Feb 14 13:58:25 backup01 logger: NetWorker device cleaned: (notice) device `/dev/nst1' (0) in jukebox `SonyJukebox2' cleaned at `Tue Feb 14 13:58:25 2006'
Feb 14 14:04:37 backup01 logger: NetWorker media: (warning) /dev/nst1 reading: read open error: drive status is
(/dev/nst0 is the removed Jukebox1 and /dev/nst1 is Jukebox2).
Any ideas why Networker cannot give us an error that indicates exactly what is wrong? The only thing we can successfully do at this point is load a cleaner. IYHO, does it look like hardware? If so, can you come up with any reasons why Networker doesn't give us more detailed reasons? And lastly, if anybody has ever seen this behavior before, can you provide direction to fix this?
We're going to the colo tomorrow to reinstall the removed unit, leave it disabled, and see if it affects operation. Physical layout is normally server->Jukebox1->Jukebox2. Currently it's just server->Jukebox2. You never know until you try.
Any insight is appreciated.
Regards... Todd
We have a single Linux box with two Sony 10 tape jukeboxes (1 drive each) directly attached. After 3 years of relatively troublefree operation, we all of a sudden had both jukeboxes start to give us problems at nearly the same time. Jukebox1 had to be removed a week ago because it was having tapes hung (eating the tapes and the tape hanging out caused it to get stuck between the slots and the drive). The only correlation between time and events my coworker (the backup manager) can make is that he performed a firmware upgrade to the Sony jukeboxes around 3 months ago, close to the time the tapes started getting eaten, but not exactly the same time. We have not ruled that out though as part of the problem. Enough setting the stage, now on to the main question.
Jukebox1 was eating tapes, so we removed it. After removal, Networker sees this and Jukebox1 is disabled. All data should get sent to Jukebox2. Initially we were able to load and label tapes (bar code enabled) with no problems. We started a normal nightly backup and we began to get some strange errors. The errors are strange and incomplete. The software is reporting a blank status of the drive. I've gone down the analytical path of "is it hardware", but I'm not seeing any scsi errors, so I don't think it is. Now it cannot load nor label tapes. The backup server has been rebooted more than once attempting to fix this strange error. Jukebox2 has been power cycled as well.
This is from /nsr/logs/messages:
Feb 14 13:28:55 backup01 logger: NetWorker media: (warning) /dev/nst1 reading: read open error: drive status is
Feb 14 13:38:22 backup01 last message repeated 2 times
Feb 14 13:48:09 backup01 last message repeated 2 times
Feb 14 13:48:10 backup01 logger: NetWorker media: (warning) /dev/nst1 reading: read open error: drive status is
Feb 14 13:58:25 backup01 logger: NetWorker device cleaned: (notice) device `/dev/nst1' (0) in jukebox `SonyJukebox2' cleaned at `Tue Feb 14 13:58:25 2006'
Feb 14 14:04:37 backup01 logger: NetWorker media: (warning) /dev/nst1 reading: read open error: drive status is
(/dev/nst0 is the removed Jukebox1 and /dev/nst1 is Jukebox2).
Any ideas why Networker cannot give us an error that indicates exactly what is wrong? The only thing we can successfully do at this point is load a cleaner. IYHO, does it look like hardware? If so, can you come up with any reasons why Networker doesn't give us more detailed reasons? And lastly, if anybody has ever seen this behavior before, can you provide direction to fix this?
We're going to the colo tomorrow to reinstall the removed unit, leave it disabled, and see if it affects operation. Physical layout is normally server->Jukebox1->Jukebox2. Currently it's just server->Jukebox2. You never know until you try.
Any insight is appreciated.
Regards... Todd
