Hipath 4K Backups Failing on save portion

[kevin906]

I don't know why the Hipath product has so many issues with backups through all releases. The stock answer-hotfixes-might take care of the problem for a while, then it's failing again. I have monitored these backups in the Unix process tables and the file structures as they progress along. Any ideas on this are welcome. This system fails on the MO-RMX "ALL" backup at the last portion "SAVE". All other parts pass. While monitoring this I observed the following data.

At the last portion of the backup there is a large file that gets zipped. They are shown below.
The file "Content.v1.2" is then supposed to get copied to the A1H6I partition and fails.
Reason shown below from the device.MO_RMX.log file.
The DATA.DATA.p.gz zipped portion completes OK.
The file copy attempt to :A1H6I: fails due to no space to allocate on the drive.
This file appears to be 139 Meg in size.
The DIS-DDSM:A1,HD,6; command indicates about 123 Meg of space left.
So it's obvious why it's failing. Not enough space.

HOW DO YOU CORRECT THIS PROBLEM???!!!
All of this runs "under the covers" and there is no manual manipulation of files available.
So how can this be resolved if you can't delete certain files that are taking up too much space?
This system was backing up with no issue until about 2 months ago.
Is there some file or files that needs to be manually nulled or removed that is part of the backup and hogging the disk space?

This system is a Version 4.0 and disk partitions are the recommended sizes.
REGEN-DDSM is below to show partition sizes.


-rw-rw-rw- 1 root root 139366400 Sep 5 14:43 Content.v1.2
-rw-rw-rw- 1 root root 139353721 Sep 5 14:13 DATA.DATA.p.gz

:A1H6H:HBR/A/Content.v1.2
gzip -S .gz *DATA*.p (RC=0)
./DATA.DATA.version.02-01-01
./DATA.DATA.pointers
./DATA.DATA.p.gz
./InfoFile
./files.tbl
./set_ok
cat FileList | pax -w >Content.v1.2
/opt/bin/uricmd HBR rmxcopy "Content.v1.2"
:A1H6I:HBR/DATA/Content.v1.2
062 Unable to copy file
rmx_create (1): can't allocate space for the file
domain_rmx (1): can't create the rmx destination file
domain_rmx (1): rmxdel call done

RMXCOPY error occurred
Exit: 145 62 at: 2013-09-05 14:43:37
Command: /opt/hbr/sys/device.MO_RMX -A MO_RMX -P 10487 -c umount
Started at: 2013-09-05 14:43:39
Exit: 0 at: 2013-09-05 14:43:40
Command: /opt/hbr/sys/device.MO_RMX -A MO_RMX -P 10413 -c umount
Started at: 2013-09-05 14:43:40
Bind directory (/.AS/BACKUP/tmp/SBACKUP/MO_RMX) removed (0)
Exit: 0 at: 2013-09-05 14:43:40
#


+---------------------+-------+--------+-------+
| A1H6I | FREE | USED | SUM |
+---------------------+-------+--------+-------+
| FNODES | 51190 | 82 | 51272 |
| SPACE (MB) | 123 | 677 | 801 |
| SPACE (64KB)| 1974 | 10842 | 12817 |
+----------------------------------------------+


REG-DDSM;
H500: AMO DDSM STARTED
ADD-DCSM:A1,1,HD,"STDH4";
ADD-DASM:A1,1,3200&992&2400&1120&32767&3200&9868,4096&4096&4096&4096&4096&4096&4096;
ADD-DLSM:A1,1,E,"DS:";
ADD-DLSM:A1,1,F,"BDA:"&"BD:"&":TMD:"&"AS:"&":AMD:"&"MP:";
ADD-DLSM:A1,1,G,":CGD:";
ADD-DLSM:A1,1,H,"MS:"&"SY:";
ADD-DLSM:A1,1,I,":SCR:";
ADD-DLSM:A1,1,J,":GLA:";
ADD-DLSM:A1,1,K,"IAG:";
ADD-DCSM:A1,6,HD,"CF1G3";
ADD-DASM:A1,6,3200&992&800&1120&12817,4096&4096&4096&4096&4096;
ADD-DLSM:A1,6,E,"DS:";
ADD-DLSM:A1,6,F,"BDA:"&"BD:"&":TMD:"&"AS:"&":AMD:"&"MP:";
ADD-DLSM:A1,6,G,":CGD:";
ADD-DLSM:A1,6,H,"MS:"&"SY:";
ADD-DLSM:A1,6,I,":MOD-SCR:";
AMO-DDSM -111 DISK STATUS
REGENERATE COMPLETED;

 

[Iamnothere]

Test

 

[Iamnothere]

Your controller 6 is properly configured.

You are probably not using the original "STEC" brand of CF card, which provides the most actual usable storage space on a 2GB CF card. The STEC provides about 1865MB(*) of usable space. The asterisk indicates that I do not recall the specific number, but this 1865MB is a close guesstimate. No other brand of CF card comes within 10MB of matching that total available amount of usable space. Since your backup is falling short about 16MB, I suspect that you are using a Kingston or Patriot 2GB CF card.

The 1865MB provided by the 2GB STEC CF card is a VERY TIGHT Squeeze! No other 2GB CF card can match the performance of the STEC.

Therefore, I suggest that you use a 4GB size (or higher) of ANY brand, which should resolve the issue. With a 4GB or higher CF card, when you initialize it, the maximum allowed amount of space is allocated for HiPath 4000 V4 backups to CF. That amount will be somewhere in the neighborhood of 1865MB - no matter what the publicized size of the CF card (such as 4GB, 16GB).

When you use any other brand of 2GB CF card, the initialization cannot match the 1865MB that is needed to perform all of the V4 backups to CF card.

Good luck.

 

[kevin906]

So the actual partition definition from DASM isn't the defining factor here? I didn't realize the system could write into a partition larger than it's set to be. I will try this and find out. Thanks,

 

[ccmjgb]

kevin906,

a very similar situation happened at a HiPath 4000 V.5 site a few months back. The Siemens TAC engineer that assisted with this mentioned that this would be taken care of by an upgrade to the latest RMX, UNIX version 5. The upgrade was completed and the back-ups are back to normal. An important note was that the TAC engineer accessed the UNIX via expert mode (comwin) and deleted older, larger back-up type files & directories in an attempt to restore the back-up procedure, but this did not work - he tried it a couple times. The only back up that did work was a manually started MO_RMX backup to the CF card. It may be that the upgrades restructure the UNIX so that the back-up directories are free of space and eventually get loaded up. Hopefully this will not happen again with this upgraded V.5 but your comment made me think that you've had this happen more than once...

 

[Iamnothere]

The DASM values are the GOSPEL. These are the settings that were designed for the STEC 2GB CF card, with its 1865MB. I'm not sure what specifically happens when the CF card is smaller than the DASM requirements. I know that some of the CF cards that we tested would fail the initialization - while others would pass the initialization, but they were not large enough to support the HiPath 4000 backups to CF card, and would subsequently fail.

The system cannot create a partition larger than what DASM calls for. But by using certain vendor's CF 2GB CF cards, there is not enough usable space for DASM to create the partitions as designed.

I have found it very difficult to locate 2GB CF cards now. I see only 16GB cards at my favorite Frys Electronics. And the 16GB price has dropped to the point where price is no longer a concern.
Whatever size you choose, the DASM values are used during the initialization.

 

[donb01]

I'm reading all this with great interest, as one or more of the techs have mumbled something about the CF card in the past, but I never knew there was a special brand. I also have trouble finding 2GB CF cards. I have bought some in the past from either geeks.com or Amazon. I have a few SanDisk ones, and most recently I think the ones I bought were some generic AOC crap or something.

How does one tell how much the card actually formatted out to? I assume you can probably access the mounted card from unix, but usually unix reports free space in blocks or something depending on what command you use before - I'm not sure if it reports in bytes at the prompt level (I know on my servers I can use the GUI to see the size in bytes). I haven't had any disk full errors ... yet, but my system is pretty small, with only like 140 stations and maybe 46 trun ports between the Cornet and PRI, so maybe that is why.

Thanks!

 

[Iamnothere]

I used a USB-based CF card reader with a Windows PC to measure the amount of usable storage. Format the CF card using Windows, then you will see the actual available RAM. The "STEC" provides more (in a 2GB CF card) than any other vendor (that we tested).