Nedd zipped files using shell script

[snehareddy]

Hi ,

I need to get zipped files as output , using shell scripts.

I am using following rule in the map:

=VALID(GET("SHL","-CMD 'zipfile.sh "+ FileName ConfigFieldsutGzipOptions + "' -TV -A"), FAIL( " Zip File Script falied to execute Successfully " + LASTERRORCODE() + LASTERRORMSG( ) )


The log file says:
<MercatorMapAudit StartTime="14:44:11 July 24, 2006">
2
3 <Platform>Mercator Command Execution Engine for HP-UX(TM) - Version 6.7(306)</Platform>
4
5 <ExecutionSummary MapStatus="Error" mapreturn="30" ElapsedSec="0.118" BurstRestartCount="0">
6 <Message>FAIL function aborted map: Zip File Script falied to execute Successfully -2Error Reading I
nput Data</Message>
7 <CommandLine>/var/opt/merccesar/maps/utils/tools/utGzipFile.hp -IF1 Input.txt -IF2 UTZIPFILE_Win.ini
</CommandLine>
8 <ObjectsFound>13</ObjectsFound>
9 <ObjectsBuilt>3</ObjectsBuilt>
10
11 <SourceReport card="1" adapter="File" bytes="17" adapterreturn="0">
12 <Message>Data read successfully</Message>
13 <Settings>/var/opt/merccesar/maps/utils/tools/Input.txt</Settings>
14 <TimeStamp>13:57:51 July 24, 2006</TimeStamp>
15 </SourceReport>
16
17 <SourceReport card="2" adapter="File" bytes="210" adapterreturn="0">
18 <Message>Data read successfully</Message>
19 <Settings>/var/opt/merccesar/maps/utils/tools/UTZIPFILE_Win.ini</Settings>
20 <TimeStamp>11:18:51 July 24, 2006</TimeStamp>
21 </SourceReport>
22
23 <TargetReport card="1" adapter="File" bytes="83" adapterreturn="0">
24 <Message>Data written successfully</Message>
25 <Settings>/var/opt/merccesar/maps/utils/tools/o1.txt</Settings>
26 <TimeStamp>14:44:11 July 24, 2006</TimeStamp>
27 </TargetReport>
28
29 <TargetReport card="2" adapter="File" bytes="0" adapterreturn="0">
30 <Message>Data written successfully</Message>
31 <Settings>/var/opt/merccesar/maps/utils/tools/o2.txt</Settings>
32 <TimeStamp>11:36:28 July 24, 2006</TimeStamp>
33 </TargetReport>
34
35 <WorkArea type="File">
36 <inputarea card="1" Path="/var/opt/merccesar/maps/utils/tools/utGzipFile.I01" TimeStamp="14:44:11 Ju
ly 24, 2006" bytes="65713"/>
37 <inputarea card="2" Path="/var/opt/merccesar/maps/utils/tools/utGzipFile.I02" TimeStamp="14:44:11 Ju
ly 24, 2006" bytes="65713"/>
38 <outputarea card="1" Path="/var/opt/merccesar/maps/utils/tools/utGzipFile.O01" TimeStamp="14:44:11 Ju
ly 24, 2006" bytes="65536"/>
39 </WorkArea>
40
41 </ExecutionSummary>
42
43 </MercatorMapAudit>


Any pointers would be appreciated

Thanks !!

 

[BocaBurger]

I think you need a PUT rather than a GET. You don't need the -A and how do you plan on passing the data to whatever is doing the zipping? Or are you passing a file name to the schell script, that will then pass the file name to the program?



BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[OllyC]

Why don't you use the GZIP adapter?

Also what does the shell script do - does it return the compressed / decompressed data using standard out?

Cheers,
Olly.

 

[snehareddy]

Hi,

I have used the GZIP adapter. There was a posting which I made previously and got some valuable suggestions from you as well.

However, this GZIP adapter is behaving peculiarly.

I am running three interfaces every hour (as per the requirement). Each interface has got two soft links , that means overall I am trying to zip 6 files in 6 different directories at same time.

The trace file says:

257 <7402-3248>: Stream type PLAIN
258 <7402-3248>: [PutPlain]
259 <7402-3248>: | ACTION_TYPE = COMPRESSTOGZIP
260 <7402-3248>: | Trying to open the file INSP_WOODSIDE/TR_WOODSIDE_HF_20060719081500.dat.gz for
writing!
261 <7402-3248>: | Unable to open the file INSP_WOODSIDE/TR_WOODSIDE_HF_20060719081500.dat.gz for
writing!
262 <7402-3248>: [PutPlain] (rc = -1) *** ERROR ***

But still the problem remains ..the zipped files are not getting formed consistently.

Failures are occuring at random..

A copy of my previous post is as below:

*********************************************************************************************
Hi,

There is a requirement wherein we need to generate zipped files as output.

We have used GZIP/ZLIB adapter do so.

The command used is:

-OAGZIP1 '-FILE<filename>.gz -STREAM PLAIN -ACTION COMPRESSTOGZIP -T'

I am trying to create the output file in a particular directory by using soft links and mentioning them with -FILE option.

By this, we are able to create the desired output files (zipped).

However, this is not working fine consistently. The map is failing randomly.
There is no particuler pattern being observed for the failure.

It is always failing with the following message:

<MercatorMapAudit StartTime="21:15:03 July 13, 2006">
+2
+3 <Platform>Mercator RUN Function for IBM(TM) RS/6000 AIX(TM) - Version 6.7.1(306_2)</Platform>
+4
+5 <ExecutionSummary MapStatus="Error" mapreturn="9" ElapsedSec="0.124" BurstRestartCount="0">
+6 <Message>Target not available</Message>
+7 <CommandLine>/merccesar/maps/local/537/ukx2diout_hf.aix -IE1S932 %lt;data%gt; -IE2S192 %lt;data%gt; -IE3S66 %lt;data%gt;
-if2 %dMercSrcHome%/config/local/537/UKX2DIOUT_HF_INITS.ini -OAGZIP1 '-FILE dly_its/IN_ITS_HF_20060713211503.dat.gz -STREAM PLAIN -A
CTION COMPRESSTOGZIP -T' -of2 %dMercSrcHome%/config/local/537/UKX2DIOUT_HF_INITS.ini -omMQS3 '-qmn %cCesarQMgr% -qn %cAuditQN%' -aeu
='%dLogs%/local' -t='%dTraceFiles%/local' -wu=%dWorkFiles%</CommandLine>
+8
+9 <SourceReport card="1" adapter="Echo" bytes="932" adapterreturn="0">
+10 <Message>Data retrieved successfully</Message>
+11 <TimeStamp>21:15:03 July 13, 2006</TimeStamp>
+12 </SourceReport>
+13
+14 <SourceReport card="2" adapter="File" bytes="192" adapterreturn="0">
+15 <Message>Data read successfully</Message>
+16 <Settings>/merccesar/config/local/537/UKX2DIOUT_HF_INITS.ini</Settings>
+17 <TimeStamp>19:15:01 July 13, 2006</TimeStamp>
+18 </SourceReport>
+19
+20 <SourceReport card="3" adapter="Echo" bytes="192" adapterreturn="0">
+21 <Message>Data retrieved successfully</Message>
+22 <TimeStamp>21:15:03 July 13, 2006</TimeStamp>
+23 </SourceReport>
+24
+25 <TargetReport card="1" adapter="GZIP/ZLIB" bytes="2553" adapterreturn="-1">
+26 <Message>Unknown Error</Message>
+27 <Settings>-FILE dly_its/IN_ITS_HF_20060713211503.dat.gz -STREAM PLAIN -ACTION COMPRESSTOGZIP -T</Settings>
+28 </TargetReport>
+29
+30 <WorkArea type="File">
+31 <inputarea card="1" Path="/merccesar/workfiles/Run_ukx2diout_hf_22284781152796341_59.I01" TimeStamp="21:15:03 July 13, 20
06" bytes="66424"/>
+32 <inputarea card="2" Path="/merccesar/workfiles/Run_ukx2diout_hf_22284781152796341_59.I02" TimeStamp="21:15:03 July 13, 20
06" bytes="66424"/>
+33 <inputarea card="3" Path="/merccesar/workfiles/Run_ukx2diout_hf_22284781152796341_59.I03" TimeStamp="21:15:03 July 13, 20
06" bytes="66424"/>
+34 <outputarea card="1" Path="/merccesar/workfiles/Run_ukx2diout_hf_22284781152796341_59.O01" TimeStamp="21:15:03 July 13, 20
06" bytes="65536"/>
+35 </WorkArea>


Please guide me if any one of you have faced similar kind of problems.

Any kind of inputs would be very helpful.

Thanks a lot !!

***********************************************************

Can you post a copy of the trace file from a failure occurring?

Possibly two maps are trying to write to the same file at the same time - is date/time named files unique enough?

Also you can specify retries in the adapter string:
-OAGZIP1R5:2 '-FILE<filename>.gz -STREAM PLAIN -ACTION COMPRESSTOGZIP -T'

Cheers,
Olly.

***************************************************************

Hi,

Surprisingly, there is no trace file getting formed when it is failing.

It is only getting formed when there is success.

1 <7402-227>: Stream type PLAIN
2 <7402-227>: [PutPlain]
3 <7402-227>: | ACTION_TYPE = COMPRESSTOGZIP
4 <7402-227>: | Trying to open the file DLY_ITS/IN_ITS_HF_20060718081500.dat.gz for writing!
5 <7402-227>: | File DLY_ITS/IN_ITS_HF_20060718081500.dat.gz succesfully opened in mode wb !
6 <7402-227>: | Preparing to write 2553 bytes
7 <7402-227>: | Number of bytes succesfully written 2553
8 <7402-227>: [PutPlain] (rc = 0) OK
9 <7402-558>: Stream type PLAIN
10 <7402-558>: [PutPlain]
11 <7402-558>: | ACTION_TYPE = COMPRESSTOGZIP
12 <7402-558>: | Trying to open the file SHIP_ITS/IN_ITS_HF_20060718111500.dat.gz for writing!
13 <7402-558>: | File SHIP_ITS/IN_ITS_HF_20060718111500.dat.gz succesfully opened in mode wb !
14 <7402-558>: | Preparing to write 332 bytes
15 <7402-558>: | Number of bytes succesfully written 332
16 <7402-558>: [PutPlain] (rc = 0) OK
17 <7402-561>: Stream type PLAIN
18 <7402-561>: [PutPlain]
19 <7402-561>: | ACTION_TYPE = COMPRESSTOGZIP
20 <7402-561>: | Trying to open the file SHIP_WOODSIDE/TR_WOODSIDE_HF_20060718111500.dat.gz for
writing!
21 <7402-561>: | File SHIP_WOODSIDE/TR_WOODSIDE_HF_20060718111500.dat.gz succesfully opened in m
ode wb !
22 <7402-561>: | Preparing to write 332 bytes
23 <7402-561>: | Number of bytes succesfully written 332
24 <7402-561>: [PutPlain] (rc = 0) OK
25 <7402-580>: Stream type PLAIN
26 <7402-580>: [PutPlain]
27 <7402-580>: | ACTION_TYPE = COMPRESSTOGZIP
28 <7402-580>: | Trying to open the file INSP_WOODSIDE/TR_WOODSIDE_HF_20060718111500.dat.gz for
writing!
29 <7402-580>: | File INSP_WOODSIDE/TR_WOODSIDE_HF_20060718111500.dat.gz succesfully opened in m
ode wb !
30 <7402-580>: | Preparing to write 602 bytes
31 <7402-580>: | Number of bytes succesfully written 602
32 <7402-580>: [PutPlain] (rc = 0) OK


As suggested I have used
-OAGZIP1R5:2 '-FILE <filename>.gz -STREAM PLAIN -ACTION COMPRESSTOGZIP -T+'

I am running three interfaces every hour (as per the requirement). Each interface has got two soft links , that means overall I am trying to zip 6 files in 6 different directories.
The results are as below:

TIME 11.15 12.15 13.15 14.15 16.15



UKSHPROUT IN-ITS :SUCCESS FAILURE SUCCESS (with R5:2) SUCCESS FAILURE
TR-WOODSIDE :SUCCESS SUCCESS SUCCESS SUCCESS FAILURE


UKIPFWDOUT IN-ITS :FAILURE SUCCESS SUCCESS (with R5:2) SUCCESS SUCCESS
TR-WOODSIDE :SUCCESS SUCCESS SUCCESS FAILURE SUCCESS


UKX2DIOUT IN-ITS :SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS
TR-WOODSIDE :SUCCESS SUCCESS SUCCESS (with R5:2) SUCCESS SUCCESS


There is no particular failure pattern being observed..

Any help/suggestion is welcome

***********************************************************************************


BocaBurger (Vendor) 18 Jul 06 14:03
I see:
-of2 %dMercSrcHome%/config/local/537/UKX2DIOUT_HF_INITS.ini -omMQS3 '-qmn %cCesarQMgr% -qn %cAuditQN%' But there does not seem to be an 2nd output card in the map.


BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!


**********************************************************************************


snehareddy (IS/IT--Management) 19 Jul 06 2:34

There is a certainly a second output card card in the map.

However this time, I can find a trace file getting formed in case of failure . The same is pasted below:


257 <7402-3248>: Stream type PLAIN
258 <7402-3248>: [PutPlain]
259 <7402-3248>: | ACTION_TYPE = COMPRESSTOGZIP
260 <7402-3248>: | Trying to open the file INSP_WOODSIDE/TR_WOODSIDE_HF_20060719081500.dat.gz for
writing!
261 <7402-3248>: | Unable to open the file INSP_WOODSIDE/TR_WOODSIDE_HF_20060719081500.dat.gz for
writing!
262 <7402-3248>: [PutPlain] (rc = -1) *** ERROR ***


But still the problem remains ..the zipped files are not getting formed consistently.

Failures are occuring at random..

*********************************************************************************************

Suggest sending test case to Support.
However you audit log shows:

+25 <TargetReport card="1" adapter="GZIP/ZLIB" bytes="2553" adapterreturn="-1">
+26 <Message>Unknown Error</Message>
+27 <Settings>-FILE dly_its/IN_ITS_HF_20060713211503.dat.gz -STREAM PLAIN -ACTION COMPRESSTOGZIP -T</Settings>
+28 </TargetReport>

I don't see a second card.
BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!


*****************************************************************************************



Since this option was not working I am trying to call a sheel script (via) map, but am still facing problems.

Any help/suggestion would be highly appreciated.

 

[OllyC]

Did IBM support come back with any useful information on why you're having problem with the GZIP adapter?

theres no reason why you can't call the gzip program directly, but as nothing will be retuned, you'll need to use the PUT function.

if you need to use the GET function (e.g. to make sure that compression is done immediately) then add an echo command on as well:
gzip myfile.txt;echo done
then check that the result is just "done" and has no error message from gzip.