Trying to read a file 2

[frangac]

Hi All, PHV

I am stuck with the following or if there is a better solution please post.

What I am trying to achive is to read a bin file with "od"

0000000 4837 7470 722b 3a30 3964 7331 3753 4654
0000020 3030 3030 6161 6161 6161 6161 6161 6161
0000040 6161 6161 6161 6161 6161 3781 0000 0000
0000060 0123 0108 92aa aa20 2020 2020 2020 2020
0000100 2020 2020 2020 2020 0604 1513 5516 0123
0000120 2621 96aa aaaa aaaa aaaa aaaa aaaa aaaa
0000140 aaaa aa00 0004 2020 2053 5350 0000 26ac
0000160 0000 0037 8100 0000 0001 2301 0892 aaaa
0000200 2020 2020 2020 2020 2020 2020 2020 2020
0000220 2006 0415 1355 4201 2326 2196 aaaa aaaa
0000240 aaaa aaaa aaaa aaaa aaaa aaaa 0000 0420
0000260 2020 5353 5000 0026 ac00 0000 5476 7470
0000300 722b 3a30 3964 7331 3753 4654 3030 3030
0000320 3030 3032 3030 3030 3030 3030 3030 3030
0000340 3030 3030 3030 3030 3230 3030 3030 3030
0000360 3030 3030 3030 3030 3030 3030 3032 3030
0000400 3030 3030 3030 3030 3030 3030 3030 3030
0000420 3030 3230 3030 3030 3030 3030 3030 3030
0000440 3030 3030

Result
=======
H 7 tpr+:09 ds17 VAL 0000
aaaaaaaa aaaaaa aaaa aaaa
7 80 xxxxxxxxxxaaaa 20060415135516 0123262196aaaaaaaa 000004 007599 S SP 000026 ac000000
7 80 0218552997aaaa 20060415214645 0860007249aaaaaaaa 000066 009703 S SP 000026 ac000000
T v tpr+:09 df17 VAL


Script

awk 'BEGIN{

HEADER=0;
RECORDS=-1;
TRAILER=0;
String1="";
cmd = "od -x FILENAME";

while (cmd | getline >0)
for (i=1; i<=NF; ++i)
if(length($i)>4)
{
continue
}
else
{
if (HEADER < 21)
{
HEADER=HEADER+1;
#cmd1=hex2ascii(substr($i,1,2)) " | cut -f2 -d\" \""
#cmd2=hex2ascii(substr($i,3,2))
#cmd1 | getline b ; close(cmd1);
#print b

}
else if (RECORDS<38)
#else if (RECORDS<24)
{
RECORDS=RECORDS+1;
#if (RECORDS < 23 )
if (RECORDS < 37 )
{
#RECORDS=RECORDS+1;
printf $i
}
else
{
if(substr($i,1,2)==37)
{
print ""
printf $i
RECORDS=0;
}
else
{
TRAILER=1;
RECORDS=38;
#RECORDS=25;
print""
print $i
#exit 99;
}
}
}
else
{
print $i
}
}
close (cmd)
}


###################################################
function hex2ascii(hex_string)
###################################################

{

#Convert any size hex string to 8-bit ASCII

result_ascii = "";
module = length(hex_string) % 2
if ( module==1 )
strcol="0"hex_string

strcol = length(hex_string) - 1;

while (strcol > 0)

{

#Capture binary 8-bit chunks starting from right to left
chunk = substr(hex_string, strcol, 2);
if ( chunk < 20 ){
print hex_string" ""is not a valid Hexa value to Convert to ASCII or is not printable\n"
exit 1
}
#Convert hex to decimal
dec = hex2uint(chunk);

#Convert decimal to ASCII character
digit = sprintf("%c", dec);

#Concatenate and resume looping
result_ascii = digit result_ascii;

strcol = strcol - 2;

}

return(result_ascii);
}


###################################################
function hex2uint(hex_string){
###################################################

# Convert hex string to unsigned integer

result = 0;
power = 0;
MSD = substr (hex_string, 1, 1);

for (strpos = length(hex_string); strpos > 0; strpos--)
{
digit = substr(hex_string, strpos, 1);
if (match(digit, /[a-fA-F]/))
{
gsub(/[aA]/, "10", digit);
gsub(/[bB]/, "11", digit);
gsub(/[cC]/, "12", digit);
gsub(/[dD]/, "13", digit);
gsub(/[eE]/, "14", digit);
gsub(/[fF]/, "15", digit);
}
result = result + digit*(16**power);
power++;
}
return (result);
}'

My script seems to be going nowhere

Please help
Many Thanks
Chris

 

[frangac]

Hi Annihilannic,

Many thanks for your help.
Code as requested.

#!/bin/ksh
#set -x

od -x $2 | awk -v opt=$1 -v opt_1=$2 'BEGIN { i = 0 }
{
for (j=2; j<=NF; j++) {
a[i++]=substr($j,1,2)
a[i++]=substr($j,3,2)
}

}
END {
for (k=0; k<i; k++) {
if (a[k] == "48" && (a[k+1] == "37"|| a[k+1] == "31")){
print "header Chris"
k+=41
}
if ((a[k] == "37") && (a[k+1] == "81" || a[k+1] == "80")) {
print "BODY"
k+=46
}
if (a[k] == "54" && a[k+1] == "76" && a[k+6]=="3a"){
print "TAIL"
#k+=71
}
}
}


Thanks Once Again
Chris

 

[Annihilannic]

Your code seems to work fine for me given the test data that you posted, maybe your data is different?

I get this output:

[tt]header Chris
BODY
BODY
TAIL[/tt]

Which awk are you using and on what platform? Maybe it doesn't like 48000 element arrays.

Annihilannic.

 

[frangac]

Hi Annihilannic,

Thanks. I have changed k+=45 instead of 46 and all is looking well but for this one thing only.
When the counter reaches k=999990 as indicated below + 45, it cannot read any further
Is there any explanation to this and how can we solve this.

999898 3781
BODY
37810116282339aaaa200601101238529038187aaaaaaaaaaa000000010000295300aabd782d562020412020313331
999944 3781
BODY
37810118538015aaaa200601101238520216578402aaaaaaaa000000030000565300edd3b91c562020422020315350
999990 3780
BODY
37800114826020aaaa203838383838aaaaaaaaaaaaaaaaaaaab4b4b4b4b4b4b4b4b4b4b450505050505050505050aa
1.00004e+06 aaaa
1.00004e+06 aaaa
1.00004e+06 aaaa
1.00004e+06 aaaa


Many Thanks
Chris

 

[Annihilannic]

I think you're starting to run into limitations of awk.. at a certain point it considers numbers to be floats. I'd recommend breaking the file up into chunks before processing, or changing the script to read it in and load the a[] array in chunks. With 999990 elements in that array it must be using a hell of a lot of memory, a test I did yesterday showed it to be using about 200MB of memory for about 400000 elements, if I recall.

I refer back to my original recommendation, this is really something that should be done in C.

Annihilannic.

 

[frangac]

Hi Annihilannic,

Thanks. I would also recommend it but before that I thought I give awk a bash just to see on how it performes. Anyway many thanks for your assistance and help.

Much appreciate it

Many Thanks
Chris