Parse file and evaluate line - Complicated! 1

[rattrap47]

I have the following data in a set of files I need to look at and evaluate:

the file has a header row, and is delimited by <fsp> (wierd, I know)

I'm looking to see if the number of rows with an TimeInventoryUptime value between the file creation date and that same date +24 hrs is the same as the line count from a another file.

---------------- Begin Data Sample --------------------
NetworkElementID<fsp>DetailedInventory<fsp>TimeAdded<fsp>TimeDiscovered
<fsp>TimeDiscoveryChecked<fsp>TimeLastPolled<fsp>TimeLastEvent<fsp>
TimeConfigUpdated<fsp>TimeConfigChecked<fsp>TimeInventoryUptime<fsp>
TimeInventoryChecked<fsp>TimeFullInventoryUpdated<fsp>UptimeTimestamp<fsp>
TimeLastBoot<fsp>UptimeAtChassisChange<fsp>TimeDeleted
10.167.105.133<fsp><fsp>2008-10-28 12:41:11.633<fsp><fsp><fsp>2008-11-10 15:15:27.826<fsp><fsp>2008-11-10 15:15:36.670<fsp>2008-08-21 07:12:01.000<fsp>2008-11-10 15:15:27.826<fsp>2008-11-10 15:15:27.793<fsp>2008-11-07 19:29:58.693<fsp><fsp><fsp><fsp>
b-234-1<fsp><fsp>2008-10-28 12:41:12.183<fsp><fsp><fsp>2008-11-10 14:06:22.860<fsp><fsp>2008-11-10 15:12:43.896<fsp>2007-11-01 00:26:32.000<fsp>2008-11-10 14:06:22.860<fsp>2008-11-10 14:06:22.830<fsp>2008-11-07 19:29:58.923<fsp><fsp><fsp><fsp>
sacag13east-1<fsp><fsp>2008-10-28 12:41:12.333<fsp><fsp><fsp>2008-11-10 14:44:26.086<fsp><fsp>2008-11-10 15:19:01.120<fsp>2008-01-07 01:53:36.000<fsp>2008-11-10 14:44:26.086<fsp>2008-11-10 14:44:26.066<fsp>2008-11-07 19:29:59.126<fsp><fsp><fsp><fsp>
--------------- End Data Sample -------------------------

Here is what I have (please pardon the mess I'm kinda new at shell coding):
------------- Begin Code -------------------------
#!/bin/sh
## Variable declaration
tgFile=""
oscount=""
eiicount=""
eiiFileDate=""
echo "Please enter the file to check:"
read tgfile

unzip "$tgFile" 'out/OperatingSystem.download' -do /home/rlevau/download/
unzip "$tgFile" 'out/ElementInventoryInfo.download' -do /home/rlevau/download/
eiiFileDate=`ls -l /home/rlevau/download/out/ElementInventoryInfo.download | awk '{print $6,$7}'`
oscount$=`wc -l /home/rlevau/download/out/OperatingSystem.download`
eiicount=ElementCount(eiiFileDate)
if oscount=eiicount
then
cp $tgfile /home/rlevau/holding/$tgfile
fi

ElementCount (FileDate)
{
CountofElements=0
file="/home/rlevau/download/out/ElementInventoryInfo.download"
string=""
search="<fsp>"
do while not eof
nawk -f
string=getline $file
n=split(string,array,search)
if TimeInventoryUptime -ge `date '+%b %e'` and -le `date '+%b %e'` + 24
then CountOfElements=CountOfElements+1
fi
loop

Return CountOfElements
}
------------- End Code ---------------------------------

Thanks In advance

 

[booke]

Does this help at all...

awk -F"<fsp>" '{print NF}' #All lines above show 16 fields
awk -F"<fsp>" '{print $10}' #Prints TimeInventoryUptime

 

[Annihilannic]

So far your script has a mix of shell code and awk code without anything separating them. You need to clearly delimit the two so that they are interpreted properly, either by putting your awk code in single quotes, e.g. nawk 'awk code here' inputfile or using nawk -f some_file_containing_awk_script.

Another challenge is that date comparisons aren't easy in shell or awk (unless you're lucky enough to be using the GNU versions like those found on Linux, because they have enhanced date/time processing). For example, the ls command you are using to pull the date of the file from returns months in Jan, Feb, ... format whereas you need them to be numeric to compare them to the data in your file.

For example, to make the file's date numeric, you could do something like this:

eiiFileDate=`date +%Y``ls -l /home/rlevau/download/out/ElementInventoryInfo.download | nawk '
    BEGIN {
        mon_numeric["Jan"]="01"
        mon_numeric["Feb"]="02"
        mon_numeric["Mar"]="03"
        mon_numeric["Apr"]="04"
        mon_numeric["May"]="05"
        mon_numeric["Jun"]="06"
        mon_numeric["Jul"]="07"
        mon_numeric["Aug"]="08"
        mon_numeric["sep"]="09"
        mon_numeric["Oct"]="10"
        mon_numeric["Nov"]="11"
        mon_numeric["Dec"]="12"
    }
    { print mon_numeric[$6] $7 }
'`

That sets the eiiFileDate to something like 20081217103600. It assumes that the file's date will be in the current year.

You will also need to calculate that date +24 hours for your comparison logic, and convert it to the same format... which isn't straightforward. Personally I tend to use perl's date calculation facilities for stuff like this.

Then you can pass in the two numbers as variables to the awk script to count the lines you're interested in, e.g.

count=`nawk -F '<fsp>' -v eiiFileDate=$eiiFileDate -v eiiFileDatePlus24hrs=$eiiFileDatePlus24hrs '
    BEGIN { count=0 }
    {
        TimeInventoryUptime=$10
        # Remove spaces, dashes and colons
        gsub("[ -:]","",TimeInventoryUptime)
        if (TimeInventoryUptime >= eiiFileDate && TimeInventoryUptime < eiiFileDatePlus24hrs) { count++ }
    }
    END { print count }
'`

I hope this gives you some ideas anyway. If you have further questions, do mention which OS version you are using as it affects the utilities that are available to you; I'm guessing Solaris judging by your use of nawk.

Annihilannic.

 

[rattrap47]

Thanks Annihilannic,
You are spot on, it is Solaris.

The info is great, things that I wouldn't have thought of.

I'll let you know what happens, the lessons might be useful for someone else as well.

 

[rattrap47]

Okay, I'm getting a " 'done' unexpected" error.
I've tried everything I can find in the on-line docs. Iknow this has to be something I'm overlooking (I could code it in VB in about 10 minutes, but it's not an option)

The collection-list file is just a list (1 column wide) of 4 character ID's (like bcbf) with the last entry being "end".

so what I'm trying to do is loop through the collection list and process the corresponding file.

Thanks again for any help!

--------------------------------- Begin Code --------------
#!/bin/sh -x
# --------------------------------------------------------------
# Shell script to verify CNC download status (Full/Incremental)
#
# Created by: Ronson E. LeVau - 31 Oct 2008
# Last edit: REL 17-DEC-08
# --------------------------------------------------------------

## Variable declaration
file=collection-list
collection=""
tgFile=""
version=""
dlType=""
oscount=""
eiicount=""


# Function to count the element inventory lines
ElementCount ()
{
eiiFileDate=""
CountofElements=0
EIfile="/home/rlevau/download/out/ElementInventoryInfo.download"
string=""
search="<fsp>"
# eiiFileDate=`ls -l /home/rlevau/download/out/ElementInventoryInfo.download | awk '{print $6,$7}'`

eiiFileDate=`date +%Y``ls -l /home/rlevau/download/out/ElementInventoryInfo.download | nawk '
BEGIN {
mon_numeric["Jan"]="01"
mon_numeric["Feb"]="02"
mon_numeric["Mar"]="03"
mon_numeric["Apr"]="04"
mon_numeric["May"]="05"
mon_numeric["Jun"]="06"
mon_numeric["Jul"]="07"
mon_numeric["Aug"]="08"
mon_numeric["sep"]="09"
mon_numeric["Oct"]="10"
mon_numeric["Nov"]="11"
mon_numeric["Dec"]="12"
}
{ print mon_numeric[$6] $7 }
'`
# string=getline < $EIfile

# nawk -f '{n=split(string,array,search)}'

# TimeInventoryUptime is the tenth field
# if [ $10 -ge `date '+%b %e'` and -le `date '+%b %e'` + 24 ]
# then CountOfElements=CountOfElements+1
# fi
count=`nawk -F '<fsp>' -v eiiFileDate=$eiiFileDate -v eiiFileDatePlus24hrs=$eiiFileDatePlus24hrs '
BEGIN { count=0 }
{
TimeInventoryUptime=$10
# Remove spaces, dashes and colons
gsub("[ -:]","",TimeInventoryUptime)
if (TimeInventoryUptime >= eiiFileDate && TimeInventoryUptime < eiiFileDatePlus24hrs) { count++ }
}
END { print count }
'`
CountOfElements=$count
Return $CountOfElements
}

## Create collection list - Use this if we want to check the whole folder
# rm -f collection-list
# ls collections/transport-????.zip > /home/rlevau/collection-list

## loop through the collection list to get the Full collections
line_count=`wc collection-list | awk '{ print $1 }'`

while [ "$collection" != end ] # `seq $line_count`
do
read collection

# Create the transport file name from the current collection ID
echo $collection # for Testing
tgfile="/home/rlevau/collections/transport-$collection.zip"
echo $tgfile # for Testing
unzip -ojq "$tgFile" 'out/download.hdr' -d /home/rlevau/download/

# Check the version
version=`grep -i 'version' /home/rlevau/download/out/download.hdr | tr -d '<Version>' | tr -d '/'`
echo $version # for Testing

if [ "$version" -ge "5.1" ] # check if 5.1
then # Check to see if it's a full download
dltype=`grep -i 'downloadingtype' /home/rlevau/download/out/download.hdr | tr -d '<DownloadingType>' | tr -d '/'`
echo $dltype # for Testing

if [ "$dltype" = "Full" ]
then
cp $tgfile /home/rlevau/holding/$tgfile
fi

# If it's not 5.1 then see if it's 5.0
else if [ "$version" -ge "5" && -lt "5.1" ];
then
# process 5.0 logic
unzip -ojq "$tgFile" 'out/OperatingSystem.download' -d /home/rlevau/download/
unzip -ojq "$tgFile" 'out/ElementInventoryInfo.download' -d /home/rlevau/download/

oscount$=`wc -l /home/rlevau/download/out/OperatingSystem.download`

ElementCount()
eiicount=
if oscount=eiicount
then
cp $tgfile /home/rlevau/holding/$tgfile
fi
fi
echo

done #<"$file"

--------------------- END CODE -------------------------

 

[PHV]

Replace this:
else if
with this:
elif

Hope This Helps, PH.
FAQ219-2884
FAQ181-2886