Need to read only last half hour

[Compscigui]

Good day to you all!

I have some REXX code that works for me right now, but I need it to work better. I am currently reading everything from a log file, and then sending that over TCP/IP to another product. Works great right now, except by the end of the day I am sending a ton of redundant data, like 50k lines worth of data that the other product has already received and digested. I would like to read only the last half hours worth of data in the log file, as I have an ESP job set to submit my JCl and REXX every fifteen minutes. This would give me some overlap, but nothing like what I have now.

Currently my code to read is:

/* Read all records from the MESSAGES DD into a stem variable */
"EXECIO * DISKR messages (STEM records."
return_code = RC
SAY records.0 ' records read'

Having trouble wrapping my brain around how to make this read only the last half hours worth of stuff. Any help would be greatly appreciated, as I am new to REXX.

 

[nclouston]

If you only read so many records at a time, say 1000, you could stop when you have reached 'current time' - 30 minutes. This assumes that the records have a time on them. Replace the * by the number of records to read, add code to take the current time (TIME()) and subtract 30 minutes, put the current processing inside a loop. Something like:

stop_time = time() - 30 /* NO - THIS IS NOT REAL CODE! */
read 1000 records
extract time from records.1
Do while extracted_time > stop_time
   do current processing
   DROP records.
   read 1000 more records
End

You should change the read all (*) to read a block at a time, anyway. All those records are being held in memory at one time which is not "a good thing".



Nic

 

[nclouston]

We need to know if your problem is resolved by the help given, or if you need further assistance.


Nic

 

[Compscigui]

It has not been resolved as of yet, because I don't know REXX very well, so pseudo code doesn't help me much right now.

 

[nclouston]

Show a sample record so that I can see its layout and work out some code for you. Or, if the time is in a fixed position in the record just let me know that position and the time format (hh:mm:ss for example).

Another thing, how many records do you generate per 30 minutes? Give a reasonable maximum plus 10% extra.


Nic

 

[nclouston]

This is the code that you may need:

/* Rexx */
/* some variables */
recs_per_half_hour = 1000	/* estimated records generated in 30 minutes */
time_difference = 30		/* length of time to go back (in minutes) */
t_start  = 34			/* start position of time field */
t_length = 8			/* length of time field */
do_more  = 1			/* loop control */
/*********/
/* Start */
/*********/
===> Any existing initial processing <===
now_minutes = Time('M')		/* Current time in minutes */ 
stop_time = now_mimutes - time_difference	/* Time we go back to */ 
Do While do_more = 1				/* Work until there are no more records */
						/* within the last 30 minutes */
						/* Read first batch of records */
   "EXECIO "recs_per_half_hour" DISKR messages (STEM records."
						/* Extract time of first record in batch */
   time_in_record = Substr(record.1, t_start,t_length)
   record_time = Time('M',time_in_record)	/* Convert to same format as stop_time */
   Do batch_ix = 1 To records.0			/* Process this batch */
      If record_time > stop_time		/* If record is within time frame */
      Then Do
===>	do current processing	<===
      End
      Else Do					/* Rest of records are too old */
	 do_more = 0			        /* Switch off processing */
	 batch_ix = record.0 + 1		/* Force end of processing of the batch */
      End
   End
   DROP records.	                        /* Get rid of this batch of records */
   If do_more = 1	                        /* Haven't gone back too far so get another batch */
   Then "EXECIO "recs_per_half_hour" DISKR messages (STEM records."	/* read next batch of records */
End
===> continue end of data set processing <===

Note the arrowed (===>) comments. This is where your existing code will go.
Also note that the 3 variables t_strat, t_length and records_per_half_hour will need to be changed to reflect reality.
Also, no provision has been made for processing records across midnight. I am not going to worry myself about that. Let your local Rexx guru do it. It is, possibly, what he is paid to do. I am not.


Nic

 

[Compscigui]

Sorry this took awhile for me to respond to, I have had the flu. Here is the current JCL and REXX that runs and grabs the log file. I will try to implement what you have responded with, but I wanted to share this just incase you see an easier way. We get between 1000 and 5000 logs each hour, depending on how busy the system is.

//DXZ4JOBC JOB (SCRUBBED),'COMPSCIGUI',             00000119
// MSGLEVEL=1,MSGCLASS=T,PRTY=10,TIME=5                                 00000201
//*EXEC CNTR3                                                           00000323
//*MAIN ORG=CNTR3,CLASS=BATCH                                           00000423
//OPERSCAN EXEC OPERLOGP                                                00000516
//OPRDATA  DD DISP=SHR,DSN=SCRUBBED.OPERLOG3                             00000723
//PRTOLOG  DD  DSN=&&OLOGOUT,UNIT=DISK,SPACE=(CYL,(100,100)),           00000821
//         DISP=(,PASS),DCB=LRECL=133                                   00000921
//OPRPARMS DD *                                                         00001221
 REPORT  PRTOLOG                                                        00001321
 TIME    0000 2359                                                      00001424
 MESSAGE ICH70001I                                                      00001521
 MESSAGE ICH408I                                                        00001621
//*                                                                     00001721
//* PARSE THE OLOG OUTPUT TO COMBINE ICH408I MESSAGES ONTO              00001821
//* A SINGLE LINE.                                                      00001921
//*                                                                     00002021
//PARSE    EXEC PGM=IRXJCL,DYNAMNBR=50,REGION=6000K,                    00002813
//            PARM='AUTHPARS'                                           00002913
//SYSTSPRT  DD SYSOUT=*                                                 00003013
//MESSAGES  DD DSN=&&OLOGOUT,DISP=SHR                                   00003118
//OUTDD     DD DSN=&&AUTHOUT,UNIT=DISK,SPACE=(CYL,(100,100)),           00003416
//          DISP=(,PASS),DCB=LRECL=500                                  00003518
//SYSTSIN   DD DUMMY                                                    00003613
//SYSEXEC   DD DSN=COMPSCIGUI.REXX.EXEC,DISP=SHR                           00003719
//*                                                                     00003816
//* UNCOMMENT TO WRITE A SECOND COPY TO THE JOB SYSOUT */               00003916
//*                                                                     00004016
//IEBCOPY  EXEC PGM=IEBGENER                                            00004116
//SYSUT1    DD DSN=&&AUTHOUT,DISP=(SHR,PASS)                            00004216
//SYSUT2    DD SYSOUT=(,)                                               00004316
//SYSPRINT  DD SYSOUT=(,)                                               00004416
//SYSIN     DD DUMMY                                                    00004516
//*                                                                     00005013
//* TRANSMIT THE MESSAGES TO THE SPLUNK SERVER USING SYSLOG             00005113
//*                                                                     00005213
//TRANSMIT EXEC PGM=IRXJCL,DYNAMNBR=50,REGION=6000K,                    00005316
//            PARM='SCRUBBED'                       00005420
//SYSTSPRT  DD SYSOUT=*                                                 00006016
//MESSAGES  DD DSN=&&AUTHOUT,DISP=SHR                                   00007016
//SYSTSIN   DD DUMMY                                                    00008016
//SYSEXEC   DD DSN=COMPSCIGUI.REXX.EXEC,DISP=SHR                           00009019

TRACE N     /* TRACE I - verbose, R - intermediate, N - normal, E - errors */
 
/* Parse and display the parameters passed by the batch job */
parse arg ipaddress port
say 'Endpoint -' ipaddress':'port 
 
/* Read all records from the MESSAGES DD into a stem variable */
"EXECIO * DISKR messages (STEM records." 
return_code = RC             
SAY records.0 ' records read'  
 
/* Initialize */
call Socket 'Initialize', 'SPLUNKER'
if src=0 then initialized = 1
else call error 'E', 200, 'Unable to initialize SOCKET'
if server='' then do
  server = Socket('GetHostId')
  if src^=0 then call error 'E', 200, 'Cannot get the local ipaddress'
end
 
/* Initialize for receiving lines sent by the server                  */
s = Socket('Socket')          /* Would add ,DATAGRAM type here if UDP */
if src^=0 then call error 'E', 32, 'SOCKET(SOCKET) rc='src
hostname = translate(Socket('GetHostName'))
if src^=0 then call error 'E', 32, 'SOCKET(GETHOSTNAME) rc='src
call Socket 'Connect', s, 'AF_INET' port ipaddress
if src^=0 then call error 'E', 32, 'SOCKET(CONNECT) rc='src
call Socket 'SetSockOpt', s, 'SOL_SOCKET', 'SO_ASCII', 1
 
crlf="0D"X          /* store a literal for the end of record character */
 
/* Loop to write records to the server   */
DO i = 1 to records.0
    sendstring = strip(records.i)                        
    call Socket 'Write', s, sendstring        
    if src^=0 then call error 'E', 32, 'SOCKET(WRITE) rc='src
    call Socket 'Write', s, crlf                                                      
    if src^=0 then call error 'E', 32, 'SOCKET(WRITE) rc='src        
END
 
/* Terminate and exit                                                 */
call Socket 'Terminate'
exit 0
 
EXIT return_code
 
socket: procedure expose src
  a0 = arg(1)
  a1 = arg(2)
  a2 = arg(3)
  a3 = arg(4)
  a4 = arg(5)
  a5 = arg(6)
  a6 = arg(7)
  a7 = arg(8)
  a8 = arg(9)
  a9 = arg(10)
  parse value 'SOCKET'(a0,a1,a2,a3,a4,a5,a6,a7,a8,a9) with src res
return res
 
/* Syntax error routine                                               */
syntax:
  call error 'E', rc, '==> REXX Error No.' 20000+rc
return
 
 
/* Halt processing routine                                            */
halt:
  call error 'E', 4, '==> REXX Interrupted'
return
 
/* Error message and exit routine                                     */
error:
  type = arg(1)
  retc = arg(2)
  text = arg(3)
  ecretc = right(retc,3,'0')
  ectype = translate(type)
  ecfull = 'RXSCLI' || ecretc || ectype
  say '===> Error:' ecfull text
  if type^='E' then return
  if initialized
     then do
       parse value Socket('SocketSetStatus') with . status severreason
       if status^='Connected'
          then say 'The status of the socket set is' status severreason
     end
  call Socket 'Terminate'
exit retc

 

[nclouston]

Sorry to hear about the 'flu. Nasty business!

2500 records per 30 minutes should be ok then.
But I do need to know the time each record was generated. Being a logfile each record should have time on it. But where? Unless this information is available it would not be possible to do what you want.

Not in rexx, anyway. You could save current the logfile and compare against the next logfile, writing out the differences and simply send the differences. When comparing the new logfile that would have to be saved and used as the 'old' ogfile next time. The compare could easily be done using JOINLEYS option of your sort productg.


Nic

 

[Compscigui]

Here is a sample of some of the log file I am working with, looks like the time starts at column 26:

N 0000000 CP3X 18220 00:00:04.91 JES3 00000281 IRR010I USERID SCRUBBED
N 4000000 CP3X 18220 00:00:04.94 TADZXXXX 00000090 HSIZ009I DATA WRITTEN TO
N 4000000 CP3Y 18220 00:00:04.95 TADZXXXX 00000090 HSIZ045I 3W00 CREATED RE
N 4000000 CP3Y 18220 00:00:04.95 TADZXXXX 00000090 HSIZ047I 3W00 USAGE MONI
N 0000000 CP3X 18220 00:00:04.96 JES3 00000090 IAT6100 (JOB99999) JOB E
S SUB=JOB9990 ORG=LOCAL
N 0000000 CP3X 18220 00:00:04.96 JES3 00000290 IAT6100 (JOB57845) JOB E
S SUB=JOB62360 ORG=LOCAL
N 0020000 CP3Z 18220 00:00:05.03 QRFWLMG 00000281 ICH70001I SCRUBBED LAST
N 0000000 CP3X 18220 00:00:05.04 DXZXXXXX 00000090 IAT2000 JOB DXZXXXXX (JO
N 0020000 CP3X 18220 00:00:05.05 DXZXXXXX 00000281 ICH70001I SCRUBBED LAST
N 4000000 CP3X 18220 00:00:05.04 TADZXXXX 00000090 HSIZ046I 0X00 DELETED RE
N 0000000 CP3X 18220 00:00:05.05 DXZXXXXX 00000090 IEF403I DXZXXXXX - START
N 0020000 CP3X 18220 00:00:05.12 ESP 00000281 ICH70001I SCRUBBED LAST

 

[nclouston]

I make it 22 but if the reords are variable length then the 4-byte RDW would make it 26.
The lines with SUB in them could be a problem because they do not have a time but I guess they relate to the previous line - just extra coding.

I really think that the alternative solution, comparring and sending differences, would be the way to go - but not as much fun as a bit of Rexx coding. (JCL coding is fun too!)


Nic