Is there a maximum size for working storage? 5

[slb0423]

We just ran into a problem where it seems that working storage section is too big, causing strange abends. Can someone tell me what the max size for working storage is? Is this a setting we can modify on our system? Thanks in advance.

 

[SiouxCityElvis]

Thanks, Tom.
Okay, so just to make sure I've got this straight.

I read my datafile, and store the name fields in to a Working Storage Table, correct? Because if I were to create another indexed File before working on the program that reads its name-fields, if I'm not mistaken a file can not have a record length longer than 65,280 according to page 8-35 in my RM/COBOL-85 manual.

So, just to make sure I understand you correctly I just want verify the following:

1) read my Indexed File, populating a table with two fields per record in Working storage with the primary key(5 bytes) and the Name-Field values.

2)Then prep the name fields for "@@" on spaces(indicating end of name in the field) and "@" for single spaces, which my occur between name values i.e. first, last, middle, alias, title, company name with multiple words, etc.
...this prep work will be my prep of each tbl-name-field.
INSPECT WS-TBL-NAME-FIELD REPLACING ALL SPACES WITH "@".

3)Then, upon receiving the input string from the user, perform a read loop through all the records in my new table residing in Working Storage.
a)INSPECT INPUT-STRING REPLACING ALL SPACES WITH "@".
b)INSPECT INPUT-STRING TALLYING CTR-1 FOR
CHARACTERS BEFORE INITIAL "@@".

.....in my loop looking at all WS-TBL-NAME-FIELDS
c)INSPECT WS-NAME-FIELD(idx) TALLYING CTR-2 FOR
ALL INPUT-STRING(1:CTR-1).
d)make a check that each time CTR-2 is incremented to move that to output variables as a match.

Does this sound correct? Table in the working storage, etc.?

Thanks.
-Elvis

 

[3gm]

Elvis -

I think you have the general gist of it down. The key is to be able to speed up your search to locate the desired record. Doing the SEARCH ALL should accomplish that.

Another concern: The table you construct for the SEARCH ALL will have to have a unique key field (the name) AND will have to be in sorted order by key. The table should use an OCCURS DEPENDING ON clause or you will have to fill unused entries with HIGH-VALUES (assuming ASCENDING key sequence).

Glenn

 

[SiouxCityElvis]

Okay, I think I understand what you mean.

I filled my table to use later in my SEARCH ALL. It has the name field which is the the ascending key and another field PIC 9(4) for a relative key to use later if necessary.
I used OCCURS 5000 times. I will implement the OCCURS DEPENDING ON code.

Here's my challenge upon trying this yesterday...
When I do a SEARCH ALL
AT END MOVE "Y" TO WS-EOF
WHEN TBL-NAME(KEY) = INPUT STRING
(success - match found code)
END-SEARCH

The part WHEN TBL-NAME(KEY) = INPUT STRING
(success - match found code)
will only find "John" matches for INPUT-STRING value "John".

I need it to find all "John" strings, eg. "John", "Johnathan", "longjohnsilver" etc.

I've done that with INSPECT, with the prep of converting cases from lower to upper, etc. But that was when I was reading through my whole datafile record by record and checking each name field. That takes too long and that is why I got started on this SEARCH ALL method of approach. Can the SEARCH ALL method find "John" for all my table fields in SEARCH ALL that have "John", "Johnathan", etc?
In other words, is there a way to combine the advantages of INSPECT with SEARCH ALL? I did not have any luck trying to do so. I'm kind of discouraged if the SEARCH ALL can only find exact matches because I was looking forward to the idea of using a binary search as it is so much faster than reading a file record by record until the end of the file.

Thanks.
-Elvis

 

[3gm]

Elvis -

You can't use binary search to do what you want to do. However, having tabled the data, you can still sequentially search the table much faster than rereading the file each time.

I'm sure there are other approaches to speeding this up, but I believe they become more complex very quickly.

Regards.

Glenn

 

[SiouxCityElvis]

Okay. Thanks.
Although other solutions might be more complex, I would still be happy to hear of them as speed is a priority for this project I'm working on.

Thanks again.
-Elvis

 

[slade]

Elvis,

You can't use the SEARCH verb in any flavor for what you want to do.

You can loop thru the table bumping the index(ss) by 1 via a PERFORM, then use INSPECT. This is time consuming in any event, but it seems the only way.

Perhaps, if you explained the business problem in a bit more detail someone could come up w/an alternative. Fer instance, could the user be coaxed into providing the 1st letter of the name, then the name fragment? E.g., j,nathan.

It could speed things up a bit. Just a thought.

Regards, Jack.

 

[k5tm]

Amarillo,

I am testing a possibility.

Can I presume that the main file (containing 5000 records) is relatively static?

from Austin...

Tom Morrison

 

[k5tm]

Also, what is the maximum length of INPUT-STRING -- the maximum length of a name you are willing to seek?

Tom Morrison

 

[SiouxCityElvis]

Tom, the maximum length of input string is 50.

There will be no updates on the file records(if that's what you mean by static). I am simply reading from my data file - all 5000 records.

Austin is a nice town.

 

[k5tm]

Ok... and I guess I should have asked another question.

Is there a practical minimum on the input string? As I read you 'requirements document' I can see that if the user types only a single character (for example, 'a') you are going to get a very large, and probably meaningless, result set.

Tom Morrison

 

[SiouxCityElvis]

Tom,

True, true. As a matter of fact, I actually thought of that. I'm sure I will do a check to see that the input string is greater than 1 character. I have to say, it'll probably be a minimum of 2 characters, since Al, BK, and things like that are all used for "also known as" type of names.

Elvis

 

[k5tm]

I think that you can use a cross reference file to good effect. Given that your 5000 record file does not change a whole lot, you can build a cross reference file in a separate job step before doing your name lookups. Assume that you have something like this:

fd  xref-file.
01  xref-record.
    02  a-unique-key.
        03  unique-num   pic s9(9) packed-decimal.
    02  name-key         pic x(50).
    02  record-containing-name pic x(5).

where a-unique-key is just that, and name-key is an alternate key, with duplicates (upgrade your RM/COBOL to eliminate the need for a-unique-key).

01  a-word      pic x(100).

Then, for each record of your 5000 (i.e. inside your READ NEXT loop):

do whatever you do to 'normalize' the NAME-FIELD (e.g. converting to upper case)
move main-file-key to record-containing-name
move 1 to I
perform until I > length of NAME-FIELD
  unstring NAME-FIELD delimited by all spaces
      into a-word count in J
      pointer I
  end-unstring
  if J > 1
     subtract 1 from J
     perform varying K from 1 by 1 until K > J
       add 1 to unique-key
       move name-field (K:) to name-key
       write xref-record
     end-perform
   end-if
end-perform

Now you can use the xref-file in a rather straightforward manner. Take the 'name' that the user is requesting, move it to the name-key field and:

MOVE INPUT-STRING(1:CTR-1) TO NAME-KEY.
START xref-file KEY = NAME-KEY SIZE CTR-1
   INVALID KEY 
      display "name does not exist"
END-START

At this point you may READ NEXT through the xref file, testing:

IF NAME-KEY (1:CTR-1) not equal INPUT-STRING(1:CTR-1)
    display "I've seen all the records containing name"
END-IF

Tom Morrison

 

[SiouxCityElvis]

Thanks, Tom. I will give this a shot tomorrow morning.
I'm running on RMCOBOL Ver 5.something.
So, my xref-file should be simplified as much as possible and built before coding this program, right? That's what you mean by "separate job step", correct?

fd xref-file.
01 xref-record.
02 a-unique-key.
03 unique-num pic s9(9) packed-decimal.
02 name-key pic x(50).
02 record-containing-name pic x(5).

Thanks.
-Amarillo

 

[SiouxCityElvis]

Tom, I forgot to ask,
What part of your example would cover the finding "JONATHAN", "NATHAN", and "NATHANIEL" as matches when the input string is simply "NATHAN". I need to find all three as matches. I think your example shows to store the name-fields from my existing Name Datafile into the xref file.
Confused a bit here.

Thanks.
-Amarillo

 

[3gm]

Tom's approach is an excellent one for many similar problems: preprocess some (relatively) static stuff to get a quick, easy data structure or file to work with. Unfortunately, it doesn't meet your needs.

A more complex approach I've considered is this:
Read your input file and concatenate all the name fields together into one large string (a table of one byte entries if you will). Keep a single separator between each name field, but otherwise, squish them down tight (squish is a technical term I learned about down South!) Searching that one large string for every occurence of "Nathan" should be fairly easy and fast to code. (Can I cop out here and leave this as an exercise for the reader?)

The problem then is how do you know which record each hit came from? You could solve that problem by building a separate table showing the starting position of each name field for each record. E.g. table entry (13) contains an integer that is the subscript of the beginning of the name field in the big string that originally came from record 13. This is trivial to do during the load phase of the large table. You could then identify which record was the source by doing a (manual) binary search of this table.

You wanted complex but fast - this should fit the bill.

Glenn

 

[k5tm]

First, yes I propose this as a separate program, run only after changes are made to the 5000 record file. Since each of the 5000 records can cause a lot of records to be created in the xref-file, it is something that you want to do in the middle of the night, for example. You are spending time doing the cross-reference to enhance (considerably) the performance of your name fragment lookups.

My proposal would write the following records into the xref-file for your example (assuming three records, with keys A, B, and C, in the original file, containing JONATHAN, NATHAN and NATHANIEL respectively):

NAME-KEY       RECORD-CONTAINING-NAME
===================================== 
AN             A
AN             C
ANIEL          B
ATHAN          A
ATHAN          C
ATHANIEL       B
EL             B
HAN            A
HAN            C
HANIEL         B
IEL            B
JONATHAN       A
NATHAN         A
NATHAN         C
NATHANIEL      B
NIEL           B
ONATHAN        A
THAN           A
THAN           C
THANIEL        B

(This is in the order of the key NAME-KEY.)

Now, in the example you use, the user enters 'nathan'. After you normalize this, it is NATHAN and you know it is 6 characters long. So when you execute

MOVE INPUT-STRING(1:CTR-1) TO NAME-KEY.
START xref-file KEY = NAME-KEY SIZE CTR-1
   INVALID KEY 
      display "name does not exist"
END-START

it will behave as if the following literal code was entered:

MOVE "NATHAN" TO NAME-KEY.
START xref-file KEY = NAME-KEY SIZE 6
   INVALID KEY 
      display "name does not exist"
END-START

At this point you key of reference (for subsequent READ NEXTs) is NAME-KEY and your file position indicator is pointed at the first instance of NATHAN, viz:

NAME-KEY       RECORD-CONTAINING-NAME
===================================== 
AN             A
AN             C
ANIEL          B
ATHAN          A
ATHAN          C
ATHANIEL       B
EL             B
HAN            A
HAN            C
HANIEL         B
IEL            B
JONATHAN       A
NATHAN         A    <=== file position after START
NATHAN         C
NATHANIEL      B
NIEL           B
ONATHAN        A
THAN           A
THAN           C
THANIEL        B

Now you start your READ NEXT loop, testing (in this case) the first 6 characters of name key for equality. You will find all three records.

Now, it is possible with my scheme to have multiple xref-records that point to the same original record. Depending on your application, you will need to take steps to eliminate duplicate references in your READ NEXT loop. For example, a record that contains "JONATHAN ANDERSAN" will produce 3 xref records that will match "AN", and there is no way for the program creating the xref-file to eliminate the duplicates because it doesn't know how long the name fragment to be looked up will be.

Tom Morrison

 

[k5tm]

Glenn:

You assert:
Unfortunately, it doesn't meet your needs.

In what way?

Tom Morrison

 

[deveshjogal]

Hi Amar

If speed is a issue then, I have an idea but these experts need to have a look at it. What from the reading I gather is u have a name field upto 200 characters per record. But its not that all 200 characters are occupied. Lets say the average will be 10 names with 10 characters each per record. If u try to create index file which will have all possible combinations from these name then what u need to do is use only a read command with an alternate index to get an exact macth. Y I am suggsting this is because u have a static file.

Lets say the name is of 10 characters so the possible combinations its generates with minimum 2 characters is
10*9/2= 45. Let me explain this lets say u have name AMAR
so the possible combinations are AM, AMA, AMAR, MA, MAR, AR
i.e. 4*3/2 = 6. So using the rough estimation u can have around 450 combination per record or for that matter about 2250000 records for all 5000 records. This will be one time task. And a single program can do that. This field will be an alternate key for the index file which will be have an incremental seqence no. as primary key which u will never use. U will also store the record no. and if required name no. within the record. So the structure will similar to following

01 in-rec.
05 unique key value 9(10) comp.
05 name-value pic x(25).
05 record-no pic 9(04).
05 name-no pic 9(03).

Name-no can help u resolve the situation when u will have 2 matches in a single name ie. lets say the name is ABCYXBC. And the user enter BC for search so this will give you two search. Or u can omit such entries where the record-no, word-no, content are same from the index file itself.

So U will have the file oraganization as dynamic. First read will be random with alternate keys and then u can use read next key word until there is a match.

I know somebody can improve on this also and we can end with a more better solution.

Devesh

 

[deveshjogal]

He Tom
more or less we have offered the same kind of solution. It took 45 minutes to type for me so I didnt see ur update.

devesh

 

[k5tm]

Glenn:

BTW, it is important to note that the older version of RM/COBOL being used by Amarillo has constraints on any data item whose length is greater than 65280. My suggestion was based upon those constraints.

Tom Morrison