Making a Map Multi-Threaded Question 1

[clk430]

I know that using wild cards for input and output should force a map to be multithreaded.

But I'm a bit confused as to what the Workspace settings should be. Does it matter if it is File vs Memory? How about the WorkFile Prefix? Create or delete? Unique or Mapname? Do these matter?

What about a run command inside the map? For expample:
RUN("MyPutFTP.mmc", "-WM -IE1S" + NUMBERTOTEXT(SIZE(TEXT(FTPData)))

Does the -WM override the IFD Workspace setting and make it Memory?

Thanks in advanced.

 

[BocaBurger]

Work area has to be memory or unique. Audit log, if used has to be unique, no tracing. All outputs have to be unique. You should not change map settings for a RUN map in the IFD, since you shoulod not be deploying RUN maps with an MSL.



BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[clk430]

I see. So, if my map is single threaded, then if I get a "source not available" soft error (the file still gets processed), then the double trigger issues you diagnosed in the following thread

http://www.tek-tips.com/viewthread.cfm?qid=1158859&page=1

technically can't happen?

 

[BocaBurger]

Untrue. Threading has nothing to do with it. If the map is single threaded, the trigger can occur, the second map is in a pending state until the first is completed.

BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[clk430]

Man, you're good. Thanks Boca.

 

[BocaBurger]

That's why I get the big paycheck. Currently it is


18" long and 6.5" high




BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[clk430]

LOL! I'm going to have to borrow that line if you don't mind.

Last question(s): If my maxthreads and watchmaxthreads in the .ini were set to 20 and 20 respectivly, AND I followed all of the rules stated above and had only one system running on the server that picks up 20 files (of the same size), renames them and places them in a folder, technically, they should all be picked up at once and placed in the folder asynchronously, right?

Does the maxthreads apply only to the .mms, .mmc or both?

 

[BocaBurger]

That joke is older than our combined ages Maxthreads applies to the .mmc (executable map). As long at you have the CPU horsepower (or enough CPUs) to run 20 threads equally, you should see 20 maps running simultaniously. There will be some delay due to overhead and directory scanning for the file trigger. FileListencount etc.

Of course the resulting names have to be unique. Sometimes a time stamp is not fast / granular enough to run 20 maps.

Best performance is usually between 2 and 4 threads (maps) per CPU, but one map can take 100% of a CPU depending on the map validation and rules.



BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[clk430]

You're that old?

I will play with the settings (maxthreads watchmaxthreads) to effectively correlate to our hardware. 12 and 8 respectively.

What are the negative ramifications of having those particular settings too high (like 20 and 20 in our case) with only two procs @ 2.8 ghz?

 

[BocaBurger]

Two procs at 2.8 GHz could run well with 12 threads. As you add more threads performance could drop. This needs to be tested. One map can take 100% of a CPU, depending on it's design (rules, functions etc.). It depends on how long the map takes to run (DB connections etc.).

BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[clk430]

In my quest to multithread some poor performing systems, I've come across some confusing system/mapping configuration that a certain former Mercator employee person setup in our company.

He created 1 .mms which has 5 or so run maps. These run maps (mostly generic in=out) are called extensively throughout all of our systems and maps.

If our volume of inbound and outbound data is very large and performance is a priority, is this bad design? Shouldn't each system system have their own generic .mmc instead of using one that every other system uses? Can having too many .mmc's hinder Mercator in anyway?

And I know, any more questions from me and you'll have to send me a BILL 18" long and 6.5" high.

 

[BocaBurger]

If the RUN map is thread safe, and you are using a current version that allows RUN map caching, there is no performance loss with RUN maps. many companies use a blob in blob out map to FTP data or use it with a target over ride to distribute data to many places. There is one big gotcha: If you have multiple developers and someone decides to create a new RUN map and uses the name of a previous one, when this is deployed, a problem can occur in the map is completely different from the first.

This has happened several times.



BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[clk430]

We are using 6.7 right now. I'll investigate weather we are able to utilize run map caching.

We do keep all of our .mmc's in one folder. We have some RUN maps with the same exact name, but in different .mms's. You're saying, if those maps are different (even slightly) problems can occur?

 

[BocaBurger]

OK, so what happens when .mmc abc.mmc from mms abc.mms gets built when abc.mmc from xyz.mms was what you are looking for?

From the 6.7.1 mercator.ini
;use the following to define the max number of RUN maps to be cached in the memory
;default value is 300, minimum value is 10
;set this value to -1 to disable RUN map caching
;RunMapCacheMaxNum=300

Just uncomment the line. Also, do not deploy a .msl with your RUN maps. This will load them into memory twice and waste RAM.



BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[clk430]

OK, so what happens when .mmc abc.mmc from mms abc.mms gets built when abc.mmc from xyz.mms was what you are looking for?"

That's exactly it. In theory, EITHER abc.mmc can get built, right? If so, we need to do some changing.

And about the .ini file, I was always cofused about the commenting. There are 'default' values that are commented out. Is it set to 300 by default even if it's commented out? If not, then I will uncomment and set it at 300.

I will test first, if there is plenty of RAM, are there any negative ramifications from uncommenting this parameter on the fly?

 

[BocaBurger]

Any change to the .ini will not take effect until the ES is restarted. If you are OK with RAM, there should be some increase in perfomance, if a RUN map is called repeatedly within one map (from a nested Functional map, for example).

The abc.mmc that gets deployed is the last one built when you deploy from the IFD. Good idea to keep all maps in one directory (makes checking problems easier), but you need to have a naming convention that prevents conflicts. You could add the developer's initials to the map name: bb_abc.mmc.

This way, if there is a problem with a map, you also know who has the source and was working on it.


BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[clk430]

Awesome! OK, gotta run. LAST question though. So, since we had the RunMapCacheMaxNum=300 parameter commented out, did that mean that we didn't have run map chaching enabled? Meaning, our run maps were single threaded (or enabled to the value set in maxthreads/maxconcurrent maps)

Thanks for all your help. Have a Merry Christmas!!

 

[BocaBurger]

Caching is not threading. RUN maps run under the thread of the calling map. The cache prevents them from being loaded from the hard drive every time that are called.

BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[clk430]

I forgot to ask, should 'skip if busy' = NO?

And if this system that I enable is to be multithreaded, and say that it is on source event to pull 5,000 files already in the trigger directory (basically, take 1/2 a day to move), will enabling multi-threading in this system mean that this will use up all the availble threads (12 as set in the .ini)? Will this system allow other systems to run if this one always has triggers to satisfy?

I know setting maxconccurent maps in the IFD to other important maps would overcome this possiblity, but what about the systems that don't have a maxconccurent setting set?

Man, I can't believe I'm thinking about this on the holidays...

 

[BocaBurger]

If there are sufficient source events to max out the maxthreads setting, then that Event Server would only run that map, unless the watchmaxthreads and or maxconncurrent say otherwise.



BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!