Mercator's FTP Speed/Voulme Capabilities

[clk430]

6.7 on Win2K.

We're bottle-necking on a certain outbound system. The application area sends us about 3000 tif images a day, that require ecryption then sending to the TP's server. I investigated and found that each map in teh system pulls one file at a time.

So I suggested multi-threading (Work area either unique or memory, Audit log unique or off, No fixed output file names, No single threaded RUN maps called by the map, Wild card for the input etc.) as the current system violates all of these rules!

The "non technical consultants" shot down my idea and said the problem is that Mercator itself is the bottle neck and can only send 1.3mb/minute and can only use 1 single threaded connection.

Wha?? Can we "multithread" the FTPput maps and FTP multiple files at once? How can we speed this up?

 

[BocaBurger]

If you are not doing any transforms in the map, (blob to blob) DS TX can FTP as fast as the OS can. FTP maps can multi-thread, but you can, (and I have seen this happen) overwhelm the FTP server or network. This would indicate the map is not the bottleneck.

The 1.3mb/minute is probably from an untuned, improperly designed system, or the bottleneck was someplace else.

One client had DS TX running on a multi-CPU AIX server and the FTP server was a single processor Windows box. It was relatively easy to crash the FTP server.

BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[clk430]

Interesting. We have a 1 proc 2.8ghz box on Win2K environment and I'm not sure what our FTP server is or it's limitations.

I guess, my question is, in a fast, capable environment, will multithreading my FTPPUT map force Mercator to send multiple transactions at once at the same rate it send one? Or will it send multiple at once, but at a much slower rate (the old bandwith phenomena)?

How many FTP connections can Merc 6.7 keep open at once?

 

[BocaBurger]

Whatever the bottleneck is, will be. Too many variables to make a guess. I/O on the local server? Network llimits? FTP server limits?

Event Server can use about 6 to 8 threads per CPU on a box of 2.8 ghz. without too much impact on speed, depending on how much transformation is being done. One map can tale 100% of a CPU if it is required by the map's design (rules & functions).



BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[eyetry]

Check your ini file ftp settings.

;IdleFTP=1
;SLimFTP=0
HLimFTP=10 ; High/Hard limit for concurrent FTP connections
;KeepFTP=0
;MinFTP=0

Also, you can tell your 'non techie' friends that DSTX does not have it's own ftp protocal. That it uses services native to the box FTP services are running on. So, the limitation you are running into could be based on TX settings but it is also limited to band width. And, band width is limited based on machine configuration on both ends of the FTP transfer as well as network configuration.

I ran into the same problem here with HTTP transactions. Even when our sniffing tool showed that the slow down was not DSTX but envolved an Oracle Stored Proc, APPC gateway and our mainframe our Java people said it was DSTX and they could write a faster system (Java people crack me up). so far they haven't and it's been 4 years.

Anyway, it wasn't a total waist as I was able to perform some tweeks that ultimately improved TX performanced as it related to other HTTP systems.

--------------------------------------------------

PS: You could setup a grinder script to perform multiple simultatious FTPs. I used it to test multithreading and HTTP map performance. It posted about 2500 http queries to the TX Event Agent. The maps processed the queries and posted a response. When the cycle was complete the tool provided statistics for my review. At the same time we ran a sniffer tool on each machine affected by the queries to see if and where performance was affected. I think Grinder is a free download. Microsoft has a tool of its own you can use too.

 

[clk430]

Eyetry,
Thanks for the tip on the grinder script. I'll look into that, as I've been looking into sniffers as well.


Our .ini settings are:
IdleFTP=1
;SLimFTP=0
;HLimFTP=0
;KeepFTP=0
;MinFTP=0

Should we comment the idleftp back? What will that do for us?

 

[eyetry]

Not sure what the default HLimFTP is. Boca should know. Worst case you can try uncommenting it and setting it to something reasonable.

Be carefull with the Idle, Keep and Min setting for FTP. As you know Idle connections get reused first. If you have a bad connection and it is idle it cold be reused. In this situation your FTP should fail. From Mercator support several years ago.....may have even been from Boca if he is who I think he is....

"IdleFTP, two high, can cause FTP's to que up and/or increase frequency of FTP failures, too low timeouts could cause FTP failures as well."

Our idle setting is '1' and seems to work well here but we aren't doing the FTP volume you are.

You'll need to play with it.

 

[BocaBurger]

I didn't write that, and I think it's backwards. If Idle is too high and the map takes too long to complete, the connection to the server can be dropped. FTP connection is opened when map starts, but data is not actually sent until the card commit occurs.

Too short and you waste time re-establishing a connection you could have re-used. Don't comment out Idle, it will leave the connection open until something fails.





BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[eyetry]

This is a good discussion point....

I understood that a: idle connections were reused before establishing new connections. b: reusing idle connections is more efficient than establishing a new connection. c: reusing idle connections runs the hazard of a potential failure due to a bad idle connection. d: keepxxx, within parameters defined in the ini file, tests idle connections before reusing and if bad established new connection.

 

[clk430]

Something crazy happened today. From 9:30am to 12:30, all FTP connections, in and out, were failing in Mercator. All the logs indicated that connections were attempeted, but failed. The firewall people saw activity to and from, but no files were sent/pulled.

We rebooted the server and sure enough, everything re-established.

It's things like this, unexplicable, that makes my job a PITA.

Have any of you experienced something like this?

 

[BocaBurger]

You are on Windows? Rebooting is SOP. Consider DS TX 8.0 on Linux running on the same hardware.



BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[clk430]

SOP? I know SOB. And I proposed DS TX 8.0, Unix/Linux. They laughed at me. But then they stopped and said we can get 8.0 on Windows 2003 with a new dual proc server (already being built). I'm in the Pacific Northwest - it's Microsoft country out here.

 

[clk430]

Back to the idleFTP discussion. Mercator/IBM support (a while back) recommended it to be between 10 and 60 to increase performance. Right now we are set at 1.

I will play with it, but it's hard the replicate the thousands of connections we use everyday in prod with dev.

So I'll start with this: What are the negative ramifications of increasing to, say 30?

 

[BocaBurger]

Only possible negative could be if the FTP server is flaky and it breaks idle connections in less than the IdleFTP setting.
Solution, fix the FTP server. Default for most is 5 minutes, idle, drop connection. You want the IdleFTP to come into play when your system is not busy, but you want to re-use conenctions when it is busy.



BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!