Perl Ftp problem

[rachanta]

Hi All:

Let me explain what we do with a ftp perl script.

The ftp script runs as cron job, searches in three directoties on remote server, put the files on local directory only if the file is not there on local server, then deletes the original file on the remote server after successful file transfer.

The script has been doing exactly what it supposed to do for almost year but some how recently there were following in consistencies.

1) it finds a file on remote server
2) try to get the file but fail to process the file.
3) even though there is no instace of the file on local server. Still it deletes the file on remote server.


I repeat the script is functioning well even till date despite of the problems.

There are no permission problems, I doubt some network issues.

I really appreciate if someone can help me in this regard.
How to diagnose the exact cause for the above issue?
is there any way to check the file is been locked on remote server?


Thanks in advance.

 

[MikeLacey]

what are you using for ftp? Net::FTP? Mike
______________________________________________________________________
"Experience is the comb that Nature gives us after we are bald."

Is that a haiku?
I never could get the hang
of writing those things.

 

[rachanta]

Yes
we use Net::FTP module.

 

[MikeLacey]

Ok, well that's good news - so it should be easy for you to check for errors after doing your "get". Are you doing that? Mike
______________________________________________________________________
"Experience is the comb that Nature gives us after we are bald."

Is that a haiku?
I never could get the hang
of writing those things.

 

[Palooka]

I encountered a similar problem once, and that turned out to be the proxy/firewall. Because of that, I had to use passive mode ftp transfers. Since your script has been running flawlessly for a year, it might be such an external error source.

Hope this helps,
Palooka

 

[rachanta]

Yes we use Net::FTP module

 

[rachanta]

Thank you very much for your response,

Mike: We log all the ftp messages and it was so apparent that the script finds a file on remote server but for some reasons we don't see the file on local server. Still it deletes the file on remote server which it supposed to do only on successful transfer.

Palooka:

I am happy to see someone facing a similar problem. first of all how to ftp in passive mode? In our case both remote and local servers are located with in the fire wall.

need further help,
Ram

 

[Palooka]

My problem was similar in the way that I had a script working fine for some time, and then suddenly it stopped working. It was a VB-script, but nevertheless the same problem. The new firewall configuration prevented ordinary ftp traffic. I solved it by setting setting the ftp transfer mode to passive in the ftp-component i was using.

In Perl, the constructor comes with a

Passive

option to enable passive mode transfers:

http://www.perldoc.com/perl5.6.1/lib/Net/FTP.html#CONSTRUCTOR

You might also want to check out this link, which is a script right up your alley, with error-handling. It's called ftp-mover, and is explained quite well:

http://www.unixreview.com/documents/s=2426/uni1018361705067/0204e.htm

Hope this helps,
Palooka

 

[MikeLacey]

Ram -- are you checking for errors after your get command? Mike
______________________________________________________________________
"Experience is the comb that Nature gives us after we are bald."

Is that a haiku?
I never could get the hang
of writing those things.

 

[rachanta]

Mike:

We log ftp get errors also,
Actually my script looks for file with a specific extension and an associated token file when found these two it will try to bring the file to local server. Every activity is duely logged.
I like to categorize my problem like below
case 1. Finds the file and a token file say .X and .X.YES this we implemented to avoid duplicate file transfers. but unable to process the file for some reasons.

case 2.
It says nothing to get from remote host even thogu there is a file in remote host.

Even this weekend we see a similar error the script executed and did not pick up anything, log says nothing to process but there was a file with above said token file. I have no clue for this strange behaviour. My script removes (a flag) the original files on remote server only on successful transfer but in the above cases it deleted the files even though there is no file on local server.
what I do is if ( -f $FILE)) then down the script delete the files on remote host.

We don't see this behaviour when we ran the ftp script manually.

#############33
Palook


Thanks for your info I will let you know if I found anything interesting with my script. As I mentioned the script was working perfectly for good number of years and if I reckon correctly started such problems very recently may be from feb 2002. I do suspect some environmental issues but can figure out the root cause. Sometime this year they changed the IP address of the remote host I don't know this has anything to do with my problems.

I appreciate your response.

Thanks,

Ram

 

[MikeLacey]

The file you are getting, how and when is it created? Could it be that it is being created just as your script is performing the 'get'? Mike
________________________________________________________________________________

"Experience is the comb that Nature gives us, after we are bald."

Is that a haiku?
I never could get the hang
of writing those things.

 

[rachanta]

Mike:

The files on remote server are copied to the respective input directories may be a hour before my ftp script starts. Is there any way to find out the file is been locked whether or not.

Palooka:

I went through the information infact I am doing something similar to that but more comprehensive way. There was a typo error in my previous response I appreciate if you ignore that, and share some ideas

Ram

 

[MikeLacey]

I don't know, using Net::FTP, of an easy way to check if the file has been locked.

In the past I've solved this problem in the following way, it will only work if you have full control of the program that creates the file in the first place.

Let's say you're creating and the transferring a file called myfile.txt.

Your originating program opens, fills and then closes a file called myfile.txt, the problem is that it may not have finished writing the file before Net::FTP does a get().

Ok --

step 1: don't create myfile.txt, create myfile.tmp and then - when you've finished - rename it to myfile.txt.

step 2: change your Net::FTP script to check for the existence of myfile.txt before it does a get(), if it doesn't there it should wait a few seconds and then check again.

Having said all that, if the file is being collected an hour after it's being created then it doesn't sound like that's your problem. Mike
________________________________________________________________________________

"Experience is the comb that Nature gives us, after we are bald."

Is that a haiku?
I never could get the hang
of writing those things.

 

[rachanta]

Mike:

I got your point but in my case as you already appreciated there is no parallel processing. On remote server the file copied (for more clarity) an hour before our ftp process starts.

Also in some instances the ftp script finds a file in the pick up directory and was not able to bring to the local server. Colud you guess under what circumstances a ftp will fail besides network problem?

Ram