FTP: Uploading only partial file

[jsparrow]

Hello,

Perl newbie needs some help! If a large file, the below FTP-perl script is only uploading part of the file. If it's a small file, the upload is successful.

Our EDI send files all get dumped to one file and then I’m taking that, renaming it, running an external process on them, then pushing to the remote server (FTP). If the file is small, or has one EDI interchange, there’s no issue. The problem arises when there are more than one interchange, and/or the file is large…it then uploads the partial file to the remote server (sometimes even a NULL file). My first thought was maybe perl is running faster then the FTP is responding and it can’t keep up. I tried installing a few pauses (SLEEP commands) but still no luck.


## Define Global Variables
my $Name;
my $xprocess;
my $newname;
my $password;
my $username;
my $remote_dir;
my $host;
my $local_dir;
my $filename;
my $log;


## Assign values to Variables
$host = "host";
$password = "Password";
$username = "username";
$remote_dir = "send";
$local_dir = "C:/Program Files/COMMERCE Connection32/VPN FILES/Send";
$log = "C:/Perl/Programs/ftp-log.txt";

## create system date
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime;
$year += 1900;
$mon += 1;
if($mon <= 9){$mon = "0$mon"};
if($mday <= 9){$mday = "0$mday"};


## Open log file
open(LOG, ">>$log");
print LOG "***********FTP SEND LOG***********\n";
print LOG "$mon-$mday $hour$min\n";

## create variable for filename
$Name = ("C:/Program Files/COMMERCE Connection32/VPN FILES/Send/send.txt");


## check to make sure the file exists
if (-e "$Name"){
print LOG "$mon-$mday $hour$min file exist $Name\n" ;
}else
{
print LOG "$mon-$mday $hour$min ###ERROR file does NOT exist $Name\n";
}

## create variable for external process - bat file. This bat file will kick off the external process that splits the data
$xprocess = "C:/Perl/Programs/xprocess-send.bat";

## system stop until xprocess is completed (bat file is finished)
system($xprocess);

print LOG "$mon-$mday $hour$min xprocess\n" ;

## create variable adding year, month, day and hour and minute, with the extension *.edi
$newname = "$mon$mday$hour$min.edi";


## rename the send.txt file to date +.edi
rename($Name, "C:/Program Files/COMMERCE Connection32/VPN FILES/Send/$newname") or die print LOG "$mon-$mday $hour$min ###ERROR problem renaming send.txt\n";

print LOG "$mon-$mday $hour$min file renamed\n" ;

## assign variable newname to filename variable - this is the file that will be uploaded to the remote server
$filename = "C:/Program Files/COMMERCE Connection32/VPN FILES/Send/$newname";


print LOG "$mon-$mday $hour$min file named $filename\n";

## check to make sure the file exists
if (-e "$filename"){
print LOG "$mon-$mday $hour$min file does exist $filename \n" ;
}else
{
print LOG "$mon-$mday $hour$min ###ERROR file does not exist $filename\n";
}

## check to make sure the file is readable
if (-r "$filename"){
print LOG "$mon-$mday $hour$min file is readable $filename\n" ;
}else
{
print LOG "$mon-$mday $hour$min ###ERROR: file is not readable $filename\n";
}


open(FILE,$filename);
close(FILE);

use Net::FTP;

## Connect to address
$ftp = Net::FTP->new($host) or die print LOG "$mon-$mday $hour$min ###ERROR Can't connect: $host\n";

print LOG "$mon-$mday $hour$min FTP session started\n";

## Attempt to login
$ftp->login($username, $password) or die print LOG "$mon-$mday $hour$min ###ERROR Can't Login $username, $password\n";

print LOG "$mon-$mday $hour$min Login successful\n";

## Change to remote directory
$ftp->cwd($remote_dir) or die print LOG "$mon-$mday $hour$min ###ERROR Couldn't change directory $remote_dir\n";

print LOG "$mon-$mday $hour$min Changed Directory: $remote_dir\n";

## Transfer file to remote directory
$ftp->put($filename) or die print LOG "$mon-$mday $hour$min ###ERROR Couldn't put $filename\n";

print LOG "$mon-$mday $hour$min Uploaded file\n";

## FTP ends
$ftp->quit;


print LOG "$mon-$mday $hour$min FTP session ends\n";

## move file
use File::Copy;

## move send file to send directory
$status = move($filename, "C:/Program Files/COMMERCE Connection32/VPN FILES/Send/Send History/$newname") or die print LOG "$mon-$mday $hour$min ###ERROR file move failed: $!";

print LOG "$mon-$mday $hour$min file moved to C:/Program Files/COMMERCE Connection32/VPN FILES/Send/Send History\n";

print LOG "$mon-$mday $hour$min ------END FTP SEND------\n";

close(LOG);

 

[PaulTEG]

Try upping the timeout (defaults to 120ms)

$ftp = Net::FTP->new($host, -Timeout=>'500') ;

HTH
--Paul

It's important in life to always strike a happy medium, so if you see someone with a crystal ball, and a smile on their face ... smack the fecker

 

[Wonboodoo]

On a quick glance your script looks OK. Questions:
1) Is it possible that the script is called before the file is fully created locally?
2) Is it possible that you can end up with multiple copies of this script running simultaneously, and one script FTPs the file and moves it before the second script completes it's FTP of the same file?

 

[jsparrow]

Added the timeout but still receiving the error "Missing Interchange trailer segment" (only when sending a large file). It seems to be pushing only 160 lines of data then drops everything else.

1) Is it possible that the script is called before the file is fully created locally?
2) Is it possible that you can end up with multiple copies of this script running simultaneously, and one script FTPs the file and moves it before the second script completes it's FTP of the same file?

- I took out all the file naming, and file moving and ran the script below after the file was fully created and still no luck. Dropped in some pauses thinking that might help - but still only sends 160 lines.

use Net::FTP;

## Connect to address
$ftp = Net::FTP->new($host, -Timeout=>'500') or die print LOG "$mon-$mday $hour$min ###ERROR Can't connect: $host\n";

sleep(5);
print LOG "$mon-$mday $hour$min FTP session started\n";

## Attempt to login
$ftp->login($username, $password) or die print LOG "$mon-$mday $hour$min ###ERROR Can't Login $username, $password\n";

sleep(5);
print LOG "$mon-$mday $hour$min Login successful\n";

## Change to remote directory
$ftp->cwd($remote_dir) or die print LOG "$mon-$mday $hour$min ###ERROR Couldn't change directory $remote_dir\n";

sleep(5);
print LOG "$mon-$mday $hour$min Changed Directory: $remote_dir\n";

## Transfer file to remote directory
$ftp->put($filename) or die print LOG "$mon-$mday $hour$min ###ERROR Couldn't put $filename\n";

sleep(10);
print LOG "$mon-$mday $hour$min Uploaded file\n";

## FTP ends
$ftp->quit;

sleep(10);

 

[MikeLacey]

Hi,

Can you send another, large, file?

I'm wondering if there are control characters in the file that are convincing Net::FTP that it's reached the end of file.

Create a big-enough text file - use notepad - and transfer that with your script.

Mike

"Deliver me from the bane of civilised life; teddy bear envy."

Want to get great answers to your Tek-Tips questions? Have a look at faq219-2884

 

[jsparrow]

Actually this solved the problem:

## Binary mode
$ftp->binary;

 

[MikeLacey]

Thought it might, it was seeing a ^Z in the file somewhere no doubt.

Mike

"Deliver me from the bane of civilised life; teddy bear envy."

Want to get great answers to your Tek-Tips questions? Have a look at faq219-2884

 

[jsparrow]

Thanks Mike!!

Maybe you have a tip on this:

Doing this it gets all the FTP files and delivers them to the directory in where my PERL script resides. I want to download the files to another local-directory.

## get all files
$ftp->cwd($remote_dir);
my @get = $ftp->ls;
print LOG "@get @files\n";
if (@get) {
foreach (@get) {$ftp->get ($_) or die print LOG "$mon-$mday $hour$min ###ERROR Can't fetch files $!\n";}
}
$ftp->quit;



I tried this but if we download more than one file, it is only putting one of the files into the $receive_file:

foreach (@get) {$ftp->get ($_, $receive_file);

 

[waiterm]

do you not have to put a hash in there because you're working through the array, maybe something like:

foreach (@get)  { $ftp->get($_,$get[$recieve_file])};

Can't remember if it's square or curly brackets, try em both.

Rob Waite

http://www.hostpipe.co.uk?src=TT

 

[MikeLacey]

errm curly I think

Sorry, I didn't reply because I've no way of testing this - I don't use Net::FTP at all now, we're all secure-shell types here nowadays...

Mike

"Deliver me from the bane of civilised life; teddy bear envy."

Want to get great answers to your Tek-Tips questions? Have a look at faq219-2884

 

[jsparrow]

Trying these works but when receiving more than one file, rather appending all the downloaded files to receive_file, it's overwriting.

foreach (@get) {$ftp->get(@get,$receieve_file};

foreach (@get) {$ftp->get($_,$get[$receive_file]);

 

[PaulTEG]

So recieve all the files, and then concatenate (append)
-Paul

 

[waiterm]

You may well have figures this out by now, but I've got one further suggestion. Try using the hash on both sides of the get() method i.e.

foreach (@get)  { $ftp->get($get[$receive_file],$get[$receive_file]) };

That way both the source file and downloaded file should hopefully be named the same on each iteration of the loop.

Rob Waite

http://www.hostpipe.co.uk?src=TT