piping data to Net::FTP 1

[fishiface]

I want to regularly transfer an enormous amount of data computed from a remote database query to a remote ftp server and don't want to find it a temporary home on my machine.

Net::FTP says that the first argument to put() can be a filehandle so I have tried using this with anonymous pipes and named pipes. My idea was that, as the database returns more rows, I write them to the pipe and Net::FPT should read them and send them straight out over the network.

The ftp is slightly complicated by the presence of a proxy but I have proved that the ftp parameters are correct: a file is transferred if I pass a valid filename to put() rather than a filehandle. With both of the examples below, however, a zero-byte file is created on the remote ftp server.

Here's my try using an anonymous pipe

[COLOR=#006600]#!/usr/bin/perl[/color]

[COLOR=#0000FF]use[/color] strict;
[COLOR=#0000FF]use[/color] Net::FTP;
[COLOR=#0000FF]use[/color] IO::Handle;

[COLOR=#0000FF]my[/color] $host = [COLOR=#808080]'ftp.myserver.co.uk'[/color];
[COLOR=#0000FF]my[/color] $user = [COLOR=#808080]'testacc'[/color];
[COLOR=#0000FF]my[/color] $pass = [COLOR=#808080]'yeahsure'[/color];
[COLOR=#0000FF]my[/color] $rdir = [COLOR=#808080]'Connect/Download'[/color];
[COLOR=#0000FF]my[/color] $file = [COLOR=#808080]'/etc/motd'[/color];

[COLOR=#0000FF]my[/color] $child_rdr   = [COLOR=#FF0000]new[/color] IO::Handle;
[COLOR=#0000FF]my[/color] $parent_wtr  = [COLOR=#FF0000]new[/color] IO::Handle;
[COLOR=#FF0000]pipe[/color]( $child_rdr, $parent_wtr ) [COLOR=#FF8000]or[/color] [COLOR=#FF0000]die[/color] [COLOR=#808080]"Can't open pipe: $!"[/color];;

$parent_wtr->autoflush([COLOR=#FF0000]1[/color]);

[COLOR=#0000FF]if[/color]( [COLOR=#0000FF]my[/color] $pid = [COLOR=#FF0000]fork[/color] ) { [COLOR=#006600]# parent simply writes to the pipe[/color]
        [COLOR=#FF0000]die[/color] [COLOR=#808080]"cannot fork: $!"[/color] [COLOR=#0000FF]unless[/color] [COLOR=#FF0000]defined[/color] $pid;
        $child_rdr->[COLOR=#FF0000]close[/color]();

        [COLOR=#FF0000]open[/color](IN,[COLOR=#808080]'/etc/motd'[/color]) [COLOR=#FF8000]or[/color] [COLOR=#FF0000]die[/color] $!; [COLOR=#006600]# simulate much data[/color]
        [COLOR=#0000FF]while[/color]( <IN> ) {
                [COLOR=#FF0000]print[/color] $parent_wtr;
        }

        $parent_wtr->[COLOR=#FF0000]close[/color]();
        [COLOR=#FF0000]waitpid[/color]($pid,[COLOR=#FF0000]0[/color]);
} [COLOR=#0000FF]else[/color] { [COLOR=#006600]# child[/color]
        $parent_wtr->[COLOR=#FF0000]close[/color]();

        [COLOR=#0000FF]my[/color] $ftp = Net::FTP->[COLOR=#FF0000]new[/color]( [COLOR=#808080]'proxy'[/color], Debug => [COLOR=#FF0000]1[/color])
               [COLOR=#FF8000]or[/color] [COLOR=#FF0000]die[/color] [COLOR=#808080]"Cannot connect to proxy: $@"[/color];
        $ftp->login( [COLOR=#808080]"$user\@$host"[/color], $pass )
               [COLOR=#FF8000]or[/color] [COLOR=#0000FF]return[/color] [COLOR=#808080]'Login failed'[/color];
        $ftp->cwd( $rdir )
               [COLOR=#FF8000]or[/color] [COLOR=#FF0000]die[/color] [COLOR=#808080]"Can't chdir to $rdir: "[/color], $ftp->message();
        $ftp->put( $child_rdr, [COLOR=#808080]'test'[/color] )
               [COLOR=#FF8000]or[/color] [COLOR=#FF0000]die[/color] [COLOR=#808080]"Put $file failed: "[/color], $ftp->message();
        $ftp->quit();

        $child_rdr->[COLOR=#FF0000]close[/color]();
        [COLOR=#FF0000]exit[/color];
}

and here's one using a named pipe

[COLOR=#006600]#!/usr/bin/perl[/color]

[COLOR=#0000FF]use[/color] strict;
[COLOR=#0000FF]use[/color] Net::FTP;
[COLOR=#0000FF]use[/color] IO::Handle;

[COLOR=#0000FF]my[/color] $host = [COLOR=#808080]'ftp.myserver.co.uk'[/color];
[COLOR=#0000FF]my[/color] $user = [COLOR=#808080]'testacc'[/color];
[COLOR=#0000FF]my[/color] $pass = [COLOR=#808080]'yeahsure'[/color];
[COLOR=#0000FF]my[/color] $rdir = [COLOR=#808080]'Connect/Download'[/color];
[COLOR=#0000FF]my[/color] $file = [COLOR=#808080]'/etc/motd'[/color];
[COLOR=#0000FF]my[/color] $pipe = [COLOR=#808080]'/tmp/ftp.pipe'[/color];

[COLOR=#006600]# system return val is backwards, so && not ||[/color]
[COLOR=#006600]#[/color]
[COLOR=#FF0000]unlink[/color] $pipe; [COLOR=#006600]# allow to fail silently - it shouldn't exist[/color]
$[COLOR=#0000FF]ENV[/color]{PATH} .= [COLOR=#808080]":/etc:/usr/etc"[/color];
[COLOR=#0000FF]if[/color]( [COLOR=#FF0000]system[/color]([COLOR=#808080]'mknod'[/color],  $pipe, [COLOR=#808080]'p'[/color])
 && [COLOR=#FF0000]system[/color]([COLOR=#808080]'mkfifo'[/color], $pipe)
) {
        [COLOR=#FF0000]die[/color] [COLOR=#808080]"mknod/mkfifo $pipe failed"[/color];
}

[COLOR=#0000FF]if[/color]( [COLOR=#0000FF]my[/color] $pid = [COLOR=#FF0000]fork[/color] ) { [COLOR=#006600]# parent[/color]
        [COLOR=#FF0000]die[/color] [COLOR=#808080]"cannot fork: $!"[/color] [COLOR=#0000FF]unless[/color] [COLOR=#FF0000]defined[/color] $pid;
        [COLOR=#0000FF]my[/color] $out = [COLOR=#FF0000]new[/color] IO::Handle;
        [COLOR=#FF0000]open[/color]( $out, [COLOR=#808080]'>'[/color], $pipe )
                [COLOR=#FF8000]or[/color] [COLOR=#FF0000]die[/color] [COLOR=#808080]"Can't open $pipe for writing: $!"[/color];
        $out->autoflush();

        [COLOR=#FF0000]open[/color](IN,[COLOR=#808080]'/etc/motd'[/color]) [COLOR=#FF8000]or[/color] [COLOR=#FF0000]die[/color] $!; [COLOR=#006600]# simulated input data[/color]
        [COLOR=#0000FF]while[/color]( <IN> ) {
                [COLOR=#FF0000]print[/color] $out;
        }

        $out->[COLOR=#FF0000]close[/color]();
        [COLOR=#FF0000]wait[/color](); [COLOR=#006600]# mop up our dead child[/color]
        [COLOR=#FF0000]unlink[/color] $[COLOR=#FF0000]pipe[/color]; [COLOR=#006600]# and tidy up[/color]
} [COLOR=#0000FF]else[/color] { [COLOR=#006600]# child[/color]
        [COLOR=#0000FF]my[/color] $in = [COLOR=#FF0000]new[/color] IO::Handle;
        [COLOR=#FF0000]open[/color]( $in, [COLOR=#808080]'<'[/color], $[COLOR=#FF0000]pipe[/color] )
                [COLOR=#FF8000]or[/color] [COLOR=#FF0000]die[/color] [COLOR=#808080]"Can't open $pipe for reading: $!"[/color];

        [COLOR=#0000FF]my[/color] $ftp = Net::FTP->[COLOR=#FF0000]new[/color]( [COLOR=#808080]'proxy'[/color], Debug => [COLOR=#FF0000]1[/color] )
                [COLOR=#FF8000]or[/color] [COLOR=#FF0000]die[/color] [COLOR=#808080]"Cannot connect to proxy: $@"[/color];
        $ftp->login( [COLOR=#808080]"$user\@$host"[/color], $pass )
                [COLOR=#FF8000]or[/color] [COLOR=#0000FF]return[/color] [COLOR=#808080]'Login failed'[/color];
        $ftp->cwd( $rdir )
                [COLOR=#FF8000]or[/color] [COLOR=#FF0000]die[/color] [COLOR=#808080]"Can't chdir to $rdir: "[/color], $ftp->message();
        $ftp->put( $in, [COLOR=#808080]'test'[/color] )
                [COLOR=#FF8000]or[/color] [COLOR=#FF0000]die[/color] [COLOR=#808080]"Put failed: "[/color], $ftp->message();
        $ftp->quit();

        [COLOR=#FF0000]exit[/color];
}

Has anyone made something similar work or can anyone see what I'm doing wrong?

Yours,

fish

["]As soon as we started programming, we found to our surprise that it wasn't as easy to get programs right as we had thought. Debugging had to be discovered. I can remember the exact instant when I realized that a large part of my life from then on was going to be spent in finding mistakes in my own programs.["]
--Maur

 

[fishiface]

Got it.

print $parent_wtr;

should be

$parent_wtr->print( $_ );

I was unthinkingly trying to use indirect object syntax, as in the more standard

print OUT $data, "\n";

but perl was interpreting the object as a real argument to print.

Object lesson 1: when coding unusual stuff, the bug is not always in the unusual bit. I was concentrating on the child process and barely glancing at the parent code.

Object lesson 2: avoid indirect object syntax and use the explicit -> operator exclusively. There are just too many pitfalls to save one key-stroke.

Yours,

fish

["]As soon as we started programming, we found to our surprise that it wasn't as easy to get programs right as we had thought. Debugging had to be discovered. I can remember the exact instant when I realized that a large part of my life from then on was going to be spent in finding mistakes in my own programs.["]
--Maur

 

[MikeLacey]

Nice...

Mike

I am not inscrutable.

Want great answers to your Tek-Tips questions? Have a look at faq219-2884

 

[MikeLacey]

That's gotta one for a Tips & Tricks FAQ

Mike

I am not inscrutable.

Want great answers to your Tek-Tips questions? Have a look at faq219-2884

 

[fishiface]

I'm wondering about creating a module. The interface could be as simple as

use Net::FTP::Pipe;

my $ftp = new Net::FTP::Pipe::Put(
     $host,
     -rpath => $remote_path, # other args all standard Net::FTP
     @std_NetFTP_opts
   ); # store parameters
...
$ftp->print( $stuff ); # first print forks and opens connection
while( my @data_row = $query->fetch() ) {
  $ftp->print( process( @data_row ) ); # print to pipe
}
$ftp->close(); # close connection, kill and wait for child process

which looks nice and useable to me.

I'm not sure to what extent it could usefully inherit any methods from Net::FTP as the Net::FTP object lives only in the child.

Yours,

fish


["]As soon as we started programming, we found to our surprise that it wasn't as easy to get programs right as we had thought. Debugging had to be discovered. I can remember the exact instant when I realized that a large part of my life from then on was going to be spent in finding mistakes in my own programs.["]
--Maur

 

[MikeLacey]

That looks neat.

Would the parent need to inherit any methods from Net::FTP?

My (simple minded) approach would be for Net::FTP:ipe to not use any methods from Net::FTP but just be a front-end for a child process which would do the FTP stuff - based upon the connection details etc it got passed from the parent.

Mike

I am not inscrutable.

Want great answers to your Tek-Tips questions? Have a look at faq219-2884

 

[fishiface]

I've been reading up on Net::FTP and I think that you are right. All the stuff I can think of that you might want to do to the handle (such as my simple-minded two-stage proxy login) can be done automatically with proper constructor arguments.

f

["]As soon as we started programming, we found to our surprise that it wasn't as easy to get programs right as we had thought. Debugging had to be discovered. I can remember the exact instant when I realized that a large part of my life from then on was going to be spent in finding mistakes in my own programs.["]
--Maur