Uploading files via FTP OK, but SFTP is not

[weberm]

I am converting some UNIX shell scripts that use FTP to upload a load file produced by a PL/SQL script to the mainframe via SFTP instead of FTP, but when I replaced FTP with SFTP, it's making the records "wrap around". Making this even stranger, I have other accounts on this UNIX server that are just fine and the data file looks fine.

Here is the data on the UNIX server:

S12079AJ100010175797112-041058 CARLOS A LOPEZ DE ALBEAR                8197           00000030
S12079AJ100020175803012-041065 ELLIS GED BODDEN                        99542          00000015
S12079AJ100030175805012-041066 ELLIS GED BODDEN                        99544          00000015
S12079AJ100040175807112-041069 ELLIS GED BODDEN                        99545          00000015
S12079AJ100050175807312-041071 ELLIS GED BODDEN                        99441          00000015
S12079AJ100060175807712-041075 ELLIS GED BODDEN                        99467          00000015

This what I get on the minaframe using SFTP:

S12079AJ100010175797112-041058 CARLOS A LOPEZ DE ALBEAR                8197           00000030..S12079
AJ100020175803012-041065 ELLIS GED BODDEN                        99542          00000015.S12079AJ10003
0175805012-041066 ELLIS GED BODDEN                        99544          00000015.S12079AJ100040175807
112-041069 ELLIS GED BODDEN                        99545          00000015.S12079AJ100050175807312-041
071 ELLIS GED BODDEN                        99441          00000015.S12079AJ100060175807712-041075 ELL
IS GED BODDEN                        99467          00000015.S12079AJ100070175811012-041077 GREENSPAN 
LAW FIRM                      7150           00000015.S12079AJ100080175823212-041092 GREENSPAN LAW FIR

I thought perhaps it's an end-of-line char problem causing the records to right-shift, but I have other accounts uploading their files fine and the test instance using the same code works fine as well.

sftp -b ftp/ftp2mf ftp@scdc.dfs.state.fl.us

put /u003/tmp/sop.dat //FTP.RCP.PS29.TRAN
quit

Any ideas?

 

[PHV]

What about this ?
[CODE ftp/ftp2mf][!]bin[/!]
put /u003/tmp/sop.dat //FTP.RCP.PS29.TRAN
quit[/code]

Hope This Helps, PH.
FAQ219-2884
FAQ181-2886

 

[Annihilannic]

What type of mainframe are we talking about here, and what type of character set does it use? ASCII? EBCDIC? And what type of line or record terminators?

I'm guessing that what PHV is suggesting would make your existing FTPs behave in a similarly undesirable fashion to sftp.

With FTP in "ascii" mode (as opposed to "binary" mode which PHV's suggestion would enable), you get some automatic character translation for free; unfortunately it varies a lot by FTP implementation and depending on the FTP client and server software in use. So by switching to sftp you may need to do this manually before or after transferring the file.

Annihilannic
[small]tgmlify - code syntax highlighting for your tek-tips posts[/small]

 

[fredericofonseca]

if this is a IBM mainframe (Z/os) look at the ftp site options - some will change the behaviour of how things are transferred regardless of the client used.
You most likely need to do it for any charset conversion between your mainframe and your unix machine - don't take it for granted that it will always work with defaults

By looking at the data it seems like the record size on the mainframe is bigger than the that of the file you sending - this would cause the shifting depending on the transfer mode.

Regards

Frederico Fonseca
SysSoft Integrated Ltd

http://www.syssoft-int.com

FAQ219-2884
FAQ181-2886

 

[weberm]

What about this ?

[CODE ftp/ftp2mf]
[red]bin[/red]
put /u003/tmp/sop.dat //FTP.RCP.PS29.TRAN
quit
[/code]

Sorry. sftp, or at least the version I am running (protocall version 3), does not like that.

 

[weberm]

Here are some more details. The files on the UNIX server are processed by PL/SQL and the mainframe is IBM (Z/os). Fredericofonseca's observation reminded me that the programs which write the file (PL/SQL on the UNIX box, COBOL on the mf) might be writing a header record that is shorter than the record size used for the rest of the file. I'm going to have to experiment with padding or omitting the header record so the record size is the same for all records in the file.

 

[weberm]

I tried skipping writing the header record (which is shorter than the data records) but that did not help.

You most likely need to do it for any charset conversion between your mainframe and your UNIX machine - don't take it for granted that it will always work with defaults

You reminded me of a UNIX command I had completely forgotten about, unix2dos. I added it to my script and it worked. I also noticed that according to the "man sftp", the put command mentions flags but I do not see them listed. I wonder if there's a flag I can use to set the record size?

put [flags] local-path [local-path]

 

[tarn]

Just a thought ... have you tried basic scp transfer?

scp <filename> user@host:<remotePath>/

Or another option (assuming you have ssh) is to zip or compress your files before transfer and then unzip/uncompress again once transferred thus encapsulating your file to maintain the format .....

Scripted would be something like:

# optional(1) 
# /usr/sbin/cksum /u003/tmp/sop.dat
# store result for use later ....

# 1 compress file to transfer
/usr/local/bin/gzip -9 /u003/tmp/sop.dat

# 2 secure copy file to remote
/usr/local/bin/scp /u003/tmp/sop.dat.gz remoteuser@remote.host:/destination/path/sop.dat.gz

# 3 send remote command to uncompress via ssh
/usr/local/bin/ssh remoteuser@remote.host "guzip /destination/path/sop.dat.gz"

# optional(2)
# /usr/local/bin/ssh remoteuser@remote.host "/usr/sbin/cksum /destination/path/sop.dat"
# compare result against earlier stored result and or take action

Just and alternative maybe ?


Laurie.

 

[chgwhat]

did you try

sftp> site I=dos J=mvs
sftp>put /u003/tmp/sop.dat //FTP.RCP.PS29.TRAN
sftp>quit

I=mvs: The line delimiters used in the file are NL (\n, 0x0a) and CR (\r, 0x0D).

I=mvs-ftp: The only line delimiter used in the file is NL (\n, 0x0a), any CR (\r, 0x0D) is passed on as data.

I=unix: The line delimiter used in the file is NL (\n, 0x0a).

I=dos: The line delimiter used in the file is LFNL (\r\n, 0x0d0a).

I=mac: The line delimiter used in the file is LF (\r, 0x0d).

http://www.ssh.com/manuals/client-user/61/sftp-site.html

Tony ... aka chgwhat

When in doubt,,, Power out...

 

[weberm]

I asked about using scp but alas, my only option is sftp.

I used the site command and it didn't accept it. I'm guessing it's b/c we're using "SFTP protocol version 3".

I did find a list of flags one can use with put, but it appears my version is too old to support it, too.

I did discover that executing unix2dos in a local shell does the trick, but there must be a better way...

! unix2dos /u003/users/tmp/sop.dat /u003/users/tmp/sop.dat
put /u003/tmp/sop.dat //FTP.RCP.TS29.TRAN 
quit

This is so annoying!

 

[weberm]

Well, it turns out the issue was resolved after tinkering with the settings on the mainframe. Go figure!