end of line character - Windows vrs Unix

[ddiamond]

In December we are going to move our DB2 from a Windows server to an AIX box. At the moment I don't have any unix boxes to play with, but I'm just trying to anticipate any problems that might arise. Every night we currently load many ascii delimited data files into our windows DB2. When we move our DB2 to AIX, I imagine we will need to FTP these files from windows to AIX. I've heard rumors that Unix uses different characters for end of line than Windows. Our Windows files use chr(13) + chr(10) for EOL. Will this cause a problem for unix? Will we need to convert the files?

 

[p5wizard]

Not a rumor - a fact!

But it shouldn't be a problem, just make sure the FTP transfers are done in mode ASCII.


HTH,

p5wizard

 

[ddiamond]

Thanks p5wizard.

 

[culleoka]

ddiamond,

I intercept files from a Windows 2K server, with ASCII data. I then edit these files in Visual Foxpro & copy back out to flat files for input to a Sun Solaris Unix server.

The Unix server kept rejecting the data.

Took forever to figure out, but after I started inserting an extra CR at the end of the record, it works fine. I'm not sure if this is a problem with the software program or the OS. But, if you have problems with Unix not taking the data files in ASCII format, try ending each record with CR+CR+LF.

culleoka

 

[ddiamond]

culleoka,

How are you copying the files to your Sun Solaris Unix server? If p5wizard is correct, FTP should be smart enough to handle the eol conversion. In your experience, is this not the case? CR+CR+LF sounds quite odd.

- Dan

 

[SumoRunner]

Yes, ftp takes care of it. But if you use Samba so you can cross-mount the file systems, use the dos2unix freeware utility in your scripts.

 

[ddiamond]

How does the perfomance of Samba compare to ftp? It sounds like Samba is more convenient, but is ftp faster for file transfers?

 

[SumoRunner]

Samba (

http://us5.samba.org/samba/)

does not transfer files. It allows a Unix machine to become a Windows SMB client and mount it's file systems and printers as a Windows share. So you just open a file manager and drag & drop. The problem is there is no ascii text conversion that way and you have to have a utility like dos2unix running in your unix process to alter the end of line chars.

 

[columb]

As well as dos2unix (which doesn't seem to come with AIX 5.1) you can use

tr -d '\r' < infile > outfile

Columb Healy

 

[ddiamond]

Does anyone know which would be faster:

1. Using Samba to mount my windows drive to AIX, running a script to convert the end of line character, and then another script to load the data from the windows mounted drive into DB2.

2. Using FTP to move the files (1 GB) from windows to AIX and then running a script to load the data from the AIX drive into DB2.

Also, I just heard there is a possibility we may be moving to Linux instead of AIX (same hardware - IBM p5). Should that make any difference?

 

[mrn]

See

http://64.233.183.104/search?q=cach...tml+convert+windows+end+of+file+to+unix&hl=en

for more options

Mike

"A foolproof method for sculpting an elephant: first, get a huge block of marble, then you chip away everything that doesn't look like an elephant."

http://www.airport-parking-site.co.uk/

 

[ddiamond]

Thanks Mike.

 

[columb]

As regards your speed question, with only 1Gb of data it's not going to be time efficient to install/configure Samba unless you need it for something else. It's not that Samba is particularly difficult or long winded to install, it's just that installing Samba will take approx 1 hour whereas FTPing a 1Gb file should take less than that.

It really doesn't matter which flavour of *nix, AIX, Linux, Solaris etc. The CR/LF issues will be the same.

Columb Healy

 

[ddiamond]

columb,

This process is going to run nightly, so the only time I'm concerned with is the process after installation. Although I agree that Samba probably isn't worth it if all I want to do is avoid FTPing 1GB worth of files every night. I'm just looking for what ever solution will run the fastest because I have a small window of time to do it in.

 

[columb]

If it's a nightly process use Samba. I don't think there will be any significant differences in speed - LAN and NIC considerations will be far more significant. Samba has ease of use. In my opinion batch driven FTP scripts tend to be a pain in the fundament with passwords in plain text in some script somewhere unless you're using SFTP which won't do CR/LF conversion.

You might also want to look at the PuTTY family. If properly configured with public/private key pairs you can use pscp to copy the files across and plink to run the conversion/insertion scripts. The reason for running the scripts from the WinTel end is that resolves the issue of 'how do you know when the transfer is complete'.

Columb Healy

 

[ddiamond]

Columb,

Thanks for your advice. I don't know if my management will want to install Samba, but I will present it as an option. I've looked at the PuTTY familly before, but it didn't seem worth it to me. A simple KSH script can wait until the files have arrive and then kick off what every processes I want to run after they have arrived.

- Dan

 

[p5wizard]

SAMBA is fine if you only need to read the files once to process them. If the disk space is sufficient on the p5, I'd suggest copying them over (putty pscp/plink as described by Columb is fine, but you would need to convert the files dos2unix after copying, that could be part of the script launched via plink)

By copying the files over at night, your network won't be overloaded should you need to reload the file(s) for whatever reason during daytime hours. The file(s) would be available in p5 filesystems.

p5 AIX or p5 LINUX does not make a difference for CR/CRLF issue.


HTH,

p5wizard

 

[ddiamond]

p5 AIX or p5 LINUX does not make a difference for CR/CRLF issue.

Is there any performance differences between AIX and LINUX?

 

[p5wizard]

Haven't got any LINUX on p5, so don't know. AIX 53 can utilize SMT, which gives a perf boost on p5 machines. Simultaneous MultiThreading uses duplicated processor registers to run 2 processor threads on one processor, effectively showing up as 2 processor cores per real processor on the chip.

I'm unsure if Linux can utilize the p5 SMT or not.


HTH,

p5wizard