Question / UNIX text file formats

[StevenB]

Hi folks,

I'm running into a text file problem, wanted to know if anyone could help.

Basically, I have some UNIX shell scripts that run a PL/SQL program. At the end of the process, it outputs several text files, which are then sent to a vendor's HTTPS site using cURL.

During testing, most of the files were sent manually, and no problems were experienced. However, once we started sending the files in an automated fashion, it seems that the different text formats of UNIX and windows has started causing problems.

Basically, when we sent the files manually from Windows machines, the files had both a CR and a LF at the end of each line, but now they only have a CR. (I'm not 100% certain that that's accurate - I'm trying to confirm right now.)

But, the idea is right - it has to do with UNIX and DOS-format text files treating end-of-line differently.

Does anyone know, is there any way I can send these files from the UNIX box using cURL, but not have them arrive in this UNIX text format?

Thanks,

Steve

 

[StevenB]

Ah, the specifics are that the UNIX version leaves off the Carriage Return (char 13).

 

[SamBones]

Depending on your version of Unix, you probably already have a utility to convert the file. For example, on Sun Solaris there are two utilities, "[tt]unix2dos[/tt]" and "[tt]dos2unix[/tt]", that make the conversion for you. I'm sure you have something like that.

 

[Ogzilal]

Hi,

The CR LF problem appears when text files are sent by FTP from Windows to unix in BINARY mode rather than ASCII mode.
Try to send text files by FTP in ASCII mode

Ali

 

[StevenB]

Unfortunately, FTP is not an option - I am required to post these files using HTTPS. I also haven't been able to find a conversion utility on my OS (unixtodos definitely does not exist). It's AIX.

However, I have managed to convert the files from UNIX to DOS format prior to sending by using awk:

awk 'sub("$", "\r")' oldfilename > newfilename

I'm currently waiting to hear back from our vendor on whether this actually worked, but so far it looks good.

I also tried using sed, but it didn't seem to work quite right. Others have reported success using this syntax:

sed -e 's/$/^r/' oldfilename > newfilename

Thanks everyone for your help!

Steve

 

[feherke]

Hi

Why the caret ( ^ ) and not backslash ( \ ) like in the [tt]awk[/tt] code ?

sed -e 's/$/[red][b]\[/b][/red]r/' oldfilename > newfilename

Feherke.

http://rootshell.be/~feherke/

 

[StevenB]

Oops, sorry. I had originally tried with the backslash like in the awk code. The carat was from when I was just messing around after it didn't work the first way.

When I used this:

sed -e 's/$/\r/' oldfilename > newfilename

I ended up with the character "r" literally printed at the end of each line of text. I don't know if perhaps it's an older version of sed? Some articles I read indicated that would only work with newer versions of sed.

 

[mrn]

Try

sed 's/$/^M/g' < oldfile > newfile

where ^M is produced be pressing

CTRL-V followed by CTRL-M

You will also have to add a CTRL-Z to the end of the file


Or perl

perl -p -e 's/$/\r/' < unixfile.txt > dosfile.txt

Mike

"Whenever I dwell for any length of time on my own shortcomings, they gradually begin to seem mild, harmless, rather engaging little things, not at all like the staring defects in other people's characters."

 

[KenCunningham]

Perhaps the AIX equivalent is dosread -a and doswrite -a?

I don't mind people who aren't what they seem. I just wish they'd make their mind up.

 

[mrn]

Nope

doswrite requires a dos formatted filesystem to write out to.

Mike

"Whenever I dwell for any length of time on my own shortcomings, they gradually begin to seem mild, harmless, rather engaging little things, not at all like the staring defects in other people's characters."

 

[KenCunningham]

So it does. Just ignore me!!

I don't mind people who aren't what they seem. I just wish they'd make their mind up.

 

[StevenB]

OK, here's the final news.

awk was able to get the job done:

awk 'sub("$", "\r")' oldfile > newfile

Earlier I was having a problem because apparently you can't pipe a file into itself. (I was trying to basically do awk 'sub("$", "\r")' myfile > myfile).

Looks great. Thanks very much for your help, everyone!

Steve

 

[PHV]

(unixtodos definitely does not exist). It's AIX.
Install dos utils from base system and I think you'll get the unix2dos and dos2unix commands.

Hope This Helps, PH.
Want to get great answers to your Tek-Tips questions? Have a look at FAQ219-2884 or FAQ181-2886