Problem creating tab delimited file

[lmbylsma]

I have the script that prints out a tab delimited file. However, when I open it in Unix it puts these ^M symbols after 2 of the columns. When I open it in windows in notepad they appear as rectangle boxes instead, and in Excel it interprets it as go to a new line, so my columns get all out of alignment.

Here is an example of the output I get as viewed from Unix:

1.37657^M 0 16.37657^M 16.37657
3.92090^M 0 19.26123^M 19.26123
10.37657^M 1 22.21257^M 22.21257
50.37657^M 1 214.23345^M 214.23345
159.56789^M 0 238.11100^M 238.11100
500.26123^M 0 287.04997^M 287.04997
6056.12612^M 1 301.06797^M 301.06797


Here is the portion of code that prints the information to the file:


open(OUT, "> $out_file&quot ;


for my $i(0..$#Array32){

print OUT "\n$Array1920[$i]\t$Array1314[$i]\t$Array32[$i]";

}

close(OUT);

Any ideas what the problem is? What baffles me is that it happens on some columns and not others. Also that first column when I used to print it as the last column it didn't have the M^'s but when I moved it to first position then it started doing the M^'s. It doesn't make any sense to me.

-Lauren

 

[chazoid]

Are you using chomp() after reading in data from the input file? My guess is that the column with the ^M is at the end of the line in the data. ^M being a carriage return control code. Chomp will remove it - something like:

while (<INP>){
chomp;

# etc...

}

 

[lmbylsma]

Yes I am using chomp. And if that was the problem, it wouldn't make sense that the M^s would appear for one of the columns when I print it out in one order and go away when I print them out in another order. Here is the full text of my code:


#!/usr/bin/perl -w
use strict;
use warnings;
use diagnostics;

#my $datafile='matt6-03.txt'; #input filename
#open(DATA, $datafile) || die qq(Can't open &quot;$datafile&quot; for input...$!\n);

my $col2=99;
my @Array13;
my @Array14;
my @Array1314;
my @Array19;
my @Array20;
my @Array32;
my @Array1920;
my @Final;
my $x19=0;
my $y19=0;
my $x20=0;
my $y20=0;
my $x13=0;
my $y13=0;
my $x14=0;
my $y14=0;
my $diff=0;



while (<>) {

chomp; #remove newline from $_

if (/^&quot;CHANNEL&quot; &quot;13&quot;/) { #set $col2 depending on whether we've seen '&quot;CHANNEL&quot; &quot;19&quot;' or '&quot;CHANNEL&quot; &quot;20&quot;'
$col2 = 4;
}elsif (/^&quot;CHANNEL&quot; &quot;14&quot;/) { #this tells us which array to push to, and is also the data we want in the second column
$col2 = 5;
}elsif (/^&quot;CHANNEL&quot; &quot;19&quot;/) {
$col2 = 1;
}elsif (/^&quot;CHANNEL&quot; &quot;20&quot;/) {
$col2 = 0;
}elsif (/^&quot;CHANNEL&quot; &quot;32&quot;/) {
$col2 = 3;

}



if (/^\d+\.\d+/) { #if we've seen proper numeric pattern, push $_ to appropriate array based on $col2

if ($col2 == 1) {
$y19=$_;
$diff=$y19-$x19;

if($diff>1.2){
push @Array19, &quot;$_\t\t$col2&quot;;
}
$x19=$_;


} elsif ($col2 == 0) {
$y20=$_;
$diff=$y20-$x20;

if($diff>1.2){
push @Array20, &quot;$_\t\t$col2&quot;;
}
$x20=$_;


} elsif ($col2 == 3) {
my @comp = split(/\t/, $_); #Remove the &quot;????&quot; and 0s
$_ = &quot;$comp[0]\t$comp[2]&quot;;
push @Array32, &quot;$_&quot;;


}elsif ($col2 == 4) {
$y13=$_;
$diff=$y13-$x13;

if($diff>1.2){
push @Array13, &quot;$_&quot;;
}
$x13=$_;

}elsif ($col2 == 5) {
$y14=$_;
$diff=$y14-$x14;

if($diff>1.2){
push @Array14, &quot;$_&quot;;
}
$x14=$_;

}

}
}

$&quot;= &quot;\n&quot;;


push (@Array1920, @Array19, @Array20); #Put @Array19 and @Array20 into @Final
push (@Array1314, @Array13, @Array14);




@Array1314 = sort { $a <=> $b} @Array1314; #sort @Array1314 numerically


my $len = 15;
my $i = 0;

@Array1920 = map { $Array1920[substr($_,$len)] } #sort @Final numerically
sort map { sprintf(&quot;%${len}s&quot;,(split /\t/)[0]).$i++ }
@Array1920;


for my $i(0..$#Array32){ #Add value in 32 only if corresponding value in 1314
if ($Array32[$i]=$Array1314[$i]){
push @Final, &quot;$Array32[$i]&quot;;
}
}



#ask user for output filename
print &quot;\nPlease enter the filename where you wish to store the results:\n&quot;;
my $out_file = <>;
chomp($out_file);



#print data to output file
open(OUT, &quot;> $out_file&quot or die &quot;Failed to open $out_file ... $!&quot;;


for my $i(0..$#Array32){

print OUT &quot;\n$Array1920[$i]\t$Array1314[$i]\t$Array32[$i]&quot;;

}

close(OUT);



print &quot;\nDone!&quot;;

 

[chazoid]

I don't have time at the moment to look over your code, but another thing to check is how you're transferring the input and output file between platforms. I don't work with unix often, but my understanding is that if you're using FTP, you want to transfer in ascii to properly translate the cd/lf's since windows uses x0D0A and unix uses just x0D (I think)
If you have a hex editor, look at your input file to see what's at the end of each line

 

[lmbylsma]

I&quot;m running the program in Unix. Then after it generates the output.txt file I open it with Pico and I see the M^'s. In windows after I transfer it, it interprets those in Excel as going to a new line. Transferring in ASCII makes no difference since the M^s are also there in Unix.

 

[raklet]

I don't know where they are coming from, but the problem is definitely related to newline characters. Unix only puts in a \n (linefeed) to indicate newline. Windows on the other hand, uses \n\r (linefeed and carriage return) to indicate newline. Windows does not know how to interpret a single \n so it print s out ^M instead. Notepad just shows the little block symbol.

 

[lmbylsma]

No, Unix is doing the M^'s, Windows is doing rectangle boxes. If Unix was interpreting it as a \n then it should be going to a new line rather than printing the M^

 

[chazoid]

You didn't mention whether or not you've been transferring files in ascii or binary - Also, do you see the ^M's when you view the input file in pico?

from a couple google searches:

&quot; I have some HTML files that have been written on a PC/Windows system. Before uploading these files to the virtual server, should I convert the text to UNIX text or leave it in another form? Basically, I am asking does my text need to be converted in order to run on the Virtual Server? &quot;

When you upload the files with your FTP client, just be sure that you upload all HTML files and CGI source code in ASCII mode. Upload all binary files such as images and compiled CGI programs in BINARY mode. The FTP client will automatically convert the text when transferred in ASCII mode.

===================

NOTE : VI is a better editor for scripts because, unlike pico, it can see the embedded end-of-line markers (^M) that windows editors leave behind at the end of each line, that invariably cause various scripts and programs to interpret them as actions, causing havoc.

 

[lmbylsma]

I already mentioned this. I do see the M^'s in Unix when using Pico. I also see the M^'s in emacs. Transferring in ASCII to windows doesn't help as the M^'s are already there in Unix.

 

[chazoid]

What I'm getting at is you may have mangled newlines in the input file which then end up in the output. Did the input file ever exist in windows, or was it transferred back and forth at any point?
In windows, if you chomp a string ending with x0D0D0A, it only removes the 0D0A, something similar could be happening on the unix box.
I suggest looking at everything in hex on both platforms to figure out what's happening.

 

[raklet]

>>No, Unix is doing the M^'s, Windows is doing rectangle boxes.

Oops. Got those backwards. Thanks for the heads up.

 

[lmbylsma]

I'm not sure on what platform the input files were generated, we received them from a colleague. I have no idea how to look at everything in hex. However, if that is the problem, that it only removes the 0D0A then how do I chomp it so that it removes the entire x0D0D0A? Alternatively, how can I convert the input files to the approrpriate format so there won't be this problem, if it is the case that they were transferred across platforms?

 

[mactonio]

try changing your print statement form
print OUT &quot;\n$Array1920[$i]\t$Array1314[$i]\t$Array32[$i]&quot;;
to
print OUT &quot;\n&quot;,$Array1920[$i],&quot;\t&quot;,$Array1314[$i],&quot;\t&quot;,$Array32[$i];

sometimes its the simple stuff

 

[lmbylsma]

Nope, that results in exactly the same thing. Still have the M^'s

 

[mactonio]

sorry can't tell why those are showing up, on another issue
this if statement:
for my $i(0..$#Array32){ #Add value in 32 only if corresponding value in 1314
if ($Array32[$i]=$Array1314[$i]){
push @Final, &quot;$Array32[$i]&quot;;
}

you have '=' instead of '==' inside of the if statement

 

[chazoid]

I'm wondering if this has anything to do with it - $&quot;=&quot;\n&quot;;
It's a special variable for 'list separator' but I don't really understand what it's purpose is. Try commenting it out and see if it makes any difference.

I'm pretty sure emacs will display hex, but if not, someone else (ircf) posted some code here recently which works very well for displaying the contents of a file in hex - try running it through this, then post some of the output here. (something containing 0d's and/or 0a's)

open F, 'matt6-03.txt';
binmode F;
undef $/;
$data = <F>;
close F;

$hex = unpack 'H*', $data;
push @list, substr $hex, 0, 2, '' while($hex ne '');
print join ' ', @list;