A few weird characters are automatically added to a string 1

[whn]

I wrote a small program using Expect.pm to read the output of 'ls -l' on a remote host and then write into a local logfile.

Somehow, certain weird char's are also automatically written into the log. This is my first time using Expect.pm and I am not sure if my problem is related to Expect.pm.

Here is a small piece of my codes:

  # @buf contains the output of 'ls -l' on a remote host 192.168.1.5
  foreach my $e (@buf) {
    $e =~ s/^\s+//;
    $e =~ s/\s+$//;
    my @tmp = split(/\s+/, $e);
    if($#tmp == 8 && $tmp[$#tmp] =~ /idx/) {
      $flist{$tmp[$#tmp]} = $tmp[4];
      if($cnt < 2) {
        print "$cnt. $server, #$tmp[$#tmp]#, $tmp[4]\n";
        print H1 "$cnt. $server, #$tmp[$#tmp]#, $tmp[4]\n";
      }
      $cnt++;
    }
  }

The screen output is:

0. 192.168.1.5, #00d786312b6a16d6_config.3.0.idx#, 393216
1. 192.168.1.5, #00d786312b6a16d6_correlations.3.0.idx#, 10616832

As you can see here, they are all CLEAN!
Below is the contents of the logfile:

$ cat info.txt
0. 192.168.1.5, #00d786312b6a16d6_config.3.0.idx#, 393216
1. 192.168.1.5, #00d786312b6a16d6_correlations.3.0.idx#, 10616832

They are also clean. However, when I open the logfile through vim. The contents looks aweful:

0. 192.168.1.5, #[COLOR=red]^[[00m[/color]00d786312b6a16d6_config.3.0.idx[COLOR=red]^[[00m[/color]#, 393216
1. 192.168.1.5, #[COLOR=red]^[[00m[/color]00d786312b6a16d6_correlations.3.0.idx[COLOR=red]^[[00m[/color]#, 10616832

Where do those weird chars (in red) come from? Does it have anything to do with Expect.pm? How to get rid of them?

And lastly, for your reference, below is the output of 'ls -l' on the remote host 192.168.1.5:

% ls -l 00d786312b6a16d6_config.3.0.idx 00d786312b6a16d6_correlations.3.0.idx
-rw-r--r-- 1 abc abc   393216 Jan 31  2011 00d786312b6a16d6_config.3.0.idx
-rw-r--r-- 1 abc abc 10616832 Jan 31  2011 00d786312b6a16d6_correlations.3.0.idx

Many thanks for you help!!

 

[dmazzini]

You might have some Unix chars that you have to take out:

To confirm it:

#copy this to one file and called it ripout.pl
# run it:
# ripout.pl file.txt
# open it again with VIM, let me know results

$file =$ARGV[0];
$/=undef;
open FH, "<$file";
$perlcode=<FH>;
close FH;
$perlcode=~ s/\r//g;
open FH, ">$file";
print FH "$perlcode";
close FH;

dmazzini
GSM/UMTS System and Telecomm Consultant

 

[feherke]

Hi

I think your [tt]ls[/tt] is so kind to colorize the list.
Try to ask it to not do it by adding the [tt]--color=never[/tt] option.


Feherke.

 

[whn]

Thank you, dmazzini & Feherke.

To dmazzini, though your solution did not work as expected, it indeed gave me some hint. So I tried following cmd:

tr -d '[[:cntrl:]]' < infile > outfile

which partially solved the problem, i.e. it removed '^[['. But those '00m' are still there. I understand that's because '00m' are treated as pure ascii's. Since these '00m' could be valid, I cannot remove them blindly using regexp.

To Feherke. Wow, that's the reason. Once I used an unaliased command: "\ls -l", the problem is solved!!

 

[whn]

To someone who might be interested....

I tried hexdump command and below is the output:

$ cat log.txt
0. 192.168.1.5, #00d786312b6a16d6_config.3.0.idx#, 393216
1. 192.168.1.5, #00d786312b6a16d6_correlations.3.0.idx#, 10616832

So, again, 'cat' output looks just fine.

$ hexdump -c log.txt
0000000   0   .       1   9   2   .   1   6   8   .   1   .   5   ,
0000010   #[COLOR=red] 033   [   [/color][COLOR=blue]0   0   m[/color]   0   0   d   7   8   6   3   1   2   b
0000020   6   a   1   6   d   6   _   c   o   n   f   i   g   .   3   .
0000030   0   .   i   d   x 033   [   0   0   m   #   ,       3   9   3
0000040   2   1   6  \n   1   .       1   9   2   .   1   6   8   .   1
0000050   .   5   ,       # 033   [   0   0   m   0   0   d   7   8   6
0000060   3   1   2   b   6   a   1   6   d   6   _   c   o   r   r   e
0000070   l   a   t   i   o   n   s   .   3   .   0   .   i   d   x 033
0000080   [   0   0   m   #   ,       1   0   6   1   6   8   3   2  \n
0000090

Both red and blue are trash. When I try the following command:

tr -d '[[:cntrl:]]' < log.txt > log.txt

The unwanted chars in red will be removed. But the ones in blue still stay.

 

[feherke]

Hi

The red and blue ones are together. They are the ANSI sequence for setting text color and attributes. As long as you remove the leading escape and [ characters up to the terminating m, the result will be clean ( no remaining garbage, no excessive removal ) :

ls --color=always | perl -pe 's/\033\[[^m]*m//g'

Feherke.

 

[whn]

Thank you, feherke!

That's very beautiful!!

Now, I have a follow-up question ---

Just to experiment, I modified my codes to do the following:

1) logon the remote host, call it host-a, through expect.pm;
2) perform the cmd 'ls -l' and save the output on host-a;
3) scp the output from host-a to the local host;
4) open the file using vi. It does not have those unwanted hidden chars!!

However, my previous implementation is like below:

# print $ssh_exp "\\ls -l $remoteDir\r"; # To avoid the hidden chars
print $ssh_exp "ls -l $remoteDir\r"; # This will have the hidden chars
my $flist = &expect_prompt($ssh_exp, $timeout, $prompt);
sub expect_prompt(){
  my ($ssh_exp, $timeout, $prompt) = @_;
  $ssh_exp->expect($timeout, '-re' , $prompt) or die "expect error" . $ssh_exp->exp_error() . "\n";
[b][COLOR=red]  my $out = $ssh_exp->before(); [/color][/b]
  return $out;
}

then the results do contain the hidden chars.

Does this mean that Expect.pm..before() needs an enhancement?

 

[feherke]

Hi

Sorry, I have absolutely no idea about expect.

I suppose any difference is related to this sentence :

With --color=auto, ls emits color codes only when standard output is connected to a terminal.

Feherke.

 

[Annihilannic]

Correct... one of the main reasons you would need to use expect is because it emulates a terminal, where normal ssh/rsh/rcmd style remote commands do not.

Annihilannic
[small]tgmlify - code syntax highlighting for your tek-tips posts[/small]