Need to delete a line while looping thrugh a file 2

[netman4u]

Hello all,

I am writing a script that (in part) opens a file for reading and writing, loops through the file line by line looking for a specific strings. If it finds any of the strings I want to delete the line and go to the next line in the file (while loop).

Here is what I have so far:

my ($profile,$server,$xfer_method,$path2log,$source,$site,$trak,$exclude) = split;

my @excludes = split (/,/,$exclude);
open (EXCLUDE, "+>$log_name")or ErrorOut ("Error opening log file $log_name: $1");
while (<EXCLUDE>) {
   foreach  my $entry(@excludes) {
   if ($_ =~ $entry) {
      delete the whole line here and go to the next line
   }
}

Here is what @excludes contains:

.exe,.js,.ico,.gif,.jpg,.css,170.6.91.240,170.6.116.233,63.66.113.77

First, how do I delete the whole line that is in $_ from the while loop? Second, after deleting the line, how do I jump directly to the next line in the while loop without going throuh the rest of my @excludes array as if ANY of the $entry exist in the line I am going to delete it so I do not need to loop through the rest (hope that explains it ok).

Thanks,

Nick


If at first you don't succeed, don't try skydiving.

 

[MillerH]

perlfaq5 - How do I change one line in a file/delete a line in a file/insert a line in the middle of a file/append to the beginning of a file?

- Miller

 

[KevinADC]

Use perls inplace editor, or Tie::File, which is what I think Millers link will recommend.

------------------------------------------
- Kevin, perl coder unexceptional!

 

[netman4u]

Thanks Miller and Kevin but unfortunately I am working for a large corporation that does not allow the download of Open Source software on their unix systems. I can use perl because it comes with the standard licensed Unix distribution but I cannot download modules.

If at first you don't succeed, don't try skydiving.

 

[MillerH]

Tie::File is a core library. Use it, abuse it, love it.

- Miller

 

[netman4u]

Thanks that helps with the line delete, but how do I jump out of the foreach loop when the first exclusion is found and jump to the next line in the while loop?

If at first you don't succeed, don't try skydiving.

 

[hungryhungryhippo]

i think flow control insturctions like next and last apply to the innermost loop. so just make "last" the next command after you deleted the line. but somehow this seems pretty obvious, i don't know if i got your post right.
greetings,
michael

 

[MillerH]

Something like the following is probably what you're looking fore:

[url=http://perldoc.perl.org/functions/use.html][black][b]use[/b][/black][/url] [green]Tie::File[/green][red];[/red]

[black][b]use[/b][/black] [green]strict[/green][red];[/red]

[gray][i]# Constants[/i][/gray]
[url=http://perldoc.perl.org/functions/my.html][black][b]my[/b][/black][/url] [blue]$log_name[/blue] = [red]'[/red][purple]foobar.txt[/purple][red]'[/red][red];[/red]
[black][b]my[/b][/black] [blue]$exclude[/blue] = [red]'[/red][purple].exe,.js,.ico,.gif,.jpg,.css,170.6.91.240,170.6.116.233,63.66.113.77[/purple][red]'[/red][red];[/red]

[gray][i]# Create the matching regex[/i][/gray]
[black][b]my[/b][/black] [blue]$list[/blue] = [url=http://perldoc.perl.org/functions/join.html][black][b]join[/b][/black][/url] [red]'[/red][purple]|[/purple][red]'[/red], [url=http://perldoc.perl.org/functions/map.html][black][b]map[/b][/black][/url] [red]{[/red][url=http://perldoc.perl.org/functions/quotemeta.html][black][b]quotemeta[/b][/black][/url][red]}[/red] [url=http://perldoc.perl.org/functions/split.html][black][b]split[/b][/black][/url] [red]'[/red][purple],[/purple][red]'[/red], [blue]$exlude[/blue][red];[/red]
[black][b]my[/b][/black] [blue]$exlude_re[/blue] = [red]qr{[/red][purple][blue]$list[/blue][/purple][red]}[/red][red];[/red]

[url=http://perldoc.perl.org/functions/tie.html][black][b]tie[/b][/black][/url] [black][b]my[/b][/black] [blue]@array[/blue], [red]'[/red][purple]Tie::File[/purple][red]'[/red], [blue]$log_name[/blue], [purple]recsep[/purple] => [red]"[/red][purple][purple][b]\n[/b][/purple][/purple][red]"[/red]
	or [url=http://perldoc.perl.org/functions/die.html][black][b]die[/b][/black][/url] [red]"[/red][purple]open Tie::File [blue]$file[/blue]: [blue]$![/blue][/purple][red]"[/red][red];[/red]

[maroon]LINE[/maroon][maroon]:[/maroon]
[olive][b]for[/b][/olive] [red]([/red][black][b]my[/b][/black] [blue]$i[/blue] = [fuchsia]0[/fuchsia][red];[/red] [blue]$i[/blue] <= [blue]$#array[/blue][red];[/red] [blue]$i[/blue]++[red])[/red] [red]{[/red]
	[olive][b]if[/b][/olive] [red]([/red][blue]$array[/blue][red][[/red][blue]$i[/blue][red]][/red] =~ [blue]$exclude_re[/blue][red])[/red] [red]{[/red]
		[url=http://perldoc.perl.org/functions/splice.html][black][b]splice[/b][/black][/url] [blue]@array[/blue], [blue]$i[/blue], [fuchsia]1[/fuchsia][red];[/red]
		[olive][b]redo[/b][/olive] LINE[red];[/red]
	[red]}[/red]
[red]}[/red]

[url=http://perldoc.perl.org/functions/untie.html][black][b]untie[/b][/black][/url] [blue]@array[/blue][red];[/red]

[tt]------------------------------------------------------------
Pragmas (perl 5.8.8) used :
[ul]
[li]strict - Perl pragma to restrict unsafe constructs[/li]
[/ul]
Core (perl 5.8.8) Modules used :
[ul]
[li]Tie::File - Access the lines of a disk file via a Perl array[/li]
[/ul]
[/tt]

- Miller

 

[MillerH]

And taking the time to actually run the script to detect syntax errors leaves us with this:

[url=http://perldoc.perl.org/functions/use.html][black][b]use[/b][/black][/url] [green]Tie::File[/green][red];[/red]

[black][b]use[/b][/black] [green]strict[/green][red];[/red]

[gray][i]# Constants[/i][/gray]
[url=http://perldoc.perl.org/functions/my.html][black][b]my[/b][/black][/url] [blue]$log_name[/blue] = [red]'[/red][purple]foobar.txt[/purple][red]'[/red][red];[/red]
[black][b]my[/b][/black] [blue]$exclude[/blue] = [red]'[/red][purple].exe,.js,.ico,.gif,.jpg,.css,170.6.91.240,170.6.116.233,63.66.113.77[/purple][red]'[/red][red];[/red]

[gray][i]# Create the matching regex[/i][/gray]
[black][b]my[/b][/black] [blue]$list[/blue] = [url=http://perldoc.perl.org/functions/join.html][black][b]join[/b][/black][/url] [red]'[/red][purple]|[/purple][red]'[/red], [url=http://perldoc.perl.org/functions/map.html][black][b]map[/b][/black][/url] [red]{[/red][url=http://perldoc.perl.org/functions/quotemeta.html][black][b]quotemeta[/b][/black][/url][red]}[/red] [url=http://perldoc.perl.org/functions/split.html][black][b]split[/b][/black][/url] [red]'[/red][purple],[/purple][red]'[/red], [blue]$exclude[/blue][red];[/red]
[black][b]my[/b][/black] [blue]$exclude_re[/blue] = [red]qr{[/red][purple][blue]$list[/blue][/purple][red]}[/red][red];[/red]

[url=http://perldoc.perl.org/functions/tie.html][black][b]tie[/b][/black][/url] [black][b]my[/b][/black] [blue]@array[/blue], [red]'[/red][purple]Tie::File[/purple][red]'[/red], [blue]$log_name[/blue], [purple]recsep[/purple] => [red]"[/red][purple][purple][b]\n[/b][/purple][/purple][red]"[/red]
	or [url=http://perldoc.perl.org/functions/die.html][black][b]die[/b][/black][/url] [red]"[/red][purple]open Tie::File [blue]$log_name[/blue]: [blue]$![/blue][/purple][red]"[/red][red];[/red]

[maroon]LINE[/maroon][maroon]:[/maroon]
[olive][b]for[/b][/olive] [red]([/red][black][b]my[/b][/black] [blue]$i[/blue] = [fuchsia]0[/fuchsia][red];[/red] [blue]$i[/blue] <= [blue]$#array[/blue][red];[/red] [blue]$i[/blue]++[red])[/red] [red]{[/red]
	[olive][b]if[/b][/olive] [red]([/red][blue]$array[/blue][red][[/red][blue]$i[/blue][red]][/red] =~ [blue]$exclude_re[/blue][red])[/red] [red]{[/red]
		[url=http://perldoc.perl.org/functions/splice.html][black][b]splice[/b][/black][/url] [blue]@array[/blue], [blue]$i[/blue], [fuchsia]1[/fuchsia][red];[/red]
		[olive][b]redo[/b][/olive] LINE[red];[/red]
	[red]}[/red]
[red]}[/red]

[url=http://perldoc.perl.org/functions/untie.html][black][b]untie[/b][/black][/url] [blue]@array[/blue][red];[/red]

[tt]------------------------------------------------------------
Pragmas (perl 5.8.8) used :
[ul]
[li]strict - Perl pragma to restrict unsafe constructs[/li]
[/ul]
Core (perl 5.8.8) Modules used :
[ul]
[li]Tie::File - Access the lines of a disk file via a Perl array[/li]
[/ul]
[/tt]

- Miller

 

[KevinADC]

Using perls inplace editor is generally much faster and also creates a backup copy of the original file:

[url=http://perldoc.perl.org/functions/my.html][black][b]my[/b][/black][/url] [blue]$exclude[/blue] = [red]([/red][url=http://perldoc.perl.org/functions/split.html][black][b]split[/b][/black][/url][red])[/red][red][[/red][fuchsia]7[/fuchsia][red]][/red][red];[/red]
[black][b]my[/b][/black] [blue]@excludes[/blue] = [black][b]split[/b][/black] [red]([/red][red]/[/red][purple],[/purple][red]/[/red],[blue]$exclude[/blue][red])[/red][red];[/red]
[red]{[/red]
   [url=http://perldoc.perl.org/functions/local.html][black][b]local[/b][/black][/url] [blue]@ARGV[/blue] = [red]([/red][blue]$log_name[/blue][red])[/red][red];[/red]
   [black][b]local[/b][/black] [blue]$^I[/blue] = [red]'[/red][purple].bak[/purple][red]'[/red][red];[/red]
   [maroon]LOOP[/maroon][maroon]:[/maroon] [olive][b]while[/b][/olive] [red]([/red]<>[red])[/red] [red]{[/red]
      [olive][b]foreach[/b][/olive] [black][b]my[/b][/black] [blue]$entry[/blue] [red]([/red][blue]@excludes[/blue][red])[/red] [red]{[/red]
         [olive][b]next[/b][/olive] LOOP [olive][b]if[/b][/olive] [red]([/red][red]/[/red][purple][blue]$entry[/blue][/purple][red]/[/red][red])[/red][red];[/red]
      [red]}[/red]
      [url=http://perldoc.perl.org/functions/print.html][black][b]print[/b][/black][/url][red];[/red]
   [red]}[/red]
[red]}[/red]

------------------------------------------
- Kevin, perl coder unexceptional!

 

[MillerH]

Kevin is right in that the in place editor is not just faster, but often by an order of magnitude. Not only that, but it often takes less code.

Nevertheless, I would still advocate the use of Tie::File for most purposes. It's just a good tool to get in the practice of using since it is so powerful. And it's generally as easy to understand as a simple array operation, therefore it doesn't take much explanation to the newly exposed.

That's just my opinion though. YMMV.

- Miller

 

[netman4u]

Very nice Kevin. I have it working!

Is the "print;" really necessary? Does it write back to the file? I'm not really sure how this code works.

If you have the time could you give a brief explination or a FAQ, etc.?

If at first you don't succeed, don't try skydiving.

 

[netman4u]

Kevin,

OMT...my program ftp's many many very large log files from many servers to the server it runs on, cleans them of the excludes and then immediately ftp's them to another series of servers.

Bottom line is by the time it is finished it will totally fill the LV I am ftping to with all the *.bak file. Is there anyway I not create the backup files?

If at first you don't succeed, don't try skydiving.

 

[KevinADC]

To not create a backup file you can try:

local $^I = '';

it may or may not work, depends on the operating system. You can also delete/unlink the backup copy right after editing the file to free up disk space. As far as I know the "print" is necessary.

------------------------------------------
- Kevin, perl coder unexceptional!

 

[KevinADC]

here is a link to understanding perls inplace editor, it focuses more on command line syntax but it's all applicble to writing scripts too:

http://www.perl.com/pub/a/2004/10/14/file_editing.html

------------------------------------------
- Kevin, perl coder unexceptional!