How do I catenate over 400 files into one file? 1

[tpbjr]

I used the CAT command but after 300+ files the command blew up. Is there a way to iterate through a folder for given wild card (myfiles*.dat for example) and combine them into one file.

Thank you for all your help

http://www.besware.com

Tom

 

[mikevh]

I haven't been able to come up with an Awk solution. As the original post has been here for several hours and no-one else has posted one either, I offer the following solution in Perl.

use strict;
use warnings;

while (my $filename = shift) {
    for my $f (glob $filename) {
        unless (open(FH, $f)) {
            warn qq(Can't open "$f"!\n);
            next;
        }
        print while <FH>;
        close(FH) || warn qq(Can't close file "$f"!\n);
    }
}

Run like so
path/to/perl scriptfile myfiles*.dat > outputfile

If anyone has an Awk solution I'd be interested to see it.

 

[vgersh99]

ls myfiles*.dat| xargs cat >> oneFATcat

vlad
+----------------------------+
| #include<disclaimer.h> |
+----------------------------+

 

[futurelet]

awk '1' myfiles*.dat >junk

Let me know whether or not this helps.

If you have nawk, use it instead of awk because on some systems awk is very old and lacks many useful features. For an introduction to Awk, see faq271-5564.

 

[futurelet]

Vlad, would [tt]cat myfiles*.dat >bigfile[/tt] work?

 

[futurelet]

I see it now. Command-line was too long.

BEGIN {
  while ( ("ls myfiles*.dat" | getline) > 0)
    ARGV[ARGC++] = $0
}
1

Run with
[tt]
awk -f lumper.awk >lump
[/tt]

 

[mikevh]

Thanks for the star.

I assumed since the OP said he was using the CAT command that he wasn't on *nix and wouldn't have ls or a shell that did globbing.

 

[futurelet]

If he doesn't have ls, he probably has something similar. Under messy-dos:
[tt]
dir /b /a-d
[/tt]
will list filenames only in "bare" format, one per line.

 

[PC888]

If ls was available, he could try along the following lines:

rm -f ../allfiles
touch ../allfiles
for i in `ls`
do
cat $i >> ../all files
done

Notice that it is important to place the cumulative file outside of the current directory, or else a recursive effect will factor in.

 

[vgersh99]

1. what's the diff between '../all files' and '../allfiles'?
2. what's the diff between '`ls`' and '*' ?
3. what file the output will end up in?
cat foo >> ../all files



vlad
+----------------------------+
| #include<disclaimer.h> |
+----------------------------+

 

[PHV]

If the real problem is 'arg list too long':
(cd /path/to/dir;ls | grep 'myfiles.*\.dat' | xargs cat) > newbigfile

Hope This Helps, PH.
Want to get great answers to your Tek-Tips questions? Have a look at FAQ219-2884 or FAQ222-2244

 

[PC888]

Oops, the space between all and files was inadvertent. Thanks Vlad.
I am not sure what the original problem was, the argument list too long, or a problem with file sizes doubling up because it was piling up on itself. You could try the following modified version and see what problem, if any, shows up.

rm -f ../allfiles
touch ../allfiles
ls myfiles*.dat > ../files.lst
for i in `cat ../files.lst`
do
cat $i >> ../allfiles
done