Joining files from three directories into a single file

[Paulova]

I know its a newbie question, but my bash knowledge is so limited. Any help with this script? Thanks.

Trying to join files from three directories into a single file like:

<directory1>
header.txt
footer.txt

<directory2>
050421.txt
050420.txt
050419.txt
...

<directory3>
050421.txt
050419.txt
...

Result would be:

### final.file start
header.txt content ## from directory1
050421.txt content ## from directory2
050421.txt content ## from directory3
050420.txt content ## from directory2
050419.txt content ## from directory2
050419.txt content ## from directory3
footer.txt content ## from directory1
### final file end

As you see file names are in format %y%m%d and must be appended to final.file ordered by date.
If name exists in both directories, file from directory2 goes before file from directory3.
Sometimes a filename exists only in directory2 and not in directory3 so next file from directory2 must be appended to final.file

 

[fredericofonseca]

Humm.

First search this forum for "sort" and "cat". You will find many examples.

Then do
man find
man sort
man cat

and read through it.

Then search these forums for "while read" and you will find some examples of looping through a list.
This will be usedfull for the "cat" (e.g. where you add the content of one file to another.)

Then with some of your own code please do comeback here with some specific questions and with any error you may have.


Also note that tek-tips is NOT an helpdesk or a place where you get your assignments done.



Regards

Frederico Fonseca
SysSoft Integrated Ltd

http://www.syssoft-int.com

 

[olded]

Frederico is right, but I'll give you a hint to get you started. Once you find all the files, redirect it to the xargs & cat command:

find ....|xargs cat > filecontents

 

[Paulova]

Thanks Frederico for your guide but I dont know how to use find command in this script. Perhaps you have another vision different from mine for this code:

#!/bin/sh
ls -1 directory2|sort -r>lines.d2
ls -1 directory3|sort -r>lines.d3
cat directory1/header.txt>final.file
cat lines.d2|while read line
do
for i in $line
do
cat directory2/"$i">>final.file
if [ -e "directory3/$line" ]; then
cat directory3/$i>>final.file
## how to delete $i from lines.d3 file here?
## i have try it with sed command with no success
fi
done
done

I am thinking in another "while read do" for files in lines.d3. But i would like not to use temp files like lines.d2,lines.d3.

To olded:
can you post a clear example using xargs please?. I dont understand how and when to use this command. (I alredy have seen man page and some another examples...)

Thanks

 

[fredericofonseca]

Paulova,

I mentioned find for two reasons.

1- You may have more files on those directories that you may not wish to have processed.
2- It will the file list including their pathname.

And also because if you work with Unix and variants you REALLY NEED to know how to use find.
So please do read the man pages and search the forums for find examples.

Your code will be cleaner if you have all the file listing within the same file, and if you then sort that file.
e.g.

allfilelist (output of find before sort)
directory2/010101.txt
directory2/010202.txt
directory3/010101.txt
directory3/010102.txt
directory3/010203.txt

Apply sort to the above file, using "/" as a field delimiter, sorting by second field (man sort will give you required info), and either output to another file or pipe directly to a while loop.

Ouput will look as
directory2/010101.txt
directory3/010101.txt
directory3/010102.txt
directory2/010202.txt
directory3/010203.txt

which is what you asked for, so your loop can just cat the file directly without the need to search for the directory3 file.


Regarding your code you asked "how to delete $i from lines.d3 file here?". You don't need to delete it as it will not be processed again.
According to your code you don't even need to retrieve the file list from directory3, but maybe that is what you wish.


My sugestion will give you also files that are on directory3 that ARE NOT on directory2. Is this what you wish?


Regards

Frederico Fonseca
SysSoft Integrated Ltd

http://www.syssoft-int.com

 

[olded]

cp directory1/header.txt > final.file
find directory2 directory3 -type f -name "*.txt" -print|xargs cat >> final.file
cp directory1/footer.txt >> final.file

After finding the files you want, the xargs presents each file found as a list of arguments to the cat command. This displays the contents of each file.

 

[Paulova]

Understood:
#!/bin/sh
cat directory1/header.txt>final.file
find directory2/* directory3/*|sort -r -t"/" -k2|xargs cat >>final.file
cat directory1/footer.txt>>final.file

Thanks Frederico and olded too.