Compare files and directories across two AIX Servers?

[thedocsta]

The problem I am faced with is that, after a legatto restore we are experiencing files and directories which have not been restored, correctly, or at all.
The original files are normally on a production server while the restored files are usually on a development server or disaster recovery server.

I need to compare directories and files within each directory across two AIX Servers!

Is there any way of doing this!

Thanks in advance!
TheDocSta

 

[PHV]

If you can NFS mount the directory tree of one server on the other then you may consider dircmp

Hope This Helps, PH.
Want to get great answers to your Tek-Tips questions? Have a look at FAQ219-2884 or FAQ222-2244

 

[thedocsta]

Hi PHV

Thanks for your reply.
I have considered this and if all else fails will probably go this route, however, I am a little concerned about the overhead repercussion on our storage devices, and production servers, some of which are located at remote sites via wan connections.

Is there not another way of doing this without having to mount the dirs as NFS mounts?

Thanking You Kindly
TheDocSta

 

[ericbrunson]

Try iterating over the trees, generating md5 sums of the files and comparing those.

 

[thedocsta]

Hi Ericbrunson

Thanks for your reply.
I am actually looking for which files(file names) are missing from the restore.
Some of the directories on the Dev/Dr Servers may have already contained files which are not on the restore tapes.
So the number of files on the Dev/Dr Servers will differ from the Prod Servers.

Thank You Kindly.
TheDocSta

 

[xmb]

use find(1) to make a list of all the files, and put that into a temp file which u copy over to the other server to check. on that you do the same

a shell function comparing both files looks like the following

fcomp() {
        grep -Ev '^/(proc|sys|dev|initrd)/' $2 |
        awk -v f=$1 'BEGIN { while ((getline <f) > 0) { if (/^\/(dev|initrd|proc|sys)\//) continue; Seen[$0] =1 } close(f) }
        { if (! Seen[$0]) print "New:", $0; else delete Seen[$0] }
        END { print "-- END --"; for (i in Seen) print "Not seen:", i }'
}

used as.. fcomp <'old' list> <new list>

the same can kinda be done with all kind of data, as permissions and file/dir/dev differencies

i have made a script that 'maps' permissions, user and group bits from one tree over the other, can be modified easily to do whatever is useful.. blink if u're interessted.

. Mac for productivity
.. Linux for developement
... Windows for solitaire

 

[thedocsta]

Hi Xmb

Thanks for your reply.

Tried to run your script after compiling a list of files and copying it to a server, on which a list of files was also complied, bot of the /var directories.
There seemed to be no return from the script!
Although the two directories varied considerably!
What am I doing wrong?
I ran the script as fcomp myfile myfile1

I appreciate the assistance.

Thank You Kindly
TheDocSta

PS. I would appreciate the other scripts you mentioned as well!

 

[xmb]

Hm, i dunno, you may not have egrep or some old awk
what system, shell and awk do u have?
try doing 'set -x' and running it again, see what it does/doesnt

you can safely get rid of the grep [place $2 at the end of the awk cmd] and the same skip part of the awk cmd, as its only intended for from-root comparision

my code tree is down at the moment, you can get the the mapfiles.sh here
i hope the commented instructions are enough, lemme know.

. Mac for productivity
.. Linux for developement
... Windows for solitaire

 

[xmb]

a) Sorry, your system is AIX as the forum entry says.. was early.. had only AIX going through my mind, but that was like, better ask than not
b) The shell shouldnt matter, my guess is your awk on AIX is old. I dunno where you can get an alternative precompiled awk for AIX, if it isnt shipped already, im sure there are some existing though.
c) i dunno if that is of any use for anyone, but i paste the mapfiles.sh script here. I guess in this kinda forums where code is common, viewing the code directly is better seen than following links, at least for anyone else besides you It aint quite small.

Best working with bash, i havent tested with anything else, *may* work with zsh/ksh. Its actually my first code where i use [ instead bash's [[, and other few such changes, in the hope to be somewhat portable.

#!/bin/sh

# /* 23.01.2005
# mapfiles.sh - map file permissions and user:group from one tree to another
# by xmb<@skilled.ch> - localhack
# V0.8 23.01.2005, 14.02.2005(minor changes/comments) */

# [ad]  more funky pro awk and bash code @ [URL unfurl="true"]http://xmb.ath.cx/code[/URL]
#       feel free to mail me about any comments, changes, or ideas
#       about this, or anything else.. new weird scripting ideas welcome
# [/ad]

# print differencies in file system structures
# generate commands to fix those, printed to stdout and to log/other file

# was created cause someone called 'man' fucked up the user:group bits of his
#  ibook running gentoo. this script 'mapped' the original ones from another
#  system back, and all were happy
# such has had to be written once already anyway, by me at least

# temp file structure
# /tmp
# /tmp/mapfiles.log             - log
# /tmp/mapfiles.log.USER        - alternative logs
# /tmp/mapfiles.list            - generated list
# /tmp/mapfiles.in.list         - read this list when updating files
# /tmp/mapfiles.restore         - generated restore commands from the last run
# /tmp/mapfiles.restore.RANDOM  - backups of the generated commands - TODO

# Examples:
# $ ./mapfiles c /var /lib /usr # <creates /tmp/mapfiles.list>
# $ scp /tmp/mapfiles.list other_machine:/tmp/mapfiles.in.list

# on the other machine either
# $ ./mapfiles u /var /lib /usr # <creates /tmp/mapfiles.restore>
# or
# $ ./mapfiles u /              # <creates /tmp/mapfiles.restore>

# second running longer, but both resulting the same, as only seen&&differ files
# get printed, along with a command to restore them
# for now, one command per line per changed file attribute

# show diffs on /lib only
# $ ./mapfiles u /lib

# common find(1) options, to be specified after specifying search paths or
# directly specified without any paths:

# -type f       -type d         -maxdepth <number>      -group <group>
# -[i]name <name>               -i[regex] <regex>       -user <user>

### ## # -

#[[ $1 == [cudh]* ]] && do=$1 && shift
do=$1 ; shift

Path=( "$@" )   # paths being the scripts arguments
[ ! $list ] && list=/tmp/mapfiles.list
[ ! $in ] && in=/tmp/mapfiles.in.list
log=/tmp/mapfiles.log reflist=$list refin=$in
commands_out=/tmp/mapfiles.restore # where the to-fix/restore commands of the last
                                   # script run will be stored

change=123 # 1 for chmod, 2 for user, 3 for group
           # this is actually equivalent to the awk field gotten from _find output
commands="chmod chown chgrp" # representing the commands to be done for the fields

[ ! $awk ] && awk=awk
[ ! $Path ] && Path=( "$PWD" $search_paths $mapfiles_paths )
[ -s $list -a $list != $reflist ] && list=$list.$RANDOM # existing list already, dont overwrite
[ -f $log -a ! -w $log ] && log=/tmp/mapfiles.log.$USER # append $USER if the log cant be written

trap    '_e aborting; _end 2' 2 # doesnt help when stopping the awk running command

_e()    { echo -e "mapfiles: $@" | tee -a $log; }
_ee()   { Xret=$1; shift; _e "$@"; _end $Xret; } # _e + _end wrapper, Xret name to not conflict
                                                 # with any vars, dunno if 'local' is portable
_find() { find "$@" -printf "%m %u %g\t%p\n"; }
_info() { _e "starting at.. $date1 - $date2"
                  _e "list = $list, log file = $log, paths =\e[1m ${Path[@]} \e[m"
                  [ $do_u ] && _e "restore commands log =\e[1m $commands_out \e[m"
                  _e
                  [ "$@" ] && _e "$@"
}
_err()  { _e "$@"; _end 1; }
_end()  { _zero $1 && x=success || x=failed; _e "$x; end: `date +%F//%T` - `date`"; exit $1; }

gimme_file() { for file; do [ -f "$file" ] && echo "$file" && return; done; return 1; }
_zero() { [ -z $1 -o $1 == 0 ] && return 0 || return $1; } # compatibility function to not use [[

### ## # -

date1=`date +%F//%T` date2=`date`

case $do in
-h*|h*|'') echo "Usage: mapfiles <c|create u|update d|diff h|help> [paths] [find args]"
                echo "       c) create index list"
                echo "    p) u) update the files depending on the list"
                echo "       d) show diffs with diff"
                echo "       D) better output than diff, TODO"
                echo "       l) show statistics about the last run, TODO"
                echo "       h) this help"
                exit 1
;;
-c|c)   _info "Creating files list"
        _find "${Path[@]}" >$list
        ret=$?
;;
-u|u|p) do_u=1 _info "Updating files from list"
                [ ! -f $in ] && _e "No input list found ($in)" && _err "Copy one over or export "\
                        "the in variable pointing to one"

                # the awk thing could be reused for other matchings, as
                # $change kinda defines the fields
                _find "${Path[@]}" |
                $awk -v list="$in" -v cmds="$commands" -v c=$change -v out=$commands_out \
                -v d1="$date1" -v d2="$date2" -v paths="`echo ${Path[@]}`" 'BEGIN {
                        split(c, C, ""); split(cmds, Cmd) # cmds = "chmod chown chgrp"
                        while ((getline < list) > 0) { file = getfile()
                                List[file] = 1; while (C[++x]) List[file, C[x]] = $C[x]; x = 0
                        } close(list) }
                        { file = getfile(); if (! List[file]) next
                                while (C[++x]) if ($C[x] != List[file, C[x]]) p(cmd(C[x])); x = 0 }
                        function getfile() { match($0, FS $4); return substr($0, RSTART +1) }
                        function cmd(n) { if (! Z) { pre(); Z = 1 }
                                return Cmd[n] " " List[file, n] " \"" file "\" \t# " $n }
                        function p(str) { print str; print str >> out } # > or >>, same
                        function pre()  { printf "" > out; p("# mapfiles.sh generated commands")
                                p("# " d1 " -- " d2 " -- paths run: " paths) }' |
                tee -a $log
                ret=$? ret=$PIPESTATUS
;;
-d|d)   file=`gimme_file $diff $list $in $reflist $refin` || _err "Couldnt find any list"
                _info "Showing differencies on $file"
                _find "${Path[@]}" | diff - $file && _e "No diffs found"
;;
esac

### ## # -

[ -z $ret ]      && _end 0
[ $ret == 130 ]  && _ee $ret "probably killed awk"
[ $ret -gt 130 -o $ret -lt 130 ] && _ee $ret "unknown return code $ret"
_end $ret

. Mac for productivity
.. Linux for developement
... Windows for solitaire

 

[thedocsta]

Hi Xmb

Thanks for your reply.

I am using AIX 5.2! Should the awk not be uptodate with this version?
I will experiment with the code and see if I can get something working!

Thank You Kindly for the scripts, they will come in handy!

Regards
TheDocSta

 

[xmb]

I was hoping for awk -W version or awk -V output, what awk that is.

. Mac for productivity
.. Linux for developement
... Windows for solitaire