rsync with hardlinks rotating snapshots code 1

[rigstars2]

Hello everyone,

I’m curious to know if my code is working as designed? The first hourly.0 performs a full backup then every snapshot after (hourly.1 —> hourly.11 afterwards is hardlinked. Now here is the part I’m confused. During one of the runs, I’ll get a full backup performed again on hourly.1 and hourly.2 which I don’t understand. If the rotate function moves 10->11 9->10 8->9 7->8 etc... Why don’t I have just 1 full copy? On the 12th run, hourly.11 would have rotated out and been deleted according to my code? Thanks in advance.

du -sch * //
hourly.0 10gb
hourly.1 20k
hourly.2. 20k
and
so
on...
—————
rotateSnapshots()
{
if [ -d “$DST/hourly.0” ]; then
rm -fr hourly.11
for i in {11..1}
do
mv "$DST/hourly.$[${i}-1]" "$DST/hourly.${i}" 2> /dev/null
sleep 1
done
fi
}

runIncremental()
{
/opt/local/bin/rsync -a --delete --link-dest="$DST/hourly.1" "$SRC" "$DST/hourly.0"
sleep 1
}

 

[rigstars2]

additional info: this is after many runs of the script... please assist. much appreciated.

sample expected ouput:
hourly.0 10gb
hourly.1 20k
hourly.2. 20k
hourly.3 10k
hourly.4. 20k
hourly.5 20k
hourly.6 10k
hourly.7. 20k
hourly.8 20k
hourly.9 20k
hourly.10 20k
hourly.11 20k

unfortunate output I get:
hourly.0 10gb
hourly.1 10gb
hourly.2. 10gb
hourly.3 10k
hourly.4. 20k
hourly.5 20k
hourly.6 10k
hourly.7. 20k
hourly.8 20k
ourly.9 20k
hourly.10 20k
hourly.11 10k

 

[SamBones]

Not sure I fully understand, but your loop goes from 11 to 1, which means the last pass moves [tt]hourly.0[/tt] to [tt]hourly.1[/tt]. That's your full backup going to [tt]hourly.1[/tt]. Then, I assume this pass or next pass does a full backup again because there's no [tt]hourly.0[/tt].

But I'm making some assumptions because you just posted a snippet of code and not the whole thing.

 

[rigstars2]

I’ll post the entire code when I get home tonight.

 

[rigstars2]

Here is the code. If you run it, you'll need a partition/volume called 'Hourly' ...
It appears to be working. Just need a second opinion on it ..because it shouldn't be creating more than 1 full backup other than the first time it runs and excluding the SRC directory. After running it more than 12 times .. it appears to be working ..?
All the 0B sizes are all hardlinks below, just as expected ..

du -sch /Volumes/Hourly/snapshot/*
860K /Volumes/Hourly/snapshot/Source
860K /Volumes/Hourly/snapshot/hourly.0
0B /Volumes/Hourly/snapshot/hourly.1
0B /Volumes/Hourly/snapshot/hourly.10
0B /Volumes/Hourly/snapshot/hourly.11
0B /Volumes/Hourly/snapshot/hourly.2
0B /Volumes/Hourly/snapshot/hourly.3
0B /Volumes/Hourly/snapshot/hourly.4
0B /Volumes/Hourly/snapshot/hourly.5
0B /Volumes/Hourly/snapshot/hourly.6
0B /Volumes/Hourly/snapshot/hourly.7
0B /Volumes/Hourly/snapshot/hourly.8
0B /Volumes/Hourly/snapshot/hourly.9
1.7M total


--------------
#!/bin/bash
set -x

SRC="/Volumes/Hourly/snapshot/Source/"
DST="/Volumes/Hourly/snapshot"

consolidate()
{

/opt/local/bin/rsync -a --delete "/Users/user1/Documents" "$SRC"

}

rotateSnapshots()
{
cd "$DST"
if [ -d hourly.0 ]; then
rm -fr hourly.11
sleep 5
for i in {11..1}
do
mv "$DST/hourly.$[${i}-1]" "$DST/hourly.${i}" 2> /dev/null
sleep 1
done
fi
}

runIncremental()
{
/opt/local/bin/rsync -a --delete --link-dest="$DST/hourly.1" "$SRC" "$DST/hourly.0"
sleep 1
}

consolidate;
rotateSnapshots;
runIncremental;
exit 0

 

[rigstars2]

And in response to your question, yes, the hourly.0 moves to hourly.1 moving the first time full backup down the line until it gets to hourly.11 and then it gets deleted per the code. it should repeat again ..with full backup starting at hourly.0, am I right?

 

[SamBones]

You should post your code between code tags; [tt][ignore]

[/ignore][/tt]. That makes it MUCH easier to read and debug (see below).

Looking at the code, it's not really clear what you're trying to do. Every time you run this script you do a full rsync, then rotate the files, then do an incremental. I don't understand what you're trying to do.

The [tt]-link-test[/tt] parameter creates hard links, not soft links. That means a [tt]du[/tt] or an [tt]ls[/tt] will report the full size of the file. The hard links give an misleading idea of how much space is actually being taken up. Each directory will report the full file sizes of the linked files, but it's actually only taking up the space on disk once.

Also, you don't need the semi-colons at the end of the function calls. They do no harm, but they aren't needed.

#!/bin/bash
set -x

SRC="/Volumes/Hourly/snapshot/Source/"
DST="/Volumes/Hourly/snapshot"

consolidate()
{
/opt/local/bin/rsync -a --delete "/Users/user1/Documents" "$SRC"
}

rotateSnapshots()
{
cd "$DST"
if [ -d hourly.0 ]; then
    rm -fr hourly.11
    sleep 5
    for i in {11..1}
    do
        mv "$DST/hourly.$[${i}-1]" "$DST/hourly.${i}" 2> /dev/null
        sleep 1
    done
fi
}

runIncremental()
{
/opt/local/bin/rsync -a --delete --link-dest="$DST/hourly.1" "$SRC" "$DST/hourly.0"
sleep 1
}

consolidate;
rotateSnapshots;
runIncremental;

exit 0

 

[SamBones]

Oh, also, why do you have [tt]sleep[/tt]s sprinkled throughout the code? All that does is slow the script down, and nothing more.

If you're worried about a command not completing before the next command (i.e. [tt]rsync[/tt], [tt]rm[/tt], [tt]mv[/tt], etc), it would be better to put a [tt]sync[/tt] in there. The [tt]sync[/tt] command will flush the buffers to disk (or at least request that they be flushed). That will add small delays while the buffers write, but those delays are for a reason, not just because you want to slow the script down for some reason.

Example:

rotateSnapshots()
{
cd "$DST"
if [ -d hourly.0 ]; then
    rm -fr hourly.11
    sync
    for i in {11..1}
    do
        mv "$DST/hourly.$[${i}-1]" "$DST/hourly.${i}" 2> /dev/null
        sync ; sync ; sync    # Multiple syncs don't hurt anything
    done
fi
}

Just keep in mind that the [tt]sync[/tt] command affects the buffers for the whole machine, so you could end up impacting performance of other things running on that machine.

 

[rigstars2]

Thanks Sam. Appreciate your feedback. Very insightful. Always learning something new here.

 

[SamBones]

You're welcome. Glad to help.