script to copy files based on available disk space

[namityadav]

I have to write a script that will be executed periodically to copy files from a remote server, and will have to do the following:

1. Check the disk space on local server, find out how much free disk space is available for the copying (We can assume that the copying can take 90% of the total free disk space on local server)
2. Use scp to copy the files (Keys are already set) from the remote machine

What we have to note is: Due to performance issues of the ssh tunnels, it'll be better to copy the files over in one go. One possible problem, in that case, is that if there are too many files on remote machine, then we should figure out how many files we can copy ( and what files are those? ).

Any suggestions?

 

[PHV]

And what have you so far ?
man cron
man df
man du

Hope This Helps, PH.
Want to get great answers to your Tek-Tips questions? Have a look at FAQ219-2884 or FAQ181-2886

 

[namityadav]

Thanks for the reply, PHV. My problem is not cron or du. My problem is that I want to achieve all that in a single ssh tunnel (a set of ';' seperated commands in ssh, perhaps). And how can I take care of the 90% disk space problem .. as this problem can happen during an already-under-execution tunnel. So I should know (while copying) how many files are safe to copy.

 

[feherke]

Hi

Give us more detailes.
[ul]
[li]How the script will know what to copy ?[/li]
[li]All files to copy are new ?[/li]
[li]If a file exist on the remote machine, will be overwrited or skipped ?[/li]
[li]What to do if the space is insufficient ?[/li]
[/ul]

Maybe you can sketch what you tried until now.

Feherke.

http://rootshell.be/~feherke/

 

[namityadav]

Hi,

Here are answers to those questions:

1. Script needs to copy all files from a specific remote directory whose names have a defined structure (like abc_*)
2. Yes, all files are new .. so sync won't help
3. Files won't exist on local machine
4. Stop copying the files from remote machine when the disk usage reaches 90% (Or already know how many files can be copied safely before disk usage would reach 90%)

I'll try to put more info about what I've tried and how I'm trying to achieve it later on today. Till then, any suggestions will be appreciated. Thanks

 

[feherke]

Hi

My first step would be this :

cango=`ssh user@server df / | tail -1 | awk '{printf "%.f",$2/100*90-$3}'`

So [tt]$cango[/tt] will store the available space until 90% will be reached ( if negative, is already over 90% ).

The next step could be something like whis :

total=0
IFS=$'\t'
find abc_* -type f -printf "%p\t%s\n" | \
while read name size; do
  if ((total+size<cango)); then
    let total=total+size
    echo $name
  fi
done > tocopy.txt

After this you will have a list of files to copy in the tocopy.txt file.

The third step will be copying them with [tt]scp[/tt]. No idea properly how, I think using [tt]xargs[/tt].

Another question : what will be the faith of those copied files and what of the not copied files ?

Feherke.

http://rootshell.be/~feherke/