Move list of files to new location 1

[theniteowl]

I am trying to figure out how to move files based on a plain text file listing one filename per line.
I have found some examples online but none seem to work for me.

I am using Solaris Unix so not all command options are the same.
The most recent attempt that came closest is:

cat files.txt | while read line; do mv "$line" /ops/data/obsolete ; done

But this comes back with the message "cannot access filename.ext" for each line in the text file.

The files.txt file is in the same folder as the files to be moved and that is the folder I am executing from. It does not seem to be a permissions issue as I tried it with superuser privledges as well.
I tried specifying the path to the file in files.txt as well but with the same result.
I have tried a number of xarg commands as well with no success.
Anyone have any ideas?

Thanks.


At my age I still learn something new every day, but I forget two others.

 

[walkingseed]

try removing the quote marks from around the $line variable.

 

[feherke]

Hi

Some questions :
[ul]
[li]What kind of shell are you using ?[/li]
[li]Are you sure there are no extra trailing whitespace characters at the end of lines ? Typically the DOS CR/LF's [tt]\r[/tt] character.[/li]
[li]Does files.txt contain paths along the file names ?[/li]
[/ul]
In meantime I would try the following debugging steps :
[ul]
[li]add [tt]echo[/tt] to see the commands as they should be executed,[/li]
[li]add [tt]IFS=''[/tt] and [tt]-r[/tt] ( if supported by your shell's [tt]read[/tt] ) to reduce some character transformation/removal[/li]
[li]redirect the output to a hex dumper like [tt]od[/tt] ( or something else available ) to more accurate check for unwanted special characters[/li]
[/ul]

cat files.txt | while [highlight]IFS=''[/highlight] read [highlight]-r[/highlight] line; do [highlight]echo[/highlight] mv "$line" /ops/data/obsolete ; done | [highlight]od -t ax1[/highlight]

If those debug steps reveal nothing to you, post some of the output, maybe somebody here will notice something.


Feherke.

http://free.rootshell.be/~feherke/

 

[theniteowl]

It is a Solaris server using I believe ksh as the shell.
All the examples I have found give me problems with the various commands not supporting the same switches as in the examples like no -t for the mv command.

I did manage to get a line that works.

 ( cat obsoletefiles.txt; echo /ops/data/obsolete ) | xargs mv

This is a bit of a work around. The text file DOES have carriage returns. It was just created in a text editor on a Windows box with one filename per line delimited by the carriage return.

To make the above line work I had to add an extra space at the end of each filename otherwise the output path was being appended directly at the end of the filename and of course then an command.

I am trying to find the simplest method to do this. We have regular file migrations to the servers and sometimes we have to obsolete some of the files that are being replaced so we have them for auditing purposes. I have a bit of Linux experience but mostly in the GUI. I can generally figure out anything I need to do but the differences in the command options have been throwing me and I do not know the available commands well enough to find other was around the issue.
In any event, we usually use FTP software for the migrations but to be able to move a file we have to first bring it down to our own device and then FTP it back up to the obsolete folder, then delete the original as the FTP software does not support move, only copy. This causes a change in the file dates which is not ideal when they have to be there for auditing purposes.
So I am trying to make as simple a method as possible to do these migrations that my non-Linux familiar co-workers will be able to handle. The need to put an extra space at the end of the filenames is one of those quirks bound to be forgotten so I need to come up with a better solution but I may need ultimately to come up with a shell script to handle things for us. I just have to learn some shell programming first.

Ideally if I could find some VBScript code that works for SSH connections I could write something that would do everything wonderfully. For now it is a matter of making it easy to move a list of files to the obsolete folder as that is the most difficult step. The rest is just FTPing files directly.

What would be a better method to delimit the filenames and how would I state the delimiter on the command line?

Thanks.


At my age I still learn something new every day, but I forget two others.

 

[theniteowl]

BTW, here is the command I used and the resultant hex.
The hex means nothing to me though.

dvemsx02:/ops/data > cat files.txt | while IFS='' read -r line; do echo mv "$line" /ops/data/obsolete ; done | od -t ax1
0000000   m   v  sp   Z   L   0   5   .   v   i   p   o   p   t  sp  cr
          6d  76  20  5a  4c  30  35  2e  76  69  70  6f  70  74  20  0d
0000020  sp   /   o   p   s   /   d   a   t   a   /   o   b   s   o   l
          20  2f  6f  70  73  2f  64  61  74  61  2f  6f  62  73  6f  6c
0000040   e   t   e  lf   m   v  sp   Z   L   0   7   .   v   i   p   o
          65  74  65  0a  6d  76  20  5a  4c  30  37  2e  76  69  70  6f
0000060   p   t  sp  cr  sp   /   o   p   s   /   d   a   t   a   /   o
          70  74  20  0d  20  2f  6f  70  73  2f  64  61  74  61  2f  6f
0000100   b   s   o   l   e   t   e  lf   m   v  sp   Z   L   0   9   .
          62  73  6f  6c  65  74  65  0a  6d  76  20  5a  4c  30  39  2e
0000120   v   i   p   o   p   t  sp  cr  sp   /   o   p   s   /   d   a
          76  69  70  6f  70  74  20  0d  20  2f  6f  70  73  2f  64  61
0000140   t   a   /   o   b   s   o   l   e   t   e  lf   m   v  sp  cr
          74  61  2f  6f  62  73  6f  6c  65  74  65  0a  6d  76  20  0d
0000160  sp   /   o   p   s   /   d   a   t   a   /   o   b   s   o   l
          20  2f  6f  70  73  2f  64  61  74  61  2f  6f  62  73  6f  6c
0000200   e   t   e  lf
          65  74  65  0a
0000204

The echo output without using od is:

 /ops/data/obsolete
 /ops/data/obsolete
 /ops/data/obsolete
 /ops/data/obsolete

At my age I still learn something new every day, but I forget two others.

 

[feherke]

Hi

I did manage to get a line that works.

( cat obsoletefiles.txt; echo /ops/data/obsolete ) | xargs mv

Not a good idea. When using [tt]mv[/tt] with multiple source files the last one, the target, has to be a directory. But there you have no control on which one will be the last.

The command lines have a given maximum length, [tt]xargs[/tt] passes as many parameters to the specified command as possible without exceding that length. If there are more parameters to pass, [tt]xargs[/tt] will execute the specified commands multiple times. And you have no control that way on where [tt]xargs[/tt] will split up the parameter list.

The text file DOES have carriage returns.

Well, then you found the solution. Use [tt]dos2unix[/tt] to remove the carriage returns. If not available on Solaris, use [tt]sed -i 's/\r$//' files.txt[/tt] .

The hex means nothing to me though.

Huh ? You only need to read it :

0000000   [highlight #fcc]m   v[/highlight]  sp   [highlight #cfc]Z   L   0   5   .   v   i   p   o   p   t  sp  cr[/highlight]
          6d  76  20  5a  4c  30  35  2e  76  69  70  6f  70  74  20  0d
0000020  sp   [highlight #ccf]/   o   p   s   /   d   a   t   a   /   o   b   s   o   l[/highlight]
          20  2f  6f  70  73  2f  64  61  74  61  2f  6f  62  73  6f  6c
0000040   [highlight #ccf]e   t   e[/highlight]  lf   m   v  sp   Z   L   0   7   .   v   i   p   o
          65  74  65  0a  6d  76  20  5a  4c  30  37  2e  76  69  70  6f

Resulting this : [highlight #fcc]mv[/highlight] [highlight #cfc]ZL05.vipopt \r[/highlight] [highlight #ccf]/ops/data/obsolete[/highlight]

The source file name is clearly invalid because the extra characters. Note that you also have to remove the trailing spaces you added at the end of lines.


Feherke.

http://free.rootshell.be/~feherke/

 

[theniteowl]

Thanks feherke I will give it a try.
I am slapping my forehead on the hex code. I am more than a bit out of it this week, my best friend of 27 years had a massive stroke on Monday and is not expected to make it. To make things worse his mother does not speak english and has not yet arrived from Hong Kong. My friend a couple months ago asked me to be his health care proxy in the unlikely event something like this ever happened and now I have the authority and moral responsibility to override the family if it comes to a disagreement in pulling the plug if no hope is found for his recovery. To say the least I have been under stress and not sleeping well. I seem to be making rational decisions but I find I keep making silly and obvious mistakes. I may have to spend tonight on a deep benadryl assisted sleep to clear my head.

I will do some testing with an enlarged list. My test file only had three filenames in it so I did not see a problem but the actual list that will be used in this afternoons migration is substantially longer and probably would have failed based on what you said above. I will use your suggestions and do some more extensive testing. If I have no luck I can always write up a quick vbs FTP script to send the files using the rename command to change the path. I did a quick test with that yesterday and it was working. Do not want to spend a lot of time as we will be moving to a system soon that automate migrations and do versioning of the files so we have a clean migration path and audit trail and none of my coworkers need to learn command line linux. We will be moving from Solaris to Red Hat soon though which would have helped. I am gonna miss ALOM though.

Thanks.

At my age I still learn something new every day, but I forget two others.

 

[theniteowl]

It worked. The sed command gave me an error saying the parameter -i was invalid which was odd as it seems to be valid according to man sed.
I was able to use dos2unix on the file though so now it works.
I found that I had to ensure there was a carriage return after the last filename before I FTPd the file and used dos2ascii or the last file in the list was not moved.

I had worried that after converting the file to unix that ftping it back to a Windows box to stage it for the migration might alter the file but I tested and it seems to work. When I open it in Notepad++ it shows me the CRLF characters just as it did prior to converting to unix though so I guess it is not showing a distinction between the two versions. I even used Beyond Compare to compare before and after versions and they match though one version fails and the other works.

Thanks for the help.

At my age I still learn something new every day, but I forget two others.

 

[feherke]

Hi

Two minor notes :
[ul]
[li]On Solaris there are some alternative tools in /usr/xpg4/bin/ which behave more similar to GNU tools. Maybe the man page referred to that one.[/li]
[li]Usually when transferring files through FTP in ascii mode, the end of line marks are converted if the local and the remote system differs.[/li]
[/ul]


Feherke.

http://free.rootshell.be/~feherke/