How to merge multiple text files? 3

[linousa]

The question is - of how to merge multiple text files, using something similar to browse feature(append from ? sdf), the problem of using it "as is", is that it allows to append only one selected file, and I can't select more than one at the time and I need to find a way to select and append 30-50 at the time. Fyi: file names are always different and need to keep them.
If someone can help with this question- RESPECT! =)

 

[Mike Lewis]

There are several ways of doing this. One that comes to mind is to use the DOS COPY command:

COPY /A File1.txt + File2.txt NewFile.txt

You can run this from VFP using the RUN command.

Mike



__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips, training, consultancy

 

[linousa]

First of all I want to keep it in VFP, and second - those names are very long and need to stay the same. If you are talking bat/cmd files and use "tokens", I need more info to accomplish it...

 

[Olaf Doschke]

The file names will stay the same, what is the problem? I think you meant to say the file names may change to whatever names are selected.
No matter how you do it, it's easy to put file names into an array or several variables and refer to them instead of hardcoded file names using name expressions, which means variable name inside paranthessis: (var)

You will need one new file to be created or one of the files is expanded with the others. You will need to decide that.

One way to do it wouthout a dos command running, even though that's fair enough and you can suppress the dos box, is to fopen() files and fread() them and fwrite() to one of them, after fseek()ing to it's end, then fclose() them.

Unless all files are longer than 2GB in total, when appended into one, then you'd have to use WinApi file functions to work on larger files.

Bye, Olaf.

 

[Mike Lewis]

Linousa, my solution was in VFP. You use the VFP RUN command to run the DOS command I showed you. And, as Olaf said, there's nothing to stop you from varying the filenames, or from using long file names. What I showed you was just an example.

But there are plenty of alternatives. Here is another example:

lcFile1 = <put the name of the first input file here>
lcFile2 = <and the second one here>
lcFile3 = <and this is the output file (which can have the same name as File1 or File2)>

lcX = FILETOSTR(lcFile1) + FILETOSTR(lcFile2)
STRTOFILE(lcX, lcFile3)

Mike


__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips, training, consultancy

 

[Mike Lewis]

Come to think of it, if you want to append File2 to an existing File1 (rather than creating an entirely new file), it would be faster to do this:

lcFile1 = <put the name of the first input file here>
lcFile2 = <and the second one here>

lcX = FILETOSTR(lcFile2)
STRTOFILE(lcX, lcFile1, .T.)

The third parameter to STRTOFILE says that the new file will be added to the end of the existing one.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips, training, consultancy

 

[Dan Freeman]

Just be aware that if the incoming txt file is too large Mike's code will blow up. The part that blows up, though, is assigning a string that is too long to a variable. You can bypass the variable entirely!

lcFile1 = <put the name of the first input file here>
lcFile2 = <and the second one here>

STRTOFILE(FILETOSTR(lcFile2), lcFile1, .T.)

 

[Mike Lewis]

That's interesting, Dan. Are you saying that, if a string is too long to be assigned to a variable, it might still be OK to return it from one function and pass it straight to another?

I'm not disagreeing with you. I would assume that, if it was too long to be explicitly assigned to a variable, it would still be too long to to be passed behind the scenes in that way. But you might well be right. I guess it will need a bit of experimenting to find out.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips, training, consultancy

 

[Dan Freeman]

Yep. It works.

Fixed a bug with this not long ago. The app was abending because it was trying to store too long of a string to a variable. But since it was coming from one text file and going to another, I just took the variable assignment out of the middle and it all worked fine.

 

[Olaf Doschke]

Well Mike,

quite a while ago I did some experiments. The 16MB limitation documented in the VFP topic about VFP system capacities is a soft limitation, not a hard one. Actually I don't remember the details of my tests, but I think it wasn't really a variable limitation. I think some functions only work reliable up to that chunk size. You can't go wrong, if you rather use the FREAD/FWRITE approach and limit yourself to 8k blocks or such. If the files are all smaller anyway, this part of the discussion really doesn't matter, just for the sake of scaling up in the long run.

Bye, Olaf.

 

[Dan Freeman]

Olaf,

I ran some performance tests when we first got Filetostr() and Strtofile() and they win over LLFNs. Every time.

Another way to skin this cat is:

Create Cursor mytemp (mytext m)
Append Blank
Append Memo mytext From (lcFilename)
StrtoFile("filename.txt",mytemp.mymemo)

Of course, you could also use ADDITIVE on APPEND MEMO, and once you've got the string in a memo field there's COPY MEMO.

Gotta love VFP. There are ALWAYS at least three ways to do any one thing.

 

[Mike Lewis]

Thanks for that, Dan and Olaf. An interesting point.

Of course, we don't know whether any of this is relevant to Linousa. We don't know if these are particularly large text files, nor whether performance is an issue (it might be a one-off job). It would be helpful if he could come back and give us some response to the suggestions he's had so far.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips, training, consultancy

 

[linousa]

Thank you very much guys for your effort! But I need, also looking for some solution that will let me to select or automatically will grab multiple files in some directory. Another words(making long story-short): I have one folder, that gets updated 10 times a day and every update uploads from 10 to 500 text files, with unique filenames. When I had 10-20 files I was fine using 'button.click' event:

append from ? sdf

Which offered me to select one by one text files. But now I need some better solution for 500 files, 5 times a day and number will only grow.

 

[JRB-Bldr]

But I need, also looking for some solution that will let me to select or automatically will grab multiple files in some directory.

A new topic should be posted separately - not tacked on to an existing, different-topic posting.

Good Luck,
JRB-Bldr

 

[Olaf Doschke]

Well, you have ADIR() and with that can programmatically determine all files in a directory you pick with one GETDIR() call, or even make that directory part of a configuration you only need to specify once, that's so simple, that's why you only got the part of the problem to append files discussed. Then ask a seperate question about how to best pick multiple files.

Append from filename SDF appends to a DBF or cursor, that was not in your specification. That also only works good with identical file types with identical columns, we also didn't know about that, though that was a guess I made anyway, but it could also be you wanted to append many gz files to a tar.

If in the end you pull together all data into a DBF, what also could be reconsidered is generating all these seperate files, why isn't there a table to be filled in the first place? You cannot only fill a DBF, but also a CSV file via ODBC or file operations from other languages and append to a file instead of generating a huge amount of files.

Bye, Olaf.

 

[Mike Lewis]

Thank you very much guys for your effort!

So have your found a solution? Did you adopt any of our suggestions? If so, which one?

In general, when you receive suggestions or answers to a problem, it's helpful if you could give some feedack, so that we can know if the suggestions were useful. That helps anyone else who might have the same problem. It's also a courtesy to those who took the trouble to reply.

But I need, also looking for some solution that will let me to select or automatically will grab multiple files in some directory.

This is a separate issue. As JRB-Bldr says, you should start a new topic for it. That way, you're likely to get a better range of replies, and, once again, it will be helpful for anyone searching for the same problem.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips, training, consultancy

 

[linousa]

Sorry, that it took me so long to reply.
I have it working using "ADIR()" and after looping "Append From" command, not sure if it's the best way of doing it, but it is working so far. Thank you everyone, if you have any additional suggestion or ideas, please feel free to write.