Extreme File Handling

[HardcoreTechnoHead]

Does anyone have a way of handling files that are larger than 2Gb with VB.

I have a Parsing Utility which has been used to parse text files (about 1Gb each) output from a mainframe billing system and insert the data into a database.

We are in the process of decomissioning the mainframe that produced the original text files and all of the original systems data has been dumped into text files for data warehousing.

The problem I face is that I now have 14x 90Gb files to parse. The utility will not read the file. I can open the file and get a handle to it, but any attempt to read produces and error 52 : Bad File Name or Number error


Dim lngFileHandle as long
Dim strReadBuffer as String * mc_lngRecordLength

lngFileHandle = Freefile
Open filename for Random Access Read Lock Read Write as lngFileHandle Len = mc_lngRecordLength

'fails here
Get #lngFileNumber, RecordNumber, strReadBuffer


I have tried opening for Input, Random and Binary and get the same results with each. Any Ideas

 

[MichaelRed]

No soloution - but one question -- does the system have sufficient ADDRESSABLE disc space for the process? I think you need ~~ 2X the file size to be able to manipulate it, so your 1G source would need ~~ 2G of EMPTY/AVAILABLE disc?



MichaelRed
m.red@att.net

Searching for employment in all the wrong places

 

[IWarez]

You can try to use the createfile api to open the file and use API file I/O to read/write.

 

[kel1981b]

[red]'Open filename for Random Access Read Lock Read Write'[/red]
I am not sure that cause your problem but you Open file for Access Read only and then Locked it for Read Write
I think you have open it to Read/Write
Something like that

Open filename for Random Access Read Write Lock Read Write

 

[chiph]

kel1981b -

The problem is that the usual VB "Open filename..." routines are limited to 2gb or smaller files.

CreateFile api would be your only chance. Unless you're able to split the file into smaller pieces via some 3rd party tool?

Chip H.

 

[dirknerkle]

You may have to use the getchunk method and parse only one block of data at a time.

 

[HardcoreTechnoHead]

In the end, I have written a Java utility to split the files into chunks that can be processed by the VB Parser.

Thanks for all your attempts to answer this. In answer to some of your questions

MichaelRed - I have almost a terrabyte of storage available, so no problem with processing needs.

dirknerkle - not sure how the GetChunk method would help me with file handling on local files. Surely you not suggesting that I process through some kind of web interface, with a 90Gb file?

 

[kel1981b]

chiph -
Read my reply one more time. I said "I am not sure that cause your problem". There is one more problem, it's not possible to open file for read only and lokck it for Read Write

HardcoreTechnoHead - did you try to use FileSystemObject?

 

[vb5prgrmr]

HardcoreTechnoHead,

I think dirknerkle was talking about using the MS ODBC Text driver with ADO to read in your files (could be wrong).

Good Luck

 

[HardcoreTechnoHead]

kel1981b

Access and Lock operate independently of each other. Access is how your code can use the file. Lock is how other programs can use the file at the same time as your program. Whilst it may not always be approriate to do so, al logical combinations of Access and Lock can be used.

For Instance, If your program only ever writes to a log file, you could "Access Write" with no lock. but If you want to read in from a file which you do not want to be changed by an external application then you would use "Access Read Lock Read Write"