Tek-Tips is the largest IT community on the Internet today!

Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!

  • Congratulations gkittelson on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!

Help! How to batch import .csv files into single Access table? 2

Status
Not open for further replies.

Lou

Technical User
Apr 29, 2004
8
US
I have 1300 .csv files, with a single column & 260,000 rows each. The file name is the name I would like to assign to the field. Is there any way to automate importing the files into Access one at a time, each time adding a new field & assigning the file name to the field name? End product would be a 260,000 by 1300 table (plus ID field).
Thanks!
 
Lou,
Not to parade on your rain, Acces has a limitation of 255 fields (columns) in a table.

CMP

BTW there are a couple of different ways you can batch import the files. Have your tried anything yet that we can build off of? CMP

[small]For the best results do what I'm thinking, not what I'm saying.[/small]
(GMT-07:00) Mountain Time (US & Canada)
 
First off, that sucks! So I'm going to have to figure out how to amalgamate them too. Eeek.

Re: code, I have the following so far to try to import the files as separate tables, but I get an error message (Runtime error 3011 The Microsoft Jet database engine could not find 'S100025.csv'. Make sure the object exists and that you spell its name correctly)

Sub BatchImportCSVFiles()

Dim InputDir, ImportFile As String, tblName As String
Dim db As Database
Set db = CurrentDb

InputMsg = "Type the pathname of the folder that contains "
InputMsg = InputMsg & "the files you want to import."
InputDir = InputBox(InputMsg)
ImportFile = Dir(InputDir & "\*.csv")

Do While Len(ImportFile) > 0
' Use the import file name without its extension as the table name.
tblName = Left(ImportFile, (InStr(1, ImportFile, ".") - 1))

'Import the file
DoCmd.TransferText acImportDelim, , tblName, ImportFile

Loop

End Sub

I hadn't got as far as trying to add a field yet

THanks for any help,
Lou
 
Dir returns the file name (e.g. 'S100025.csv') but to tell Access where to find it you need the full path and file
Code:
DoCmd.TransferText acImportDelim, , tblName, [red]InputDir & "\" & [/red]ImportFile

The more significant question in all this is

What is the purpose of a 1,300 column table?

I can understand 1,300 columns in a database but 1,300 pieces of information that are all attributes of a single entity just doesn't compute.
 
Thanks for the soln - it imports the first file but none after that - I guess I need to change the name of the file that was just imported so it is no longer recognised by the module. How would i do that?

Re: 1300 columns, each csv file is a presence/absence species distribution. I have 1300 species. The rows are geographic locations. In doing this, I'm trying to generate an input file for another program, that reqiures a table format of no. location x no. features.

Is there any other way (other than Access) that I can have a 1300 column table?

Thanks :):)
 
Lou,
Lou said:
...generate an input file for another program, that reqiures a table format of no. location x no. features...

What file type options does another program give you for importing?

CMP

[small]For the best results do what I'm thinking, not what I'm saying.[/small]
(GMT-07:00) Mountain Time (US & Canada)
 
Lou,
This is typed and untested but it should work. It assumes that all the files have the same number of lines and that the data needs to be seperated by commas.
Code:
Sub AggregateFile()
On Error GoTo AggregateFile_Exit
'[b]Change this to the real directory[/b]
Const cstFilePath As String = "C:\"
Dim intIn As Integer, intOut As Integer, intTemp As Integer
Dim strFileName As String
Dim strIn As String, strTemp As String
'make an output directory
MkDir cstFilePath & "Out"
'create the output file
intOut = FreeFile
Open cstFilePath & "Out\Temp.csv" For Output As #intTemp
'Get the first file and make sure it exists
strFileName = Dir(cstFilePath & "*.csv")
If strFileName = "" Then GoTo AggregateFile_Exit
'Open the file
intIn = FreeFile
Open cstFilePath & strFileName For Input As #intIn
'Write data from the first in file to the out file
Do
  Line Input #intIn, strIn
  Write #intTemp, strIn
Loop Until EOF(intIn)
intTemp = FreeFile

'Do the rest of the files
Do While strFileName <> ""
  Reset
  strFileName = Dir
  Open cstFilePath & "Out\Temp.csv" For Input As #intTemp
  Open cstFilePath & strFileName For Input As #intIn
  Open cstFilePath & "Out\All.csv" For Output As #intOut
  Do
    Line Input #intTemp, strTemp
    Line Input #intIn, strIn
    'If you need double quotes around the fields
    'add them below ... & Chr(34) & strIn & Chr(34)
    Write #intOut, strTemp & "," & strIn
  Loop Until EOF(intIn)
  Kill cstFilePath & "Out\Temp.csv"
  FileCopy cstFilePath & "Out\All.csv", cstFilePath & "Out\Temp.csv"
  Kill cstFilePath & "Out\All.csv"
Loop

AggregateFile_Exit:
Reset
Exit Sub

AggregateFile_Error:
Stop
End Sub

When it's all done you should have a file [tt]Out\Temp.csv[/tt] with all the data.

Hope this helps,
CMP

[small]For the best results do what I'm thinking, not what I'm saying.[/small]
(GMT-07:00) Mountain Time (US & Canada)
 
> I guess I need to change the name of the file that was just imported so it is no longer recognised by the module. How would i do that?

In your code, after a CSV file is processed, move it to another directory, perhaps named "Parsed". That way you will still have the file in case you need it, and it will be the same file name.

Hope that helps alittle...
 
Just out of curiosity I ran a few tests with this (with credit to CautionMP).
Code:
Public Function AggregateFile(ByVal FilePath As String, ByVal FilePattern As String) As Double

Dim intIn                       As Integer
Dim intOut                      As Integer
Dim intTemp                     As Integer
Dim nLines                      As Long
Dim FileName                    As String
Dim strIn                       As String
Dim strTemp                     As String
Dim tm                          As Double

tm = Timer

On Error GoTo AggregateFile_Error
If Right(FilePath, 1) <> "\" Then FilePath = FilePath & "\"

'make an output directory
If Len(Dir(FilePath & "Out", vbDirectory)) = 0 Then MkDir FilePath & "Out"

FileName = Dir(FilePath & FilePattern)
If FileName = "" Then
    MsgBox "No files found for " & FilePath & FilePattern
    GoTo AggregateFile_Exit
End If

'create the output file
With New FileSystemObject
    .CopyFile FilePath & FileName, FilePath & "Out\Temp.Txt", True
End With

'Do the rest of the files
FileName = Dir
Do While FileName <> ""

    intTemp = FreeFile: Open FilePath & "Out\Temp.Txt" For Input As #intTemp
    intIn = FreeFile:   Open FilePath & FileName For Input As #intIn
    intOut = FreeFile:  Open FilePath & "Out\All.Txt" For Output As #intOut

    nLines = 0
    Do Until EOF(intIn) Or EOF(intTemp)
        nLines = nLines + 1
        If Not EOF(intTemp) Then Line Input #intTemp, strTemp
        If Not EOF(intIn) Then Line Input #intIn, strIn

        If Len(strTemp) = 0 Then
            Exit Do
        ElseIf Len(strIn) = 0 Then
            Print #intOut, strTemp & " ?"
        Else
            Print #intOut, strTemp & " " & strIn
        End If
        strTemp = ""
        strIn = ""
    Loop

    Close

    With New FileSystemObject
        .DeleteFile FilePath & "Out\Temp.Txt"
        .MoveFile FilePath & "Out\All.Txt", FilePath & "Out\Temp.Txt"
    End With

    FileName = Dir

Loop

AggregateFile_Exit:
If Dir(FilePath & "Out\All.Txt") <> "" Then Kill FilePath & "Out\All.Txt"
AggregateFile = Timer - tm
Exit Function

AggregateFile_Error:
MsgBox Err.Number & " - " & Err.Description
Close
GoTo AggregateFile_Exit
End Function
It takes 221 seconds on my machine (P4 2.8 GH; 1 GB Ram) to process 100 files of 220,000 lines each.

I also took a look at the ResNet stuff and it seems that columns are positionally dependent ... that is, the interpretation of a column depends on its position in the input array. That means that you must not only combine the files but you must combine them in the correct order.

"Dir" however doesn't guarantee that they will be in the order that you want. Do you have some way to impose a sequence on the ordering of files?
 
Hi again,
Thanks to both CMP and Golom for their help on this. I haven't managed to get the code to work yet - should I be running is as a module in Access or directly in Visual Basic (I have version 6)?

Re: the positional accuracy - only the first 15 columns matter, after that it isn't important. I have 1 file with the first 15 columns in, which could be specified as the file to append to, perhaps? or as the first file to add through the code (although couldn't this just be achieved by naming that file so that it would be picked first, assuming that the code will run through the folder in alphabetical order?)....

Thanks again - getting there :)
 
it imports the first file but none after that
Do While Len(ImportFile) > 0
' Use the import file name without its extension as the table name.
tblName = Left(ImportFile, (InStr(1, ImportFile, ".") - 1))
'Import the file
DoCmd.TransferText acImportDelim, , tblName, InputDir & "\" & ImportFile
[!]ImportFile = Dir[/!]
Loop

Hope This Helps, PH.
Want to get great answers to your Tek-Tips questions? Have a look at FAQ219-2884 or FAQ181-2886
 
Lou,
A module in Access will work, since the piece that I wrote is so vanilla (not Access specific) I actually used Excel to type and error check the code. I did a quick re-visit and this should compile in VB6 as well.

A quick note on the code from Golom, it uses the [tt]FileSystemObject[/tt] (FSO) for some of the file operations. If you don't have a reference in your project to one of the libraries that contain the [tt]FileSystemObject[/tt] object (like me) you will get an error when you try to compile or run the code.

To correct just add a reference to your project, I normally use the Microsoft Script Host Object Model (wshom.ocx), but if memory serves me correctly there are two other librarys that will expose the FSO.

Hope this helps,
CMP

[small]For the best results do what I'm thinking, not what I'm saying.[/small]
(GMT-07:00) Mountain Time (US & Canada)
 
Hi CMP,
I have it in Access and have been toggling through some breaks, as it starts to run, gets as far as creating the output directory but doesn't actually import the files.

All I needed to change was the input directory right?

I've pasted what I've been using below:

Sub AggregateFile()
On Error GoTo AggregateFile_Exit
'Change this to the real directory
Const cstFilePath As String = "C:\PhD\Data\SAUP\2003SpeDistributions"
Dim intIn As Integer, intOut As Integer, intTemp As Integer
Dim strFileName As String
Dim strIn As String, strTemp As String
'make an output directory
MkDir cstFilePath & "Out"
'create the output file
intOut = FreeFile
Open cstFilePath & "Out\Temp.csv" For Output As #intTemp
'Get the first file and make sure it exists
strFileName = Dir(cstFilePath & "*.csv")
If strFileName = "" Then GoTo AggregateFile_Exit
'Open the file
intIn = FreeFile
Open cstFilePath & strFileName For Input As #intIn
'Write data from the first in file to the out file
Do
Line Input #intIn, strIn
Write #intTemp, strIn
Loop Until EOF(intIn)
intTemp = FreeFile

'Do the rest of the files
Do While strFileName <> ""
Reset
strFileName = Dir
Open cstFilePath & "Out\Temp.csv" For Input As #intTemp
Open cstFilePath & strFileName For Input As #intIn
Open cstFilePath & "Out\All.csv" For Output As #intOut
Do
Line Input #intTemp, strTemp
Line Input #intIn, strIn
'If you need double quotes around the fields
'add them below ... & Chr(34) & strIn & Chr(34)
Write #intOut, strTemp & "," & strIn
Loop Until EOF(intIn)
Kill cstFilePath & "Out\Temp.csv"
FileCopy cstFilePath & "Out\All.csv", cstFilePath & "Out\Temp.csv"
Kill cstFilePath & "Out\All.csv"
Loop

AggregateFile_Exit:
Reset
Exit Sub

AggregateFile_Error:
Stop
End Sub


Thanks,
Lou
 
Lou,
It would help if I wrote the error handler correctly. The file path also needs to end in [tt]\[/tt] since I didn't add the punctuation when I open the files.
Code:
Sub AggregateFile()
On Error GoTo AggregateFile_E[b]rror[/b]
'Change this to the real directory
Const cstFilePath As String = "C:\PhD\Data\SAUP\2003SpeDistributions[b][red]\[/red][/b]"
...

Sorry about that,
CMP

[small]For the best results do what I'm thinking, not what I'm saying.[/small]
(GMT-07:00) Mountain Time (US & Canada)
 
OK, thanks. Now it's perceiving an error, as it's making the Out directory, but it still jumps straight from

MkDir cstFilePath & "Out"

to Stop (under AggregateFile_Error)

I have breaks at:
intout=Freefile
Open cstFilePath & "Out\Temp.csv" For Output as #intTemp

and various other places through the code, but the error seems to be at the MkDir line (even though it makes the Directory).

I fixed this by changing the location of the \ to
MkDir cstFilePath & "\Out"

This enabled me to get a couple of lines further, to:
Open cstFilePath & "\Out\Temp.csv" For Output As #intTemp
but then it hits an error again
I had also changed the following line to
strFileName=Dir(cstFilePath & "\*.csv")

Any ideas?
Thanks,
Lou
 
MkDir cstFilePath & "Out"

is probably raising an error because cstFilePath & "Out" already exists.

Try
Code:
If Len(Dir(cstFilePath & "Out", vbDirectory)) = 0 Then MkDir cstFilePath & "Out"
 
Status
Not open for further replies.

Part and Inventory Search

Sponsor

Back
Top