Automating uploads

[Dweezel]

This isn't a question relating directly to php, but I'm hoping someone here may have tackled a similar problem.

I have a client who is emailed large numbers of csv files everyday. I need to parse the data in these files using a php script, extracting particular pieces of information and entering them into a mysql database. The data is then to be displayed in graphs also generated from php scripts.

None of this is a problem as I've done this kind of thing before. However, getting the files to the php scripts is where I need help. If my client saves all of the csv's (there will be hundreds every day) to a file on his computer, is there a way that I can automate the upload of the files?

When I've done this kind of thing before I've created a simple file input form field for the client, to browse to the data file and upload it. In this case this just isn't practical with the number of files that need to uploaded every day.

I'm looking for any ideas and suggestions please.

 

[BabyJeffy]

Maybe zip them all into one big file, then use the file upload idea you have used in the past? I'm sure there will be an unzip solution (free?) available in php script somewhere... and if not, you could run a shell script on the server to do it. Then you have your million csv files ready for parsing!

Hope that's what you are after

Cheers,
Jeff

[tt]Jeff's Page [/tt][tt]@[/tt][tt] Code Couch

http://www.codecouch.com/jeff/

[/tt]

 

[sleipnir214]

I had to do something similar once (process files that arrived by email).

I was using a Linux box as the server for the domain which held the address to which the files were being emailed and I was using qmail as the MTA. qmail can be configured that when email arrives for a certain recipient, instead of attempting to put the mail in a mailbox, it invokes a program an feeds the email to the program.

I then wrote a perl script to process the data in the mail.

I don't know what your environment is like, but you can automate PHP similarly on a unix-like OS with qmail, postfix or sendmail.

Want the best answers? Ask the best questions!

TANSTAAFL!!

 

[Dweezel]

Thanks for the replies. I know that my client has a remote host and no shell access. I'm guessing this is going to throw a spanner in the works if I were to attempt your idea Sleipnir. Reading the data directly from the emails would be an ideal solution though.

Looking at the php manual I see that I can unzip using php but it requires the ZZIPlib libray to be installed. I'll have to check with my clients host.

 

[KarveR]

I'd ask, where do all the csv's come from, what system - I do something similar for a client (spam him with csv's), if requested I'd do mysql dumps instead.

______________________________________________________________________
There's no present like the time, they say. - Henry's Cat.

 

[Dweezel]

The csv's are created by monitoring equipment that records water flow rates through pipes. My client supplies this equipment to water utility companies.

As to how this data is passed from monitoring equiment to computer and then emailed I'm not sure at the moment. I'm meeting with him tomorrow afternoon and will get the full sp then.

 

[Dweezel]

I've just got back from a meeting with my client that has shed some light on how the data files are being transfered.

Originally one of my clients' associates said that he belived the files were being sent via email. It turns out that the files are sent via SMS from the monitoring equipment to a computer in his office.

I've suggested installing apache and php on this computer and using the php ftp functions to get the data to the remote server. Does this sound like a good idea?

Also, as the computer is running a Windows OS and I therefore can't set up a cron job, would it be possible to set the ftp script to run using windows scheduled tasks?

 

[Dweezel]

I've just found some ftp software that I can set up with a scheduled ftp transfer so ignore the questions above.

Thanks for looking.