Changing Host - PHP MySQL question 1

Dweezel · Jan 17, 2007

I'm unhappy with my current host and want to find another in the near future. One of the problems I'm having is with a php script which parses many csv files into a database. It runs as a cron job twice daily. The problem is that it seems to timeout after about 30 seconds without completing its task. The fact that only one cron job can be set, and that it will only run a maximum of 2 times a day is also a headache. If the timeouts can't be avoided, then I could easily process the data in batches with seperate scripts if I could set multiple cron jobs.

My site relies heavily on MySQL and PHP. The server I'm on at the moment is running PHP version 4.3.10 and MySQL version 3.23.56. I'm wondering which versions of PHP and MySQL I'll need to avoid in order for my site to work? I'm guessing the answer won't be as simplistic as the question.

Any advice appreciated.

jpadie · Jan 17, 2007

i don't think it has anything to do with the versions you are on. have you tried simply adding a

Code:

set_time_limit(0);

to the beginning of the script?

Dweezel · Jan 17, 2007

By 'in order for my site to work' I wasn't talking about this script. I meant for my scripts and database functions to work in general when I change PHP/MySQL versions.

If this set_time_limit function works it may solve the problem though and I won't need to change providers.

Thanks. I'll give it a go and get back to you.

Dweezel · Jan 17, 2007

That didn't work mate. I tried it like this:

Code:

set_time_limit(100);

Thinking along similar lines I also tried some other functions:

Code:

ini_set("maximum_execution_time","100");
ini_set("maximum_input_time","100");

None of them worked. The manual says that they won't work if PHP is in safe mode. I've checked the phpinfo file and safe mode is definitely disabled on the server, so this won't be causing the problem.

It also states that the first ini_set I've used (max_execution time) returns false on failure, and returns the old value as set in the php.ini on success. In my script it returns the value '30' which is what maximum_execution_time is set to in the php.ini, yet the script is still getting cut off. No errors or feedback is given at all. The script just stops working.

Any ideas?

jpadie · Jan 17, 2007

can you share your csv import code and the config values from phpinfo()?

Dweezel · Jan 18, 2007

Thanks again for your help with this jpadie. Although the files have a csv extension the values are not comma-seperated. Apart from the headers (which I use to identify each file) the rest of the file is one piece of data per line, so I've used fgetss rather than fgetcsv. This is the import code:

Code:

//Get the names of all files in the directory
$opdir = opendir($dirname);


//Loop through the files in the directory
while($file = readdir($opdir))
{


 

    //Open a file
    $fp = fopen("../data/$file","r");




   
    //Loop through the file one line at a time
    while(!feof($fp))
      {


       $data = fgetss($fp,1000);

       
       //database input functions e.t.c go here
      

      }


}

If you copy and paste the code below into a html file and view it in a browser you can see all of the php settings on the server:

Code:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "DTD/xhtml1-transitional.dtd">
<html><head>
<style type="text/css"><!--
body {background-color: #ffffff; color: #000000;}
body, td, th, h1, h2 {font-family: sans-serif;}
pre {margin: 0px; font-family: monospace;}
a:link {color: #000099; text-decoration: none; background-color: #ffffff;}
a:hover {text-decoration: underline;}
table {border-collapse: collapse;}
.center {text-align: center;}
.center table { margin-left: auto; margin-right: auto; text-align: left;}
.center th { text-align: center !important; }
td, th { border: 1px solid #000000; font-size: 75%; vertical-align: baseline;}
h1 {font-size: 150%;}
h2 {font-size: 125%;}
.p {text-align: left;}
.e {background-color: #ccccff; font-weight: bold; color: #000000;}
.h {background-color: #9999cc; font-weight: bold; color: #000000;}
.v {background-color: #cccccc; color: #000000;}
i {color: #666666; background-color: #cccccc;}
img {float: right; border: 0px;}
hr {width: 600px; background-color: #cccccc; border: 0px; height: 1px; color: #000000;}
//--></style>
<title>phpinfo()</title></head>
<body><div class="center">
<table border="0" cellpadding="3" width="600">
<tr class="h"><td>
<h1 class="p">PHP Version 4.3.10</h1>
</td></tr>
</table><br />
<table border="0" cellpadding="3" width="600">
<tr><td class="e">System </td><td class="v">###</td></tr>
<tr><td class="e">Build Date </td><td class="v">############# </td></tr>
<tr><td class="e">Configure Command </td><td class="v"> './configure' 'i386-redhat-linux' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/usr/com' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--prefix=/usr' '--with-config-file-path=/etc' '--disable-debug' '--enable-pic' '--disable-rpath' '--enable-inline-optimization' '--with-bz2' '--with-db3' '--with-curl' '--with-dom=/usr' '--with-exec-dir=/usr/bin' '--with-freetype-dir=/usr' '--with-png-dir=/usr' '--with-gd' '--enable-gd-native-ttf' '--with-ttf' '--with-gdbm' '--with-gettext' '--with-ncurses' '--with-gmp' '--with-iconv' '--with-jpeg-dir=/usr' '--with-mm' '--with-openssl' '--with-png' '--with-pspell' '--with-regex=system' '--with-xml' '--with-expat-dir=/usr' '--with-zlib' '--with-layout=GNU' '--enable-bcmath' '--enable-debugger' '--enable-exif' '--enable-ftp' '--enable-magic-quotes' '--enable-safe-mode' '--enable-sockets' '--enable-sysvsem' '--enable-sysvshm' '--enable-discard-path' '--enable-track-vars' '--enable-trans-sid' '--enable-yp' '--enable-wddx' '--without-oci8' '--with-imap=shared' '--with-imap-ssl' '--with-kerberos=/usr/kerberos' '--with-ldap=shared' '--with-mysql=shared,/usr' '--with-pgsql=shared' '--with-snmp=shared,/usr' '--with-snmp=shared' '--enable-ucd-snmp-hack' '--with-unixODBC=shared' '--enable-memory-limit' '--enable-bcmath' '--enable-shmop' '--enable-versioning' '--enable-calendar' '--enable-dbx' '--enable-dio' '--enable-mbstring' '--enable-mbstr-enc-trans' '--enable-cgi' '--enable-cli' '--disable-force-cgi-redirect' </td></tr>
<tr><td class="e">Server API </td><td class="v">CGI </td></tr>
<tr><td class="e">Virtual Directory Support </td><td class="v">disabled </td></tr>
<tr><td class="e">Configuration File (php.ini) Path </td><td class="v">/etc/php.ini </td></tr>
<tr><td class="e">PHP API </td><td class="v">#######</td></tr>
<tr><td class="e">PHP Extension </td><td class="v">######### </td></tr>
<tr><td class="e">Zend Extension </td><td class="v">###########</td></tr>
<tr><td class="e">Debug Build </td><td class="v">no </td></tr>
<tr><td class="e">Thread Safety </td><td class="v">disabled </td></tr>
<tr><td class="e">Registered PHP Streams </td><td class="v">php, http, ftp, https, ftps, compress.bzip2, compress.zlib   </td></tr>
</table><br />
<table border="0" cellpadding="3" width="600">
<tr class="v"><td>
<a href="[URL unfurl="true"]http://www.zend.com/"><img[/URL] border="0" src="/php.php?=PHPE9568F35-D428-11d2-A769-00AA001ACF42" alt="Zend logo" /></a>
This program makes use of the Zend Scripting Language Engine:<br />Zend Engine v1.3.0, Copyright (c) 1998-2004 Zend Technologies
    with Zend Optimizer v2.5.7, Copyright (c) 1998-2004, by Zend Technologies
</td></tr>
</table><br />
<hr />
<h1><a href="/php.php?=PHPB8B5F2A0-3C92-11d3-A3A9-4C7B08C10000">PHP Credits</a></h1>
<hr />
<h1>Configuration</h1>
<h2>PHP Core</h2>
<table border="0" cellpadding="3" width="600">
<tr class="h"><th>Directive</th><th>Local Value</th><th>Master Value</th></tr>
<tr><td class="e">allow_call_time_pass_reference</td><td class="v">On</td><td class="v">On</td></tr>
<tr><td class="e">allow_url_fopen</td><td class="v">On</td><td class="v">On</td></tr>
<tr><td class="e">always_populate_raw_post_data</td><td class="v">Off</td><td class="v">Off</td></tr>
<tr><td class="e">arg_separator.input</td><td class="v">&amp;</td><td class="v">&amp;</td></tr>
<tr><td class="e">arg_separator.output</td><td class="v">&amp;</td><td class="v">&amp;</td></tr>
<tr><td class="e">asp_tags</td><td class="v">Off</td><td class="v">Off</td></tr>
<tr><td class="e">auto_append_file</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">auto_prepend_file</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">browscap</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">default_charset</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">default_mimetype</td><td class="v">text/html</td><td class="v">text/html</td></tr>
<tr><td class="e">define_syslog_variables</td><td class="v">Off</td><td class="v">Off</td></tr>
<tr><td class="e">disable_classes</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">disable_functions</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">display_errors</td><td class="v">On</td><td class="v">On</td></tr>
<tr><td class="e">display_startup_errors</td><td class="v">Off</td><td class="v">Off</td></tr>
<tr><td class="e">doc_root</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">docref_ext</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">docref_root</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">enable_dl</td><td class="v">On</td><td class="v">On</td></tr>
<tr><td class="e">error_append_string</td><td class="v">&lt;/font&gt;</td><td class="v">&lt;/font&gt;</td></tr>
<tr><td class="e">error_log</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">error_prepend_string</td><td class="v">&lt;font color=ff0000&gt;</td><td class="v">&lt;font color=ff0000&gt;</td></tr>
<tr><td class="e">error_reporting</td><td class="v">2039</td><td class="v">2039</td></tr>
<tr><td class="e">expose_php</td><td class="v">On</td><td class="v">On</td></tr>
<tr><td class="e">extension_dir</td><td class="v">/usr/lib/php4</td><td class="v">/usr/lib/php4</td></tr>
<tr><td class="e">file_uploads</td><td class="v">On</td><td class="v">On</td></tr>
<tr><td class="e">gpc_order</td><td class="v">GPC</td><td class="v">GPC</td></tr>
<tr><td class="e">highlight.bg</td><td class="v"><font style="color: #FFFFFF">#FFFFFF</font></td><td class="v"><font style="color: #FFFFFF">#FFFFFF</font></td></tr>
<tr><td class="e">highlight.comment</td><td class="v"><font style="color: #FF9900">#FF9900</font></td><td class="v"><font style="color: #FF9900">#FF9900</font></td></tr>
<tr><td class="e">highlight.default</td><td class="v"><font style="color: #0000CC">#0000CC</font></td><td class="v"><font style="color: #0000CC">#0000CC</font></td></tr>
<tr><td class="e">highlight.html</td><td class="v"><font style="color: #000000">#000000</font></td><td class="v"><font style="color: #000000">#000000</font></td></tr>
<tr><td class="e">highlight.keyword</td><td class="v"><font style="color: #006600">#006600</font></td><td class="v"><font style="color: #006600">#006600</font></td></tr>
<tr><td class="e">highlight.string</td><td class="v"><font style="color: #CC0000">#CC0000</font></td><td class="v"><font style="color: #CC0000">#CC0000</font></td></tr>
<tr><td class="e">html_errors</td><td class="v">Off</td><td class="v">Off</td></tr>
<tr><td class="e">ignore_repeated_errors</td><td class="v">Off</td><td class="v">Off</td></tr>
<tr><td class="e">ignore_repeated_source</td><td class="v">Off</td><td class="v">Off</td></tr>
<tr><td class="e">ignore_user_abort</td><td class="v">Off</td><td class="v">Off</td></tr>
<tr><td class="e">implicit_flush</td><td class="v">Off</td><td class="v">Off</td></tr>
<tr><td class="e">include_path</td><td class="v">.:/usr/share/pear</td><td class="v">.:/usr/share/pear</td></tr>
<tr><td class="e">log_errors</td><td class="v">Off</td><td class="v">Off</td></tr>
<tr><td class="e">log_errors_max_len</td><td class="v">1024</td><td class="v">1024</td></tr>
<tr><td class="e">magic_quotes_gpc</td><td class="v">On</td><td class="v">On</td></tr>
<tr><td class="e">magic_quotes_runtime</td><td class="v">Off</td><td class="v">Off</td></tr>
<tr><td class="e">magic_quotes_sybase</td><td class="v">Off</td><td class="v">Off</td></tr>
<tr><td class="e">max_execution_time</td><td class="v">30</td><td class="v">30</td></tr>
<tr><td class="e">max_input_time</td><td class="v">-1</td><td class="v">-1</td></tr>
<tr><td class="e">memory_limit</td><td class="v">16M</td><td class="v">16M</td></tr>
<tr><td class="e">open_basedir</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">output_buffering</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">output_handler</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">post_max_size</td><td class="v">8M</td><td class="v">8M</td></tr>
<tr><td class="e">precision</td><td class="v">14</td><td class="v">14</td></tr>
<tr><td class="e">register_argc_argv</td><td class="v">On</td><td class="v">On</td></tr>
<tr><td class="e">register_globals</td><td class="v">On</td><td class="v">On</td></tr>
<tr><td class="e">report_memleaks</td><td class="v">On</td><td class="v">On</td></tr>
<tr><td class="e">safe_mode</td><td class="v">Off</td><td class="v">Off</td></tr>
<tr><td class="e">safe_mode_exec_dir</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">safe_mode_gid</td><td class="v">Off</td><td class="v">Off</td></tr>
<tr><td class="e">safe_mode_include_dir</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">sendmail_from</td><td class="v">####</td><td class="v">######</td></tr>
<tr><td class="e">sendmail_path</td><td class="v">########</td><td class="v">##########</td></tr>
<tr><td class="e">serialize_precision</td><td class="v">100</td><td class="v">100</td></tr>
<tr><td class="e">short_open_tag</td><td class="v">On</td><td class="v">On</td></tr>
<tr><td class="e">SMTP</td><td class="v">localhost</td><td class="v">localhost</td></tr>
<tr><td class="e">smtp_port</td><td class="v">25</td><td class="v">25</td></tr>
<tr><td class="e">sql.safe_mode</td><td class="v">Off</td><td class="v">Off</td></tr>
<tr><td class="e">track_errors</td><td class="v">Off</td><td class="v">Off</td></tr>
<tr><td class="e">unserialize_callback_func</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">upload_max_filesize</td><td class="v">8M</td><td class="v">8M</td></tr>
<tr><td class="e">upload_tmp_dir</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">user_dir</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">variables_order</td><td class="v">EGPCS</td><td class="v">EGPCS</td></tr>
<tr><td class="e">xmlrpc_error_number</td><td class="v">0</td><td class="v">0</td></tr>
<tr><td class="e">xmlrpc_errors</td><td class="v">Off</td><td class="v">Off</td></tr>
<tr><td class="e">y2k_compliance</td><td class="v">Off</td><td class="v">Off</td></tr>
</table><br />
<h2><a name="module_Zend Optimizer">Zend Optimizer</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">Optimization Pass 1 </td><td class="v">enabled </td></tr>
<tr><td class="e">Optimization Pass 2 </td><td class="v">enabled </td></tr>
<tr><td class="e">Optimization Pass 3 </td><td class="v">enabled </td></tr>
<tr><td class="e">Optimization Pass 4 </td><td class="v">enabled </td></tr>
<tr><td class="e">Optimization Pass 9 </td><td class="v">disabled </td></tr>
<tr><td class="e">Zend Loader </td><td class="v">enabled </td></tr>
<tr><td class="e">License Path </td><td class="v"><i>no value</i> </td></tr>
</table><br />
<h2><a name="module_bcmath">bcmath</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">BCMath support </td><td class="v">enabled </td></tr>
</table><br />
<h2><a name="module_bz2">bz2</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">BZip2 Support </td><td class="v">Enabled </td></tr>
<tr><td class="e">BZip2 Version </td><td class="v">1.0.2, 30-Dec-2001 </td></tr>
</table><br />
<h2><a name="module_calendar">calendar</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">Calendar support </td><td class="v">enabled </td></tr>
</table><br />
<h2><a name="module_ctype">ctype</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">ctype functions </td><td class="v">enabled </td></tr>
</table><br />
<h2><a name="module_curl">curl</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">CURL support </td><td class="v">enabled </td></tr>
<tr><td class="e">CURL Information </td><td class="v">libcurl 7.9.8 (OpenSSL 0.9.6b) (ipv6 enabled) </td></tr>
</table><br />
<h2><a name="module_dba">dba</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">DBA support </td><td class="v">enabled </td></tr>
<tr><td class="e">Supported handlers </td><td class="v">gdbm cdb cdb_make db3 inifile flatfile  </td></tr>
</table><br />
<h2><a name="module_dbx">dbx</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">dbx support </td><td class="v">enabled </td></tr>
<tr><td class="e">dbx version </td><td class="v">1.0.0 </td></tr>
<tr><td class="e">supported databases </td><td class="v">MySQL
ODBC
PostgreSQL
Microsoft SQL Server
FrontBase
Oracle 8 (oci8)
Sybase-CT </td></tr>
</table><br />
<table border="0" cellpadding="3" width="600">
<tr class="h"><th>Directive</th><th>Local Value</th><th>Master Value</th></tr>
<tr><td class="e">dbx.colnames_case</td><td class="v">unchanged</td><td class="v">unchanged</td></tr>
</table><br />
<h2><a name="module_dio">dio</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">dio support </td><td class="v">enabled </td></tr>
</table><br />
<h2><a name="module_domxml">domxml</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">DOM/XML </td><td class="v">enabled </td></tr>
<tr><td class="e">DOM/XML API Version </td><td class="v">20020815 </td></tr>
<tr><td class="e">libxml Version </td><td class="v">20419 </td></tr>
<tr><td class="e">HTML Support </td><td class="v">enabled </td></tr>
<tr><td class="e">XPath Support </td><td class="v">enabled </td></tr>
<tr><td class="e">XPointer Support </td><td class="v">enabled </td></tr>
</table><br />
<h2><a name="module_exif">exif</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">EXIF Support </td><td class="v">enabled </td></tr>
<tr><td class="e">EXIF Version </td><td class="v">1.4 $Id: exif.c,v 1.118.2.29 2004/11/10 01:44:58 iliaa Exp $ </td></tr>
<tr><td class="e">Supported EXIF Version </td><td class="v">0220 </td></tr>
<tr><td class="e">Supported filetypes </td><td class="v">JPEG,TIFF </td></tr>
</table><br />
<h2><a name="module_ftp">ftp</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">FTP support </td><td class="v">enabled </td></tr>
</table><br />
<h2><a name="module_gd">gd</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">GD Support </td><td class="v">enabled </td></tr>
<tr><td class="e">GD Version </td><td class="v">bundled (2.0.28 compatible) </td></tr>
<tr><td class="e">FreeType Support </td><td class="v">enabled </td></tr>
<tr><td class="e">FreeType Linkage </td><td class="v">with freetype </td></tr>
<tr><td class="e">GIF Read Support </td><td class="v">enabled </td></tr>
<tr><td class="e">GIF Create Support </td><td class="v">enabled </td></tr>
<tr><td class="e">JPG Support </td><td class="v">enabled </td></tr>
<tr><td class="e">PNG Support </td><td class="v">enabled </td></tr>
<tr><td class="e">WBMP Support </td><td class="v">enabled </td></tr>
<tr><td class="e">XBM Support </td><td class="v">enabled </td></tr>
</table><br />
<h2><a name="module_gettext">gettext</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">GetText Support </td><td class="v">enabled </td></tr>
</table><br />
<h2><a name="module_gmp">gmp</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">gmp support </td><td class="v">enabled </td></tr>
</table><br />
<h2><a name="module_iconv">iconv</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">iconv support </td><td class="v">enabled </td></tr>
<tr><td class="e">iconv implementation </td><td class="v">glibc </td></tr>
<tr><td class="e">iconv library version </td><td class="v">2.2.5 </td></tr>
</table><br />
<table border="0" cellpadding="3" width="600">
<tr class="h"><th>Directive</th><th>Local Value</th><th>Master Value</th></tr>
<tr><td class="e">iconv.input_encoding</td><td class="v">ISO-8859-1</td><td class="v">ISO-8859-1</td></tr>
<tr><td class="e">iconv.internal_encoding</td><td class="v">ISO-8859-1</td><td class="v">ISO-8859-1</td></tr>
<tr><td class="e">iconv.output_encoding</td><td class="v">ISO-8859-1</td><td class="v">ISO-8859-1</td></tr>
</table><br />
<h2><a name="module_imap">imap</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">IMAP c-Client Version </td><td class="v">2001 </td></tr>
<tr><td class="e">SSL Support </td><td class="v">enabled </td></tr>
<tr><td class="e">Kerberos Support </td><td class="v">enabled </td></tr>
</table><br />
<h2><a name="module_ldap">ldap</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">LDAP Support </td><td class="v">enabled </td></tr>
<tr><td class="e">RCS Version </td><td class="v">$Id: ldap.c,v 1.130.2.10 2004/06/01 21:05:33 iliaa Exp $ </td></tr>
<tr><td class="e">Total Links </td><td class="v">0/unlimited </td></tr>
<tr><td class="e">API Version </td><td class="v">2004 </td></tr>
<tr><td class="e">Vendor Name </td><td class="v">OpenLDAP </td></tr>
<tr><td class="e">Vendor Version </td><td class="v">20023 </td></tr>
</table><br />
<h2><a name="module_mbstring">mbstring</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">Multibyte Support </td><td class="v">enabled </td></tr>
<tr><td class="e">Japanese support </td><td class="v">enabled </td></tr>
<tr><td class="e">Simplified chinese support </td><td class="v">enabled </td></tr>
<tr><td class="e">Traditional chinese support </td><td class="v">enabled </td></tr>
<tr><td class="e">Korean support </td><td class="v">enabled </td></tr>
<tr><td class="e">Russian support </td><td class="v">enabled </td></tr>
<tr><td class="e">Multibyte (japanese) regex support </td><td class="v">enabled </td></tr>
</table><br />
<table border="0" cellpadding="3" width="600">
<tr class="h"><th colspan="2">mbstring extension makes use of "streamable kanji code filter and converter", which is distributed under the GNU Lesser General Public License version 2.1.</th></tr>
</table><br />
<table border="0" cellpadding="3" width="600">
<tr class="h"><th>Directive</th><th>Local Value</th><th>Master Value</th></tr>
<tr><td class="e">mbstring.detect_order</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">mbstring.encoding_translation</td><td class="v">Off</td><td class="v">Off</td></tr>
<tr><td class="e">mbstring.func_overload</td><td class="v">0</td><td class="v">0</td></tr>
<tr><td class="e">mbstring.http_input</td><td class="v">pass</td><td class="v">pass</td></tr>
<tr><td class="e">mbstring.http_output</td><td class="v">pass</td><td class="v">pass</td></tr>
<tr><td class="e">mbstring.internal_encoding</td><td class="v">ISO-8859-1</td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">mbstring.language</td><td class="v">neutral</td><td class="v">neutral</td></tr>
<tr><td class="e">mbstring.substitute_character</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
</table><br />
<h2><a name="module_mysql">mysql</a></h2>
<table border="0" cellpadding="3" width="600">
<tr class="h"><th>MySQL Support</th><th>enabled</th></tr>
<tr><td class="e">Active Persistent Links </td><td class="v">0 </td></tr>
<tr><td class="e">Active Links </td><td class="v">0 </td></tr>
<tr><td class="e">Client API version </td><td class="v">3.23.58 </td></tr>
<tr><td class="e">MYSQL_MODULE_TYPE </td><td class="v">external </td></tr>
<tr><td class="e">MYSQL_SOCKET </td><td class="v">/var/lib/mysql/mysql.sock </td></tr>
<tr><td class="e">MYSQL_INCLUDE </td><td class="v">-I/usr/include/mysql </td></tr>
<tr><td class="e">MYSQL_LIBS </td><td class="v">-L/usr/lib/mysql -lmysqlclient  </td></tr>
</table><br />
<table border="0" cellpadding="3" width="600">
<tr class="h"><th>Directive</th><th>Local Value</th><th>Master Value</th></tr>
<tr><td class="e">mysql.allow_persistent</td><td class="v">On</td><td class="v">On</td></tr>
<tr><td class="e">mysql.connect_timeout</td><td class="v">60</td><td class="v">60</td></tr>
<tr><td class="e">mysql.default_host</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">mysql.default_password</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">mysql.default_port</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">mysql.default_socket</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">mysql.default_user</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">mysql.max_links</td><td class="v">Unlimited</td><td class="v">Unlimited</td></tr>
<tr><td class="e">mysql.max_persistent</td><td class="v">Unlimited</td><td class="v">Unlimited</td></tr>
<tr><td class="e">mysql.trace_mode</td><td class="v">Off</td><td class="v">Off</td></tr>
</table><br />
<h2><a name="module_ncurses">ncurses</a></h2>
<table border="0" cellpadding="3" width="600">
<tr class="h"><th>ncurses support</th><th>enabled</th></tr>
<tr><td class="e">ncurses library version </td><td class="v">5.2 </td></tr>
<tr><td class="e">color support </td><td class="v">yes </td></tr>
</table><br />
<h2><a name="module_openssl">openssl</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">OpenSSL support </td><td class="v">enabled </td></tr>
<tr><td class="e">OpenSSL Version </td><td class="v">OpenSSL 0.9.6b [engine] 9 Jul 2001 </td></tr>
</table><br />
<h2><a name="module_overload">overload</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">User-Space Object Overloading Support </td><td class="v">enabled </td></tr>
</table><br />
<h2><a name="module_pcre">pcre</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">PCRE (Perl Compatible Regular Expressions) Support </td><td class="v">enabled </td></tr>
<tr><td class="e">PCRE Library Version </td><td class="v">4.5 01-December-2003 </td></tr>
</table><br />
<h2><a name="module_posix">posix</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">Revision </td><td class="v">$Revision: 1.51.2.2 $ </td></tr>
</table><br />
<h2><a name="module_pspell">pspell</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">PSpell Support </td><td class="v">enabled </td></tr>
</table><br />
<h2><a name="module_session">session</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">Session Support </td><td class="v">enabled </td></tr>
<tr><td class="e">Registered save handlers </td><td class="v">files user mm  </td></tr>
</table><br />
<table border="0" cellpadding="3" width="600">
<tr class="h"><th>Directive</th><th>Local Value</th><th>Master Value</th></tr>
<tr><td class="e">session.auto_start</td><td class="v">Off</td><td class="v">Off</td></tr>
<tr><td class="e">session.bug_compat_42</td><td class="v">On</td><td class="v">On</td></tr>
<tr><td class="e">session.bug_compat_warn</td><td class="v">On</td><td class="v">On</td></tr>
<tr><td class="e">session.cache_expire</td><td class="v">180</td><td class="v">180</td></tr>
<tr><td class="e">session.cache_limiter</td><td class="v">nocache</td><td class="v">nocache</td></tr>
<tr><td class="e">session.cookie_domain</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">session.cookie_lifetime</td><td class="v">0</td><td class="v">0</td></tr>
<tr><td class="e">session.cookie_path</td><td class="v">/</td><td class="v">/</td></tr>
<tr><td class="e">session.cookie_secure</td><td class="v">Off</td><td class="v">Off</td></tr>
<tr><td class="e">session.entropy_file</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">session.entropy_length</td><td class="v">0</td><td class="v">0</td></tr>
<tr><td class="e">session.gc_divisor</td><td class="v">100</td><td class="v">100</td></tr>
<tr><td class="e">session.gc_maxlifetime</td><td class="v">1440</td><td class="v">1440</td></tr>
<tr><td class="e">session.gc_probability</td><td class="v">1</td><td class="v">1</td></tr>
<tr><td class="e">session.name</td><td class="v">PHPSESSID</td><td class="v">PHPSESSID</td></tr>
<tr><td class="e">session.referer_check</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">session.save_handler</td><td class="v">files</td><td class="v">files</td></tr>
<tr><td class="e">session.save_path</td><td class="v">/tmp</td><td class="v">/tmp</td></tr>
<tr><td class="e">session.serialize_handler</td><td class="v">php</td><td class="v">php</td></tr>
<tr><td class="e">session.use_cookies</td><td class="v">On</td><td class="v">On</td></tr>
<tr><td class="e">session.use_only_cookies</td><td class="v">Off</td><td class="v">Off</td></tr>
<tr><td class="e">session.use_trans_sid</td><td class="v">On</td><td class="v">On</td></tr>
</table><br />
<h2><a name="module_shmop">shmop</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">shmop support </td><td class="v">enabled </td></tr>
</table><br />
<h2><a name="module_sockets">sockets</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">Sockets Support </td><td class="v">enabled </td></tr>
</table><br />
<h2><a name="module_standard">standard</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">Regex Library </td><td class="v">System library enabled </td></tr>
<tr><td class="e">Dynamic Library Support </td><td class="v">enabled </td></tr>
<tr><td class="e">Path to sendmail </td><td class="v">###########</td></tr>
</table><br />
<table border="0" cellpadding="3" width="600">
<tr class="h"><th>Directive</th><th>Local Value</th><th>Master Value</th></tr>
<tr><td class="e">assert.active</td><td class="v">1</td><td class="v">1</td></tr>
<tr><td class="e">assert.bail</td><td class="v">0</td><td class="v">0</td></tr>
<tr><td class="e">assert.callback</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">assert.quiet_eval</td><td class="v">0</td><td class="v">0</td></tr>
<tr><td class="e">assert.warning</td><td class="v">1</td><td class="v">1</td></tr>
<tr><td class="e">auto_detect_line_endings</td><td class="v">0</td><td class="v">0</td></tr>
<tr><td class="e">default_socket_timeout</td><td class="v">60</td><td class="v">60</td></tr>
<tr><td class="e">safe_mode_allowed_env_vars</td><td class="v">PHP_</td><td class="v">PHP_</td></tr>
<tr><td class="e">safe_mode_protected_env_vars</td><td class="v">LD_LIBRARY_PATH</td><td class="v">LD_LIBRARY_PATH</td></tr>
<tr><td class="e">url_rewriter.tags</td><td class="v">a=href,area=href,frame=src,input=src,form=fakeentry</td><td class="v">a=href,area=href,frame=src,input=src,form=fakeentry</td></tr>
<tr><td class="e">user_agent</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
</table><br />
<h2><a name="module_tokenizer">tokenizer</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">Tokenizer Support </td><td class="v">enabled </td></tr>
</table><br />
<h2><a name="module_wddx">wddx</a></h2>
<table border="0" cellpadding="3" width="600">
<tr class="h"><th>WDDX Support</th><th>enabled</th></tr>
<tr><td class="e">WDDX Session Serializer </td><td class="v">enabled </td></tr>
</table><br />
<h2><a name="module_xml">xml</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">XML Support </td><td class="v">active </td></tr>
<tr><td class="e">XML Namespace Support </td><td class="v">active </td></tr>
<tr><td class="e">EXPAT Version </td><td class="v">expat_1.95.2 </td></tr>
</table><br />
<h2><a name="module_yp">yp</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">YP Support </td><td class="v">enabled </td></tr>
</table><br />
<h2><a name="module_zlib">zlib</a></h2>
<table border="0" cellpadding="3" width="600">
<tr><td class="e">ZLib Support </td><td class="v">enabled </td></tr>
<tr><td class="e">Compiled Version </td><td class="v">1.1.3 </td></tr>
<tr><td class="e">Linked Version </td><td class="v">1.1.4 </td></tr>
</table><br />
<table border="0" cellpadding="3" width="600">
<tr class="h"><th>Directive</th><th>Local Value</th><th>Master Value</th></tr>
<tr><td class="e">zlib.output_compression</td><td class="v">Off</td><td class="v">Off</td></tr>
<tr><td class="e">zlib.output_compression_level</td><td class="v">-1</td><td class="v">-1</td></tr>
<tr><td class="e">zlib.output_handler</td><td class="v"><i>no value</i></td><td class="v"><i>no value</i></td></tr>
</table><br />
<h2>Additional Modules</h2>
<table border="0" cellpadding="3" width="600">
<tr class="h"><th>Module Name</th></tr>
<tr><td>sysvsem</td></tr>
<tr><td>sysvshm</td></tr>
</table><br />
<h2>Environment</h2>
<table border="0" cellpadding="3" width="600">
<tr class="h"><th>Variable</th><th>Value</th></tr>
<tr><td class="e">PATH </td><td class="v">/usr/local/bin:/usr/bin:/bin </td></tr>
<tr><td class="e">DOCUMENT_ROOT </td><td class="v">############</td></tr>
<tr><td class="e">HTTP_ACCEPT </td><td class="v">*/* </td></tr>
<tr><td class="e">HTTP_ACCEPT_ENCODING </td><td class="v">gzip, deflate </td></tr>
<tr><td class="e">HTTP_ACCEPT_LANGUAGE </td><td class="v">en-gb </td></tr>
<tr><td class="e">HTTP_CONNECTION </td><td class="v">Keep-Alive </td></tr>
<tr><td class="e">HTTP_HOST </td><td class="v">########## </td></tr>
<tr><td class="e">HTTP_UA_CPU </td><td class="v">x86 </td></tr>
<tr><td class="e">HTTP_USER_AGENT </td><td class="v">Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727) </td></tr>
<tr><td class="e">REMOTE_ADDR </td><td class="v">#############</td></tr>
<tr><td class="e">REMOTE_PORT </td><td class="v">#########</td></tr>
<tr><td class="e">SCRIPT_FILENAME </td><td class="v">##############</td></tr>
<tr><td class="e">SERVER_ADDR </td><td class="v">#############</td></tr>
<tr><td class="e">SERVER_ADMIN </td><td class="v">##########</td></tr>
<tr><td class="e">SERVER_NAME </td><td class="v">########### </td></tr>
<tr><td class="e">SERVER_PORT </td><td class="v">######### </td></tr>
<tr><td class="e">SERVER_SOFTWARE </td><td class="v">Apache/1.3.29 (Unix)  (Red-Hat/Linux) Chili!Soft-ASP/3.6.2 mod_ssl/2.8.14 OpenSSL/0.9.6b PHP/4.3.10 FrontPage/5.0.2.2510 </td></tr>
<tr><td class="e">UNIQUE_ID </td><td class="v">####################</td></tr>
<tr><td class="e">GATEWAY_INTERFACE </td><td class="v">CGI/1.1 </td></tr>
<tr><td class="e">SERVER_PROTOCOL </td><td class="v">HTTP/1.1 </td></tr>
<tr><td class="e">REQUEST_METHOD </td><td class="v">GET </td></tr>
<tr><td class="e">QUERY_STRING </td><td class="v"><i>no value</i> </td></tr>
<tr><td class="e">REQUEST_URI </td><td class="v">/php.php </td></tr>
<tr><td class="e">SCRIPT_NAME </td><td class="v">/php.php </td></tr>
</table><br />
<h2>PHP Variables</h2>
<table border="0" cellpadding="3" width="600">
<tr class="h"><th>Variable</th><th>Value</th></tr>
<tr><td class="e">PHP_SELF </td><td class="v">/php.php </td></tr>
<tr><td class="e">_SERVER["PATH"]</td><td class="v">/usr/local/bin:/usr/bin:/bin</td></tr>
<tr><td class="e">_SERVER["DOCUMENT_ROOT"]</td><td class="v">########</td></tr>
<tr><td class="e">_SERVER["HTTP_ACCEPT"]</td><td class="v">*/*</td></tr>
<tr><td class="e">_SERVER["HTTP_ACCEPT_ENCODING"]</td><td class="v">gzip, deflate</td></tr>
<tr><td class="e">_SERVER["HTTP_ACCEPT_LANGUAGE"]</td><td class="v">en-gb</td></tr>
<tr><td class="e">_SERVER["HTTP_CONNECTION"]</td><td class="v">Keep-Alive</td></tr>
<tr><td class="e">_SERVER["HTTP_HOST"]</td><td class="v">##########</td></tr>
<tr><td class="e">_SERVER["HTTP_UA_CPU"]</td><td class="v">x86</td></tr>
<tr><td class="e">_SERVER["HTTP_USER_AGENT"]</td><td class="v">Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727)</td></tr>
<tr><td class="e">_SERVER["REMOTE_ADDR"]</td><td class="v">############</td></tr>
<tr><td class="e">_SERVER["REMOTE_PORT"]</td><td class="v">#####</td></tr>
<tr><td class="e">_SERVER["SCRIPT_FILENAME"]</td><td class="v">##############</td></tr>
<tr><td class="e">_SERVER["SERVER_ADDR"]</td><td class="v">########</td></tr>
<tr><td class="e">_SERVER["SERVER_ADMIN"]</td><td class="v">#########</td></tr>
<tr><td class="e">_SERVER["SERVER_NAME"]</td><td class="v">###########</td></tr>
<tr><td class="e">_SERVER["SERVER_PORT"]</td><td class="v">########</td></tr>
<tr><td class="e">_SERVER["SERVER_SOFTWARE"]</td><td class="v">Apache/1.3.29 (Unix)  (Red-Hat/Linux) Chili!Soft-ASP/3.6.2 mod_ssl/2.8.14 OpenSSL/0.9.6b PHP/4.3.10 FrontPage/5.0.2.2510</td></tr>
<tr><td class="e">_SERVER["UNIQUE_ID"]</td><td class="v">##############</td></tr>
<tr><td class="e">_SERVER["GATEWAY_INTERFACE"]</td><td class="v">CGI/1.1</td></tr>
<tr><td class="e">_SERVER["SERVER_PROTOCOL"]</td><td class="v">HTTP/1.1</td></tr>
<tr><td class="e">_SERVER["REQUEST_METHOD"]</td><td class="v">GET</td></tr>
<tr><td class="e">_SERVER["QUERY_STRING"]</td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">_SERVER["REQUEST_URI"]</td><td class="v">/php.php</td></tr>
<tr><td class="e">_SERVER["SCRIPT_NAME"]</td><td class="v">/php.php</td></tr>
<tr><td class="e">_SERVER["PHP_SELF"]</td><td class="v">/php.php</td></tr>
<tr><td class="e">_SERVER["argv"]</td><td class="v"><pre>Array
(
)
</pre></td></tr>
<tr><td class="e">_SERVER["argc"]</td><td class="v">0</td></tr>
<tr><td class="e">_ENV["PATH"]</td><td class="v">/usr/local/bin:/usr/bin:/bin</td></tr>
<tr><td class="e">_ENV["DOCUMENT_ROOT"]</td><td class="v">############</td></tr>
<tr><td class="e">_ENV["HTTP_ACCEPT"]</td><td class="v">*/*</td></tr>
<tr><td class="e">_ENV["HTTP_ACCEPT_ENCODING"]</td><td class="v">gzip, deflate</td></tr>
<tr><td class="e">_ENV["HTTP_ACCEPT_LANGUAGE"]</td><td class="v">en-gb</td></tr>
<tr><td class="e">_ENV["HTTP_CONNECTION"]</td><td class="v">Keep-Alive</td></tr>
<tr><td class="e">_ENV["HTTP_HOST"]</td><td class="v">################</td></tr>
<tr><td class="e">_ENV["HTTP_UA_CPU"]</td><td class="v">x86</td></tr>
<tr><td class="e">_ENV["HTTP_USER_AGENT"]</td><td class="v">Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727)</td></tr>
<tr><td class="e">_ENV["REMOTE_ADDR"]</td><td class="v">##############</td></tr>
<tr><td class="e">_ENV["REMOTE_PORT"]</td><td class="v">#########</td></tr>
<tr><td class="e">_ENV["SCRIPT_FILENAME"]</td><td class="v">###############</td></tr>
<tr><td class="e">_ENV["SERVER_ADDR"]</td><td class="v">###########</td></tr>
<tr><td class="e">_ENV["SERVER_ADMIN"]</td><td class="v">###############</td></tr>
<tr><td class="e">_ENV["SERVER_NAME"]</td><td class="v">##############</td></tr>
<tr><td class="e">_ENV["SERVER_PORT"]</td><td class="v">###########</td></tr>
<tr><td class="e">_ENV["SERVER_SOFTWARE"]</td><td class="v">Apache/1.3.29 (Unix)  (Red-Hat/Linux) Chili!Soft-ASP/3.6.2 mod_ssl/2.8.14 OpenSSL/0.9.6b PHP/4.3.10 FrontPage/5.0.2.2510</td></tr>
<tr><td class="e">_ENV["UNIQUE_ID"]</td><td class="v">################</td></tr>
<tr><td class="e">_ENV["GATEWAY_INTERFACE"]</td><td class="v">CGI/1.1</td></tr>
<tr><td class="e">_ENV["SERVER_PROTOCOL"]</td><td class="v">HTTP/1.1</td></tr>
<tr><td class="e">_ENV["REQUEST_METHOD"]</td><td class="v">GET</td></tr>
<tr><td class="e">_ENV["QUERY_STRING"]</td><td class="v"><i>no value</i></td></tr>
<tr><td class="e">_ENV["REQUEST_URI"]</td><td class="v">/php.php</td></tr>
<tr><td class="e">_ENV["SCRIPT_NAME"]</td><td class="v">/php.php</td></tr>
</table><br />
<h2>PHP License</h2>
<table border="0" cellpadding="3" width="600">
<tr class="v"><td>
<p>
This program is free software; you can redistribute it and/or modify it under the terms of the PHP License as published by the PHP Group and included in the distribution in the file:  LICENSE
</p>
<p>This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
</p>
<p>If you did not receive a copy of the PHP license, or have any questions about PHP licensing, please contact license@php.net.
</p>
</td></tr>
</table><br />
</div></body></html>

jpadie · Jan 18, 2007

hi

can you give me an idea of how many files you process each job (assuming no timeouts) and what the average size of each file is?

i'm assuming you are writing each field to a blob or longtext field in mysql or similar.

what do you do with the datafiles once they have been 'uploaded'? are they then deleted or are is the database write cumulative?

lastly ... what is the timeout error message you get (assuming you log the output of the operation)? are you running the cron job remotely by calling this script over an http connection or is the cron job established from the local *nix server on which the script resides? (i'm trying to rule out a browser timeout rather than a script execution timeout).

Dweezel · Jan 18, 2007

I think I may have solved the main problem, and it's one of your questions in your last repy that did it. You said:

I'm assuming you are writing each field to a blob or longtext field in mysql or similar.

I'm guessing this was because the size of the limit number in fgetss was 1000. Now 1000 was just a number I pulled out of the air when putting this script together. I assumed that the figure of 1000 just denoted a maximum number of characters or numbers.

For instance, if there were only 10 digits in one of the lines of the file then I assumed fgetss would identify the 10 digits followed by a new line character and the 1000 figure would have no bearing.

I've just increased the number from 1000 to 10000 and the script only parses about a third of the files, and is even slower. I then decreased the number to 120 and all of the files in the directory have been parsed successfully.

Now the only lines I'd actually need the maximum 120 character limit for are the first 7 in each file. For the rest a limit of 12 is sufficient. So using the while loop I set up a conditional statement:

Code:

$i = 1;
 while(!feof($fp))
  { 
    if($i < 8)
    $data = fgetss($fp,120);
    else
    $data = fgetss($fp,12);


   $i++
  }
[code]

I expected this to make the execution time even faster. Instead the script failed at about the same point as when I used to have the limit set to 1000. Very odd.


To answer your other questions: There are about 60 of these files at the moment. There are 7 lines at the top of each file that relate to the identity of the file, time/date of the first data sample recorded, and also a 'sample number' which records the number of samples. New data is appended to each file once a day, adding 96 lines of data to each file (One data sample every 15 minutes for 24 hours). At the moment the files range from about 1000 lines to about 30000.

 The script is set up to ignore each line of data returned from fgetss except the newly uploaded section which is entered into a MySQL database. The data is stored in a 'float(5,6)' field.

The cron job which runs the script once every 12 hours is run on the remote server where the script resides. I can also run the script manually in a browser. 

I'm afraid I don't now how to log the output of the operation. The only feedback I have at the moment is an echo statement that prints the name of each file as it is parsed which I can see when I run the script in a browser. Could you point me in the right direction as to what I need to do to log the output of the operation. I've just had a google but can't find anything relevant.


What do you think of this system? And why do you think my conditional statement above had the opposite affect to that which I expected?

Thanks for all your help jpadie.

jpadie · Jan 18, 2007

the conditional does what i think you're intending it to do. i can't find the fault at the moment so let's do some debugging.

for the moment, please revert back to a non-conditionalised fgetss but OMIT the length parameter. from php 4.3 onwards this should cause it to read the line whatever it's length.

i don't know what's in each file so you probably will pickup cruft but we can handle that later.

you talk about the script discarding data. can you share that code as it may be introducing overhead?

for the logging, at the beginning of the script add the following:

Code:

$flog = fopen ("./testlog.txt", "wb");
function addToLog($data){
 global $flog;
 fwrite ($flog, $data.'\r\n');
}
function closeLog(){
 global $flog;
 fclose($flog);
}

add this to the end of your code

Code:

closeLog();

and for the logging itself i would add the following within your code

Code:

[red]
set_time_limit(0);
error_reporting (E_ALL);
$start=date();[/red]
while($file = readdir($opdir))
{

[red]
   addToLog(date("H:i:s \t opening file $file \r\n");
 [/red]

    //Open a file
    $fp = fopen("../data/$file","r");
   [red]addToLog(date("H:i:s \t opened file $file \r\n");
[/red]


   
    //Loop through the file one line at a time
    while(!feof($fp))
      {


       $data = fgetss($fp);
       [red]addToLog(date("H:i:s") . " \t read ".strlen($data)." characters \r\n");
      [/red] 
       //database input functions e.t.c go here
       
      [red]addToLog(date("H:i:s") ." \t database write successful \r\n");
      addToLog(date("H:i:s") . " \t elapsed time is " .date() - $start ." \r\n");[/red]
      }


}

for the purposes of this exercise i'd also recommend pointing the app at a different table

Code:

create table `newtablename` like `oldtablename`

will produce a table with the same schema BUT i'd also recomment updating a field that is setup as a LONGTEXT and not a float. this will help us analyse whether the issue is a coding problem or a true timeout. From what you have said, 30000 lines across 60 files should NOT cause a timeout. why not include the db write language so we can see whether any optimisation could be done there too?

Dweezel · Jan 18, 2007

I've got to leave my computer until tomorrow, but I'll do all of that in the morning and post up the results.

Thanks a lot jpadie.

Dweezel · Jan 25, 2007

Just to update on this jpadie. I've been away from the computer for almost a week, but I've used your logging system described above to help me sort this script out.

Removing the second parameter from the fgetss function just gave me an error message saying 'incorrect parameter count'. From experimenting with the second parameter and inspecting the log file, I found that although increasing it's size slows the scipt down, it doesn't change the length of the string returned. For instance, if I set the second parameter to 100, then 1000 and then 10000, all of these would still produce "read 12 characters" in the logfile. Set at 10000 the script only gets through about 25 files before it cuts out.

The date() function didn't work on my server. I got the same error as with fgetss: 'incorrect parameter count'. I used print_r(gettimeofday()) instead, which had the added advantage of giving me the time in microseconds. I used this to optimise my code, and it can now process about 3 times as much data before it cuts out.

The core problem still remains though. The script is still stopping abruptly without any error messages. Adding 'error_reporting (E_ALL);' made no difference to this. It can't be a php timeout as with my previous version of the script it was cutting off at 23 - 24 seconds and now it's 35 - 36 seconds.

It's not a database issue either. The database communication in this script is split into two parts. Part one is getting information such as 'site name','unit number' e.t.c from the first 5 lines of each file, and comparing that with information on the database in order to match the file to the correct database table.

I timed this using gettimeofday() and this whole process takes between 1 and 2 thousandths of a second.

Part two is the inputing of the new data from the bottom of the file. If there is no new data to input from any of the files, the script still stops abruptly without error messages. Infact, I can remove all of the code from this section of the script and just have the the fgetss iterating through the lines of data and doing nothing. The script still terminates early.

Where do you think I should go from here? Do you think contacting my hosting provider would be the best move? If I could only get some kind of error message I'd have something to work with.

Thanks again for your help.

jpadie · Jan 25, 2007

my guess is that there is something in one of the files that kills it. although i am guessing too that "stop abruptly" means mid-process rather than after one particular file.

fgetss - sorry i forgot you are using an old version of php hence the need for the length parameter
date() - that's very wrong. the second param has always been optional. can you double check the error as this may point to a faulty installation of php

check the log we created. does the script always fail whilst reading the same file?
also try a straightforward fread in place of fgetss, then log te result
then strip_tags the output and log the result

the log should help identify where the point of failure is.

jpadie · Jan 25, 2007

Dweezel

i had meant to follow up this post earlier (and did) but for some reason my post didn't stick.

i wanted to suggest adding some gateways in to make sure that the script is not failing when reading items in the directory that are not true files. i've partially marked in the code below the bits that i have added but i have also changed the date calls to a microtime call. the gettimeofday function returns an array which is not much use for rapid logging. the reason why they were failing above is that the brackets were not closed around some of them.

Code:

<?
set_time_limit(0);
error_reporting (E_ALL);
//[red]
$start=microtime();
$opdir = realpath('../data/');
//[/red]
while($file = readdir($opdir))
{
//[red]
//insert some condition gateways
	if ($file = '.' || $file == '..'){
		break(1); //avoid the directory markers
	}
	//belt and braces
	if (!is_readable($opdir.$file)){
		addToLog(microtime() . "\t file $file is not readable");
		break(1); //go to next instance of the while loop
	}
	if (!is_file($opdir.$file)){
		addToLog(microtime() . "\t file $file is not a file");
		break(1);
	}
	
	//end of belt and braces [/red]
	addToLog(microtime()."\t opening file $file \r\n");
 

    //Open a file
    $fp = fopen([red]$opdir.$file,"rb"[/red]);
   addToLog(microtime()."\t opened file $file \r\n");



   
    //Loop through the file one line at a time
    while(!feof($fp))
      {
       //[red]
       $data = fread($fp, 2048);
       addToLog(microtime() . " \t read ".strlen($data)." characters \r\n");
       $strippedData=strip_tags($data);
       addToLog(microtime() . " \t stripped tags.  New length is  ".strlen($strippedData)." characters \r\n");
       //[/red]
       //database input functions e.t.c go here
       
      addToLog(microtime() ." \t database write successful \r\n");
      addToLog(microtime() . " \t elapsed time is " .(microtime() - $start) ." \r\n");
      }


}
?>

Tek-Tips is the largest IT community on the Internet today!

Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!

Changing Host - PHP MySQL question 1

Dweezel

Technical User

jpadie

Technical User

Dweezel

Technical User

Dweezel

Technical User

jpadie

Technical User

Dweezel

Technical User

jpadie

Technical User

Dweezel

Technical User

jpadie

Technical User

Dweezel

Technical User

Dweezel

Technical User

jpadie

Technical User

jpadie

Technical User

Similar threads

Part and Inventory Search

Sponsor