Program Stops after 10 Minutes 2

[chvol]

I am trying to run my PHP program that uses class Snoopy to access the HTML of web pages. It goes through about 300 pages for 10 minutes, then stops. A program that uses only CPU, periodically writing the current time, runs for hours with no problem. If I can’t fix it, an alternate solution would be for another program to start it up, and restart it after it stops, if possible.

Charlie chvol@aol.com

 

[sleipnir214]

How are you running the script? Through a web browser? From a command shell?



Want the best answers? Ask the best questions!

TANSTAAFL!!

 

[chvol]

Internet Explorer

 

[sleipnir214]

If you're running PHP through a web server, there is a limit to how long PHP will run. This limit is set by the runtime configuration directive max_execution_time



Want the best answers? Ask the best questions!

TANSTAAFL!!

 

[DRJ478]

You can use a combination of PHP with client side code to keep the script running. Set a particular number of pages to be processed, a number lower than when you experience the timeout. As long as there are pages to be processed print a client side script, e.g. JavaScript, to reload the page. This will reload until all pages are done - then don't output the reloader code.
As an interesting note on the side:
The maximum execution time does not include time PHP is waiting for the system to execute commands. That means you could theoretically put the script into sleep mode for longer than the maximum processing time. Only CPU time counts. If there's a timeout with that it is either the client or the server who abandon the connection.
When the client drops the connection the process might still run and turn into a runaway process (also called "spinner").

 

[chvol]

I don’t know what hardly any of this means.

1. You can use a combination of PHP with client side code to keep the script running.

This doesn’t seem to be saying for me to do anything, only promising that the problem can be fixed.

2. Set a particular number of pages to be processed, a number lower than when you experience the timeout.

You mean how many URLs I pass to snoopy before it stops? That ranges from 136 to 346. 1 is less than these numbers.

3. As long as there are pages to be processed print a client side script, e.g. JavaScript, to reload the page.

I don’t know JavaScript (I can’t do it in PHP?). How do I “print a client side script”?

4. This will reload until all pages are done - then don't output the reloader code.

The whole process is passing the same URL to snoopy with different ?variable=value at the end, and saving part of the HTML that is returned. This # 4 seems to be referrring to # 3, but since I don’t know what # 3 means then I can’t do # 4.

The rest doesn’t seem to be something that I am supposed to do.

Charlie chvol@aol.com

5. As an interesting note on the side:
The maximum execution time does not include time PHP is waiting for the system to execute commands. That means you could theoretically put the script into sleep mode for longer than the maximum processing time. Only CPU time counts. If there's a timeout with that it is either the client or the server who abandon the connection.
When the client drops the connection the process might still run and turn into a runaway process (also called "spinner").

 

[chvol]

1. max_executiontime=0
2. no set_time_limit
3. no error messages
4. the time it takes is random (even when rerun)
5. Remember: the program with only CPU use runs indefinitely - for hours.
6. It is going through thousands of URLs

 

[DRJ478]

<personal note>
Maybe you should tone your writing down a bit if you want people to help you. I find your comments offensive. IMO this is not an appropriate way to respond to freely offered help.
</personal note>

Suggestions:
You are running the script from a browser. The browser receives output from your PHP script - or waits for it. Maybe your script doesn't produce output. We don't know since there's no code posted.
The first suggestion was to send output to the browser that triggers the reloading of the page. This makes sense when you invoke (whatever snoopy is) the class a finite number of times, e.g. 100 URLs at a time.
Once the URLs are processed keep track of the number of processed URLs via a session variable. Then output some JavaScript to the browser that reloads the page.
Here's an example How do I "print" a client side script?{:

print "<script>document.location.reload();</script>";

Alternatively you can do it in PHP if you make sure there is no output sent before redirecting. You could use the header() function to redirect the script to itself.

header("Location: ".$_SERVER['PHP_SELF']);

 

[chvol]

I don't know what is wrong with saying that you don't know what something means, so I will ignore the previous post and try one last time on this forum. Can anyone tell me explicitly how to solve my problem of my PHP program stopping after a while? All I know how to do is to start up Apache web server, use a file editor to create PHP programs, and run them using Internet Explorer.

If the procedure needed is complex or goes outside of what I can do, perhaps it would be better if you emailed me directly so that I can quickly ask questions when things don't work.

Charlie chvol@aol.com

 

[chvol]

Get a book and spend some of your own brain power to solve your problem."

That's a hell of a thing to say on a forum where people post problems. Gad.

"All the information needed has already been provided."

If that were true, I wouldn't have asked the follow-up question.

"Carefully consider the title of this web site:
Technical work forums for computer professionals"

I guess you didn't notice that I was a programmer.

Do your own thing and let others do theirs. Live and let live.

 

[Wishdiak]

chvol,

It looks like a likely solution was already posted in thread434-850844. Your question appears (both in that thread and in this one) to be specific to issues that you are having with Snoopy.

AFAIK, Tek-tips does not have a forum specifically for Snoopy. Sourceforge does.

I would consider that a useful resource, and attempt to gain some knowledge from folks with first-hand knowledge of Snoopy. You may even catch the attention of one of the lead developers. Otherwise, you may need to take a different approach to your program.

The kindness of strangers is not limitless, and probably never will be.



Wishdiak

http://www.wishdiak.com

 

[chvol]

I am trying to find out how to "send output to the browser that triggers the reloading of the page". That doesn't sound like it is particular to snoopy. I was also told to "run the script from the command line", which doesn't sound like it is particular to snoopy.

Those who would like to help are welcome to. Those who don't are welcome to not. I don't hold it against them if they choose not to. But they shouldn't try to interfere with those who do like to help others.

 

[ascikey]

There are many dying arts that need rekindling in your case its communication. I really feel that your writing style toward a long standing, very very helpful and sincere member of this forum is not called far. I do know this, had you asked your questions in a polite manner DRJ487 and most others for that matter would have helped you achieve your goal. You obviously feel that what you have said/the way is acceptable so I stress your writing style.

 

[Moonspell]

Hmm, I see you have decided to leave out the reasons for running this script in this thread. tut tut tut. I think you will find no more answers in this thread than the other one! might i suggest you try another project that does not involve ripping stuff off google! there are plenty of helpful people here but your attitude stinks. also if you can't do elementary javascript then you aint no programmer!

 

[chvol]

Hmm, I see you have decided to leave out the reasons for running this script in this thread. tut tut tut. I think you will find no more answers in this thread than the other one! might i suggest you try another project that does not involve ripping stuff off google! there are plenty of helpful people here but your attitude stinks. also if you can't do elementary javascript then you aint no programmer!"

Think what you wish. It is not up to me to burst anyone's bubble.

My program works fine with a number of other search engines which display no copyrights nor Terms of Service. Apparently it was Google's doing. I am fine now.

People who say to go get a book and ridicule people based on their technical skills already have a home. Those lines are straight out of "Nick the Computer Guy" on Saturday Night Line.

To each his own.

"The people are not free until the information is free."

C.

 

[DRJ478]

People who say to go get a book and ridicule people based on their technical skills already have a home.

Thanks a lot. I'm sorry you feel ridiculed, that was not my intention. Again, I should also say that had you read the posts more carefully you had all the answers right there.
There is an explicit example of the very clientside code needed to reload the page. But you keep asking the question. There is also an explicit example of how to achieve the same with PHP code.

You are right, people post their problems in this forum. But usually they are at least a bit appreciative when they are given help and they will communicate in the professional way one would expect from a professional progammer. There's nothing wrong asking appropriate follow up questions rather than giving sarcastic answers.

Live and let live - absolutely. Consider your own answers to my reply and tell me that that they were written in that spirit. Feel free again to flag my post as inappropriate and have it removed. So much for freedom - some people just can't take a pointer to the truth. I wish you luck, you'll need it with such an attitude.

 

[jimoblak]

"The people are not free until the information is free."

If chvol truly believed in the 'free' babble he is spouting, he would have freely posted his code for review so that a solution to his problem could be found.

Apparently it was Google's doing.

chvol has learned that Google purposefully blocks his leeching efforts. It was apparently not his coding that was in in error. It was his misunderstanding of copyright law and how he did not realize search engines protect their data from leeches like him.

I would not take any further issue with chvol. He has been given all the information he needs now.

- - picklefish - -
Why is everyone in this forum responding to me as picklefish?

 

[Moonspell]

Think what you wish. It is not up to me to burst anyone's bubble.

do you really think you can burst the bubble of someone who has been a programmer for the last 20+ years? judging from the use of language in your posts I have been programming for longer than you have been alive! I not only program in PHP but also C++, Javascript, VBScript, Visual Basic, Pascal, Prolog, Cobol etc. you sound like a bit of a script kiddie with very little technical knowledge to back up your big mouth.

I am fine now.

I doubt that very much!

 

[chvol]

1. "There is an explicit example of the very clientside code needed to reload the page. But you keep asking the question. There is also an explicit example of how to achieve the same with PHP code."

No. I didn't repeat the question - I asked how to do what the answer called for. Do you realize that there are people who don't have as much experience in certain areas of the use of PHP as others?

2. "If chvol truly believed in the 'free' babble he is spouting, he would have freely posted his code for review so that a solution to his problem could be found"

As if I'm hiding it? It's many pages of code (which I did post elsewhere, actually) and the problem is (was) just calling Snoopy multiple times. Do you really want to see the code that parses the HTML and sorts a list of English phrases based on the counter that it retrieves? That has nothing to do with the problem.

3. "do you really think you can burst the bubble of someone who has been a programmer for the last 20+ years? judging from the use of language in your posts I have been programming for longer than you have been alive! I not only program in PHP but also C++, Javascript, VBScript, Visual Basic, Pascal, Prolog, Cobol etc. you sound like a bit of a script kiddie with very little technical knowledge to back up your big mouth"

I have been programming database applications for 31 years and have developed approximately 20 major applications. I also wrote a 184 page book on advanced programming techniques which was very well received. I was asked by a 10,000 member national organization of programmers and users to develop an advanced programming 1/2 day seminar for their national convention (which I accepted.)

I taught myself PHP a year ago and know only a limited amount about the finer points of internet programming. I thought that I could get some help here.

4. "I doubt that very much"

I have found 3 search engines that have no copyright notice or Terms of Service and which don't have that problem. What could be better?

All 4 of these quotes illustrate the basic problems:

A. Living in a world of programmers at the same level of experience and assuming everyone knows the jargon and that the only questions are of the form "Which command/facility do I use?" rather than "How do I do that?"

B. Feeling a special status in being able to talk that lingo, putting you above others.

C. Being presumptuous.

You should audition for "Nick the Computer Guy" on Saturday Night Live!

 

[Wishdiak]

I have been programming database applications for 31 years and have developed approximately 20 major applications. I also wrote a 184 page book on advanced programming techniques which was very well received. I was asked by a 10,000 member national organization of programmers and users to develop an advanced programming 1/2 day seminar for their national convention (which I accepted.)

The way I look at it, there are three ways to go about getting help with writing a program.

1. You have someone else write it for you. Unless you are on very good terms with someone possessing the skills to do so, you are going to have to pay money for this.

2. You have someone teach you how to write it. Again, unless you are on very good terms with someone possessing the skills to do so, you are going to have to pay money for this.

3. You write 95% or better of the program yourself, using your own skills, and ask for help. In this instance, money may or may not change hands, depending on the amount of help you need.

While most people on this forum fall into the third category, and are looking for help with the syntax of a specific function or command, chvol seems to fall into one of the first two categories.

The suggestion that he obtain a book on the subject of PHP programming to solve a PHP programming problem seems to have offended him on a deep level.

This, of course, begs the question: How can he claim to "have been programming database applications for 31 years" while rejecting the very sound advice that to learn specifics about any given subject, one must open a book?

To use an analogy, if you were rebuilding an engine, and you came across some part that you did not recognise, you might call a repair shop. The mechanic that you speak with may or may not be able to help you determine what the part is. If he does, it will be in the "lingo" of engines and mechanics, and if you don't understand the "lingo", he will certainly encourage you to bring the engine in to the shop, and then charge you for his time. He will not participate in an argument about how his advice, and his time, should be provided to you free-of-charge. He will not care about your years of database programming, or your planning a half-day seminar. He will be a professional that you hire to solve a specific problem.

You can either do it yourself, or if you can't you can pay someone with the right technical skills to do it for you. It works for both cars and programming.



Wishdiak

http://www.wishdiak.com