This is a really basic question so for those of you with patience I appreciate any input.<br><br>I have written a kind of primitive search engine for urls with three letter domain names. <br>It runs a loop which creates all possible alphabetic combinations of three letters and calls a URL with that name then parses out the meta-keywords from the HTML (if they exist). It then puts the keywords in a database along with the URL.<br>This database I can then query from an asp page. And I get all the entries containing the keywords I enter. <br><br>It works fine and is chugging along on my dos prompt. However, although I've hammered out all the exceptions so that the program is running, some URLs are slow and it tends to get stuck, sometimes for over a minute waiting for a response before going to the next URL.<br><br>My question is, how do I create a timer so that the attempted read to the URL skips to the next one after say 15 seconds. I've used simple threading for applets but I'm not sure how I would do this for something like this.<br><br>Here's where the code where this should happen:<br><br><br>...<br>try<br> {<br> <font color=red>URL u = new URL(str);<br> InputStream input = u.openStream();<br> InputStreamReader reader = new InputStreamReader(input);<br> BufferedReader buffreader = new BufferedReader(reader);<br> while ((str=buffreader.readLine()) != null)<br> {<br> this.HTML += str;<br> if (linecounter>5000) break;<br> linecounter++;<br> }</font><br> } <br>catch(UnknownHostException e)<br> {String unknownhosterror = str+": UNKNOWN HOST EXECPTION: * * * S K I P P I N G U R L * * *";} <br>catch(MalformedURLException e) {System.err.println(e);}<br>catch(IOException e) {System.err.println(e);}<br>...<br><br><br>I'm not sure if the delay is occuring when it opens the input stream or when it actually reads the lines. Either way any help appreciated. <p>--Will Duty<br><a href=mailto:wduty@radicalfringe.com>wduty@radicalfringe.com</a><br><a href= > </a><br>
Tek-Tips is the largest IT community on the Internet today!
Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!
-
Congratulations IamaSherpa on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!
creating a timeout
- Thread starter wduty
- Start date
well you get the current date in milliseconds before going into a loop... then only continue the process the loop if the new current date (the date in milliseconds of that particular loop) is less than 15 seconds * 1000. <p>Liam Morley<br><a href=mailto:lmorley@wpi.edu>lmorley@wpi.edu</a><br><a href=] :: imotic :: website :: [</a><br>"light the deep, and bring silence to the world.<br>
light the world, and bring depth to the silence.
light the world, and bring depth to the silence.
- Thread starter
- #3
That is exactly the kind of obviousness which I have to kick myself for missing. I've even used this technique (before my lobotomy) in other situations! Thank you Mr. Morley! <p>--Will Duty<br><a href=mailto:wduty@radicalfringe.com>wduty@radicalfringe.com</a><br><a href= > </a><br>
hey, any time 
) <p>Liam Morley<br><a href=mailto:lmorley@wpi.edu>lmorley@wpi.edu</a><br><a href=] :: imotic :: website :: [</a><br>"light the deep, and bring silence to the world.<br>
light the world, and bring depth to the silence.
) <p>Liam Morley<br><a href=mailto:lmorley@wpi.edu>lmorley@wpi.edu</a><br><a href=] :: imotic :: website :: [</a><br>"light the deep, and bring silence to the world.<br>light the world, and bring depth to the silence.
Dear wduty,<br><br>You might also consider going down to the Socket layer in Java and setting the timeout value, i.e.:<br><br> Socket.setSoTimeout<br><br><br>public synchronized void setSoTimeout(int timeout) throws SocketException<br><br><br>Enable/disable SO_TIMEOUT with the specified timeout, in milliseconds. With this option set to a non-zero timeout, a read() call on the InputStream associated with this Socket will block for only this amount of time. If the timeout expires, a java.io.InterruptedIOException is raised, though the Socket is still valid. The option must be enabled prior to entering the blocking operation to have effect. The timeout must be > 0. A timeout of zero is interpreted as an infinite timeout.<br><br>Since:<br><br>JDK 1.1 <br><br>Good luck<br>-pete
- Thread starter
- #6
But you mean using a socket object (instead of a url object like I'm doing now)? I can do this but I'll have to wait until this thing is done reading the entire three letter permutation set. It started at "<A HREF=" TARGET="_new"> this afternoon and it's currently on "<A HREF=" TARGET="_new"> (which is about 3300 sites read). It should get to "<A HREF=" TARGET="_new"> by sometime late tonight. Thing is, it's only reading the HTML, so datawise there's very little actual data being transferred. I'm assumming I could read many urls (or sockets) simultaneously but I'm wondering where the limit is and how to go about it so that I can continue working on my computer without it interfering. I'm thinking it would be possible to do by having several concurrent threads each with an assigned batch of urls. Would this be possible?<br> <p>--Will Duty<br><a href=mailto:wduty@radicalfringe.com>wduty@radicalfringe.com</a><br><a href= > </a><br>
Well my understanding is that under HTTP1.1, you can only visit two sites under the same domain simultaneously... but I haven't heard of a limit on the number of domains. If you look back, you might be able to find Fenris' thread which was all about using multiple threads of the same class to read files simultaneously, I'm not sure if that would help.. it concerned an array of files, which you don't necessarily have here, but it might be interesting nonetheless. <p>Liam Morley<br><a href=mailto:lmorley@wpi.edu>lmorley@wpi.edu</a><br><a href=] :: imotic :: website :: [</a><br>"light the deep, and bring silence to the world.<br>
light the world, and bring depth to the silence.
light the world, and bring depth to the silence.
- Thread starter
- #8
I'll check it out. My impression from seeing firsthand how long it takes to read urls in a linear way suggests that sites like altavista and excite must use some crazy threading scheme and probably multiple machines to run their robot programs. It is interesting to see how the gathering of search-engine information works even at this ultra-simple level.<br>Another thing I would never have noticed is how much the tendency to memorable naming schemes is reflected in existing url names. I can see as the prompt reads off urls that there are a very large number of "unknownHostException" returns towards the end of an alphabetic cycle (where the names are all "*zq", "*zr", "*zs" etc.) vs how responsive urls are at the beginning where names include more vowels. <p>--Will Duty<br><a href=mailto:wduty@radicalfringe.com>wduty@radicalfringe.com</a><br><a href= > </a><br>
- Status