I am using VB.NET 2002 and am working in a windows form
I am trying to "Crawl" or index a search engine that has a database of just files, not websites. When you perform a search your results will be links to files to download. What i would like to do is create a program to index all files within the search engines database.
My question is how do i go about doing this the most effective way? I can see just trying every possible search string then placing the results in a database and filtering out any duplicate addresses i get, but this seems like a poor method. The restrictions on the search engine are a minimum of 3 characerts, so i can't just put in a "*" or "***" to get all files, it must be 3 characters other than special characters.
Could anyone recommend a better approach?
Thanks.
I am trying to "Crawl" or index a search engine that has a database of just files, not websites. When you perform a search your results will be links to files to download. What i would like to do is create a program to index all files within the search engines database.
My question is how do i go about doing this the most effective way? I can see just trying every possible search string then placing the results in a database and filtering out any duplicate addresses i get, but this seems like a poor method. The restrictions on the search engine are a minimum of 3 characerts, so i can't just put in a "*" or "***" to get all files, it must be 3 characters other than special characters.
Could anyone recommend a better approach?
Thanks.
