AxWebBrowser unable to Navigate

[mavalon]

I'm trying to create an app that navigates and downloads images from web sites. (This is my first Windows App, so perhaps I'm in over my head.)

I have quite a few problems working with AxWebBrowser and mshtml.HTMLDocument. I'll just post one problem here:

Navigation is extremely quirky. I had it working to some extent: It would navigate and download images from some page while skipping over other pages. Now, it's only downloading the first page. Here's some relevant code:

 Public Function Crawl()
        If LvUrls.Items(lviIndex).Checked Then

            Dim doc As mshtml.HTMLDocument
            Try

                doc = ReadWebDoc(DownloadLink, LvUrls.Items(lviIndex))

                If Not doc Is Nothing Then

                   [COLOR=green]' get files[/color]
                    LoadFiles(doc)

                    [COLOR=green]' get links[/color]
                    LoadLinks(doc)

                    doc.close()
                    doc = Nothing
                End If
            Catch ex As Exception
                Dim m As String = ex.Message
            End Try
        End If

    End Function

 Public Sub LoadLinks(ByVal MyHtml As mshtml.HTMLDocument)
        Dim x As Int32 = (MyHtml.links.length - 1)
        'Dim f As Int32 = (MyHtml. - 1)
        Dim y As Int32 = 0

        If CurrLevel < MaxLevel Then
            CurrLevel += 1
            For y = 0 To (x)
                Try
                    [COLOR=green]' here i add links to my listview and
                    ' a separate thread loops through the 
                    ' listview items and crawls each
                    ' one:  Crawl() above[/color]
                Catch ex As Exception
                    Dim m As String = ex.Message
                End Try
            Next y

        End If

    Public Function ReadWebDoc(ByVal DownloadLink As String, ByVal Lvi As ListViewItem) As mshtml.HTMLDocument
        [COLOR=green]' wait to load doc[/color]
        Dim doc As New mshtml.HTMLDocument
        Try
            Lvi.SubItems(3).Text = "Downloading..."
            myBrowser.Navigate(DownloadLink)
            [COLOR=green]' the first page works here, but all
            ' subsequent pages never get "busy" here:[/color]
            Do Until Not myBrowser.Busy
            Loop
            doc = DirectCast(myBrowser.Document, mshtml.HTMLDocument)
        Catch ex As Exception
            Lvi.SubItems(3).Text = ex.Message
        End Try
        Return doc
    End Function

 

[gjconely]

Try using this instead of using a webbrowser. You will then have to parse through the URLs and get what you want. You can get the images using the same call if you want.

Public Function GetPageHTML2(ByVal URL As String, Optional ByVal TimeoutSeconds As Integer = 5) As String
' Retrieves the HTML from the specified URL,
Dim objRequest As Net.WebRequest
Dim objResponse As Net.WebResponse
Dim objStreamReceive As System.IO.Stream
Dim objEncoding As System.Text.Encoding
Dim objStreamRead As System.IO.StreamReader

Try
' Setup our Web request
objRequest = Net.WebRequest.Create(URL)
objRequest.Timeout = TimeoutSeconds * 1000
' Retrieve data from request
objResponse = objRequest.GetResponse
Application.DoEvents()
objStreamReceive = objResponse.GetResponseStream
objEncoding = System.Text.Encoding.GetEncoding("utf-8")
objStreamRead = New System.IO.StreamReader(objStreamReceive, objEncoding)
' Set function return value
Application.DoEvents()
GetPageHTML2 = objStreamRead.ReadToEnd()
' Check if available, then close response
If Not objResponse Is Nothing Then
objResponse.Close()
End If
Catch
' Error occured grabbing data, simply return nothing
Return ""
End Try
End Function



Greg Conely

http://www.ImageMentor.net

http://www.gregsdirectory.com