In desperate need of Threading/Queueing Help 1

[mavalon]

I'm new to threading, and am completely baffled.

Currently I have something like the following:

lbl.Text = "start sending to queue"
for i = 0 to 100
     [COLOR=green]' do some work[/color]

     SyncLock MyQueue.SyncRoot
          MyQueue.Enqueue(MyFunctionThatTakesForeverToComplete)
          _waitQueue.Set()
     End SyncLock

     [COLOR=green]' do other work that does not require 
     ' MyFunctionThatTakesForeverToComplete to complete[/color]
next
lbl.Text = "everything's queued"

Currently, the loop requires MyFunctionThatTakesForeverToComplete to complete before proceding to the next iteration. What can i do to get around this? Am I even on the right track?

 

[ThatRickGuy]

Is MyFunctionThatTakesForeverToComplete thread safe? Can you run that method in multiple threads with out I/O or memory issues?

Is MyQueue some kind of handy .Net v2 threading tool?

-Rick

VB.Net Forum forum796 forum855 ASP.NET Forum
I believe in killer coding ninja monkeys.

 

[mavalon]

I'm still using v1.1.

MyFunctionThatTakesForeverToComplete downloads files. Oftentime very large files.

Public Function DownloadFile()
        Try
            Dim addr As String = lvi.SubItems(1).Text
            Dim target As String = lvi.SubItems(2).Text
            Dim path As String = lvi.SubItems(3).Text
            If addr.Length > 0 Then
                Dim wc As New Net.WebClient
                If Directory.Exists(target) Then
                    lvi.SubItems(4).Text = "Downloading"
                    wc.DownloadFile(addr, target)
                End If
                wc.Dispose()
                wc = Nothing
            End If
        Catch ex As Exception
            lbl.Text = ex.Message.ToString
        End Try
    End Function

 

[ThatRickGuy]

Okay, so I'm waiting on a deployment compile, so here's what I thought of.

A few notes, 1: use SyncLock at the smallest possible scope, in this code, I'm using it only to protect the m_DownloadList so that it can not be accessed by two different threads at the same time. 2: Do not update the GUI from any thread other than the primary thread. You can get away with it in some situations in 2k2 and 2k3, but 2k5 won't let you, and it's a dangerous practice.

Also, this has not been compiled, so I'm sure there are type-os and mistakes.

public class DownloadManager
  private structure DownloadQueueEntry
    Source as string
    Target as string
  end structure

  private m_DownloadList as new arraylist
  private m_CurrentSource as string = string.empty
  private m_CurrentTarget as string = string.empty
  private sl_DownloadList

  public readonly property QueueLength as integer
    get
      return m_DownloadList.length
    end get
  end property

  public readonly property CurrentSource as string
    get
      return m_CurrentSource
    end get
  end property

  public readonly property CurrentTarget as string
    get
      return m_CurrentTarget
    end get
  end property

  public sub AddToDownloadQueue(Source as string, Target as string)
    dim dqeNew as downloadqueueentry
    dqeNew.Source = source
    dqeNew.Target = target

    synclock sl_DownloadList
      m_DownloadList.add(dqeNew.source)
    end synclock

    if m_DownloadList.length = 1 then 
      Dim tDownload As New Thread(AddressOf StartDownload)
      tDownload.Start()
    end if
  end sub

  private sub StartDownload
    dim QueueEmpty as boolean = false
    while Not QueueEmpty
      Try
        synclock sl_DownloadList
          m_CurrentSource = ctype(m_DownloadList.item(0), downloadqueueentry).source
          m_CurrentTarget = ctype(m_DownloadList.item(0), downloadqueueentry).Target
        end synclock
    
        If CurrentSource.Length > 0 Then
          Dim wc As New Net.WebClient
          If Directory.Exists(CurrentTarget) Then
            lvi.SubItems(4).Text = "Downloading"
            wc.DownloadFile(CurrentSource, CurrentTarget)
          End If
          wc.Dispose()
          wc = Nothing
        End If
      Catch ex As Exception
        debug.writeline(ex.Message)
      End Try

      synclock sl_DownloadList
        m_DownloadList.removeat(0)
        if m_DownloadList.length = 0 then QueueEmpty=true
      end synclock
    end while
  end sub
end class

VB.Net Forum forum796 forum855 ASP.NET Forum
I believe in killer coding ninja monkeys.

 

[ThatRickGuy]

To build apon that code, you could also raise an event just before the end while (say like a "file complete" event!)

To use that code you could do something like

...'your form's class
dim m_DownloadManager as new DownloadManager

public sub btnDownloadClick
  m_DownloadManager.AddToDownloadQueue(lblTarget.text, lblSource.text)
  system.threading.thread.wait(250)
  me.statusbar.text = "Downloading: " & m_DownloadManager.source & " (" & m_DownloadManager.QueueLength & ")"
end sub

public sub FileComplete handles m_DownloadManager.DownloadComplete
  system.threading.thread.wait(250)
  if m_DownloadManager.QueueLength = 0 then
    me.statusbar.text = "All downloads completed" 
  else 
    me.statusbar.text = "Downloading: " & m_DownloadManager.source & " (" & m_DownloadManager.QueueLength & ")"
  end if
end sub

VB.Net Forum forum796 forum855 ASP.NET Forum
I believe in killer coding ninja monkeys.

 

[mavalon]

It's nearly midnight (here in asia), so I'll have to try out your suggestions tomorrow and let you know how it goes. For now let me just thank you for your feedback. It's greatly appreciated!

 

[mavalon]

Your code makes perfect sense. It appears to be working now (for the most part) thanks to your code samples. I modified it to have 5 concurrent downloads. I still have other issues (such as one large file throwing a webclient exception), but that's for another thread if i can't find the solution floating about on the internet.

Thanks so much for your assistance.

 

[tperri]

set the WebClient.TimeOut property to something lasrger th an the default value (which I forget what it is right now)

ex:
wc.TimeOut = 100000

 

[mavalon]

FYI: "TimeOut is not a member of System.Net.Webclient" is the error I get with the previous code.

In fact, there do not appear to be many "members" at all.

 

[ThatRickGuy]

You might want to check with the HTML/ASP/ASP.Net forums on this. I beleive there is some standard file size limit that HTTP hits. I think there is a javascript hack that can get around large file sizes, but I don't know how they apply to the .Net web control. There are also limits that can be placed on the web server.

If you need to transfer large files, you may want to look into an FTP solution. We use the open source solution from

http://www.enterprisedt.com/

for our FTP processes.

-Rick

VB.Net Forum forum796 forum855 ASP.NET Forum
I believe in killer coding ninja monkeys.