What's the best way to deal with this?

[nickd87]

Hello

I am writing an application in VB6 that listens for incoming sockets that will send data to my application, and then it will do something with that data.

The data being received is variable in length but always begins with a "header" containing, among other things, the length of the variable part of the packet.

I will be listening for sockets on a control array of 500 Winsock controls. Obviously only one DataArrival event will fire at any one time, but the client will connect multiple sockets and send data on them simultaneously.

What is the best way to ensure data integrity? At the moment, as soon as the DataArrival event fires I am removing the data from that winsock's buffer and appending it to the value in a global variable. This is working OK at the moment but is not desirable as a) one bit of bad data will stuff up the entire string of data in the variable and b) if another DataArrival event fires before the end of the entire data is received on a previous socket, the new data will be inserted into the middle of that packet on the global variable.

What's the best way to deal with a situation like this? I read that .PeekData until the correct length packet is received is poor programming practice.

Also, is there any way to pause until a Winsock control's .State property changes to sckConnected without looping DoEvents statements?

Thanks

Nick

 

[GhostWolf]

You could use an array of 500 temporary strings, keyed the same as the array of controls. Load the received string to the appropriate element, and process each element when the entire string's been received.

 

[nickd87]

and process each element when the entire string's been received.

Thanks - are you saying I should .PeekData until the whole string has been received?

 

[GhostWolf]

Uh, good question! I'm an old-fashioned kinda guy, (read unenlightened, if you prefer), who hasn't tried that new-fangled stuff.

I just figured you could check the element's contents, after appending the latest DataArrival, to see whether you have what you need to process that string - and adjust the element's contents accordingly.

 

[strongm]

The suggestion is that if you have 500 Winsock controls you should have 500 global variables. Since your Winsocks are a control array, your global storage can be a string array. Then

>as soon as the DataArrival event fires I am removing the data from that winsock's buffer and appending it to the value in a global variable

becomes

as soon as the DataArrival event fires I am removing the data from that indexed winsock's buffer and appending it to the value in the matching indexed global variable

 

[nickd87]

OK I see what you are saying, but what if the DataArrival event fires again with a new lot of data before I've had a chance to process the data already in that string?

I'm not "doing stuff" with the data received from the DataArrival event. I am running a timer every 250ms and processing it all in batch then. Is that stupid? Is there any advantage/disadvantage to "doing stuff" with the data from within the DataArrival event?

 

[GhostWolf]

I can't say "stupid" at all; for what you're doing it may be the perfect way to handle processing.

I tend to be a little more anxious, though, and think I'd set up a separate subroutine to process the data array interactively, e.g.

Public Sub CheckElement(byval prmIndex as Integer)
Dim wrkStr as String
  ' assuming that each string you need to process
  ' is 20 bytes long
  While Len(gblDataArray(prmIndex)) >= 20
    wrkStr = left(gblDataArray(prmIndex),20)
    ' do what you need with that string
    gblDataArray(prmIndex) = _
        Replace(gblDataArray(prmIndex), wrkStr, "")
  Wend
End Sub

Then, at each DataArrival event all you'd need to do is load the appropriate gblDataArray element, then pass that element's index to the processing subroutine.

Come to think of it, that subroutine would allow you to process in batch too - by removing the parameter and placing the While/Wend inside a For/Next loop, i.e.

  For a = 1 to 500
    While Len(gblDataArray(a)) >= 20
      ...
    Wend
  Next a

As to what will happen if the DataArrival event fires while you're doing this processing, I'm hoping one of the more knowledgeable members can answer that because it's a bit of guidance I need too. I honestly don't know whether the arrival events will buffer and be available when processing returns from the called subroutine.

 

[tedsmith]

No single Computer with 500 winsocks can really send simultaneously. They must all be in turn even if separated only by microseconds.
Same with a network. A wire can only be high or low!

If you had 500 computers all sending at random apparently simultaneously, they will each only send to the indexed buffer in each of the 500 Winsocks at the receiving end so they cant get mixed up.
I would even tie the index to the port number, (Say 1000+Index)

The repetition rate and size of the data packet is important. If a slow network and many computers send at once, a second packet sent immediately could easily be lost.
The server may not have acquired all the first packet before the second one is sent.
It is important that the sending end doesn't try to send another packet before the previous one has been sent otherwise data will be lost. (If you are writing the sending end, Set a flag when sending and use the Sub Winsock_SendComplete to clear the flag to allow subsequent sendings.)

If your data packet never exceeds 8k bytes you can use the following. Each Buffer can than be processed independently.

Dim Buffer(500)
Private Sub Winsock_DataArrival(Index As Integer, ByVal bytesTotal As Long)
Winsock(Index).GetData Buffer(Index)
Exit Sub

But if the data exceeds 8k the Winsock will fire more than once and other packets for another socket might have been been received in the meantime if your network or computer is fast. (A picture might fire a dozen times)
So I would write to a stream to acquire long data.
You'd have to determine how long the packet was and process the data in a second sub when the whole packet has been received:
This leaves the Sub DataArrival free to receive all the other sockets and the streams can't get mixed up.

'References needs Microsoft DAO library
Dim StreamRx(20) As ADODB.Stream
Dim PacketSize as long

Sub Form_Load
For Index=1 to 500
  Load Winsock(Index)
  Set StreamRx(Index) = New ADODB.Stream 'setup new streams to get next Data from each client
  StreamRx(Index).Open
  StreamRx(Index).Type = adTypeText 'if data is text
Next
end sub

Private Sub Winsock_DataArrival(Index As Integer, ByVal bytesTotal As Long)
Dim Buffer
On Error GoTo WDError
Winsock(Index).GetData Buffer
If Len(Buffer)< 20(Say) Then PacketSize(Index)= (Part of beginning of buffer that tells you this)
If StreamRx(Index).Size = PacketSize then
    ProcessArrivedData Index 
    Exit Sub 'dont read any more of this transmission
End If
StreamRx(Index).Write Buffer 'continue to accumulate buffer for this winsock
End Sub

Sub ProcessData (Index as integer) 
StreamRx(Index).Position = 0
MyData(Index)= StreamRx(Index).Read(adReadAll)
If StreamRx(Index).State <> 0 Then StreamRx(Index).Close
Set StreamRx(Index) = New ADODB.Stream 'setup new stream to get next packet
StreamRx(Index).Open
StreamRx(Index).Type = adTypeText 'if data is text
End Sub

Exit Sub

A problem might occur if you get corrupted data such as someone pulling out a plug in the middle of a packet so have On Error Gotos that clear, & remake the Stream otherwise you might get stuck in an endless loop.

 

[HarleyQuinn]

Just to stop any questions regarding references I think the comment in Ted's code should read:

'References needs Microsoft ADO library

Regards

HarleyQuinn
---------------------------------
Carter, hand me my thinking grenades!

You can hang outside in the sun all day tossing a ball around, or you can sit at your computer and do something that matters. - Eric Cartman

Get the most out of Tek-Tips, read FAQ222-2244: How to get the best answers before posting.

http://lessthandot.com

 

[tedsmith]

Yes, a fruedian typo!

Also Dim MyData(500) should be in Declarations but Dim Buffer in Dataarrival.

If you are sending pictures it is better to put it in a propertybag send it as a byte array otherwise Dim everything else as string.(See other posts on this)

 

[nickd87]

Thanks Ted. I'll give it a bash.

 

[dilettante]

Another point you may not be aware of is that for each socket (or Winsock control) there is an internal receive buffer. If this buffer gets full before you can empty it and process the contents, the TCP logic beneath the socket will "push back" on the sender and stall the sending of more data until there is buffer space once again. This is essentially what TCP congestion control is about.

Thus how quickly you can take data and proces it does not determine whether there will be data loss.

This has led to some exploitation and thus controversy over time as well: Fixing the unfairness of TCP congestion control.

In the same way you normally don't concern yourself with data corruption. TCP provides timeout, sequence, and error checking. One may safely think of TCP streams as being as reliable as disk I/O streams in terms of data fidelity.

 

[tedsmith]

Agreed but the loss of data can be caused by not handling the Winsock control properly, such as trying to feed the data into the send end faster than the network can transmit it.

 

[nickd87]

Before I report back on how I'm going with this, I just want to see if anyone has any comments on one of my original questions in this thread:

Is there any way to pause until a Winsock control's .State property changes to sckConnected without looping DoEvents statements?

 

[tedsmith]

Ban DoEvents at all costs!
You should only have Winsock Input statements inside DataArrival Subs.

You dont have to loop because the DataArrival Subroutine only is activated when the Winsock receives some data.
The incoming data is gathered in the Winsock internal buffer and the Sub Winsock_DataArrival only fires when it has been accepted.
If you have this buffer length any longer than the normal length of the packet, then a shorter packet of data would never be received (which can easily happen) or worse still get added to the start of the next packet.
There is nothing wrong with this Sub Dataarrival firing hundreds of times as it all happens much much faster than even a 1 ghz network can send to a fast computer.

If you are receiving big pictures and put a tiny delay of a few milliseconds in the start of the Dataarrival you can make sure each chunk you receive is the maximum of 8k but this only slows down the whole reception anyway and delays short messages as well.

Important to have only one Sub DataArrival (Index,Length) in your whole application and index the sockets whether you create and destroy them or have them in the first place. Use the index to indicate who the client is.

 

[nickd87]

I'm asking if I can, after trying to .Connect, pause until its state changes to sckConnected.

The part I'm trying to get right now is the unsolicited sending of data to the client. I don't necessarily care if it is receiving data at this point. The application I am writing is a server that brings up sockets to clients, sends data, then closes.

You helped me get the receiving of data right, but now I'm concerned with the sending.

 

[tedsmith]

Looping is not good. I would enable a normally not enabled say 1 to 5 second Timer if the connection was not successful, but have a limit on the number of times to try otherwise you'll choke the network if the server is off line.

When you .Connect, if the server senses the client, the following sub fires in the Server

Dim ClientActive(500) as Boolean

Sub Winsock_ConnectionRequest(Index As Integer, ByVal requestID As Long)
  Winsock(Index).Accept requestID
  ClientActive(Index)=True
End Sub
If you need it, ClientActive(Index) tells the server which client is connected.

In the Client you have for your sending:-

Dim TimeOut as integer

Sub MyConnectAndSendRoutine()
  'Connect or Send if Connected
  If Winsock.State = sckConnected then 
     Winsock.SendData MyData 'send if previously connected 
  Else
     If Winsock.State <> 0 Then Winsock.Close 'cancel any previous errors
     Winsock.Connect ServerIPAddress, ServerPort
     Timer1.Enabled=True
  End If
End If

Sub Timer1_OnTimer()
  'Send data a second or so later
  If Winsock.State = sckConnected then 
     Timer1.Enabled=False 'successful
  Else
     MyConnectAndSendRoutine 'this sends the actual data or try again if still not connected.
     If Timeout < 5 then
         TimeOut=Timeout+1
     Else
         Timer1.Enabled=False'too many attempts so stop trying    
         TimeOut=0
     End If
  End If
End Sub

Sub Winsock_SendComplete()
Winsock.close
Timeout=0
End If

To make the server ready for the next reception from that client, in the server you need to use:

Sub Winsock_Close(Index As Integer)
Winsock(Index).listen
End If

If your data is a picture more than 8k then you can use the Winsock.SendComplete to send a code of a few more bytes to tell the Server to only process the data when the whole picture is received by the DataArrival Buffer otherwise you never get the whole pic!

 

[tedsmith]

if you don't want the short initial delay you can have the Server's Sub ConnectionRequest send back a few bytes.
When the Client receives this at the Client Sub DataArrival, it knows it is connected and can then send the Mydata.

Sub Winsock_ConnectionRequest(Index As Integer, ByVal requestID As Long)
  Winsock(Index).Accept requestID
  Winsock(Index).Senddata "OK"
End Sub

At the Client

Dim MyData
Dim DataSent as Boolean
Put data into MyData

Sub MyConnectAndSendRoutine()
  'This is the only routine you have to call
  DataSent=False
  If Winsock.State <> 0 Then Winsock.Close 'cancel any previous errors
  Winsock.Connect ServerIPAddress, ServerPort
End If

Private Sub Winsock_DataArrival(ByVal bytesTotal As Long)
'Automatically sends as soon as connection established
Dim Buffer()
Winsock(Index).GetData Buffer
If Buffer="OK" then 
   Winsock.SendData MyData 'send as soon as connection established
   Winsock.Close
   DataSent=true
End If
End Sub

You still need a timer to test DataSent a few seconds later to see if the data was sent then try again 5 times if not like my previous example.