Binary string of floats over socket 1

[Bones3]

Would anyone know where to start troubleshooting this? I'm sending many single-precision floating point numbers over a socket connection, and every once in a great while a number will come by that appears to take up 5 bytes. Of course all the normal ones take up 4 bytes. The reason this appears to me to be the problem is because I found a few of these troublemaking numbers and tested sending them by themselves. 5 bytes appeared in the receiving buffer while any other number only caused 4 bytes to appear.

In the receiving code I assume a float is 4 bytes, and when I get 5, every number after that weird one is incorrect. So even if it were an extremely rare problem, since I am sending so much information it causes devastating trouble each time.

Anyways, my question is where should I start looking for the problem? Is it likely to be at the receiving end or is it with my TCL code...

fconfigure $clientSocket -buffering none
fconfigure $clientSocket -buffersize 10
fconfigure $clientSocket -encoding binary

....

set testStr {-0.0879111}
set binstr [binary format f* $testStr]
puts -nonewline $clientSocket $binstr

and three other numbers that don't work: 0.134918, -0.134903, 0.0879

-Bones

 

[Bong]

What happens if you make the -buffersize equal to a multiple of 4?

_________________
Bob Rashkin

 

[Bones3]

No, didn't do anything special. I tried changing the buffer size on both ends of the connection. On the receiving end when I changed it to 4 bytes my test message said:

:::4
:::1

instead of :::5!

Interesting idea though. I've been suspicious of those buffers adding stuff in to make themselves happy...

But when I tinker with the number the first byte that arrives does change in value when the least significant digit changes and the last byte received changes value when the sign changes. So it appears that at least that the first and last of the 5 bytes are part of the number. I am so confused

-Bones

 

[Bones3]

Here is some of my Java code on the other end. I'm convinced that I did something stupid because 4 bytes is 4 bytes is 4...

ByteBuffer inputBffer = ByteBuffer.allocate(BUFFER_SIZE);
try {
  isConnectionClosed = fromWavescopeChannel.read(inputBffer); // SocketChannel.read()
  byte[] inputArray = inputBffer.array();
  System.out.println(":::" + inputBffer.position());
  
  for (int i = 0; i < (BUFFER_SIZE-inputBffer.remaining()); i++)
    System.out.println(inputArray[i]);
} catch (IOException e) {}

-Bones

 

[Bong]

Well, I see now that you set -buffering to none so the buffersize is irrelevant. You might play with that but you probably don't want buffering anyway.

I don't really see what you're doing. Are you sending a number from Tcl to Java, or the other way around? Are you converting the number to a binary string before sending? If so, why?

_________________
Bob Rashkin

 

[Bones3]

Well, I see now that you set -buffering to none so the buffersize is irrelevant. You might play with that but you probably don't want buffering anyway.

Hmm, true...

From tcl to Java. I'm converting the number to a binary string first because, otherwise, tcl sends text. To send text it would be a byte for each character (in the case above that would be 11 bytes) whereas a float would be 4 bytes. When I send several numbers this might make a big impact on the network performance.

I actually do have it working with text, I just want there to be less information sent over the network cable. And less time spent for Java having to parse text.

-Bones

 

[Bong]

OK but if you're sending a binary string, you're still sending text. So, I assume what you're sending is 4 (1 byte each) characters. Then, on the Java side, you read in those 4 bytes (characters) as a binary number. Right so far?

Where along the line have you checked the integrity of the value? That is, are the 4 bytes going into the socket the right 4 bytes? How about coming out?

_________________
Bob Rashkin

 

[Bones3]

So, I assume what you're sending is 4 (1 byte each) characters."

It is my assumption as well. I've read this several times in some Tcl documentation regarding the f option for the binary commmand:

This form is the same as c except that it stores one or more one or more single-precision floating in the machine's native representation in the output string. This representation is not portable across architectures, so it should not be used to communicate floating point numbers across the network. The size of a floating point number may vary across architectures, so the number of bytes that are generated may vary.

However, I went ahead with it because when I tested the value I received in Java; it was exactly 4 bytes per number and was indeed the same number (after a little to big endian conversion). (I also went ahead with it because I know of no other way to accomplish this goal...)

Later, when I used this for the real application, I found that sometimes apparently more than 4 bytes are sent. But in my mind I think that numbers must always be the same size; perhaps I'm wrong? Because if the size could change how on earth could anyone ever know where one began and the other ended?

"Where along the line have you checked the integrity of the value?" Doesn't TCP do some of that behind the scenes? The only checking I know how to do is before the sending socket and right at the receiveng socket. Before it sends, I try to use the binary scan command in Tcl to make sure the number being stored in the binary string is the same one I hardcoded into the test program. Then without further checking the binary string is written to the socket. Then in Java I check the size of the information coming in and the value of the number from the SocketChannel.

isConnectionClosed = fromWavescopeChannel.read(inputBffer); // SocketChannel.read()
byte[] inputArray = inputBffer.array();

This is just a Java SocketChannel that is connected. I take the information out and store them as bytes or characters. Anything before getting these bytes I don't know how to do an integrity check.

-Bones

 

[Bong]

I suggest you play around with binary format and binary scan. I know I always have to hold my mouth just right each time I use them. Just in the Tcl shell (tclsh) I did the following:

% set f 4.1234
4.1234
% set b [binary format f $f]
åò?@
% binary scan $b f f2
1
% set f2
4.12340021133
%
% set f -0.134903
-0.134903
% set b [binary format f $f]
?$
?
% binary scan $b f f2
1
% set f2
-0.134902998805

It seems to be working ok, noting that one of the 4 characters in the second instance must be "linefeed-like".

_________________
Bob Rashkin

 

[Bones3]

It would make sense that Tcl has an underlying algorithm for converting every newline character into \r\n, thus going from a 4 byte segment of number to a 5 byte one. And I believe it does:

fconfigure -translation option

But I thought I had tinkered with this command before. I think more experimentation is in order when I get the time.

Thank you for your help! I'm going to have to move on to other things for a while so that I at least appear to be accomplishing something.

-Bones

 

[Bong]

I think -translation binary on the fconfigure will suppress any interpretation of linefeeds. I would have thought that -encoding binary would do much the same thing but I've never used it.

_________________
Bob Rashkin

 

[Bones3]

Yes, that was the problem. -translation binary worked. I could have sworn that I tried that before, but apparently not!

Thank you so much.

-Bones

 

[Bones3]

I honestly don't know what -encoding binary does, but without it a simple little to big endian conversion doesn't reformat the numbers properly. Without it the number doesn't appear to be formatted properly on the Java side.

-Bones