Using tclsh & expect with binary data

[rickrich]

Using tclsh & expect with binary data.

Here is a test program xxx.tcl from my real "tkcid" to capture Caller ID (CID) information from a modem. It was written in July 2001.

[pre]
#!/usr/bin/tclsh

package require Expect

exp_internal 1

set VOICEMSG "/usr/local/lib/tkcid.au"
set VOICEMSG "xxx.au"

set fp [open $VOICEMSG "r"]
fconfigure $fp -translation binary -encoding binary
fconfigure stdout -translation binary -encoding binary
fconfigure stdin -translation binary -encoding binary

# skip header
read $fp 24

set data [read $fp]
close $fp

set DLE [binary format H 10]
regsub -all "$DLE" "$data" "$DLE$DLE" voicemsg

exp_send -raw $voicemsg

[/pre]

The source is in:

http://www.rkkda.com/tmp/xxx.tcl

The binary file is in

http://www.rkkda.com/tmp/xxx.au

.

If I do:

$ ./xxx.tcl > zzz
send: sending "\u00fe\u00ff\u00ff\u00fe\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u007f\u00ff\u00ff\u007f\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u007f\u007f~~~\u007f~\u007f\u007f~\u007f\u00ff~\u007f\u00ff\u00ff\u007f\u007f\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00fe\u00ff\u00ff\u00fe\u00fe\u00fe\u00fe\u00fe\u00fe\u00ff\u00fe\u00fe\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u007f\u007f\u007f\u007f\u007f\u007f\u007f~~~\u007f\u007f\u007f\u007f\u007f\u007f\u007f\u007f\u007f\u007f\u007f\u007f\u00ff\u00ff\u007f\u00ff\u00ff\u00ff\u00ff\u007f\u007f\u00ff\u007f\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00fe\u00fe\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00fe\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u007f\u00ff\u00ff\u007f~~~~~\u007f\u007f~\u007f\u007f~~~~~~~~\u007f~\u007f\u007f\u00ff\u00ff\u00ff\u00ff\u007f\u00ff\u00ff\u00ff\u00ff\u00ff\u00fe\u00ff\u00fe\u00ff\u00ff\u00fe\u00fe\u00fe\u00fe\u00fe\u00ff\u00fe\u00fe\u00fe\u00fe\u00fe\u00ff\u00ff\u00ff\u00ff\u007f\u007f\u00ff\u00ff\u00ff\u00ff\u007f\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u007f\u00ff\u00ff\u00ff\u00ff\u00ff\u007f\u007f\u007f\u007f\u007f\u007f\u007f\u007f\u007f\u00ff\u007f\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\n" to { exp0 }

Looks good, I think. But when I hexdump the file I get:
[pre]
$ xod zzz

Dump: zzz

Offset: 0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef

00000000: c3 be c3 bf c3 bf c3 be c3 bf c3 bf c3 bf c3 bf | ................ |
00000010: c3 bf c3 bf c3 bf c3 bf c3 bf c3 bf c3 bf 7f c3 | ................ |
00000020: bf c3 bf 7f c3 bf c3 bf c3 bf c3 bf c3 bf c3 bf | ................ |
00000030: c3 bf c3 bf c3 bf c3 bf c3 bf c3 bf c3 bf c3 bf | ................ |
00000040: c3 bf c3 bf c3 bf c3 bf c3 bf 7f 7f 7e 7e 7e 7f | ............~~~. |
00000050: 7e 7f 7f 7e 7f c3 bf 7e 7f c3 bf c3 bf 7f 7f c3 | ~..~...~........ |
00000060: bf c3 bf c3 bf c3 bf c3 bf c3 bf c3 bf c3 bf c3 | ................ |
...
00000080: bf c3 bf c3 be c3 bf c3 bf c3 be c3 be c3 be c3 | ................ |
00000090: be c3 be c3 be c3 bf c3 be c3 be c3 bf c3 bf c3 | ................ |
000000a0: bf c3 bf c3 bf c3 bf 7f 7f 7f 7f 7f 7f 7f 7e 7e | ..............~~ |
000000b0: 7e 7f 7f 7f 7f 7f 7f 7f 7f 7f 7f 7f 7f c3 bf c3 | ~............... |
000000c0: bf 7f c3 bf c3 bf c3 bf c3 bf 7f 7f c3 bf 7f c3 | ................ |
000000d0: bf c3 bf c3 bf c3 bf c3 bf c3 bf c3 be c3 be c3 | ................ |
000000e0: bf c3 bf c3 bf c3 bf c3 bf c3 bf c3 bf c3 bf c3 | ................ |
00000110: bf c3 bf c3 be c3 bf c3 bf c3 bf c3 bf c3 bf c3 | ................ |
00000120: bf c3 bf 7f c3 bf c3 bf 7f 7e 7e 7e 7e 7e 7f 7f | .........~~~~~.. |
00000130: 7e 7f 7f 7e 7e 7e 7e 7e 7e 7e 7e 7f 7e 7f 7f c3 | ~..~~~~~~~~.~... |
00000140: bf c3 bf c3 bf c3 bf 7f c3 bf c3 bf c3 bf c3 bf | ................ |
00000150: c3 bf c3 be c3 bf c3 be c3 bf c3 bf c3 be c3 be | ................ |
00000160: c3 be c3 be c3 be c3 bf c3 be c3 be c3 be c3 be | ................ |
00000170: c3 be c3 bf c3 bf c3 bf c3 bf 7f 7f c3 bf c3 bf | ................ |
00000180: c3 bf c3 bf 7f c3 bf c3 bf c3 bf c3 bf c3 bf c3 | ................ |
00000190: bf c3 bf c3 bf c3 bf 7f c3 bf c3 bf c3 bf c3 bf | ................ |
000001a0: c3 bf 7f 7f 7f 7f 7f 7f 7f 7f 7f c3 bf 7f c3 bf | ................ |
000001b0: c3 bf c3 bf c3 bf c3 bf c3 bf c3 bf c3 bf c3 bf | ................ |
000001c0: 0a | .
[/pre]
All the bytes with the high bit set are converted to two byte Unicode!!! Hex FE goes to C3 BE, etc.

It used work around 2001-2003??? on linux. But then Unicode was invented and it does not work anymore.

Does anyone know how I can get the binary data back again???


References:

http://www.rkkda.com/sw/tkcid.tar.gz