Expect inserting "?"s question marks in send stream

[richwd]

My code snippet:

set send_slow {2 0.5}

set shell_commands {
	"for FILE in *.*\r"
	"do\r"
	"if \[\[ \$FILE =~ ^\[A-Z0-9]{32}\[.]\[A-Z0-9]{8}\[.]jpg$ ]]; then\r"
	    ":\r"
	"else\r"
	    "echo \"Fail: \$FILE\"\r"
	"fi\r"
	"done\r"	
}

#Send the commands, need to use "-s" to slow it down so "?"s don't get inserted into sent stream.
foreach command $shell_commands {
    exp_send -s $command
    expect "> "
}

I have this really annoying problem where question marks get inserted in the code I'm sending with expect like this; f?r. It's supposed to be "for".

# f?r FILE in *.*
-bash: f?r: command not found

After searching I found that you can slow down the speed at which you send things with by using "send_slow". Jeez, just what I want, to bog down my program. Why is this happening? Isn't there anyway to fix this besides slowing down our programs? It ruins regular expressions so the don't work and now I can't depend on my program working correctly at all. It just seems so dumb.

BTW as you can tell. I'm trying to send bash commands to a linux box. To check filenames in a directory. Is there a better way?

 

[PHV]

Why not simply use ssh ?

Hope This Helps, PH.
FAQ219-2884
FAQ181-2886

 

[richwd]

What do you mean? I am. I spawn ssh then send these commands. This is just a snippet not the whole program. Yes I can ssh to it and look manually but I'm trying to automate it.

 

[PHV]

I meant something like this:
ssh yourLinuxBox 'for FILE in *.*;do [[ $FILE =~ ^[A-Z0-9]{32}[.][A-Z0-9]{8}[.]jpg$ ]] || echo "Fail: $FILE";done'

Hope This Helps, PH.
FAQ219-2884
FAQ181-2886

 

[richwd]

Because I have to do a lot more things. Don't you open a connection to ssh, do a bunch of things then close the connection? Or do I prepend "ssh2" before every command I send?

It's a bigger program than just one command.

 

[richwd]

What really pisses me off is that Expect is hacking my code and inserting characters without telling me. I spent a lot of time looking thru my code trying to find my bug why my program wasn't working and it wasn't my code at all. It was Expect inserting questions marks. Now I can't write a reliable program using Expect. How can I get it to stop doing this?

 

[Annihilannic]

You can include many lines of shell script in an ssh command, with appropriate quoting, or even pipe it in from standard input. As you can see, even PHV's small example based on your code is more than one command.

I would definitely go that route normally; the only time I would use expect is when running a programme at the remote side which only works when communicating directly with a terminal.

Sorry, I can't say why expect is behaving that way for you; I haven't seen that before. What operating system distributions/versions are you using at each end, what version of expect, and what type of connection between them?

Annihilannic
[small]tgmlify - code syntax highlighting for your tek-tips posts[/small]

 

[richwd]

I still don't follow. This is what I'm doing in a nutshell:

spawn ssh2.exe yourLinuxBox

proc1
proc2
proc3
.
.
.
#Then when you're done close the session.
exp_send "exit\r"
exp_close
exp_wait


Are you saying I should instead do:

ssh yourLinuxBox proc1
ssh yourLinuxBox proc2
ssh yourLinuxBox proc3
.
.

or...
ssh yourLinuxBox {
my whole program
}
????

 

[Annihilannic]

You're getting warm at the end there, but taking PHV's code as an example, you can spread it over multiple lines for readability:

ssh yourLinuxBox '
    for FILE in *.*
    do 
        [[ $FILE =~ ^[A-Z0-9]{32}[.][A-Z0-9]{8}[.]jpg$ ]] || echo "Fail: $FILE"
    done

    # any other commands ....

    # no need for an "exit" here
'

The only thing you need to be careful of is not to include any single quotes in your code that may terminate the quoted script prematurely (unless you know you want to for some reason).

Alternatively you can store your programme in another script file, and do something like:

ssh yourLinuxBox <mywholeprogramme.sh

Annihilannic
[small]tgmlify - code syntax highlighting for your tek-tips posts[/small]

 

[richwd]

Hmm... Interesting. But why do all the examples I find talk about spawning ssh? And why no exit at the end? Does the connection just end after the last brace? Are you saying don't use Expect at all? Then how do I get output back? So confusing.

 

[Annihilannic]

When you invoke any command shell, say sh, and pipe some commands into it, it runs them and then exits when it hits EOF (end of file) on the input stream. If you run it with the commands embedded in the invocation, e.g. sh -c 'date ; uname -a', the result is the same; it runs those commands and then terminates.

ssh is based on rsh (remote shell), which is the same as a normal shell except it runs on a remote host. Otherwise the semantics are the same.

The output is sent to standard output and/or standard error as per any other shell programme, and you can handle them the same way, e.g.

ssh yourLinuxBox 'some commands' >standardoutput.log 2>standarderror.log

"spawn"ing ssh is Expect terminology; as mentioned earlier I would only use that when absolutely necessary; i.e. when an application which bypasses standard input and communicates directly to a terminal (which Expect emulates), or a situation where you had to use programme logic to change the commands/inpup supplied based on the output received. An example of the former is the passwd utility on some operating systems. An example of the latter are programmes that may prompt you to confirm an action in some situations, say overwriting an existing file, but not others.


Annihilannic
[small]tgmlify - code syntax highlighting for your tek-tips posts[/small]

 

[richwd]

Ok, I'd like to try that but isn't it going to prompt me for password every time I send an ssh command? I know what you're going to say, set up passwordless authentication with keys. That would be good for one time but I may have to test many machines at work and don't want to have to go thru that with every machine I want to test.

 

[richwd]

My program is working fine. I just want to know why Expect is sticking question marks in my code and how to stop it. It's REALLY annoying.

 

[Annihilannic]

Sorry, I can't say why expect is behaving that way for you; I haven't seen that before. What operating system distributions/versions are you using at each end, what version of expect, and what type of connection between them?

Annihilannic
[small]tgmlify - code syntax highlighting for your tek-tips posts[/small]

 

[richwd]

Sending machine is:

Host Name: WDMV-QA-TEST-PC
OS Name: Microsoft Windows 7 Professional
OS Version: 6.1.7601 Service Pack 1 Build 7601
OS Manufacturer: Microsoft Corporation
OS Configuration: Standalone Workstation
OS Build Type: Multiprocessor Free
Registered Owner: WDMV
Registered Organization:
Product ID: 00371-868-0000007-85531
Original Install Date: 9/29/2011, 2:43:59 PM
System Boot Time: 10/13/2011, 10:00:14 AM
System Manufacturer: HP-Pavilion
System Model: RY866AA-ABA m8034n
System Type: X86-based PC
Processor(s): 1 Processor(s) Installed.
[01]: x64 Family 15 Model 67 Stepping 2 Auth
~2600 Mhz
BIOS Version: Phoenix Technologies, LTD 5.01, 2/2/2007
Windows Directory: C:\Windows
System Directory: C:\Windows\system32
Boot Device: \Device\HarddiskVolume1
System Locale: en-us;English (United States)
Input Locale: en-us;English (United States)
Time Zone: (UTC-08:00) Pacific Time (US & Canada)
Total Physical Memory: 1,919 MB
Available Physical Memory: 833 MB
Virtual Memory: Max Size: 3,837 MB
Virtual Memory: Available: 2,377 MB
Virtual Memory: In Use: 1,460 MB
Page File Location(s): C:\pagefile.sys
Domain: WORKGROUP
Logon Server: \\WDMV-QA-TEST-PC
Hotfix(s): 55 Hotfix(s) Installed.
[01]: KB971033
[02]: KB2305420
[03]: KB2393802
[04]: KB2425227
[05]: KB2476490
[06]: KB2479943
[07]: KB2484033
[08]: KB2488113
[09]: KB2491683
[10]: KB2492386
[11]: KB2502285
[12]: KB2503665
[13]: KB2506212
[14]: KB2506928
[15]: KB2507618
[16]: KB2508272
[17]: KB2509553
[18]: KB2510531
[19]: KB2511250
[20]: KB2511455
[21]: KB2515325
[22]: KB2518869
[23]: KB2522422
[24]: KB2529073
[25]: KB2532531
[26]: KB2533552
[27]: KB2536275
[28]: KB2536276
[29]: KB2539635
[30]: KB2541014
[31]: KB2544521
[32]: KB2544893
[33]: KB2545698
[34]: KB2547666
[35]: KB2552343
[36]: KB2555917
[37]: KB2556532
[38]: KB2559049
[39]: KB2560656
[40]: KB2562937
[41]: KB2563227
[42]: KB2563894
[43]: KB2564958
[44]: KB2567053
[45]: KB2567680
[46]: KB2570791
[47]: KB2570947
[48]: KB2572077
[49]: KB2579686
[50]: KB2586448
[51]: KB2616676
[52]: KB958488
[53]: KB976902
[54]: KB976932
[55]: KB982018
Network Card(s): 1 NIC(s) Installed.
[01]: NVIDIA nForce 10/100 Mbps Ethernet
Connection Name: Local Area Connection
DHCP Enabled: Yes
DHCP Server: 192.168.1.1
IP address(es)
[01]: 192.168.1.100
[02]: fe80::fc56:c094:4982:2539

Receiving machine is Linux 2.6.32.11:

# cat /proc/cpuinfo
processor : 0
cpu : APM82181
clock : 800.000008MHz
revision : 28.128 (pvr 12c4 1c8
bogomips : 1600.00
timebase : 800000008
platform : PowerPC 44x Platform
model : amcc,apollo3g
Memory : 256 MB

I'm not an expert so let me know any commands to get the info you want. They are both on the same router using ethernet.

Someone I spoke to told me;

"Expect isn’t substituting the “?”. That’s typical behavior from a terminal driver that gets a character that it doesn’t like, doesn’t know how to display, etc. I would first check for a parity mismatch – that’s the most likely cause. For example, your sending machine is sending characters without parity and the receiving machine has parity checking turned on.

However, the fact that send_slow improves things makes me think there’s something else going on at the driver level. But check the parity first."

Terminal settings? Is that ssh2? How do I check the settings?

 

[Annihilannic]

Well, parity shouldn't be an issue when connected via Ethernet, only if it was a serial connection.

I haven't used Expect under Windows; is it a Windows native version that you run from the Windows command shell, or part of a Unix-like environment such as Cygwin? In either case expect -v should tell you the version.

The Linux box seems to be some kind of embedded system, is that correct? When logged in to it, try running stty -a, echo $TERM and locale to display the terminal and internationalisation settings.

In any case it's sounding like an environment that would be difficult for me to reproduce. :-(

If send_slow is working for you, you may have to stick with that; you could try tuning it to use some not-quite-so-slow values and see how fast you can make it before the funny characters start appearing again. It could be that the end system simply can't process the input fast enough, although you'd think an 800MHz PPC CPU should handle it no problems!

Annihilannic
[small]tgmlify - code syntax highlighting for your tek-tips posts[/small]

 

[richwd]

I'm not sure where I got this, maybe I downloaded it or maybe someone gave me a copy but this is what I'm using: SSHSecureShellClient-3.2.9 on Windows.
The Linux box has: OpenSSH_5.1p1 Debian-5, OpenSSL 0.9.8m.

Expect 5.43.2
Tcl 8.5

# stty -a
speed 38400 baud; rows 25; columns 80; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = <undef>;
eol2 = <undef>; swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R;
werase = ^W; lnext = ^V; flush = ^U; min = 1; time = 0;
-parenb -parodd cs8 -hupcl -cstopb cread -clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl ixon -ixoff
-iuclc -ixany -imaxbel -iutf8
opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
isig icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt
echoctl echoke

# echo $TERM
vt100

locale didn't give anything.

Ya I'm confused about the terminal part. Is that some low level hardware that get's used no matter what the protocol? I thought it was just for serial too but I'm not a network expert.