BatchFile Output redirection > with Wscript.shell

[stanlyn]

Hi,

I'm having an issue running a batch file from Wscript.shell that has a redirector ">" to a file within. The batch file works perfectly if run from Explorer by double clicking it or command line with cmd. When running it via a VFP program, it creates an empty file (0kb). Here is the batch file contents

whois.exe  -v  yahoo.com >C:\temp\output.txt

Using VFP's run command produces the same result.

Also, breaking the redirect parameter out of the .bat file and passing it as an external parameter also produces the same result.

The only way that I've been able to get a non-empty output.txt file is outside of VFP at the command line or Explorer.

Environment is Windows 10pro64 and VFP9sp2.

Any ideas?

Thanks, Stanley

 

[Mike Lewis]

Stanley,

I'm not sure how to do this with wscript.shell, but with the raw ShellExecute() function, you can hide the target application's window (the black Dos window in this case) by passing zero as the last parameter. This parameter determines the window state (minimised, maximised, etc) of the window, with 0 being invisible.

I assume that there is something similar in wscript.shell.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[Mike Lewis]

As I mentioned above, I am not very familiar with wscript.shell, but I see that, as well as the Exec method, it also has a Run method that might meet your needs. This supports both a window state parameter and a WaitOnReturn parameter.

So, I have understood this right, you could so something like this:

oWScript = CreateObject('Wscript.Shell')
oWscript.Run(<your command goes here>, 0, .T.)

The 0 means that the window (the black Dos box) will be hidden, and the .T. says to wait until execution finishes before returning control to your application.

Perhaps Chris can confirm if I have understood this right.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[atlopes]

Stanley

Without going off-topic, wouldn't you consider using a Whois REST API to get rid of WScript or ShellExecute challenges, together with the dependency on the availability of an external command?

 

[Chris Miller]

Mike,

unfortunately only exec allows you to get a process object that has StdOut.

Stanlyn,

I don't see how FileToStr so bad, when whois queries can sometimes take half a minute. Even a fast whois call takes longer than reading the output file.
As you can use different names per call by using Sys(2015), for example, or simply numbering them, you can also avoid parallel processing issues.

To use stdout otherwise needs to use Windows API function CreateProcess with a STARTINFO structure that has handles for stdin stdout you have to create yourself, which has other prerequisites.

So in short, it's complicated.

I think atlopes has read the text coming back from sysinternals whois, that points out usage of a whois service from

https://www.markmonitor.com/

https://domains.markmonitor.com/domains/setup/restapi/

It offers whois at

https://domains.markmonitor.com/domains/restapi/v2/whois/www.google.com,

but this only works after logging in and having a user session, so an account with markmonitor.

Chriss

 

[Mike Lewis]

unfortunately only exec allows you to get a process object that has StdOut.

I hadn't realised that, Chris. Thanks.

But, Stanley, I still think you could use FILETOSTR() to get the output into a variable. It would surely only take a tiny fraction of the time that that WhoIs takes. And you wouldn't have to worry about pausing the process after running the batch file, nor about using StdOut.

Mike



__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[Chris Miller]

I have gotten it to bypass the whole writing to a file, filetostr() and strtofile() methods as they are too costly in i/o.

Mike, I think filetostr really isn't a big I/O issue, too. If you're concerned about that a RAM disk could also help.
But maybe turning to a web service for something web related is even better, and whois is very web based, isn't it?

In the end using createprocess with the structure and stdin/out handles and all related to it will be much code to replace a few lines you'd have with an XMLHTTPrequest, which by default won't have any UI.

And Stanlyn, to answer the RUN question: No, RUN starts the commandline you give it and goes off, you don't even get the confirmation of it working. ShellExecute returns a value indicating that, at least. CreateProcess returns a process handle that enables a lot more to do, like WaitForSingleObject to wait for completion of the process.

Chriss

 

[stanlyn]

Hi,

Looks like converting it back to my previous "run" with shellexecute() and > output is in order as I had it working including a variable wait time by trying to get a lock on the file. If unsuccessful getting the lock, I wait 2 seconds and try again until I get the lock which tells me the whois process has finished, and ok to continue processing the results.

I just finished a 6 hour run where the process did 7485 requests before hanging. So now, I need to get a timeout built into it to wait a max time before aborting the current domain query and continuing with others.

This is the cleanest, but takes focus from all other apps when the dos box appears making it unusable.

oWScript = CreateObject('Wscript.Shell')
zz = 'whois -v ' + lcDomainName
oWhoisProcess = oWScript.Exec(zz)
lcOutput = oWhoisProcess.Stdout.ReadAll() 

If .not. EMPTY(ALLTRIM(lcOutput))
lcData = lcOutput
....

This is my previous code using shellexecute() which is slower...

Declare Integer ShellExecute In SHELL32.Dll ;
	INTEGER handle,;
	STRING @sFile,;
	STRING @lp,;
	STRING @Dir,;
	STRING @dir1,;
	INTEGER ncmd

lcWhoIsStr = "C:\Users\SlaAdmin\Desktop\DomStatus\Apps\whois.exe -v "+ Alltrim(lcDomainName) + " > C:\temp\output.txt"
Strtofile(lcWhoIsStr, 'C:\temp\getdomain.bat')

llAbort = .F.
lnTimeout = 0
lnError = ShellExecute(0, "open", "c:\temp\getdomain.bat", "", "", 0)
* lnProcessID = ?????????? process id of the executed whois command the just ran
lnHandle = Fopen("c:\temp\output.txt",12)

Do While lnHandle = -1
   If lnTimeout < 30
      =Inkey(2)
      lnTimeout = lnTimeout + 2
      lnHandle = Fopen("c:\temp\output.txt",12)
   Else 
      && ??? ABORT by killing lnProcessID
      llAbort = .T.
      exit
   Endif
Enddo

if llAbort = .T.
    skip 1 in 'domainlist'
    llAbort = .F.
    loop
endif

Fclose(lnHandle)
lnTimeout = 0
... continue

I need to get the process id of the command that is running the whois command so I can kill the process when timeout occurs. See the ???? above for suggestions on how to handle.

Thanks, Stanley

 

[Chris Miller]

No, you only get a process handle from CreateeProcess.

You'd need to query all running processes and see whether they are run by you, for example identified by their commandline.

Chriss

 

[Chris Miller]

Since you're now finally revealing you do this for a batch of domains, look here:

https://whois.whoisxmlapi.com/bulk-whois-lookup

Chriss

 

[Chris Miller]

And one more detail: I see whois has the usage syntax

Usage: whois [-v] domainname [whois.server]
-v: Print whois information for referrals
-nobanner: Do not display the startup banner and copyright message.

So you can also change the server that is used for the whois. And I find a list of such servers here:

https://www.nirsoft.net/whois_servers_list.html

I think you could get more stable results from using multiple servers instead of fixing an issue you can avoid that way.

Chriss

 

[stanlyn]

Chriss, I am aware of the option of applying different tld and whois servers to the queries.

Thanks Chriss for the link to

https://whois.whoisxmlapi.com/bulk-whois-lookup.

Unfortunately, they are not affordable due to the volume as well as not being able to programmatically re-query based on some special criteria.

Also thanks for showing the -nobanner option. I've seen it a lot of times, but never registered with me until now.

You'd need to query all running processes and see whether they are run by you, for example identified by their commandline.

I suppose one could query the resource process list for "whois.exe" and delete it from it by id. Will search google for whats involved and how-to from within vfp.

Thanks, Stanley

 

[Chris Miller]

process list for "whois.exe"

Well, if you run multiple whois.exe in a batch you better know which one is hanging. The commandline will tell you, it's one column/property of the process list you can get:

*EDIT: missed to create loWMIService first:
loWMIService = GetObject("winmgmts:\\.\root\CIMV2")
RUN /N0 whois.exe -v google.com >whois.google.txt 
RUN /N0 whois.exe -v yahoo.com >whois.yahoo.txt 
RUN /N0 whois.exe -v bing.com >whois.bing.txt 
loProcesses = loWMIService.ExecQuery("Select * From Win32_Process Where name LIKE '%whois%'")
For Each loProcess IN loProcesses
   ? loProcess.Commandline
Endfor

A Process object also has a Terminate method, but a little caution, if the process terminates itself while you iterate the process list, the method will fail. So as you expect the processes to finish by themselves, every access also even to the name or commandline should be done with TRY to catch any errors, which then indicate the process finished normal.

Chriss

 

[stanlyn]

Chriss,
I like the/your idea of batching a 1000 whois "run" requests with a nowait. Then stop and gather the results and stuff the tables with their data and delete the files. My current serial, (one after the other) process is working as intended, but too slow. Its doing about 120-150 per minute, and I have millions to go.

New batch way, I'll move the data to sql (so others at different ip addresses can help process to help with the bandwidth rate limits) which creates the record in the domain table and use it's pk as the name for the output file. This allows getting the data where it belongs.

Questions...
1. You see any issues with running 1000 whois requests all at once?
2. Would you expect a large performance boost?
3. Any thoughts or considerations I need to consider?

Thanks, Stanley

 

[Chris Miller]

I wouldn't run 1000, though it's mainly the web - the server you request - that does most of the work. I would let the number grow until the average response time sinks, I think then you're at the best load.
Yes, the parallel processing (not multithreading) is the easiest to increase the throughput.

Think about bottlenecks.

I guess your internet connection is high speed, maybe working from a domain or in the cloud even better than any home DSL or cable connection. It's more likely how many processes burden the harddrive used to save all the outputs. Maybe the number of paralle stdout pipes (thats what stdout is, a pipe), in the end also a file system limit.

I guess the turnover time from start to finish will be a good measure for a healthy state, as long as that's not going up with further parallel processes you're good to go for more. Just remember a server still also has to do other tasks and stay responsive to them, too. So perhaps have a test for how many processes bring the performance down and then choose only 50% of that to have 50% resources left for everything else.

It's not easy as whois queries can take quite long, so one such query highers the average although it's not a big local burden. I have no best idea, but I would perhaps try to figure out when the output file is created and how and when it grows from it's initially empty state to the final size. That results in a write speed measurement and the moment that breaks down means you're burdening the file system with too many parallel writes.

Surely it's worth considering raid and or multiple drives to have better parallel writing of the outputs.

And in this range of millions that's also speaking for rather making use of stdout and dig into createprocess details about that.


Chriss

 

[stanlyn]

Hi Chriss,

The run command is not behaving as expected...

SET STEP ON 

m.lcDomainName = 'yahoo.com'
lnPK = 30000
lcOF = " > C:\temp\" + ALLTRIM(STR(lnPK ,8,0)) + ".txt"

m.lcWhoIsStr = "C:\Users\SlaAdmin\Desktop\DomStatus\Apps\whois.exe -v -nobanner " + Alltrim(m.lcDomainName) + lcOF
RUN /N0 &lcWhoIsStr

RUN /N0-4 will not create the output file while RUN without the /N switch does.
Any idea?

Thanks

 

[stanlyn]

Also,
Using macro substitution doesn't work either, like...

m.lcWhoIsStr = "run /n C:\Users\SlaAdmin\Desktop\DomStatus\Apps\whois.exe -v -nobanner " + Alltrim(m.lcDomainName) + lcOF
&lcWhoIsStr

 

[Rick C. Hodgin]

In the past before we switched to 7-Zip's open source 7z.dll implementation, we used the command line utility 7za.exe. To capture the output I wrote a DLL in C++ which would create the STDIO and STDERR handles and launch 7za.exe connected directly to those (using Win32 calls). I would then enter a loop while the app was running and constantly read from both handles and send any data they emitted the data back to the function that launched the external app using a BINDEVENT(thisForm.HWnd, message...) feature with a message indicating how many bytes were ready to send. It would then call back into the DLL to retrieve the content, then return.

This method allowed us to provide real-time feedback as some of the ZIP operations would take a minute or two, and it was able to give the user some idea of activity.

If you'd like an interface that can launch, monitor, and provide real-time capture of any output to STDIO, let me know. I'll extract that code into a simple DLL and make it available.

--
Rick C. Hodgin

 

[Chris Miller]

You could do

lnPK = 30000
lcOF = " >C:\temp\" + ALLTRIM(STR(m.lnPK ,8,0)) + ".txt"
lcDomainName = 'yahoo.com'
lcWhoIsStr = "C:\Programming\whois.exe -v -nobanner " + Alltrim(m.lcDomainName) + m.lcOF
RUN /N0 cmd.exe /c &lcWhoIsStr

Or go Rick's route. You don't need an extra DLL, you need to establish a console session as vfp9.exe isn't one, it's a windows (GUI) process. Just like any exe you start with RUN /N isn't a console. Unless its cmd.exe, which explicitly establishes a console session. That's how the above solution works, this starts a cmd.exe for each start of a whois.exe, though.

To be able to establish stdin and stdout and then use them with createprocess, you have to do more than just establish CreateProcess with a STARTINFO struct that has stdin and sdout handles. You also need to establish a console that can establish stdin and stdout handles. And while all that is quite complicated, I just found an example doing all this:

https://github.com/VFPX/Win32API/blob/master/samples/sample_477.md

This does what I do using cmd.exe, too but with control about stdin and stdout that only uses something like one cmd.exe process to run the commands in it. So I wonder how many things it does in parallel, maybe none, ie all whois calls are sequentially done.

Chriss

 

[stanlyn]

Rick,
Thanks for the reply and offer. You are way over my head as I know nothing about C++ and calling their dlls. Also have never used bindevent before, but am aware of it. I still may have to start learning it if I cannot get this fully solved.

Chriss,
Adding the cmd.exe to the .bat file fixed most of the issues. Performance is almost 10 times better now as 1000 lookups and processing takes about 1.4 minutes whereas before it was taking about 2 seconds per row, or 120 rows per minute.

I would be good with this if I had a few issues fixed...
1. This is a really bad one, (show stopper)... All tests were done by first zapping the domains table so everything starts new. The source table is also in index order and always starts at the top.

The system runs perfectly processing 500 records at a time (they are running in parallel) until about 15000 rows are done, then it all stops and the machine hangs up and is entirely unresponsive, then hit the reset switch. Now, zero everything out again and start over, and this time it gets to 18500 rows and it hangs again. I changed the 500 batch size to 100 and it still hangs. It hangs at different places (row count) as I first thought it was something with the data, but I've ruled that out. I also ran it once with resource monitor and task manager's performance tab in focus on one of the monitors. CPU runs between 45-75%, memory at 50%, and processes are going and then back down thru the entire run and nothing hits any limits. It is what you would expect to see... BTW, I also ran rests using 1000 per batch and also did not max anything. Performance seams better at about 500. Also, nothing revealing in the event viewer.

2. I had to disable testing of whether a file can be opened because in all modes the handle returned was always -1 when using fopen() in the pause section. So in order to see some test results, I disabled it completely, until I figure out why a -1. The files are not opened in any other process as it runs fine without checking if the file can be opened. Assume for a moment that VFP is having an issue with deleting a file that is opened already, it would generate an error, and NOT kill the entire system.

Below are some screenshots.

The speed I've gotten so far will make the job doable as it will take about 33 hours to do a million lookups. I will be moving it to sql server with as many w10 clients doing the work.

Any ideal on how to deal with two issues?

Thanks, Stanley

 

[Chris Miller]

The simplest solutiojn is exit and restart. You can try to fiddle around with SYS(1104) and do garbage collection.

But unloading the whole EXE and runtime is the easiest way to get back to a clean state and restart after doing a batch of say 5000.
If you start this with task manager you can set it up to kill the process when it runslonger than say 10 minutes.

So give the EXE a starting parameter with which record to begin or let it locate the first record that has a status field "unprocessed" or lastprocessdate < DATE() or anything like that. Only update that if you have the result whois file and verified it is a valid new complete whois response and no error message or whatever else could come back.

I would personally also split the two tasks to run the whois requests and to receive the responses in two exes. Afteer you did a batch of RUN /N0 your process can quit. Another process only needs to knwo which directory to watch for incoming files.

And I wouldn't run 500 in parallel, I'd orient myself at number of cores, though as I said earlier most part of the job is done by the servers you use, you mostly wait. But 500 processes are a lot in parallel, no matter if the mostly wait. And the whois.exe will mostly wait, but still be a processid each.

You'll better know how that effects the overall turnaround, but I'd guess the lower number of processes mean after the initial lag time of the firs reponses coming back you'll keep the number ofprocesses up at the same number and will have the same overall throughput. You'll teach me otherwise.

Chriss