Reading web-based log with XMLHTTP 2

[herbstgy]

Greetings, Everyone...

I'm new to Visual Foxpro, still learning the ropes, please help me out.
The environment is VFP 9 SP 2 on Win7.
I need to read a web-browser based log, and insert it into a database.

I was able to open the webpage from VFP, and read it (so far so good )
by this code:
(I've found it here, guys... most of my VFP knowledge comes from this place )

lcReqstr="http..." && (the opening url comes here. I wouldn't bother you with it, it's more than 500 bytes long with the parameters)
lcRespstr=""

loreq=CREATEOBJECT("MSXML2.XMLHTTP.6.0")
loreq.open("GET",lcReqstr,.t.)
loreq.send()
DO WHILE loreq.readystate<>4
	DOEVENTS
ENDDO
lcRespstr=loreq.responsetext

* ...processing responsetext

My trouble is the following:
If I open this web based log reader in a browser, it dumps the arriving messages continously.

When I try to read it through XMLHTTP, it waits up until 3000 messages arrive together and only dumps it then. If I read earlier logs, they arrive in reasonable time. If I'd read realtime log, it waits 3000 messages together, which is unacceptable with realtime watching.
Until all 3000 messages arrive, loreq.readystate property stuck in 3 (interactive) and loreq.responsetext is empty.
Is there a way to break or pause or put on hold the reading? I thought I should read the log in every 5-10 seconds or in every 50-100 pieces in a cycle and process it.
When I use the Abort method, it breaks the reading, but throws away everything and responsetext property stays empty.
How could I read this log piece by piece as I need it?

 

[JRB-Bldr]

Welcome to the forum.

Generally speaking, it is up to the Web Service side to control how much information to make available to the VFP query.
Not always, but quite often this is done via one of the parameters that is included within the URL.

Your VFP code will just get all the information that is made available to it by the Web Service call.
You most like already know this, but you can check what the Web Service returns by manually entering the URL into your Browser and viewing the results.

How could I read this log piece by piece as I need it

I don't know if one or more of the parameters that you send in your URL can control the amount of information presented, but if not, then you might want to contact the Web Service developers and see if they can modify something on their side to support your needs.

Good Luck,
JRB-Bldr

 

[mplaza]

Using WinHttp you can have access to data as soon as it is available:

Clear
owinhttp  = Create('winhttp.winhttprequest.5.1')
owinhttph = Create("winhttpH")
Eventhandler(owinhttp,owinhttph)

? 'downloading content...'

url = '[URL unfurl="true"]http://www.tek-tips.com'[/URL]


Try

	With owinhttp
		.setTimeOuts(2000,30000,30000,60000)
		.Open('GET',url,.F.)
		.setrequestheader('Connection','keep-alive')
		.Send()
		strtofile(owinhttp.responseText,'test.txt')
	endwith

	modify file 'test.txt'

Catch

	Aerror(ae)
	If ae(1,1) # 1429
		Display Memory Like ae
	Endif

Endtry


*****************************************************
Define Class winhttph As Session OlePublic
*****************************************************

	Implements iwinhttprequestevents In winhttp.winhttprequest.5.1
	bytesrec = 0
	cData = ''


*-------------------------------------------------------------------------------------
Procedure iwinhttprequestevents_onresponsedataavailable(Data As variant) As void
*-------------------------------------------------------------------------------------

	This.bytesrec = This.bytesrec+Len(m.Data)
	This.cData = This.cData+m.Data

	? 'Read ',len(m.data),' bytes Total Bytes received:',This.bytesrec, left( strconv(m.data,1),50),'...'

	* put your break condition here ( bytes read, time out, chunks read etc )

*--------------------------------------------------------------------
Procedure iwinhttprequestevents_onresponsestart(Status As Number, contenttype As String) As void
*--------------------------------------------------------------------


*--------------------------------------------------------------------
Procedure iwinhttprequestevents_onerror(errornumber As Number, errordescription As String) As void
*--------------------------------------------------------------------
if errorNumber # -2147024890
	? 'ERROR',errornumber,errordescription,'url:'+url
endif

*--------------------------------------------------------------------
Procedure iwinhttprequestevents_onresponsefinished() As void
*--------------------------------------------------------------------

********************************************
Enddefine
********************************************

Marco Plaza
@vfp2nofox

 

[Olaf Doschke]

Marco has the solution.

I just want to add what you would need to use and why that's not an alternative to VFP:

With MSXML2.XMLHTTP.6.0 you would need to be able to use the ResponseStream, but VFP isn't capable of doing so because it exposes the IStream Interface, but not the IDispatch interface. In VFP you can only use OLE objects and COM Servers implementing the IDispatch interface. So you found one limitation of VFP.

In regard to streams, Calvin Hsia once wrote a blog article about its usage via a DLL he wrote for usage in conjunction with GDI+ streams:

https://blogs.msdn.microsoft.com/calvin_hsia/2006/02/17/use-an-istream-object-to-avoid-disk-access/

This DLL is giving you a VFPStream.Cstream COM Server, which itself can handle a stream. So, finally, streams for VFP. Well, only half ways. You may give it a try, but as far as I see the VFPStream.Cstream is only capable to work on streams it creates itself. You still have the major problem VFPs access to the ResponseStream is not existent, you don't even get something like a handle to that MSXML2.XMLHTTP ResponseStream you could forward to VFPStream.Cstream. So you would need another helper DLL at least or an extension to Calvin Hsias DLL. It would perhaps be interesting to get his code and extend it.

But for this specific case, it's much simpler to take Marcos solution. There are very many ways to make HTTP requests and handle their responses. If one of the OS doesn't work directly, I'd look into two solutions being bindings to libcurl, one from Carlos Alloatti, one from Craig S. Boyd.

Once again

http://www.ctl32.com.ar/libcurl.asp

is down, but

https://www.sweetpotatosoftware.com...ction-update-http-post-ftp-commands-and-more/

is available and you find HTTPGet with two callback possibilities one to only display progress giving you total bytes and bytes so far as just the numbers so you can compute the percentage of progress and a trace callback giving you actual raw bytes received, also letting you know what type of data was received (HTTP response headers vs response body, for example).

Bye, Olaf.

 

[herbstgy]

thank you guys for the answers.

@Marco,

I see you used a completely different approach... and it WAY beyond my knowledge.
Unfortunately I know nothing about event handling and creating...
I tried your example, and it work as it should, and I also see the comment where should I put the break condition, but I don't know what to write.
I tried

If this.bytesrec > 100000

return​

endif

but it didn't work.

how can i return from the onresponsedataavailable event?

 

[Olaf Doschke]

You don't need to break. This event happens, you get data in whatever chunks winhttp.winhttprequest.5.1 offers in its onresponsedataavailable event and this code is just binding (subscribing) to the same event.

Do you want to stop receiving after 100000 bytes? Then you'd need to act with owinhttp and call its Abort() method to stop receiving.

But if the URL you request is an endless stream (streaming API endpoints sometimes are), the connection is set to be kept alive via .setrequestheader('Connection','keep-alive'), there's nothing todo, these events will continue to happen and you react to new bytes coming in via the Data parameter.

The thing you will want to change perhaps, is the Open() call making an asynchronous request, like this:

Clear
owinhttp  = Create('winhttp.winhttprequest.5.1')
owinhttph = Create("winhttpH")
Eventhandler(owinhttp,owinhttph)

? 'downloading content...'

url = '[URL unfurl="true"]http://www.tek-tips.com'[/URL]


Try

   With owinhttp
      .setTimeOuts(2000,30000,30000,60000)
      .Open('GET',url, [highlight #FCE94F].T.) && async[/highlight]
      .setrequestheader('Connection','keep-alive')
      .Send()
   endwith

Catch

   Aerror(ae)
   If ae(1,1) # 1429
      Display Memory Like ae
   Endif

Endtry

? "Waiting for Events"
Read Events

Now, if you want to return from this even before the first packet of data arrives, you can do so instead of Read Events, but you need to keep the variable owinhttp and owinhttph alive, eg within _screen properties.

Bye, Olaf.

 

[mplaza]

Hi, glad it worked. As Olaf already pointed out,
you should use the winhttp abort method:

If this.bytesrec > 100000
 this.abort()
endif

that will end the coonection and your code will resume after the "send()" command.





Marco Plaza
@vfp2nofox

 

[Olaf Doschke]

OK, I see what you aimed for here, Marco.

@herbstgy: As I hinted already you can start the request asynchronously and return to your normal code instead of having a READ EVENTS here.
The abort might not be necessary, you could have a constant stream of data arriving. Marcos code does accumulate it in a cData property via This.cData = This.cData+m.Data and so aftera certain threshold like 100000 bytes are received but better yet after it contains a closing HTML tag or closing XML tag of a message you want to process, you simply callback into a method for processing the data and strip off the processed data from cData, waiting for further data.

Typically async methods have a callback method parameter, such calls are usually not usable from VFP, as they need a pointer to the address of a callback method, but in this case you have your callback in the iwinhttprequestevents_onresponsedataavailable method and can simply call into whatever code you want from that point. You turn the logic around, you don't wait for the response in the code using this construct, you let this construct call into a method expecting the response data as its parameter. Otherwise and as Marco has done it, you're not making use of the asynchronous nature of the request and you're wasting time with waiting, while your application could already be back into waiting for any other events like clicks or key presses to do whatever it does besides waiting for the web request response.

Bye, Olaf.

 

[herbstgy]

Gentlemen, Thank You for all your help!

It works!

Slowly I start to understand how it works...
As you all said, eventually it didn't needed to use the abort method. I just process the incoming data inside the onresponsedataavailable event, and everything comes just fine...

@Olaf,

it doesn't need to break the stream exactly at 100000 bytes, it was just a test to see if I can stop it at all... the whole stream (3000 messages in one) is around 8 megabytes heavily javascripted html table.
Unfortunately I can't use the asynch mode. It would be easier to control the program flow, as it returns the program control right after the Send method, but it scrambles the rows in the log. You can imagine my face when I saw that the program at last works fine, but messes up the order of the lines.

 

[Olaf Doschke]

> but it scrambles the rows in the log
Well, that's something perhaps needing addressing of how you keep objects in scope (alive) and how you receive data and concatenate it.
I don't think the response will come in random order.

Databytes will surely not come in exactly cut off at end tags, so you have to pull out HTML/XML tags as you need, but keep the rests, which are already the start of next items.

Bye, Olaf.

 

[mplaza]

Hi herbstgy, glad it worked as expected.

Take Olaf's advice in case you want to use the async mode.




Marco Plaza
@vfp2nofox