Problem reading data from internet 1

[tedsmith]

I am having problems reading data from a URL.
This data is details and GPS locations of a number of vehicles (Buses)
I can read it with Firefox opening with Notepad and see it is binary with some printable text.
If I open with Internet Transfer or Webbrowser I get a blank screen
If I open with IExplorer (ver 6 in my XP machine) I get the "Cant open" notice.

Is there a way to open it with Winsock and read to a stream in Vb6, erasing the stream as I read each vehicle? What port would I use?

Do I have to send it some code once connected to start it downloading?

This is the URL:

https://gtfsrt.api.translink.com.au/feed

This is the sender's description "The feed is an actual continuous stream which is in Protocol Buffer (Googles simplified XML platform) and has to be queried continuously"

I would have no trouble decoding and handling the data in vb6 once I could receive it.

I am not very familiar with HTML so any help would be appreciated.

 

[strongm]

Add a referencwe to the Microsoft WinHTTP Services library, then use this code as a starting point:

[blue]Option Explicit

Private Sub Command1_Click()
    Dim binaryresult() As Byte
    Dim stringresult As String
    binaryresult = GetDataFromURL("[URL unfurl="true"]https://gtfsrt.api.translink.com.au/feed")[/URL]
    stringresult = StrConv(binaryresult, vbUnicode)
    [green]' Do what you will with the data ...[/green]
End Sub

[green]' returns binary data[/green]
Function GetDataFromURL(strURL As String) As Byte()
    With New WinHttpRequest
        .Open "GET", strURL
        .Send [green]'send an HTTP request[/green]
        GetDataFromURL = .ResponseBody  [green]' get the binary data[/green]
    End With
End Function[/blue]

 

[tedsmith]

Thanks.
That works OK in my Windows 7 machine with reference added to winhttp.dll. Looks very neat and simple.
I also was able to read data using the wininet.dll This works in my Win7 machine using IExplorer8 but not in XP (which is not really a problem in this case)
It looks a bit cludgy having to assemble the string the way it does but it works at the same speed (limited by the internet anyway to .05 seconds.)

I first declared InternetOpen InternetOpenUrl InternetReadFile and InternetCloseHandle
Private Function GetHTMLFromURL(sUrl As String) As String
Dim s As String
Dim hOpen As Long
Dim hOpenUrl As Long
Dim bDoLoop As Boolean
Dim bRet As Boolean
Dim sReadBuffer As String * 2048
Dim lNumberOfBytesRead As Long
hOpen = InternetOpen(scUserAgent, INTERNET_OPEN_TYPE_PRECONFIG, vbNullString, vbNullString, 0)
hOpenUrl = InternetOpenUrl(hOpen, sUrl, vbNullString, 0, INTERNET_FLAG_RELOAD, 0)
bDoLoop = True
While bDoLoop
    sReadBuffer = vbNullString
    bRet = InternetReadFile(hOpenUrl, sReadBuffer, Len(sReadBuffer), lNumberOfBytesRead)
    s = s & Left$(sReadBuffer, lNumberOfBytesRead)
    If Not CBool(lNumberOfBytesRead) Then bDoLoop = False
Wend
If hOpenUrl <> 0 Then InternetCloseHandle (hOpenUrl)
If hOpen <> 0 Then InternetCloseHandle (hOpen)
GetHTMLFromURL = s
End Function

 

[tedsmith]

On Windows XP, with the GetDataFromUrl method I get an error message when the .send line tries to execute.

Run time error '-2147012739(80072f7d)
An error occurred in the secure channel support

This does not happen with Windows 7.

The Reference is the same version 5.1

Is it because XP has IE version 6 while my Win7 has version 8?
Is there anyway to make it run in XP?
The client's machine where it will run has Server2003 installed. Will it run in that I wonder?

 

[dilettante]

Unlike your wininet.dll approach above, WinHTTP is completely independent of Internet Explorer and its various component DLLs. It is a separate and lighter weight HTTP stack of its own.


Caution, wild guess:

Microsoft has patched a number of Schannel security holes recently along with replacing several of the SLL/TLS ciphers.

While Windows Server 2003 is approaching end of extended support (July 2015), the server in question should have been patched this month. However if this customer has disabled Windows Update they may not have the patches installed and thus may be using ciphers no longer supported by the server at the remote end.

One of these patches: Vulnerability in Schannel Could Allow Remote Code Execution (2992611)

Getting caught up on patches might do the trick, but it would be better to get off this nearly dead OS and onto something with a longer support horizon.

As for Windows XP, it is past dead and such machines have not been safe to connect to the Internet for many months now. Retiring these spam relay zombied machines is a big favor to the entire Internet community.

 

[tedsmith]

Unfortunately both versions do not work on my old Xp machine. Neither will Ie6 open or save the file. Only Mozilla will do this on the old machine
It is something to do with the request transmitted by my Ie6 which seems to be the same as the inet version. When the same code is executed in a Win7 machine (with Ie8 installed) both versions work.

Alternatively I wondered if there was any way of finding out the characters that need to be sent, it could be done using a winsock. That way it would be OS independent.

I am probably naive but I was trying to get the subject to the most basic level but can't find any info on that level in plain English that does that. Info seems to refer to higher level solutions using a language like C or python or whatever.

 

[strongm]

Try reregistering rsabase.dll

 

[strongm]

>Alternatively I wondered if there was any way of finding out the characters that need to be sent, it could be done using a winsock

Well, sure. You could write your own HTTP user agent. But that isn't quite as straightforward as it sounds. And then you'd have to write an SSL/TLS layer in order to establish an encryption tunnel between your agent and the host (you may note they are delivering their data over HTTPS). It's really quite a lot of work, writing it all from 'the most basic level'. It's certainly not something I'd try and do.

 

[tedsmith]

Strongm you are a genius.
My computer didn't even have rsabase.dll in it so I downloaded it. (hopefully the genuine one)

So Internet transfer control works OK however WinHTTP give an error on send "an error occurred in the secure channel support" -2147012739(80072f7d)

 

[tedsmith]

I experimented with using Winsock by adding ,80 and get a response OK in the form of a small text notice but no data. I tried an old example for HTTP1 but of course got no data either.
Is there a simple code that enables the encryption I should be sending to receive HTTPS data?
Where would I find out what code is required or is that not available to the public?

 

[tedsmith]

2hours later -
I am SO frustrated! My test of Inet now does not work again!
I know it definitely did because I still have the file it saved!
I wondered if perhaps I was really reading my cache instead of the real file?

 

[dilettante]

If it is a failing hard drive I'd expect you to see many other issues. Or perhaps you had a shutdown error? That leaves malware as the next likely suspect.

Files like that don't just randomly disappear.

 

[strongm]

>Is there a simple code that enables the encryption

Well ... yes - telling an HTTP user agent (in my example that would be winHTTP) that the protocol is HTTPS, which is just HTTP tunnelled through an SSL/TLS encrypted tunnel.

Drop the HTTP user agent and you are pretty much on your own - you pretty much have to start here:

http://tools.ietf.org/html/rfc5246

and

https://tools.ietf.org/html/rfc6101.

Some additional related links can be found here:

http://www.rfc-base.org/rfc-6101.html

(as you can see there's no simple 'give me an encrypted tunnel, please' option - that's what the HTML user agent provides)

You might want to look at the OpenSSL Toolkit project, which provides an SDK to implement "the Secure Sockets Layer (SSL v2/v3) and Transport Layer Security (TLS) protocols". There is a Windows release available. Not designed for use from VB, though (and, although based on Open Source code, there is a commercial license for this release. An older completely free version is available here) All this is pretty hard-core stuff; good luck. I can't guarantee you'll ever get it working. I've never tried ...

 

[tedsmith]

Thanks, I'll look at it later (or maybe give up!)

It wasn't any of my files that disappeared, it is my app that no longer will read the data. The file I have is the file I made using my app which proves it did "work" originally.

 

[strongm]

I suppose we can always try another HTTP user agent (but, like dilettante, I rather suspect that it is XP that is the problem). Add a reference to Microsoft XML library (latest version on your XP machine should be 6). Then this following modification:

[blue][green]' returns binary data[/green]
Function GetDataFromURL(strURL As String) As Byte()
    With New [b][COLOR=#EF2929]XMLHTTP60[/color][/b] [green]' could also try ServerXMLHTTP60[/green]
        .Open "GET", strURL
        .Send 'send an HTTP request
        GetDataFromURL = .ResponseBody  ' get the binary data
    End With
End Function[/blue]

 

[dilettante]

The XMLHTTPxx classes are wrappers on the IE-related WinInet stack. ServerXMLHTTPxx classes wrap WinHTTP.

 

[mattKnight]

Has anyone considered that this is fallout from POODLE

i.e translink.com.au have turned off SSLv3

Take Care

Matt
I have always wished that my computer would be as easy to use as my telephone.
My wish has come true. I no longer know how to use my telephone.

 

[tedsmith]

Forgive me but can you explain POODLE and why it still works on machines with ie8 installed?

 

[mattKnight]

You've not heard of poodle?

http://en.wikipedia.org/wiki/POODLE

In short, A vulnerability was found in SSL V3.0 which potentially allows the encryption to be broken. SSLv3 is quite old and has been superceded so most system administrators have remeditated the security risk by turning off / preventing SSLv3 encryption at the server.

I think the main people who this affects are XP users, because XP does not support the newer encryption protocols. for example XP only supported certain certificate types from SP3.

hence, it is entirely independent of browser,that is IE8 would connet if the OS is W7, but not if the OS is XP

Honest, time to ditch XP!

Take Care

Matt
I have always wished that my computer would be as easy to use as my telephone.
My wish has come true. I no longer know how to use my telephone.

 

[strongm]

Ah yes. Good point.

>It is something to do with the request transmitted by my Ie6

No, I think it is something to do with what is returned by the webserver, and how IE6 responds to that. Specifically a pragma controlling caching. You can bypass this by adding a registry setting:

[li]Start Registry Editor.[/li]
[li]For a per-user setting, locate the following registry key:[/li]HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings

[li]For a per-computer setting, locate the following registry key:[/li]HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings

[li]On the Edit menu, click New>DWORD Value and add "BypassSSLNoCacheCheck" with a value of 1[/li]