Getting information via hyperlink - what can VFP do to avoid "dead" links? 5

[german12]

It is easy to get a Internet-page opened by VFP, however when you have a table full of internet-adresses it could be, that a link can not be reached because it does not exist in the net anymore.
Could it be to have a code snippet in VFP which avoids to open such dead links - perhaps with a message for the user, and of course with automatic deletion of that records which are wortheless?

I have no idea...

Klaus



Peace worldwide - it starts here...

 

[Mike Lewis]

There is no way of recognising a dead link just by examining the URL. You would have to actually navigate to the URL and then compare the resulting page with what you were expecting.

For example, if your link table stores the page title, then you can check that against the contents of the <title> tag in the returned page (which would typically contain something like "Invalid address" or "Error 404"). But that wouldn't be completely reliable: the title might have changed, or you might have not recorded it correctly in the first place.

If and when you have detected the dead link, the rest of the operation is straightforward. The message to the user and the deletion of the record within the table would be normal VFP, and completely within your own control.

Mike



__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[GriffMG]

I think deleting the record is probably the wrong approach, partly because sometimes links are 'off-line' for periods (maintenance, service interruptions).
Better to have a retry after an hour or so, repeated a few times before the record is marked as 'off-line' and a user notified.

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you. ​

 

[Mike Lewis]

In the absence of any way of doing this reliably in VFP, you might consider using a dead-link detection utility or web-based service, of which there are many in existence. Just search for something like "dead link checker" or "404 detection".

In the case of a web-based tool, it should be possible to automate it from within VFP.

But take note of Griff's point about sites going temporarily off line.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[Olaf Doschke]

To summarize what Mike and Griff said: You only know once you tried and getting no response can just be a temporary problem. And to add more information: It may not even be a problem there but a problem on resolving a name with an NS lookup from a DNS Server. So not even doing this first step yourself, an nslookup (see shell command nslookup) will give you a definitive answer. It would only give you the IP address (or not) for the domain, not for a specific page there or a download or whatever the URL requests.

Besides all that resources may come from the cache and sometimes non-working versions do come from not well-written sites in terms of invalidating caches, so your cached page version loads a cached Javascript framework version making requests to non-existing URLs and even fails to update itself.

Whether a specific URL (eg a product link, an article link) is still working is needing a page request anyway, and then it can also be cache or a failing content network provider for a URL to fail. Your minimum request can be an HTTP HEAD request, but when you ask because trying to navigate has a long timeout: Well, a HEAD request has the same timeout. There is no more direct and instant way to see whether a URL works or not. Well, you can take a timeout in your hands by making an asynchronous request, set a timer to judge the URL as (currently) dead after YOUR timeout, but then set this too low and you'll also not see a healthy page coming back.

Bye, Olaf.

Olaf Doschke Software Engineering

https://www.doschke.name

 

[Mike Lewis]

Here's something that may help.

As explained earlier, you might check a URL (using one of the tools that I mentioned above), but find that it either returns an error or it times out. That doesn't tell you whether the page in question is really a dead link or whether the server is down.

There is a site called "Is it down right now" that will answer that question. You can automate it within VFP by navigating to

https://www.isitdownrightnow.com/

followed by the domain name, for example:

[pre]

https://www.isitdownrightnow.com/www.tek-tips.com

[/pre]

You can then scrape the returned page (within VFP) to see if the server is up or down.

It's sounds a bit cumbersome, but you should be able to automate the whole thing, which is your aim.

Mike


__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[Olaf Doschke]

One more topic is the http ststus codes. You'll usually get 200 (OK) for a working URL, 404 (not found) is another very famous status code, but there are lots more.

https://de.wikipedia.org/wiki/HTTP-Statuscode

A good site would not just remove a resource and respond with 404 but would perhaps know what was available and has moves (3xx redirect) or really is removed and respond with 410 (Gone), which would be a sure sign to remove a URL.

These states are lost, when you let a browser or Webbrowser control handle the navigation, they won't feed back to you every detail about a routing, a URL bar of a browser also won't reflect redirects, at least not in all cases. the browser will have displays of 404 to 500 or other errors, if the site doesn't have an extra redirection for such cases.

So handling the level of HTTP yourself gets you a bit more information, but for example a request via an already higher level component like xmlhttprequest won't feed back redirects, it will follow them automatically.

Bye, Olaf.

Olaf Doschke Software Engineering

https://www.doschke.name

 

[Nigel Gomm]

Mike,

i like that !


but i'm needing ".html" at the end of the site name?

i.e. [pre]

https://www.isitdownrightnow.com/www.tek-tips.com.html

[/pre]

n

 

[Mike Lewis]

Nigel.

You're right. I don't know why I didn't spot that. But it does seem to be consistent, even with sites that don't have any .HTML pages.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[Mike Lewis]

Come to think of it, it would be very useful at this stage if Klaus could come back to the discussion, in particular to let us know exactly how he is currently following the links. That might help us with coming up with a solution.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[atlopes]

Klaus,

You can query for a web resource and look for whatever information the server, transport, and communication layers send back to you.

CLEAR

LOCAL HTTP AS WinHttp.WinHttpRequest

* read the page from the website
m.HTTP = CREATEOBJECT("WinHttp.WinHttpRequest.5.1")
* in a reasonable amount of time
m.HTTP.SetTimeouts(0, 10000, 30000, 30000)
* try to get it
m.HTTP.Open("Get", "[URL unfurl="true"]https://example.com",[/URL] .F.)
* signal "I'm a browser", just in case
m.HTTP.SetRequestHeader("User-Agent", "Mozilla/VFP")

LOCAL Result AS String
LOCAL Ops AS Exception

TRY
	m.HTTP.Send()

	DO CASE
	CASE m.HTTP.Status = 404
		m.Result = "Resource not found"

	CASE m.HTTP.Status >= 500
		m.Result = "Server not available"

	CASE m.HTTP.Status >= 400
		m.Result = "Access to resource not possible"

	OTHERWISE
		m.Result = "Resource available"
	ENDCASE

CATCH TO m.Ops

	m.Result = TEXTMERGE("General error: <<CHRTRAN(STREXTRACT(m.Ops.Details, 'WinHttpRequest:'), CHR(13), '')>>")

ENDTRY

? m.Result

(most of the code can be explained by replies above, in particular, by Olaf's second intervention).

 

[Olaf Doschke]

Well, almost,

it does not play a big role, but my site redirects all HTTP requests to https and that redirect is by HTTP status 301, but WinHttp.WinHttpRequest follows and gets the page.
Whereas sites like

https://apitester.com/

will show you this redirect step.

It's obviously not important for deciding dead vs working links but in detail, you could actually update a link list when the first response status is a permanent redirect, to, well change the URL to that new URL.

Bye, Olaf.


Olaf Doschke Software Engineering

https://www.doschke.name

 

[german12]

First of all - thanks to all of you guys for your help so far.
I referred to a code-snippet many years ago suggested by Olaf Doschke here

LOCAL loHyperlink,lUrl
STORE this.caption to lUrl
loHyperlink = CREATEOBJECT("hyperlink")
loHyperlink.navigateto(lUrl)

A table with a field myurl filled by many drags and drops from various internet-pages I found is the basis to populate the captions on commandon a form by a loop.
It works - but from time to time some links are not available anymore (page not found, code 404 and so on).

A solution could perhaps also be, that after a link is considered as "dead" to deactivate a record by an entry in a certain column in the table .....so that it is not lost forever.

Regards
Klaus






Peace worldwide - it starts here...

 

[Mike Lewis]

Klaus, I'm a little puzzled by your code. If I'm not mistaken, loHyperlink.navigateto(lUrl) will open the URL in your browser (and I think always in Internet Explorer, not your preferred browser, but that's not important). Opening it in a browser in this way does not allow you to do anything with it programmatically. Instead, the user will see the error page on your screen. So why would you want to display a message to the user to tell them what they can already see? And why would you then want some automatic way of deleting the relevant row from your table?

Apologies if I have misunderstood all this. But it might help if you could explain what you are aiming for in a little more detail.

Mike



__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[Olaf Doschke]

It must have been one of perhaps 4 times I ever mentioned the hyperlink class of VFP. Makes me wonder what else I told about it.
But I don't see it returning the info you say, as Mike say it just opens the IE. What good is it for testing a URL or getting captions for command buttons.

There are some FFC classes like the _hyperlinkcommandbutton. Did you use that? In the end that is based on the hyperlink class. and in hyperlinkbase is a validation of the format of a URL, ie that it begins with a proper protocol like HTTP: or ftp:, but not a check of the availability of the URL. I also see

I don't know, we must have talked at cross purposes and when you ended up even creating your own code based on hyperlink, this just does start IE And browse to a URL, so it adds hyperlinks to the userinterface of VFP without just enabling them in editboxes in texts, but as button captions, too.

Bye, Olaf.


Olaf Doschke Software Engineering

https://www.doschke.name

 

[german12]

Mike and Olaf: you wrote:
If I'm not mistaken, loHyperlink.navigateto(lUrl) will open the URL in your browser (and I think always in Internet Explorer, not your preferred browser,but that's not important
or
But I don't see it returning the info you say, as Mike say it just opens the IE.

May be not important, but FIREFOX is my favourite browser - and that browser opens here in a normal way the Internet-Page which were requested by me when I click any button in my form.
I am familiar with that browser and do not want to install the MS Internet-Explorer extra for that reason.

All I wanted was, not to look on a homepage which is not reachable (for what reason ever), but instead of that just have a short-time message on my form titled "no access".
which could disappear without having to be clicked away, and I could continue or try with other buttons/adresses, just to stay comfortable.

Sounds too lazy, but I am lazy.

Thanks again - I had so often very good advices here - and I am very pleased that this forum still exists, although I have resigned meanwhile.
Stay healthy!
Klaus





Peace worldwide - it starts here...

 

[Olaf Doschke]

Well, you now have code that could do the check from atlopez. I would replace "Get" with "Head" so you don't load the full HTML just to see the URL is up and working.
It's just that the hyperlink button will turn up a browser right away, so you'd have to reroute the click event to not do so immediately, only after the check.
It will just feel awkward, as even working links then will have a shortly longer lag time for the check.

Bye, Olaf.

Olaf Doschke Software Engineering

https://www.doschke.name

 

[german12]

I tried the code by atlopez - and it worked fine so far.
That is what I wanted.
Additionally I learned much around my question.
Thanks to all of you.
Klaus


Peace worldwide - it starts here...

 

[Olaf Doschke]

Thanks for the feedback and given stars. I was a bit puzzled when you said you resigned, I think you meant you retired, right? There's no reason to resign with a solution so close at hand.

I just see a lot of work if you want to do this by changing the buttons on your forms one by one, and a potential problem with UAC when you've installed to default program files and now would change class code.

First, let's clear one thing up: Regarding IE or other browsers, nobody said you must install or use IE. I mean, you're seeing Firefox start, so that's evidence enough you don't need IE. But it's also evident you don't just use the pure hyperlink class, which only uses IE. I think Mike and I both saw IE coming up, and I saw it coming up though it's not my standard browser and I bet it's not Mike's too. This is what the help topic on it says:

Note that the Hyperlink object is supported only in Microsoft Internet Explorer. Use the Visual FoxPro Hyperlink Button Foundation Class, Hyperlink Image Foundation Class, or Hyperlink Label Foundation Class foundation classes in the Using the Component Gallery for browser independent navigational capabilities.

So it becomes more evident taking into account your standard browser is added functionality in the FFC classes, not in the native hyperlink class. You actually must use the foundation classes, therefore. These "browser independent navigational capabilities" are done by determination of your standard browser from the registry and only automating IE, if you don't set another browser. Code is in _hyperlinkbase.navigateTo().

Now that you want to change the click behavior to not directly navigate but first check, there is the point to change. It's just potentially problematic to do so with UAC and file redirections and VirtualStore and VFPs file pairs and split of them one remaining in HOME(), the other redirected.

So, if you clear that up, before I'd recommend how to best tackle this, that'd be good. If you already solved your problem just disregard this.

Bye, Olaf.

Olaf Doschke Software Engineering

https://www.doschke.name

 

[german12]

Hi Olaf, thanks for the correction.
I am german as you(?)
Of course I wanted to write "retired" instead of "resigned" (in deutsch: Ich bin Rentner)

However am already 78 - and my English (school-knowledge only) is getting weaker now.
No - I do not resign too early.

My paintings in watercolor see here:

http://www.dawawas.de/erlebnis/aquarelle-von-klaus-briesemeister/code/tgb5p8n73t/u/87n6qj1ivqlj/

(no registration needed, just click on "Bilder ansehen")
and because I also play chess for many years one can learn not give up too early.

Again many thanks for your professional advices - I won't hesitate to ask here again when I have a problem.

There is no better forum than this one I think.

Up to 2002 I had so often got amazing answers here - and that saved me a lot of time, when I had to deliver forecasts and cost analyses in my company.
By the way; at that time we used also LOTUS 1-2-3 for calculation purposes. EXCEL came later. IBM was first.

Back then, IBM made a huge mistake in letting MICROSOFT develop the operating system for personal computers instead
to do it with their own resources and to integrate the Lotus Smart Suite, which was already very good (better than the early EXCEL applications) at the time, into it.

Schönen Gruß aus Lingen/Ems/Niedersachsen
Western Germany

Klaus




Peace worldwide - it starts here...