VFP and Google Web service 1

[johann59]

I need some advice. I'm using VFP 9. I have a list that contains US addresses. Want to be able somehow to run those addresses through a Google search and have Google return the correct addresses. Most of the time those addresses have misspellings. Examples:

a) 123 Any Steet, Anytown CA 90000 (It should return 123 Any Street, Anytown CA 90000)

b) 123 E Califonia Blvd, Anaheim CA 92801 (It should return 123 E California Blvd, Anaheim CA 92801)

I do this right now manually. For example, I enter the "123 Any Steet CA 90000" in Google, and if such an address does exist, it will return the correct address. It's not a problem when I have 10 or 20 addresses, but when I have hundreds of addresses it becomes an issue.

So my question is:

First, is this possible using VFP? Is there such a Web service that I can use with VFP? And if so, what would be the best way to go about it? A form with a grid, a standalone prg? I need VFP to somehow run all the addresses I might have in the table or list at once.

Or should I go the .net route instead?
Or maybe it's not doable that way I'm proposing it because Google will limit my number of requests that I do at one time? I have no idea.

Hope someone can point me in the right direction. As always any advice will be greatly appreciated.

JM

 

[Mike Lewis]

My first thought on seeing your question is that Google Search is a very blunt instrument for this type of application. A dedicated geo-coding tool would be much better. I once successfully used MapInfo for precisely the same task that you are proposing. But that was a desktop application, and it was a long time ago. I expect that there are better on-line geo-coding tools available now (see, for example:

https://geoservices.tamu.edu/Services/Geocode/OtherGeocoders/).

If you do decide to stay with Google Search, you should be able to automate it quite easily from within VFP, either by using the Microsoft Web Browser ActiveX control (that comes free with VFP), or by automating Internet Explorer.

There used to be something called Google Search API, but that was withdrawn some years ago. At the time, Google said they would be replacing it with Google Custom Search, but as far as I can see that's something quite different.

And then there is the question of legalities. Google do not allow automated tools to run their searches. It's difficult to give an exact reference to this, as Google seem to have multiple overlapping Terms of Service, and it's not always clear which ones applies to which of their products. But I did find

https://www.google.com/intl/en/mobile/xhtml/terms_of_service.html,

which states "You may not send automated queries of any sort to Google's system without express permission in advance from Google."

Whether Google would worry about an automated exercise involved just a few hundred searches is another matter - as is the question of whether they can legally enforce any contract term that you haven't explicitly agreed to. But perhaps this is not the place to discuss that.

Finally, I suggest that your question about whether to use a grid or a stand-alone PRG is completely irrelevant at this stage. First figure out how to get your hands on the data. Deciding on the user interface can come later.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[JRB-Bldr]

It might be worth your while to do a a Google Search for: "address search API's" which turns up quite a few 'finds'

And, Yes - utilizing something like this is certainly do-able in VFP

Good Luck,
JRB-Bldr

 

[atlopes]

I don't know how this can escalate to many queries, but I think it will be easy to test.

With no error handling, here it goes:

CLEAR

LOCAL Google AS MSXML2.ServerXMLHTTP60
LOCAL Address AS String
LOCAL FormattedAddress AS String

m.Address = INPUTBOX("Enter an address:","Ask Google to validate an address")

m.Google = CREATEOBJECT("MSXML2.ServerXMLHTTP.6.0")

m.Google.Open("GET","[URL unfurl="true"]http://maps.googleapis.com/maps/api/geocode/xml?sensor=false&"[/URL] + "address=" + URLEncode(m.Address,.T.),.F.)
m.Google.Send()

m.FormattedAddress = m.Google.Responsexml.Selectnodes("//formatted_address").item(0).text

? "Original:",m.Address
? "Returned by Google:",m.FormattedAddress

RELEASE m.Google

* This is an auxiliary function to URL encode a text, optionally reencode it in UTF-8 previously (as required by the Google API)
FUNCTION URLEncode (Text AS String, UTF8 AS Boolean) AS String

	LOCAL OriginalText AS String
	LOCAL Encoded AS String
	LOCAL CharIndex AS Integer
	LOCAL CharAt AS Character
	
	IF m.UTF8
		m.OriginalText = STRCONV(STRCONV(m.Text,1),9)
	ELSE
		m.OriginalText = m.Text
	ENDIF

	m.Encoded = ""
	FOR m.CharIndex = 1 TO LEN(m.OriginalText)
	
		m.CharAt = SUBSTR(m.OriginalText,m.CharIndex,1)
		
		IF ASC(m.CharAt) >= 128 OR m.CharAt $ '#/:=&"%' OR m.CharAt <= " "
			m.CharAt = "%" + STRCONV(m.CharAt,15)
		ENDIF
		
		m.Encoded = m.Encoded + m.CharAt
	
	ENDFOR
	
	RETURN m.Encoded
ENDFUNC

Also to note: the response is full of other information (like geo location and administrative divisions...).

 

[Mike Lewis]

I've been thinking a bit more about Google's Terms of Service. I've now seen several other places where they explicitly prohibit automated tools from using their services.

While Google are highly unlikely to take any sort of legal action against you - especially given the relatively low number of searches you are proposing - they might well take some technical action to block you. They could put a "throttle" on your searches (in effect, to slow greatly them down), or they might send a a Captcha screen for you to fill in (which, of course, you wouldn't be able to do from within your program).

For that reason, I repeat what I said earlier about looking for a dedicated geo-coding tool. And of course there are also other search engines available (but they probably have similar terms of service).

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[johann59]

Thanks Mike & JRB for the advice, and specially to atlopes. Found the code useful and will try and incorporate it to what I finally plan to do. Another step in my learning process. Thank you very much.

Mike, we do have a geo-coding software. We use it to code several thousands of records, but almost always end up with several addresses that were not coded for the reasons I specified earlier, so I try and 'fix' as many of the left over addresses as I can, hence the 'tool' I want to create. But you do have a point of being mindful about Google's Terms of Service.

Anyways, again, thanks to all.

JM

 

[Mike Lewis]

Johann, this isn't a criticism of Atlopes' code, but (as far as I can see), it uses the Google maps API, which requires you to obtain an API key from Google*. That's OK, but when you do that, you explicitly agree to their terms of service, which limits you to a certain number of requests per second, and a certain number per day. The limits are quite high, and you can always purchase credits for a very much higher number. But the point is that you cannot circumvent them.

Sorry to keep banging on about terms of service, but the API terms are explicit, in that you agree to abide by them - unlike the ToS for Google Search.

My reference for the above:

https://developers.google.com/maps/documentation/geocoding/usage-limits.

See also the link to

https://developers.google.com/maps/documentation/geocoding/policies,

which mentions certain other terms and conditions.

(* Atlopes: Correct me if I am wrong about this.)

Johann, I take your point about your existing geo-coding software returning slightly incorrect addresses in some cases. But won't you have the same problem with Google? It's likely that any address database is going to be less than 100% accurate. On the other hand, I suppose that if you put a given address through two different checks, there's a good chance that the errors will cancel each other out.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[Mike Gagnon]

I have programmed something similar in the past, and I found that you cannot send more than 25 requests in one given minute.


Mike Gagnon

If you want to get the best response to a question, please check out FAQ184-2483 first.

 

[johann59]

You're right Mike. Sorry about my lack of knowledge in respect to the use of 'Google Maps API'. My mistake. Did not think this all the way through.

I just read the terms of service and it does say:

Users of the standard API:

-2,500 free requests per day, calculated as the sum of client-side and server-side queries.
-50 requests per second, calculated as the sum of client-side and server-side queries.

If I decide to go ahead and with the code, I will certainly take these limits into consideration.

Thanks Mike. Another lesson learned.

JM

 

[Mike Gagnon]

If the client is aware of these limits and they don't want to pay for a license, then slow down the process that makes requests with INKEY() or something.


Mike Gagnon

If you want to get the best response to a question, please check out FAQ184-2483 first.

 

[atlopes]

Request limits aside, I think the most obvious limitation of Google's ToS is the obligation to use this data with a visual, displayed, map...

 

[Mike Gagnon]

I use this service to build an HTML/Javascript page to display in a foxpro form using the webbrowser activex to show the best optimized route for a delivery system, and it even shows live construction warnings on the map. But to each their needs.


Mike Gagnon

If you want to get the best response to a question, please check out FAQ184-2483 first.

 

[Olaf Doschke]

atlopes, why? No.

For example address autocompletion:

https://google-developers.appspot.c...examples/full/places-autocomplete-addressform

Where is there a map? You can use the API with map display, but you are not forced to.

Bye, Olaf.

 

[Mike Lewis]

Olaf, I think Atlopes had this in mind:

https://developers.google.com/maps/documentation/geocoding/policies[/URL]]The Google Maps Geocoding API may only be used in conjunction with displaying results on a Google map. It is prohibited to use Google Maps Geocoding API data without displaying a Google map.

Why this should be I don't know. Maybe it's because Google collects advertising revenue from businesses that are displayed on their maps.

Mike


__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[atlopes]

Olaf, I was not talking about technical feasibility, that was already established, just referring to Google's ToS.

Displaying a map

The Google Maps Geocoding API may only be used in conjunction with displaying results on a Google map. It is prohibited to use Google Maps Geocoding API data without displaying a Google map.

 

[Olaf Doschke]

Ok, that's a point. Nevertheless, the topic which started this thread is something not needing a map and the API is usable without any map display. You just have the rate limits.

Bye, Olaf.