Capture data from a formated text 2

[SitesMasstec]

Hello colleagues!

I have, at last, been using CURSOR, thank you.

In the small program bellow I send a request to an API provider:

xmlHttp = CREATEOBJECT("MSXML2.ServerXMLHTTP")

* xmlHttp.open("GET", "[URL unfurl="true"]https://api.postmon.com.br/v1/cep/13041-620",[/URL] null)
* Substitituído pelas 3 linhas abaixo:
myCEP="33236-434"  && Value (in the real program it will be a variable)
CEPparaAPI="[URL unfurl="true"]https://api.postmon.com.br/v1/cep/"+myCEP[/URL]
xmlHttp.open("GET", CEPparaAPI, null)

xmlHttp.setRequestHeader("Content-Type", "text/xml")
xmlHttp.send("")
result = xmlHttp.responseText
? result

The result is:

This is Ok.

Then, I created a Cursor to populate with data read from API, that is, I want to "consume" the data:

CREATE CURSOR my_Endereco;
(   my_bairro c(22),;
    my_cidade c(20),;
    my_logradouro c(34),;
    my_estado c(2) )


    my_bairro = STREXTRACT( result, ["bairro": ], [,] )
    my_cidade = STREXTRACT( result, ["cidade": ], [,] )

*   etc etc

? my_bairro 
? my_cidade

* Comands to populate the Cursor will be here

But I have not got any result to my_bairro and my_cidade ...
It seems something wrong with STREXTRACT parameters I had informed.


Thank you,
SitesMasstec

 

[ggreen61]

Take a look at the class at the following link:

https://github.com/VFPX/nfJson

It has methods to extract directly to a cursor. No need to reinvent the wheel... Also, there are a lot of classes/solutions at the VFPx site:

https://vfpx.github.io/

Greg

 

[Chris Miller]

You just have to wait for the result with:

xmlHttp.open("GET", CEPparaAPI, null) 
...
xmlHttp.send() && no need to send "", don't use the body parameter.
Do While xmlHttp.readyState<>4 
   Doevents
EndDo
result = xmlHttp.responseText

Or

xmlHttp.open("GET", CEPparaAPI, .F.) && request synchronous, i.e. wait for the result
...
xmlHttp.send() && no need to send "", don't use the body parameter.
result = xmlHttp.responseText

Then you have the result and can extract data from it. It doesn't fail on STREXTRACT, it fails on your code not waiting for the result to come back. After a send() you don't automatically wait for the HTTP response, so not waiting and reading the responseText property, you get an empty string, as nothing was received yet.

I think you tested all this from the command window and it worked there. Well, you automatically will need to wait, as long as the http request runs VFP will be less responsive and so when you manually execute line by line, the responseText was there.

As Greg already pointed out, this is JSON and there are libraries like nfJSON, he mentioned. I think it's fairly simple to get the data you want with STREXTRACT. Your STREXTRACT calls will give you the strings you want but include the quotes, so you would need to fine-tune this. Or... follow Gregs' advice and use a library like nfJSON. Maybe even something like West Windws Web Connection, which makes HTTP handling overall easier, not just the detail aspect of handling JSON.

I'd like to also tell you about other wrong expectations you have judging from your code: If you set the request header "Content-Type" to "text/xml" you expect the server to respond with XML, not JSON, as it does. But notice, what you actually specify is what you send, not what you want. You actually can't. The HTTP protocol itself gives you the header 'Accept' to indicate what result types you will accept. But then this also is never forcing the server to serve what that, you just tell the server a preference. It might respond with something you don't specif in the Accept header anyway.

The actual Cotent-Type you get back is "application/json". You have to look into ? xmlHttp.getResponseHeader("Content-Type") to see what you get. I don't know the API you use and how it is defined, there might also be ways to get XML, but not this way.

There can be APIs that can respond with XML or JSON and even more different result types, but just like I explained recently in another thread, the server keeps control, you can't force something on the server, not by the HTTP protocol itself. Otherwise, the HTTP protocol would be broken in its security conception, if clients have too much control to force something on the server.

There also are typical other ways to tell an API with different result types what you want, usually within the request URL as a parameter.

If you don't want to dig into such details and you realize there's much more to know to become productive with HTTP APIs, understand why things happen as they do and how to get to what you want, then you really should rather look into Web Connection and similar libraries that pull this up on a higher level, i.e. they are implemented with more detail knowledge of the HTTP protocol and how to arrive at what you actually want and give you simpler ways of using the web and process the results you get from APIs.

Chriss

 

[SitesMasstec]

Hello Greg!

I have just downloaded nfJson you advised me, and will dive into it, thank you.

I, for now, am going to just consume Postmom data, AND I would like to extract the data using STREXTRACT, if it is possible, in VFP 9 to use a single command, like

 my_bairro = STREXTRACT( result, ["bairro": ], [,] )

I haven't got the result for my_bairro in the above command.

If it is possible to use a single STREXTRACT command, I would like to learn how, because I tried many ways, VFP Help and books and haven't got the result I need (my_bairro = Aeronautas, in my post).


Thank you,
SitesMasstec

 

[SitesMasstec]

I made some changes in the STREXTRACT command and included WAIT TIMEOUT 5 to wait for the server response, as Chriss advised:

xmlHttp = CREATEOBJECT("MSXML2.ServerXMLHTTP")

* xmlHttp.open("GET", "[URL unfurl="true"]https://api.postmon.com.br/v1/cep/13041-620",[/URL] null)
* Substitituído pelas 3 linhas abaixo:
myCEP="33236-434"  && Valor será lido do programa
CEPparaAPI="[URL unfurl="true"]https://api.postmon.com.br/v1/cep/"+myCEP[/URL]
xmlHttp.open("GET", CEPparaAPI, null)

xmlHttp.setRequestHeader("Content-Type", "text/xml")
xmlHttp.send("")     && NÃO PRECISA (Chriss)

WAIT TIMEOUT 5
result = xmlHttp.responseText
WAIT TIMEOUT 5

CLEAR
? result

my_bairro = UPPER(STREXTRACT(result, ["bairro": ], [,] ))
my_cidade = UPPER(STREXTRACT(result, ["cidade": ], [,] ))
my_logradouro = UPPER(STREXTRACT(result, ["logradouro": ], [,] ))
my_estado = UPPER(STREXTRACT(result, ["estado": ], [,] ))

? my_bairro 
? my_cidade
? my_logradouro
? my_estado

Please, see the result: (the variable my_estado is empty!):

Thank you,
SitesMasstec

 

[Chris Miller]

SitesMasstec,

I did give you two solutions, bt not advised a WAIT command, not at all. Please read again and use one of the solutions I gave.

Your last STREXTRACT doesn't work, because there is no comma after "estado": in the JSON, it's the last information. Your STREXTRACT would only work if you add a comma to the result variable or don't set an end delimiter for that STREXTRACT. Now think about future changes, where further JSON may be added, then you don't want to read the rest of the JSON after "estado":, but only to the next element, so I'd add a comma to result: result=result+"," Also notice, that JSON is more complex than that, the closing delimiter of an element could also be one of the closing brackets ] or }, so a library reading JSON would be a better use.

You're overlooking a lot of things, also the quotes around the extracted strings are not part of the values, they are there to indicate the value data type is string. I hope you'll recognize at least some of this after you had a break or even slept over this and look at it again refreshed.

But overall, I'll recommend you to use Web Connection to cover both code for web requests and json parsing, it removes the need to look into the many details you all overlook. You still overlook them, even if we already told them to you.

Chriss

 

[atlopes]

SitesMasstec,

XML is one of the formats the API can return. It may make things easier for you.

LOCAL HTTP AS MSXML2.ServerXMLHTTP60
LOCAL XML AS MSXML2.DOMDocument60
LOCAL CEP AS String

m.CEP = "33236-434"

m.HTTP = CREATEOBJECT("MSXML2.ServerXMLHTTP.6.0")

m.HTTP.Open("GET", "[URL unfurl="true"]https://api.postmon.com.br/v1/cep/"[/URL] + m.CEP + "?format=xml", .F.)

m.HTTP.Send()

m.XML = m.HTTP.responseXML

? "Bairro:", m.XML.selectNodes("result/bairro").item(0).text
? "Cidade:", m.XML.selectNodes("result/cidade").item(0).text
? "Logradouro:", m.XML.selectNodes("result/logradouro").item(0).text
? "Estado:", m.XML.selectNodes("result/estado").item(0).text

 

[Chris Miller]

And this is the code to get at all information in the JSON with West Wind Web Connection, after starting it (Do launch.prg):

loHttp = CREATEOBJECT("wwHttp")
lcResult = loHttp.get("[URL unfurl="true"]https://api.postmon.com.br/v1/cep/33236-434")[/URL]

loJSON = CREATEOBJECT("wwJsonSerializer")
loResult = loJson.deserializejson(lcResult)
? loResult.bairro
? loResult.cidade
? loResult.logradouro
? loResult.estado

* and more
? loResult.cidade_info.area_km2
? loResult.cidade_info.codigo_ibge
? loResult.estado_info.area_km2
? loResult.estado_info.codigo_ibge
? loResult.estado_info.nome

You see how all the details about HTTP protocol usage are taken away from you, you don't create MSXML2.ServerXMLHTTP but wwHTTP and then have a GET method that simply returns the response text. Also, a JSON parser class turns the JSON string into a Foxpro object with all the elements of it, including substructures like the two info subobjects.

Chriss

 

[SitesMasstec]

Yes, Chriss, I changed the last line, as you had advised me:

my_estado = UPPER(STREXTRACT(result, ["estado": ] ))

... and the result is what I expected:
my_estado: MG

Of course I will have to dive into JSON, and I also will search about Web Connection.

Thank you,
SitesMasstec

 

[SitesMasstec]

Now, another problem arose:

When I gave another postal code, special characters weren't shown:

The code for request the answer from API is:

	xmlHttp = CREATEOBJECT("MSXML2.ServerXMLHTTP")

	CEPparaAPI="[URL unfurl="true"]https://api.postmon.com.br/v1/cep/"+myCEP[/URL]
	xmlHttp.open("GET", CEPparaAPI, null)

	xmlHttp.setRequestHeader("Content-Type", "text/xml")
	xmlHttp.send("")     && NÃO PRECISA (Chriss)

I tried to changed the Request Header line with:

xmlHttp.setRequestHeader("content-type: application/json; charset=UTF-8", "text/xml")

... with the same bad result.


Thank you,
SitesMasstec

 

[Chris Miller]

Put aside how wrong this setRequestHeader is formulated, it's useless.
I already told you that the Content-Type header is only there for you to describe the content of what you post, not what you want in the response. You can't enforce it so you have to finally see what the content of the response is by using getResponseHeader('Content-Type'). Besides that, the encoding of a response is not the content-type, but the
And then convert that to VFPs ANSI type strings. In this case, I don't think you'll get there as simply as with a STRCONV() call, as the JSON contains Unicode characters specified with \U plus a hexadecimal number. That also won't change if the encoding is UTF-8. To translate that you need to declare a function.

Also, I told you that you don't send a body, so this also doesn't need a content type. I didn't tell you that calling send is unnecessary. I told you that sending an empty string is unnecessary. You put a comment "NÃO PRECIS (Chriss)" in your code, but not what I told you to change in that code, remove the "":

xmlHttp.send()     && NÃO PRECISA (Chriss)

Don't send anything, neither "" nor null, nor .f. nor 0. Nothing.

So, if you make a comment at something I recommended, then please also change it to what I recommended and don't just add this comment. It becomes quite pointless to even just try to help you if you get things so wrong.

There is a large group of Spanish and Portuguese-speaking VFP developers and you should find a forum they use. Because I sense you have big understanding problems simply because of the natural language barrier.

Chriss

 

[SitesMasstec]

Hello, colleagues!

Thank you, atlopes, your XML code performed as expected:

LOCAL HTTP AS MSXML2.ServerXMLHTTP60
LOCAL XML AS MSXML2.DOMDocument60
LOCAL CEP AS String


myCEP=thisform.txtCccep.Value

IF SUBSTR(myCEP,1,1)<>" "

	m.HTTP = CREATEOBJECT("MSXML2.ServerXMLHTTP.6.0")

	m.HTTP.Open("GET", "[URL unfurl="true"]https://api.postmon.com.br/v1/cep/"[/URL] + myCEP + "?format=xml", .F.)

	m.HTTP.Send()

	WAIT WINDOW "" TIMEOUT 5

	m.XML = m.HTTP.responseXML

*   Leitura das variaveis e transformação em letras maiúsculas
	my_logradouro = UPPER(m.XML.selectNodes("result/logradouro").item(0).text)
	my_bairro = UPPER(m.XML.selectNodes("result/bairro").item(0).text)
	my_cidade = UPPER(m.XML.selectNodes("result/cidade").item(0).text)
	my_estado = UPPER(m.XML.selectNodes("result/estado").item(0).text)

	thisform.txtCende.Value = my_logradouro
	thisform.txtCbair.Value = my_bairro
	thisform.txtCcida.Value = my_cidade
	thisform.txtCesta.Value = my_estado
ENDIF

The result:

In spite of the fact there is no UTF-8 command in HTML/XML directives, the result presented international characters!

I will dive in Chriss advices in order to better understand, because maybe I will have to use text (JSON?) in my application and not XML.

I tried to use this also, but it presented parameters error:

xmlHttp.setRequestHeader("Accept: application/json;charset=utf-8")

Thank you,
SitesMasstec

 

[atlopes]

SitesMasstec,

Good to know that you have it working.

A few things to note.

1. No need to use a [tt]WAIT WINDOW[/tt] command. The HTTP call is not asynchronous and will only return upon completion or error.

2. By the way, consider handling runtime errors. For instance, what happens if a CEP is invalid? Or if the server is unreachable?

3. There are a few libraries to work with JSON and XML for VFP. All of these would help you process your requests and the server responses. Besides what Chriss already recommended, you may have a look at VFPX JSON and XML-related projects.

For instance, this is how one could use XMLSerializer to consume the CEP API:

LOCAL XS AS XMLSerializer
LOCAL CEP AS Object

m.XS = CREATEOBJECT("XMLSerializer")

m.CEP = m.XS.GetSimpleCopy(m.XS.XMLtoVFP("[URL unfurl="true"]https://api.postmon.com.br/v1/cep/33236-434?format=xml"))[/URL]

? m.CEP.result.bairro
? m.CEP.result.logradouro
? m.CEP.result.cidade
? m.CEP.result.estado

Since MSXML2 is already in your system, using it directly is more straightforward and does not add other dependencies if you use XML instead of JSON.

4. The [tt]setRequestHeader[/tt] method requires two parameters. It should be something like this:

xmlHttp.setRequestHeader("Accept", "application/json")

Nevertheless, how a specific service negotiates the content that it serves may or may not use request headers. In this case, as it seems, format specification is set in the query string, defaulting to JSON.

 

[Chris Miller]

General setRequestheader syntax is ith 2 parameters:

xmlHttp.setRequestHeader(bstrHeader, bstrValue)

The header is something like "Content-Type", "Accept", no colon and no value, that is provided in the second parameter. You have a working example of a call in the code you posted yourself: xmlHttp.setRequestHeader("Content-Type", "text/xml"). It has no effect, though.

I told you twice and I tell it a third time: The header "Content-Type" of your request does not tell the server to respond with that content-tyype, it tells the server the content-type of what you send. But you send nothing, so it's totally useless to set this. With an "Accept" header you tell the server what you accept as response. The server can still also reply with other content types.

Atlopes gave you the way to get XML, by adding ?format=xml to the URL. This is documented here:

https://postmon.com.br/

Para obter o retorno em formato XML, basta acrescentar ?format=xml no final da requisição.

This is even documented in Portuguese. We can only help you as much as you also understand what we tell you.

Chriss

 

[SitesMasstec]

Chriss:

As I had posted here a few hours ago today, I used the XML approach as atlopes had advised me, I got the expected result (response from Postmon API using international characters).

It is the first time I am using API in my VFP applications.

Of course, I will read your advices again, in order to have a wider understanding of APIs.

Thank you both for you help and patience.


Thank you,
SitesMasstec

 

[SitesMasstec]

It was all ok, since I am using XML approach, as I had said in my last post here.

But now a problem flourished, if I enter a code (myCEP, like a Zip code) that doesn't exist:

Part of the program that process the 'myCEP' code:

	LOCAL HTTP AS MSXML2.ServerXMLHTTP60
	LOCAL XML AS MSXML2.DOMDocument60
	LOCAL CEP AS String

	myCEP=thisform.txtCccep.Value    && Code entered

	IF SUBSTR(myCEP,1,1)<>" " 
	        m.HTTP = CREATEOBJECT("MSXML2.ServerXMLHTTP.6.0")

		m.HTTP.Open("GET", "[URL unfurl="true"]https://api.postmon.com.br/v1/cep/"[/URL] + myCEP + "?format=xml", .F.)

		m.HTTP.Send()

		m.XML = m.HTTP.responseXML
	        thisform.txtCende.SetFocus
    
*  Reading variables and transform in uppercase (ERROR OCCURS IN FIRST LINE BELLOW)
		my_logradouro = UPPER(m.XML.selectNodes("result/logradouro").item(0).text)
		my_bairro = UPPER(m.XML.selectNodes("result/bairro").item(0).text)
		my_cidade = UPPER(m.XML.selectNodes("result/cidade").item(0).text)
		my_estado = UPPER(m.XML.selectNodes("result/estado").item(0).text)
	ENDIF

Thank you,
SitesMasstec

 

[atlopes]

SitesMasstec,

When the CEP does not exist, after [tt]m.HTTP.send()[/tt] the value of [tt]m.HTTP.status[/tt] will be 404, so you should test for that.

The server may also be unavailable at any point, so you should also put a [tt]TRY .. CATCH[/tt] in place to handle that, at least around [tt]m.HTTP.send()[/tt].

These were already pointed out above (see note #2).

 

[SitesMasstec]

atlopes:

I tested the 2 status codes:

...
m.HTTP = CREATEOBJECT("MSXML2.ServerXMLHTTP.6.0")
	       
HTTP.Open("GET", "[URL unfurl="true"]https://api.postmon.com.br/v1/cep/"[/URL] + myCEP + "?format=xml", .F.)

m.HTTP.Send()
		
EncontrouCEP=""   
IF m.HTTP.Status=200
    EncontrouCEP="Sim"    && Found
ENDIF

IF m.HTTP.Status=404
    WAIT WINDOW AT 20,40 "CEP não encontrado" TIMEOUT 3
    EncontrouCEP="Nao"    && Not found
    thisform.txtCende.SetFocus
ENDIF
		
IF EncontrouCEP="Sim"     && if Found, do all commands
	m.XML = m.HTTP.responseXML
        thisform.txtCende.SetFocus
	    
*	Leitura das variaveis e transformação em letras maiúsculas
	my_logradouro = UPPER(m.XML.selectNodes("result/logradouro").item(0).text)
	my_bairro = UPPER(m.XML.selectNodes("result/bairro").item(0).text)
	my_cidade = UPPER(m.XML.selectNodes("result/cidade").item(0).text)
	my_estado = UPPER(m.XML.selectNodes("result/estado").item(0).text)
...

Now, I will try to use the TRY...CATCH:

TRY
   m.HTTP.Send()
CATCH WHEN m.HTTP.Status<>200
   WAIT WINDOW "Erro" TIMEOUT 5
   EncontrouCEP="Nao"    && Not found / or not connected
FINALLY
   EncontrouCEP="Sim"    && Found!
ENDTRY

Thank you,
SitesMasstec

 

[Chris Miller]

Your code works, SitesMasstec.

But you didn't look whether m.HTTP.Status=404 in the code you posted erroring.
In case of status 404 there is no XML, but HTML in the response and therefore trying to readd XML nodes with m.XML.selectNodes fails.

You have to make the same error handling checks of the response status everywhere you send the http request. Overall it seems ou copy&paste code in several places and added error handling in some places, but not others. Do a class instead and use it everywhere, then you will just need to program error handling once.

Chriss

 

[SitesMasstec]

Yes, Chriss, I am conviced I need classes about using Web Services.

So, in the last few days I have searched at Internet sites, about API/REST/JSON/XML.

I have heard JSON is the most popular, but do you think XML is easier for Visual FoxPro developers?

Please see what "What's New in Nine" book says in a chapter:

Well, this book was published in 2005.


Thank you,
SitesMasstec