how to procss an XML file of 352 megs? 2

[peteschulte]

Hello Coders,
We use filetostr() and VFP 8 limits strings to 16,777,184 chars. Any suggestions on how to break up this file or re-set VFP or ... ?

xmlcontent = FILETOSTR(xmlfile)
lcXML = xmlcontent 

loXMLAdapter = CreateObject("XMLAdapter")

loXMLAdapter.LoadXML(lcXML,.F.,.T.)

Thanks,

Pete S

 

[DSummZZZ]

You could try using low-level IO instead. fopen(), fread(), fgets() and so on.

This is off the top of my head and untested, but you get the idea:

STORE '' TO lcXML, cTemp
xmlcontent = FOPEN(xmlfile)
DO WHILE !FEOF(xmlcontent )
   cTemp = FGETS(xmlcontent)
   lcXML = lcXML + cTemp 
ENDDO 
=FCLOSE(xmlfile)

-Dave Summers-

Even more Fox stuff at:

http://www.davesummers.net/foxprolinks.htm

 

[Dan Freeman]

You answered your own question, almost. VFP limits STRING memvars, not return values from functions. So take the memvar out of the picture.

Create cursor Foo (MyXML M)
Append Blank
Replace Foo.MyXML With FILETOSTR(xmlfile)
* Have at it!
loXMLAdapter = CreateObject("XMLAdapter")
loXMLAdapter.LoadXML(Foo.MyXML,.F.,.T.)

 

[peteschulte]

Thanks Dan and Dave!

Pete S

 

[Olaf Doschke]

While you still can work on such a large XML file via XMLADapter, I assume it takes much longer. There's a totally different strategy working on such large XML instead of loading the whole file into an object representation the IXMLDOMElement of the XMLAdapter is.

Unfortunately the subject is too big for a simple post, but you'd rather use the SAX (simpla API for XML) that will process this file as a stream instead of the whole thing at once.

http://en.wikipedia.org/wiki/Simple_API_for_XML

The file will be read in what compares to FREAD, element by element down the XML hierarchy and if a certain node type get's read it triggers an event. It's more code you'll need to write, although it's called "Simple" API to XML. But for performance reasons it can be good.

Bye, Olaf.

 

[Mike Lewis]

Olaf,

I'm glad you mentioned SAX. I've got an application that uses the XML Adapter to parse large amounts of XML, and the performance is terrible. I've wondered if SAX would be a better approach.

The trouble is that I can't find any useful documentation on how to use it. I understand that it can be done with Microsoft XML Core Services (Msxml2.DOMDocument), but I can't see anything in the object model that's specific to SAX.

Do you know of any implementations of SAX that would work under VFP? Or where I can find some documentation of the relevant parts of Core Services? I can't see anything on MSDN.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro tips, advice, training, consultancy
Custom software for your business

 

[peteschulte]

Thanks again, Olaf. The discussion of SAX is interesting for future use. We were fortunate in chopping the file into parts, creating tables for each in different folders, and appending.
Peter

Pete S

 

[Olaf Doschke]

Look into MSXML6, eg the MSXML2.SAXXMLReader and MSXML interfaces, eg IVBSAXContentHandler offers start/endDocument and start/endElement events.

You need to implement a few interfaces and several methods of these, so it will get rather lengthy code. In principle you can of course implement interfaces in VFP7+. The next part is how to make use of them.

I admit I haven't done this thing, I only know it's faster by design. wOOdy has mentioned it a few times.

I'd rather look for ActiveX XML Parser, wOOdy has mentioned one from Chilkat Software.

Bye, Olaf.

 

[Mike Lewis]

Thanks for that, Olaf. I'll do a bit more searching.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro tips, advice, training, consultancy
Custom software for your business