Updating large XML files

[CodingIsFun]

Hi all experts,

I have been given a task to update a few elements in a 7 gigabite xml file. In the past for small files I would use xpath.

Do to the large size, Im wondering ahead of time if there are any downfalls with xpath and large file sizes. I will do experiments, but I would also like to hear the experts opinion. Any help would be great.

I am going to be looking for a specific tag by an attribute id. When there is a match I will be adding attribute as a flag.

thanks in advance.

 

[chiph]

This file is too large to load into memory and use XPath queries. You'll have to treat it like a stream, and use one XmlReader to read it, and a XmlWriter to write it out to a 2nd file. In between you would insert your changes as you see the points in the file where they need to go.

Chip H.


____________________________________________________________________
If you want to get the best response to a question, please read FAQ222-2244 first

 

[CodingIsFun]

Is there a way to read in the text in chunks? Possibly parse in text for every 5,000 > characters. The xmlfile Im reading has no carriage returns so going line by line is not a good solution?

 

[chiph]

If you know the structure of the file, use XmlTextReader and XmlTextWriter instead. You'd just call XmlTextReader.ReadElement() until you got to the elements you care about, make your changes, and keep going.

But otherwise, you can call ReadChars() and do the XML parsing yourself -- just remember it'll not be an entire XML file or fragment -- it's just a bunch of characters at that point, so you'll have to handle all the entity escaping, etc. yourself.

Chip H.


____________________________________________________________________
If you want to get the best response to a question, please read FAQ222-2244 first

 

[CodingIsFun]

I know the structure of the file, so I can go down that path. The XmlTextReader.ReadElement() is not available for my text reader, is this available in 2005?

 

[chiph]

Yes, it's been in the framework since v1.0

Here's the docs for v2.0:

http://msdn2.microsoft.com/en-us/library/system.xml.xmltextreader(VS.80).aspx

Note that for v2.0, Microsoft's recommendation is to use the XmlReader and not the derived class XmlTextReader.

Chip H.


____________________________________________________________________
If you want to get the best response to a question, please read FAQ222-2244 first

 

[JurkMonkey]

My question is - what are you doing with a 7gb xml file?

Time to throw that data into a db!