ReadXML OutOfMemoryException

[MdotButler]

I have a large XML file containing one table and many records/fields in the following format:

<?xml version="1.0" standalone="yes"?>
<NewDataSet>
  <Table>
    <BD_U_FL>L</BD_U_FL>
    <BD_PK>7117</BD_PK>
    <BD_B_DOCU>B2005061000003122             </BD_B_DOCU>
    <BD_SEQ>01</BD_SEQ>
    <BD_BEG_DT>2005-05-02T00:00:00-04:00</BD_BEG_DT>
    <BD_PLC>11</BD_PLC>
    <BD_PROC>99204</BD_PROC>
  </Table>
...
...
  <Table>
    <BD_U_FL>L</BD_U_FL>
    <BD_PK>7118</BD_PK>
    <BD_B_DOCU>B2005061000003122             </BD_B_DOCU>
    <BD_SEQ>02</BD_SEQ>
    <BD_BEG_DT>2005-05-02T00:00:00-04:00</BD_BEG_DT>
    <BD_PLC>11</BD_PLC>
    <BD_PROC>90471</BD_PROC>
  </Table>
</NewDataSet>

I am trying to read the XML file to create a dataset and get the System.OutOfMemoryException error. The code follows:

Dim myDataset As New DataSet
Dim myXMLFile As XmlReader = XmlReader.Create('xxx.xml', New XmlReaderSettings())
myDataset.ReadXml(myXMLFile)

I am assuming it is trying to read the entire XML file into memory which is not realistic. The XML contains 200K records and is 1.5GB in size.

My question to the group is, how can I read the XML file in segments or buffers? If that is not an option, how can I read the XML file one record at a time?

TIA
Mark

 

[MrDataGuy]

Throwing out some ideas: (Note I have never worked with such a large XML file, so I do not know what the limits are.)

If this a one time thing why not hand split it into several files? Just as a check I would play around a little and see how large/small a file can be read. Also could the XML file size reading be a function of the amount of (free) RAM on your computer, e.g. could a computer with more (Free) Ram read a larger file?

If XML files of this size need to be read over an over again I wonder if you need to create a pre-processor that reads the XML as plain text and created multiple XML files.

Lion Crest Software Services
Anthony L. Testi
President

http://www.LionCrestSoftwareServices.com