How to query BIG files effieciently??

[1076rama]

Hi all,

I have 2 XML files (around 170 MB each) and my aim is to query both (both have some data in common) files efficiently. one file has lot of IDREF and IDREFS. One file uses DTD and the other uses XSD. Initially I thought of using DOM but I came to know that it requires lot of memory. I think SAX has limited support for IDREF etc. Is there any way to access these files efficiently?? (can I use XPATH and XSLT?)

Thanks,
Sai.

 

[Nosferatu]

Try using expat library; it's entirely C based and it's pretty fast.
DOM, as you said, it's not very handy with large files (don't want to say useless).
However, working with expat will involve writing code - anywhere between some code and a lot of code, depending on how the manipulations should be. [red]Nosferatu[/red]
We are what we eat...
There's no such thing as free meal...
once stated: methane@personal.ro

 

[chiph]

I think you're going to have to run some benchmarks.

DOM will consume a lot of memory.
SAX requires parser to run through the file every time you query.

Neither is really good, but one may be better than the other for your purposes.

Chip H.