Data merge optimization?

[rgoerss]

I'm running into some serious issues with my data merges in InDesign CS2 - The software sucks memory until it is using 350MB/512MB of system memory and inflates the page file to 1.2GB, at which point the program just begins to hang.

These aren't terribly large merges by any measure... Anything over a few hundred records (Fourteen of which fit on each page, so we're only talking 20-ish pages) causes it to stall. I'm wondering if anyone has found solutions to this that are not documented data merge options.

So far I have tried:

Hiding the windows while the data merge takes place.
Setting a maximum number of records per document (To no effect).

This is only proof of concept... The final merges will be thousands of records, so to encounter problems with just a few hundred is rather discouraging. I think the main thing that is hurting me is the sheer number of objects on each page... We're talking about 30 text objects per record, 14 records on each page - so figure about 420 objects on each page. I can't find a way to convert all of the text objects to a single vector object (They are dynamically generated by VB code, so I can't import from Illustrator or anything). I don't even know if that would help.

I'm wondering if a computer with 4GB+ of system memory would help, or if the problem would just continue to spiral out of control with 100,000+ record merges requiring 30GB of system memory to execute properly.

If anyone has any similar experiences to share or recommendations, I would very much appreciate it.

 

[jmgalvin]

First, you might make sure that you have the latest update - think it's 4.02 - and see if that does anything. You might have to trash preferences when you do the update so read teh Read Me that comes with it.

Also, be very cognizant of fonts. Try to use as few as possible. Any font problems can casue all sorts of heartache, especially if a font file is damged in any way.

Your RAM is light. If you're on a pc, you might want to check what processes are running in the backround. You could be running out of RAM and, since a merge is very disk intensive, you could be running into virtual memory clogs. A restart, and having nothing else but ID running, might help, but I still think you're about 1 gig of RAM short.

Using OSX 10.3.9 on a G4

 

[rgoerss]

Interesting advice... I've tried running as "light" as possible. Unfortunately, I have to have the Visual Basic app running while the merge occurs since this is a fully automated process. I killed all unnecessary processes including virus scanners and the like.

Fonts are probably killing me, here. The project *requires* a large number of different character styles. This is not a typical mail merge; I'm actually dynamically creating laser printed tickets with all kinds of varying fonts and formating. There are something like 15 different character styles involved and I can't get away from that. It would defeat the whole purpose of the development project.

Another unfortunate side effect that I can't seem to avoid is that when I export to a PDF, it automatically opens Acrobat. The alternative is to not export (I am running multiple consequetive merges), but then I am stuck with tons of memory taken up by big merged InD documents open. I wish I could disable Acrobat automatically opening the PDFs that I export. If there's a way to disable this, I haven't found it.

I'm running 4.0.2 and double checked the Adobe website to make sure it is the most recent release - It is.

I think the bottom line is that 512MB of system memory is far too little for the task at hand. I had the IT guy order me 3GB of memory, so we'll see what happens then. If it improves but doesn't work, I have room for another 1GB on top of that... If 4GB of system memory isn't enough, then shame on Adobe for creating such a horribly inefficient means for merging data. In that case, I think my only hope is to get into the InDesign Server program as a partner and pray that the "Server" version without the GUI is more efficient than the desktop publishing version.

Thanks for replying!

 

[jmgalvin]

If you're exporting to pdf from indesign check the first export window. There's a check box to open the pdf after export. Uncheck it. If you're working on a pdf export preset, check that to see if you have the check box marked.

On the fonts, 15 isn't that bad but you might try to see if the fonts are ok. It's not uncommon for font files to get damaged when any program using that font crashes. Also, try disabling any auto activation of fonts. That doesn't alswys work well. I juast had disasters today with Linotype Font Explorer and autoactivation.

I use ID 3, but I think that ID 4 has some sort of font activation.

Using OSX 10.3.9 on a G4

 

[rgoerss]

Ah, okay got the export issue resolved. Thanks!

As for the fonts, they're fine. The only time I run into trouble is if I try to merge too many records. The automatic proofs and the smaller merges work just fine (Using the same fonts).

I increased my page file (VM) and larger merges (4000 records -- still way too small) are now working, but they're taking about 20 hours to complete. Hoping the memory increase will resolve that issue. We'll see!

 

[rgoerss]

It still isn't working, now with 4GB of system memory. If anybody has any other ideas, do post them.

 

[rgoerss]

Just updating for the sake of keeping a record for future reference:

I was unable to find a solution that allowed me to merge anything over 250 records with reliable results in a timely manner. Merges over 300 records would stall or hang for hours upon hours, sometimes overnight.

I was able to work around this issue by breaking down my merges into batches of roughly 200 to 250 records, which seems to be the sweet spot. I export the resulting workflows as PDF documents and then, using the Acrobat 7.0 Pro library, merge all of the resulting PDF documents into one master workflow. The PDF merge is actually very quick, and the result is that it takes roughly 2 minutes for each merge of ~250 records. The 20 hours that it used to take to merge 4,000 records is down to about 35 minutes all told.

I also ran into a problem with InDesign crashing if I executed an export command too quickly after the merge. Adding a 10 second delay using the system timer (Sloppy, I know) alleviated the problem.

Bottom line: The InDesign library and data merge process have great potential, but relatively poor implementation. It's workable, but requires a lot of development just to avoid the many pitfalls.

 

[CopyJockey]

This may or may not have an affect on your problem with the system hanging. I use XM-Pie with CS1 on a G4 with 1gig of RAM. When i "merge" my data with the graphics, i get very large PDF files. They are much larger than normal PDFs of equal page length. One thing i have found is instead of merging and having the output file go to the data server, i save the file locally to the G4's drive. Many times during a large merge i will run into network hiccups and that will cause the system to hang.