XFRX - Are newer versions any faster... 3

[GriffMG]

I've been using xfrx for a very long time, 2007 I think, and am currently using (I think) version 191.9

I have a project where I need to produce a very large number of reports (call it 2 million) in PDF format and
it looks like it will take about 10 days to complete... which is a tiny bit too long.

As part of my efforts to improve this, I am wondering if anyone has tried any of the more recent versions
before I invest in updating mine.

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

There is no place like G28 X0 Y0 Z0
​

 

[Chris Miller]

Here's how to set process afinity by code:

Declare INTEGER GetCurrentProcess In Kernel32

Declare INTEGER GetProcessAffinityMask In Kernel32 ;
   INTEGER hProcess, STRING @lpProcessAffinityMask, STRING @lpSystemAffinityMask

Declare INTEGER SetProcessAffinityMask In Kernel32 ;
  INTEGER hProcess, INTEGER dwProcessAffinityMask
  
Declare Integer GetLastError in WIN32API
     
Local lnProcessHandle, lcPA, lcSA, lnPA, lnSA, lnCPU
lnProcessHandle = GetCurrentProcess()
lcPA = Space(2)
lcSA = Space(2)

Clear
If GetProcessAffinityMask(lnProcessHandle,@lcPA, @lcSA) = 1
   ? 'Affinity Masks (Process, System):'
   ? CreateBinary(lcPA), CreateBinary(lcSA)
   
   * Translate System affinity mask string to number
   * It will have bits set for all available cores, i.e. for 4 cores it will be 0h0F00 = 15
   lnSA = CToBin(lcSA,"2RS")
   [highlight #FCE94F]lnCPU = 2 && 0..3 for 4 cores[/highlight]
   lnPA = Bitand(Bitset(0,lnCPU),lnSA) && Bitand with lnSA ensures no unavailable CPU core bit is set
   * Could also set multiple CPUs by setting multiple bits, i.e. Affinity Mask=15 would mean no specific CPU core affinity.
   
   If lnPA>0 and SetProcessAffinityMask(lnProcessHandle,lnPA) = 1
      ? 'OK: Process affinity set to CPU '+Alltrim(Str(lnCPU))
   Else
      If lnPA=0
         ? 'Error: CPU number not available according to System Affinity Mask.'
      Else 
         ? 'Error:',GetLastError()
      Endif
   EndIf
 
   If GetProcessAffinityMask(lnProcessHandle,@lcPA, @lcSA) = 1
      ? 'Affinity Masks (Process, System):'
      ? CreateBinary(lcPA), CreateBinary(lcSA)
   EndIf
EndIf

The highlighted line is where you set the CPU core number (between 0 and max cores-1).


Chriss

 

[stanlyn]

Chris,
The code you posted is VFP code, so does VFP complain when switching Affinity on the already running app? How do you turn it on for a process?

Anyone else...
Does anyone know if the xfrx sdk can create a searchable pdf when outputting to pdf?

Is the text output from the OCR process available so it can be inserted into a table for searching?

Thanks,
Stanley

 

[mJindrova]

Does anyone know if the xfrx sdk can create a searchable pdf when outputting to pdf?

Yes, PDF from XFRX are searchable, if text was not converted to picture.

mJindrova

 

[tomk3]

Hello,

excel
maybe these are faster and do not use automation :

https://github.com/ggreen86/XLSX-Workbook-Class

or tools by Vilhelm :

http://praisachion.blogspot.com/search?q=excel

there are several .Net libs which may be used with westwind bridge.

For PDF
Did you try foxypreviewer ?
Or printing using bullzip (automation) ?

Regards
tom

Maybe you can do it on a "long weekend" where friday / monday are no working days (Easter)
We had to to this sometime on very big migrations.

 

[GriffMG]

I have 140 of these to do in total, that is nearly three years of ordinary weekends, and I do not know how many
Easters! B-)

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

There is no place like G28 X0 Y0 Z0
​

 

[Chris Miller]

Stanlyn,

my code about SetProcessAffinityMask does exactly what manually setting CPU affinity in the Task Manager does. And that is in general allowed and poses no problems, yes.

And regarding PDFs, there are only very few PDF solutions that work badly by just converting every page printed to an image. Even the worst PDF printer drivers will generate PDFs with text,i.e. searchable. Foxypreviewer does, too. Foxypreviewer just has a property lPDFasImage that will generate images for all the pdf pages and so the whole PDF is composed of pictures only. It defaults to .f.

Even just the question is pointing out you have a totally wrong idea about PDF generation. Or what did you use so far? Printing to a TIFF file that is by default just iamages of the pages and then converting that to PDF? I mean, you have to work hard to get a PDF that's only composed of images and thus is not searchable.

The usual way texxt is embedded within PDFs is just as that, as text. And fonts are embedded within the PDF, too. So unlike HTML with CSS pointing out fonts it would like to use, PDFs just have the fonts they use and so become device independently the same look, not depending on installed fonts. But it's really just text in PDFs, unless you really enforce everything is images.

I don't know, maybe you have a scanner with a feature to turn multi page scans into PDF by the press of a scanner button. Those will be composed of images, if not OCR is used in advance. And you will then sometimes have tiny images in the generated PDFs when letters werre not recognized. But if you start from an FRX or a XLS, there's text in that and that simply stays text in a PDF, if you don't force it to become an image of the text. Who does that, will have his reasons, but really the PDF capabilities of reproducing the same look are there for decades, I don't even remember times when this was not the case.

Chriss

 

[Chris Miller]

I have 140 of these to do in total

I don't know what you mean. Has the outset changed to 140 times the amount of data to report?

Chriss

 

[GriffMG]

I don't have one single export that needs to be done in a 'weekend' I have 140 projects I need to export in total, most of which must be done
in a 'weekend'...

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

There is no place like G28 X0 Y0 Z0
​

 

[Chris Miller]

I don't have one single export

That was very clear already.

140 reports don't seem so many, unless every report has hundreds of pages. We don't know anything about any measures.
This makes it impossible to give any more specific help.

You have not even come up with what part of reporting takes the longest time. I think you even have trouble measuring that.

Chriss

 

[GriffMG]

No Chriss, I'm quite capable of measuring each bit - profiling it - but until the customer
pulls the trigger, so to speak, I can only tinker.

I only asked about XFRX because that seemed like a good candidate for reducing the time - being as I'm
using an old version... we'll see when I get an instruction to proceed what else can be done.



Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

There is no place like G28 X0 Y0 Z0
​

 

[Chris Miller]

No Chriss, I'm quite capable of measuring each bi

Fine. If you could tell us we could concentrate on that bit.

I might be a bit impatient while you think about what to ask. You know how you could get better help.

Chriss

 

[mJindrova]

Even just the question is pointing out you have a totally wrong idea about PDF generation. Or what did you use so far? Printing to a TIFF file that is by default just iamages of the pages and then converting that to PDF? I mean, you have to work hard to get a PDF that's only composed of images and thus is not searchable.

Print from report listener to PDF virtual printer driver create PDF where each text is image. It's a base feature of report listener.

mJindrova

 

[Chris Miller]

Print from report listener to PDF virtual printer driver create PDF where each text is image.

If you go to a PDF printer driver you don't need to go through a report listener.

And the PDFs I created with PDFCreator or Bullzip PDF don't have images of the report controls that are just numbers, text, or dates, it all translates to searchable text in the PDF.

If you just wanted to give an example of how texts turn into images, okay, noticed.

Chriss

 

[GriffMG]

@Martina

Is there an option to have searchable text instead of images?

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

There is no place like G28 X0 Y0 Z0
​

 

[mJindrova]

Hmm, I will try explain better.
VFP has two report engines:

80 - this engine use GDI API for rendering and is compatible with report engine from VFP 8, but support part of new features from VFP 9. Print to virtual PDF printer create PDF where text is text:
[pre]SET REPORTBEHAVIOR 80
REPORT FORM ... TO PRINTER PROMPT
[/pre]

90 - this engine (aka report listener) use GDI+ API.
Print to virtual PDF printer create PDF where text is image:
[pre]SET REPORTBEHAVIOR 90
REPORT FORM ... TO PRINTER PROMPT
[/pre]

Why XFRX or FoxyPreview can "print" PDF where text is text? Because "print" not is "print" but "export". In both cases is derived report listener is wraper only. Gets information about objects (from bse report listener) and create PDF.


mJindrova

 

[GriffMG]

Thank you Martina

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

There is no place like G28 X0 Y0 Z0
​

 

[Chris Miller]

I can't confirm that, I've done REPORT FORM in REPORTBEHAVIOR 90 and the output PDFs were searchable.



Chriss

 

[Chris Miller]

Here's a sample PDF done with the latest version of PDFCreator
Just by running:

CD Home()
CD tools/filespec
Set Reportbehavior 90
Set Printer To Name "PDFCreator"
Report Form 90frx.frx

It's not only searchable, you can also select, copy and paste text from it.

Chriss

 

[Sime Atanasov]

Hi Griff.

Aside from the fact the data is going through an excel format, my experience has been that Bullzip (or any other) pdf driver works reasonably fast. As Tom and Chriss have mentioned it too.

For more than ten years, once a month my program would send around 700 emails, each with 3 PDF file attachments. Small files, usually 2-3 pages long, but still around 2000 PDF files. The process takes about 40 minutes, from data being gathered and calculated, "printed" to FRX native report, and then again "printed" to a fixed file name specified in the Bullzip option. That file is renamed and moved to a folder, from where it's emailed along a body text and the other two pdf attachments created the same way. Plus, there are several seconds of timeout between any of those 700 emails to make sure the sequential nature is preserved. Initially, I had to slow the process down so the SMTP server and the ISP don't consider it as spamming (and it's not a spam, just statements)

Sime

 

[vernpace]

Griff,

If you are running many reports in succession, call SYS(1104) before each report. I was in a similar situation awhile back and calling SYS(1104) frequently reduced total completion time by around one third.