Index in LL9.5: Speed? Indexed documents? 2

[DrDDT]

All,

A few questions about indexing in Livelink 9.5

- Is there a way to determine how fast livelink indexes, and how long it took to finish a complete re-index?
- How do I know that the re-index is finished, and not stalled or in error??
- How many documents are in my index?
In 9.1 we did a search for '*' in Enterprise-all versions, but now we get a:

Livelink Error: Error in search broker
[A term in your query yielded more than the maximum allowable number of results. Please refine your query.]

I thought this was a search firewall issue, but the firewall is gone in 9.5?

 

[appnair]

The enterprise query will give you the total rawhits .To enable that this is what needs to be done.in your ini file

[SearchOptions]
wantCacheResults=FALSE
showTotalHits=RAW

the enterprise versions have never worked for me so i don't know.In 9.1Sp3 we could increase the firewall but I don't see an option in 9.5 SP1
Other well known ways are

select * from kini where inisection ='OTIndex'

.If reindex is complete then you will get -1 or if you get a positive number that is the dataid being indexed.Having so many problems after the upgarde I'm not sure any of this s gonna help.also a look at the mesage pool(status.prv) is a folder of particular interest.If it has messages going thru it it kind of tells it is building.

Well, if I called the wrong number, why did you answer the phone?
James Thurber, New Yorker cartoon caption, June 5, 1937

 

[davidt50]

Because of the change in the search engine in 9.5, the firewall value has been dropped altogether. It was problematic at best.

Just a note about:

select * from kini where inisection ='OTIndex'

if the value is not -1 then it will be a decreasing value now in 9.5. The extractor runs in descending dataid order so during a re-index will start with the newest items first then proceed to the the oldest. So to just look at this value to determine the speed of the re-index could be misleading since a re-index is a low priority operation as compared to handling the notify changes. During a busy day it may look like nothing is going on

In addition, during a re-index the notify table (DTreeNotify) is handled before continuing with the re-index. This allows normal business operations to continue as expected during a re-index. This is a significant change from previous versions.

 

[appnair]

May I ask you the source of this Info,if this is what we are seeing I would like to ask OT for this.Is this proprietary or is it available in the white paper "indexing and searching in 9.5"

Well, if I called the wrong number, why did you answer the phone?
James Thurber, New Yorker cartoon caption, June 5, 1937

 

[davidt50]

I am not sure what you are asking:

1/ is this the case in 9.5 witht eh change to the extractor?

or

2/ is this an added feature of the extractor that will be added?

And yes I have a very reliable source and there should be a 9.5 white paper coming out in the early new year.

 

[appnair]

I was asking about the extractor operation.In 9.1 systems when you do a re-index the oldest documents goes through the extractor and the dtreenotify documents are processed after a re-index.Or specifically it looks at the MAX(dataid) in dtree and holds that for re-indexing.
Is this right.I just talked to OT and they said what you were saying.They reversed this order.Newer ones will be given priority.
Subsequently newer items will be returned in search first

Well, if I called the wrong number, why did you answer the phone?
James Thurber, New Yorker cartoon caption, June 5, 1937

 

[davidt50]

I think this makes for better behaviour from a business point of view.

WDYT? Does this operation make more sense?

 

[DrDDT]

Thanks all, for your answers!

One more question about tuning the indexing process:

What I see now is this:

Pending Processed
Enterprise Extractor Process Enterprise Document Conversion Active 278 249

Enterprise Document Conversion Enterprise Update Distributor Active 6 243

So the extractor is much faster than the conversion.
Is there any way to tune this? Can we offload some of the work to other machines?

We now have 2 dedicated login servers, 1 admin server and an Oracle server. The admin server is doing all the indexing itself, and uses 100% CPU all the time.
Maybe this is not the 'best' setup for LL9.5?

 

[davidt50]

If you are pegging the server 100% then definitely, it is time to off load the indexing across to multiple machines. I can see the extractor/livelink, document conversion and the Update Distributor running on the existing machine. As for the Index engines and search engines they should be moved to their own machine(s) again to lower the resources required by any single resource. The new indexing system in 9.5 was designed to address many issues encountered with the previous search engine, on Windows the 2GB ram limit that led to SE130’s, or had to be tweaked by use of the firewall setting.

All of this really depends on the load of the system you intend to have.

In the short term, if this is a re-index and if possible there is a way to limit the resource contention. Is to stop the Update Distributor, stop the document conversion and let the extractor finish its job. Then turn on the document conversation and let it "have" the system. Once it has done with the backlog start the Update Distributor.

When going to 9.5 OT recommends a performance audit or architecture/scaling engagement to help clarify the needs moving forward. I would contact the support team and let them help with the distribution planning.