Creating a second Dataflow

[cristimitulescu]

Hi,

We have a fairly large 9.7.1 system and we seem to be adding more content at times then the current Enterprise dataflow can process.

I have created multiple document converter processes, multiple index engines, have spread processes across multiple servers, use dedicated relatively high end NetApp NAS for the ipool location and index partitions but still fail to stay on top of it at times.

The bottleneck seems to be the Update Distributor. I don't believe you can have multiple Update Distributors and not sure that the Update Distributor can be multithreaded. The servers where the Update Distributor and index partitions are hosted do not have CPU or memory issues, the latency of the index volumes on the NetApp is very low averaging 1.5ms(fair amount of ssd cache) so for the life of me I don't see why we get big queues of pending messages at the update distributor.

I am now trying to explore the possibility of creating a second Dataflow and assign it either an existing branch of the folder/tree structure or create a new tree under the Enterprise and assign it to the new Dataflow.

However I haven't come across much information on how you set this up. Does anyone have documentation or experience with multiple dataflows?

Thank you,
Cristi

 

[appnair]

You might want to engage OT prof
Svcs for money While it can be done
By painful self studying it
Will never be anything close
To what an experienced OT search
Person could provide alternatively
Try others who may be ex OT who
Could help you.Almost any thing except
Multiple extractions(avoid it if
You can) you will be able to do
Without any real problems

Well, if I called the wrong number, why did you answer the phone?
James Thurber, New Yorker cartoon caption, June 5, 1937
Certified OT Developer,Livelink ECM Champion 2008,Livelink ECM Champion 2010

http://www.tek-tips.com/faqs.cfm?fid=2884

http://www.linkedin.com/in/appunair

 

[cristimitulescu]

I will be placing a support call but was trying to do some of my own research. So far Support as good as they are they also seem biased towards keeping the architecture extremely simple in the interest of making supporting the thing easier.

Strangely enough I have been asked in the past by their experienced search support personnel why I don't keep all the admin processes on a single server and the dataflow ipools on local disk. This seemed a strange bit of advice when we had 70+ index engines at the time and the supported OS was win32 only with limited max RAM. I don't believe you can give it more than 64GB RAM using /PAE.

 

[appnair]

For what its worth I have always made my data_flow local and my indexes on network devices.I have had no problems with data_flow on SAN/NAS but it is increasingly hard to get OT support as when they see that not local they seem to cry foul.I have had no problems other than setup nuaances on doing this.You setup everything local once then you update the index things using just windows file copies.As I said everything except multiple extractors I have tried(even that too) and have had good success.Not sure where your bottle neck is.

I have been told by OT experts that if you think that the data growth for a year will need about 8 partitions I would do it 2 at a time
That is to help with hydration and an even loading.Were you having all your partitions pre-created to save you time ?

Well, if I called the wrong number, why did you answer the phone?
James Thurber, New Yorker cartoon caption, June 5, 1937
Certified OT Developer,Livelink ECM Champion 2008,Livelink ECM Champion 2010

http://www.tek-tips.com/faqs.cfm?fid=2884

http://www.linkedin.com/in/appunair

 

[ggriffit]

This also seems to be discussed at [URL unfurl="true"]https://knowledge.opentext.com/knowledge/cs.dll?func=ll&objId=26740821&objAction=viewincontainer&ViewCommentsByReply&ListView&ListViewID=26740821[/url]

Greg Griffiths
Livelink Certified Developer & ECM Global Star Champion 2005 & 2006

http://www.greggriffiths.org/livelink/

 

[ggriffit]

This also seems to be discussed at [URL unfurl="true"]https://knowledge.opentext.com/knowledge/cs.dll?func=ll&objId=26740821&objAction=viewincontainer&ViewCommentsByReply&ListView&ListViewID=26740821[/url]

Greg Griffiths
Livelink Certified Developer & ECM Global Star Champion 2005 & 2006

http://www.greggriffiths.org/livelink/