LAPI Bulk upload

[hanreb]

Hi,

I am trying to upload a large number of documents into livelink, thru my java program, which uses LAPI.
I see a real performance drawback here.

Here are some data that I can provide.

Besides the data down I have calculated the time at different points and found that most of the time is consumed in calling LAPI functions.

1)Time taken to upload 30 .pdf Documents



No.of documents uploaded – 30

Time taken for upload – 48.585seconds

Time taken to upload 1 doc on an average - 1.6 seconds



2) Time taken to upload 204 .pdf Documents



No.of documents uploaded – 204

Time taken for upload – 627.087 seconds = 10min 45 seconds

Time taken to upload 1 doc on an average – 3.07 seconds





3) Time taken to upoad 629 .pdf documents



No.of documents uploaded – 629

Time taken for upload – 6764.603 seconds =112 min = 1.87 hours

Time taken to upload 1 doc on an average – 10.75 seconds

Below is my issue.

Now if I put say 1200 .pdf documents in a batch, it takes even 11 hours for all these 1200 .pdf documents to be uploaded.

You can assume that I have a dedicated LLServer.

Where do you think I am going wrong?
Could you please advice.


Thanks,
Sunu

 

[appnair]

Does the user of your lapi program have Sysadmin privs if they don't then permissions check is slowing you down.
Are the documents to be uploaded local to the server are are you uncing them from somewhere.How bg are these on average
How many Categories are added/inherited per object ?

Is your db oracle ?
Can you run threadtimings logs on your server to see where the potential bottleneck?

Do you have another lapi program such as explorer that you can baseline the same conditions ?


Well, if I called the wrong number, why did you answer the phone?
James Thurber, New Yorker cartoon caption, June 5, 1937

 

[appnair]

What I said should only be taken as potential places to look for.10 seconds sound a lot but that is probably how your business rules are made.Rather than saying lapi or livelink is at fault I would first try to understand how your system behaves.Not all livelinks work the same.It is prone to all kinds of network iniiated problems such as a poorly configured network,the webserver being used excessively,not enough livelink threads to service etc.The only way IMHO is to conduct a performance evaluation by OT or a partner and to get a clean bill of health,then we start pointing fingers

Well, if I called the wrong number, why did you answer the phone?
James Thurber, New Yorker cartoon caption, June 5, 1937

 

[hanreb]

Hi appnair,
Here are my answers in line


1.Does the user of your lapi program have Sysadmin privs
User has the Native Admin Account.I mean Application Admin privileges

if they don't then permissions check is slowing you down.

2. Are the documents to be uploaded local to the server are are you uncing them from somewhere.
We have the files to be uploaded in a remote server from which we access the livelink server.

3. How bg are these on average
File says varies from 14 K to 2999K
On an average may be 500 K, bcos the files with 4 digit size are not more than 10.

4. How many Categories are added/inherited per object ?
One category per object, and each category has 14 attributes.

5. Is your db oracle ?
Yes
6. Can you run threadtimings logs on your server to see where the potential bottleneck?

Could you explain a little more on how to do this ?

7. Do you have another lapi program such as explorer that you can baseline the same conditions ?

We have the livelink explorer module, which we have not used.
Can you please explain more on this.

Thanks,
Sunu

 

[appnair]

1)The only thing that speeds the process is perhaps your lladmin who has access to users and group can check whther the lapi user account has SA on.It is abitwise priv on the users and groups screen.Or are you sing the account called "Admin" to connect ?
6)In the llserver that replies to your lapi requests under options of the opentext.ini file you want to put WantSummaryTimings=TRUE.Then re-start livelink server after a while you will start seeing a file called thraedtimings1.csv for each ll thread.Inside that you will see that LL registers the time componenets for each transaction.If you do not have access to livelink performance analyzer you may want to do some excel macro to group those and anlyze yourself.How to analyze and group that is too much to explain here but the KB has it.
7)Livelink explorer is a lapi program that has a server component and a client component.The client when properly connected appears in the windows explorer with which you can drag/drop files without going thru the web GUI.You could possibly get a baseline by highlighting all those files on that remote server and drop them in livelink.If you have mandatory stuff on categories you are out of luck the category screen will popup for each document.


Well, if I called the wrong number, why did you answer the phone?
James Thurber, New Yorker cartoon caption, June 5, 1937

 

[hanreb]

Hi appnair,
I am using the account called "Admin"
Does that slow down the process.
If so, could you tell me which is the sysadmin that you are referring to ?

Thank You,
Sunu

 

[appnair]

No 'Admin' is the best account .It has SA.If you want to know a bit more there are these privs to a users account.Usaully most users will have
Login
PA (Public Access)
CCMU(can create users)
CCMG (can create groups)
UA (user administrator)
SA (system admin)

If a user's account has SA on it then livelink overrides the permissions checks.Since you are using Admin your code is being executed with relatively no node permission checks.

Your system needs analysis if that is this slow.Can't really pinpoint anything thru this discussion.Open a ticket with OT ,prvide them your code,let them test it in thir system and let them advice you if it is your code or your system

Well, if I called the wrong number, why did you answer the phone?
James Thurber, New Yorker cartoon caption, June 5, 1937

 

[davidt50]

Another approach could be to get the LAPI app to do multiple requests to differing threads and/or instance to divide the load.

You may want to check network settings. There has been a number of times where network packet sizes etc can affect upload performance. Check with OT-CS and they should be able to provide details on what to check.

 

[davidt50]

Forgot to add, if your application can move the documents to a Livelink server and do the upload locally, it should be significantly faster.