"resource out of service"

[cuneyt2919]

Hi,
Has anybody her got this error with Genesys 6.5. environment?

The problem is, lots of ADN get this error on their desktop application. And so the agent can not login to CTI environment. The genesys vendor says"its related with the switch vendor."

We can fix it, but thinking of the whole call center, we get this error ~10 desks per day...

If there is a genesys expert here or a CSTA expert , please help me..
thx...

 

[fidde]

You get this for example if you go into programing modeon the DTS, maybe that could be the answer. To go deeper the AL trace showing the problem would help.

 

[cuneyt2919]

Hi it's not seen on tracelogs by Applicaiton Link side
And here is the answer by genesys side, from Ticket archive..
*********************************************************
.........do not see any attempt by Tserver to send a CSTA request across the Applink.

The MD110 Engineers cannot see any CSTA message of the Applink side either.

To me, this indicates that the Tserver is maintaining some state information for the ADN/ODN, as it reports immediately that the "resouce is out of service"

I also noticed in the Tserver log that the following message appears

"write is checked"

It may be that the Tserver is buffering the requests to the Applink for some reason, or may be not.

This problem results in agents not being able to login or go ready/not ready.

The number of extensions this impacts goes up to at least 50 over a few days, after which the customer is forced to restart the Tserver and Applink.

The MD110 Engineers have noticed that the following occurs as they see it on the MD110 and Applink.

Agent goes ready/not ready 30 times in one minute, works fine
Agent goes ready/not ready 45 times in one minutes, resource error occurs.
Tserver reports extension as being ready.
MD110 and Applink report extension as being not ready.
Sending a request from Desktop application to Tserver to go ready does not result in any message being sent to Applink.
Tserver reports resource out of service immediately.

Doing the above, they can simulate the problems we see easily on the lightly loaded TServer at ST2.

The main problems we see are at the Tserver at PHP 30th and 31st floor ,we are using the Tserver at ST2 for testing.

DN Information
ODN = 8269
ADN = 21269
ACD = 1898
Login = 6000
Password = 6000

We restarted the Tserver and the Applink, the retested.

I sent multiple ready/not ready requests as fast as possible, but could not get the "resource out of service" message using the Genesys Deskotp 5.1 client

I did see "write is checked" messages with up to 145 write checks at one stage.

We then tested using the in house CIC softphone.

After going ready/not ready for 30 seconds we got the resource out of service message after about 30 seconds.

I did notice that the CIC softphone does not buffer requests, the agent can press ready/not ready at will without waiting for any acknowledgement from Tserver.

The log file "event_request_sequence.log" shows the events and requests coming from the CIC Softphone.

I can see a sequence of 4 RequestAgentReady messages one after the other that do not result in an EventAgentReady being sent back to the CIC Softphone.

There is no indication in the log that the reference ID for these requests ever gets responded to.

This shows up as......

05/21/03 17:09:06.750 Trace ts_st2 TServer_ST2 GCTI-00-04541 Message RequestAgentNotReady received from 476 ( regular DesktopToolkitX )
05/21/03 17:09:07.734 Trace ts_st2 TServer_ST2 GCTI-00-04541 Message RequestAgentNotReady received from 476 ( regular DesktopToolkitX )
05/21/03 17:09:08.640 Trace ts_st2 TServer_ST2 GCTI-00-04541 Message RequestAgentNotReady received from 476 ( regular DesktopToolkitX )
05/21/03 17:09:09.640 Trace ts_st2 TServer_ST2 GCTI-00-04541 Message RequestAgentNotReady received from 476 ( regular DesktopToolkitX )


There are other instances of the Softphone sending unacknowledged messages to Tserver.

Is this a problem with the softphone or with the Tserver in that it is not forcing the serialization of events or losing events.

We then retested using five softphones and getting the users to send events about on every two seconds.

After 60 seconds, 2 extensions had come back with "resource out of service" -- see log file TServer_ST2.five_phones_over_one_minute_8275_and_8272.log
for extensions 8275 and 8272.

Finally we tested the same scenario as above (5 phones) using Genesys Softphone 5.1. We did not see any instance on the "resource" error, but instead got
generic state errors -- these could be fixed by changing the ADN and/or ODN into ready or not ready.

So, from all of this, is there something inherent in the Tserver that causes the "resource out of service" errors or can we conformtably resolve these by altering the softphone?

Is there some protection mechanism in the Tserver for multiple events that causes this error. ?



Answer/Resolution


This problem has relation to stability of reporting from the Application Link. Please note that customer configured all Agent devices with Switch Specific type 10. For such devices T-Server does not start monitor on Application link until Request Agent Login is requested from the application..

Before the RequestAgentLogin all requests except RequestRegisterAddress and RequestQueryAddress will be rejected with error "Resource Out Of Service" without interaction with the switch (because monitor is not started for that device).

All requests became acceptable only after RequestAgentLogin. And monitor will be stoped after Agent is requested RequestAgentLogout.

Therefore, to resolve this issue Agent should perform RequestAgentLogin regardless of actual Agent status is reported in Attribute Extension in event Registered (i.e. force request Login).

For the reference please see chapter "Limitation and Restrictions" in T-Server for Ericsson MD110 Reference Manual :
1. From Application Link service pack 4, Genesys cannot guarantee Ready/NotReady status for ODNs/ADNs at T-Server start/restart. With Application Link using service pack 4 or earlier, the switch fails to report the initial state for device and agent work modes at T-Server startup.

If customer is using Switch Specific type 10 for Agent DNs this paragraph became applicable not only to Startup scenarios, but also to Login-Logout scenarios.


PS. The situation with handling of incorrect reporting of initial Agent state on device might be improved in T-Server version 6.5.304.02. This version was built specifically for this customer. But anyway - force Request Login is the best solution.

Customer upgraded to HotFix TServer 6.5.304.02 and retested okay.

 

[fidde]

Well I do not think this is a Ericsson problem. The way you force it could indicate that the tserver do not handle the request responses IE when you make a request you need to wait for an acknowlege before you make the next request. If you do not do this it will result in a failure.

The other thing I could think of is that it could be a problem between AL and MD. What version is MD and AL. It also would help to see tha AL log from the test mentioned above then it would be easy to see what causing the problem (I hope).

 

[cuneyt2919]

Hi fidde,
md110 is bc10 cn144 (latest patch level) and AL is 4.0 SP9 HF3...

I saw this error in some times, such as,

1.When i send ready, not ready request lots of times in a small period,to Application Link, Tserver gives "resource out of service" errors... Genesys says, this is because of switch can not answer to this requests...
2. When i add a new ADN or remove an ADN on ODN, i mean after changing some configurations on ODN,i saw that T server can not start a new monitor on ODN...


do you know how many requests can be sent to Application Link per second..Is it related with Application link hardware or MDcapacity or what? I have 24 lim and 16 NIU cards and we're adding 8 more this week... but will it fix the problem?

 

[fidde]

Well, I have as a guideline to use one NIU for 200 extensions if it's a call centre and 1000 for normal extension. And on the MD side I have never had problems. And for AL I have not seen any prolem (I think I heard a maximum of 8000 extensions). How many extension do you have 16NIU sounds alot....and with my calculation you can judge yourself if it will help.

The only limitation that is important is the one I tried to explain above.

If you press en Not Ready on the phone, this will be sent to AL that sends it to MD. When MD has executed the request it sends a answer back to MD saying OK/NOK. This will AL send to the requesting client.

The softwar needs to wait for this confirmation before it sends a new request. This is a limitation in the MD as it can not process more then one request at the time. If you do not do this AL will return errors.

 

[cuneyt2919]

Hi fidde,
1 NIU per 200 extensions, sounds good...
But Our MD is a very busy switch.. 3 million calls per month, we have 400 or 500 extensions (agents) for call center and nearly 1200 monitored DNs such as IVR ports+ODNs+ACD groups..
most of calls are answered by IVR as banking applications. So i think, md can not response to CTI system as it should ... Most of times i see 1 minute interval errors on CSTA ports, also MIS ports for CCM...
I'm not sure whether it's related with network issue or md issue..

I asked also people here, if BC10 supports the latest SP for AL (AL4.0 SP10) But still no answer... maybe latest SP can help us about status problems sucjh as "ready state "not ready state" of agents and this kind of problems.

 

[fidde]

Well I do not think it's capacity problem in MD. But There is a problem with the NIU board. It's importent that you configure the switch port in MD to be 10 Mb half duplex. Also I use to recomend that you use a seperat "NIU lan" IE you put dual NIC interface in the applink server and have only that and the NIU bords in a seperat lan. That way you do not have to wory for brodcast storms and such things.

So from what you tell me I think the problem is that you lose the connection between AL and a NIU board and then you get this problem. When agent logout the monitor is restarted (maybe over a new link) and everyting works again.

So first I check the switch configuration and if that do not help move to a seperat lan. Also check the firmware on the NIU. For AL SP9/SP10 is not official supporting BC10 so I think you will have a hard time getting a answer. Best would ofcourse be if you could upgrade MD to BC11 but I see your problem there.

Let me know how it goes....

 

[cuneyt2919]

Hi again
we're trying to upgrade to BC12 but Genesys is still not supporting bc12 so we could not upgrade it.But we 're adding new magazines to system,and 8 NIU2s will be added to bc10 system.. We tried one of the NIU2 card and no problem occurs on BC10..

 

[doktor]

Hi cuneyt2919

We have installed a NIU board in each LIM containg agents for the Call Centre in a Genesys installation.
All these NIU boards were placed in a LINK group.
That helped a lot concerning the siganlling towards the Application Link server.
I can see that you also did this.
Our customers are using BC11 SP12, with no problems.
Another hint could be to spread your analogue or digital IVR lines over more LIMs to avoid INTERLIM signlling and trafic.
A good advise is to have your CSTA NIU boards separated from other functions as ACD/CCM, Intercept (*23*), Voice mail (Onebox) and logins.
The CSTA NIU should run CSTA services. All other service should be installed on other NIU boards.

Best regards diktor

 

[cuneyt2919]

Hi doktor and all,
16 NIUs are used for CCM and AL also..
After genesys ERS implementation we don't need CCM as it is now, (We will use CCM but with less importance)


So what happens when i use only 2 or 3 NIU cards for CCM, and all of NIUs for Application Link,Do you think, switch will be healthy?
I have doubts about it, because, when only 3 lim's NIU cards will be used for CCM , all MIS datas will go through this 3 lim.. And PCM links will be busy of these 3 lims...

What do you think so?

[tt]

<ACFUP;
ACD MIS INDIVIDUAL DATA
LGRP IODEV LIM STATUS
CCM LIM1ETH 1 CONNECTED
LIM2ETH 2 CONNECTED
LIM3ETH 3 CONNECTED
LIM5ETH 5 CONNECTED
LIM7ETH 7 CONNECTED
LIM9ETH 9 CONNECTED
LIM12ETH 12 CONNECTED
LIM14ETH 14 CONNECTED
LIM15ETH 15 CONNECTED
LIM17ETH 17 CONNECTED
LIM19ETH 19 CONNECTED
LIM21ETH 21 CONNECTED
LIM22ETH 22 CONNECTED
LIM23ETH 23 CONNECTED
<CSTLP;
COMPUTER SUPPORTED TELECOMMUNICATIONS APPLICATIONS LINK GROUP DATA

LGRP ACTIVE JOBS IODEV LIM STATUS

APPLINK 1983 LIM1ETH 1 CONNECTED
LIM2ETH 2 CONNECTED
LIM3ETH 3 CONNECTED
LIM4ETH 4 CONNECTED
LIM5ETH 5 CONNECTED
LIM7ETH 7 CONNECTED
LIM9ETH 9 CONNECTED
LIM11ETH 11 CONNECTED
LIM12ETH 12 CONNECTED
LIM14ETH 14 CONNECTED
LIM15ETH 15 CONNECTED
LIM17ETH 17 CONNECTED
LIM19ETH 19 CONNECTED
LIM21ETH 21 CONNECTED
LIM22ETH 22 CONNECTED
LIM23ETH 23 CONNECTED

END


[/tt]



Rgrds

 

[fidde]

For your info so are we running BC12 with genesys though it's not formal tested of genesys but it works without problem. Even if it's a smal system compared to yours.

 

[cuneyt2919]

Hi, now this is what i mean

351 ACD MIS, FAULTY COMMUNICATION CHANNEL
DATE TIME ALP NOIF UNIT EQU BRDID INF1
18AUG04 11:53:04 47 2 DCTPP 001-0-70-04 106 4
RDATE RTIME
18AUG04 11:53:16
351 ACD MIS, FAULTY COMMUNICATION CHANNEL
DATE TIME ALP NOIF UNIT EQU BRDID INF1
18AUG04 11:53:36 49 2 DCTPP 002-0-70-04 106 4
RDATE RTIME
18AUG04 11:53:48
351 ACD MIS, FAULTY COMMUNICATION CHANNEL
DATE TIME ALP NOIF UNIT EQU BRDID INF1
18AUG04 11:53:36 53 2 DCTPP 012-0-60-04 106 4
RDATE RTIME
18AUG04 11:53:49
351 ACD MIS, FAULTY COMMUNICATION CHANNEL
DATE TIME ALP NOIF UNIT EQU BRDID INF1
18AUG04 11:53:36 54 1 DCTPP 017-0-70-04 106 4
RDATE RTIME
18AUG04 11:53:48
351 ACD MIS, FAULTY COMMUNICATION CHANNEL
DATE TIME ALP NOIF UNIT EQU BRDID INF1
18AUG04 11:53:36 55 1 DCTPP 021-0-70-04 106 4
RDATE RTIME
18AUG04 11:53:49
351 ACD MIS, FAULTY COMMUNICATION CHANNEL
DATE TIME ALP NOIF UNIT EQU BRDID INF1
18AUG04 11:53:37 50 2 DCTPP 009-0-70-04 106 4
RDATE RTIME
18AUG04 11:53:49
351 ACD MIS, FAULTY COMMUNICATION CHANNEL
DATE TIME ALP NOIF UNIT EQU BRDID INF1
18AUG04 11:53:38 57 1 DCTPP 023-0-70-04 106 4
RDATE RTIME
18AUG04 11:53:48
351 ACD MIS, FAULTY COMMUNICATION CHANNEL
DATE TIME ALP NOIF UNIT EQU BRDID INF1
18AUG04 11:53:39 51 2 DCTPP 007-0-70-04 106 4
RDATE RTIME
18AUG04 11:53:49
351 ACD MIS, FAULTY COMMUNICATION CHANNEL
DATE TIME ALP NOIF UNIT EQU BRDID INF1
18AUG04 11:53:39 52 2 DCTPP 005-0-70-04 106 4
RDATE RTIME
18AUG04 11:53:49
351 ACD MIS, FAULTY COMMUNICATION CHANNEL
DATE TIME ALP NOIF UNIT EQU BRDID INF1
18AUG04 11:53:39 58 1 DCTPP 022-0-42-04 106 4
RDATE RTIME
18AUG04 11:53:48
351 ACD MIS, FAULTY COMMUNICATION CHANNEL
DATE TIME ALP NOIF UNIT EQU BRDID INF1
18AUG04 11:53:39 59 1 DCTPP 014-0-42-04 106 4
RDATE RTIME
18AUG04 11:53:49
351 ACD MIS, FAULTY COMMUNICATION CHANNEL
DATE TIME ALP NOIF UNIT EQU BRDID INF1
18AUG04 11:53:39 60 1 DCTPP 019-0-70-04 106 4
RDATE RTIME
18AUG04 11:53:48
351 ACD MIS, FAULTY COMMUNICATION CHANNEL
DATE TIME ALP NOIF UNIT EQU BRDID INF1
18AUG04 11:53:40 48 2 DCTPP 003-0-70-04 106 4
RDATE RTIME
18AUG04 11:53:49
351 ACD MIS, FAULTY COMMUNICATION CHANNEL
DATE TIME ALP NOIF UNIT EQU BRDID INF1
18AUG04 11:53:40 61 1 DCTPP 015-0-70-04 106 4
RDATE RTIME
18AUG04 11:53:49

 

[fidde]

Yep that's it you have a network problem between AL server and NIU board. Follow the suggestion I wrote earlier:

Well I do not think it's capacity problem in MD. But There is a problem with the NIU board. It's importent that you configure the switch port in MD to be 10 Mb half duplex. Also I use to recomend that you use a seperat "NIU lan" IE you put dual NIC interface in the applink server and have only that and the NIU bords in a seperat lan. That way you do not have to wory for brodcast storms and such things.

So from what you tell me I think the problem is that you lose the connection between AL and a NIU board and then you get this problem. When agent logout the monitor is restarted (maybe over a new link) and everyting works again.

So first I check the switch configuration and if that do not help move to a seperat lan. Also check the firmware on the NIU. For AL SP9/SP10 is not official supporting BC10 so I think you will have a hard time getting a answer. Best would ofcourse be if you could upgrade MD to BC11 but I see your problem there.

Let me know how it goes....

 

[cuneyt2919]

Hi fidde,
I asked to our network dept. guys to check out whether there is a problem on the switch ports at that time, and they checked and told me that there is no error related with network issue..
So regardless of network, what should be done?
For exp.
i checked NIU firmwares..
[tt]
<CNBIP:BPOS=7-0-70;

DEVICE BOARD REVISION INFORMATION
DATE: 20AUG04 TIME: 14:36:01

BPOS BOARD/BRDID PRODUCT NUMBER REVISION STATUS ADD
/IND

007-0-70 NIU ROF1375396/1 R3A
CAA 1580001 R15A EXE
CAA 1580001 R9A OLD

END

<CNBIP:BPOS=3-0-70;

DEVICE BOARD REVISION INFORMATION
DATE: 20AUG04 TIME: 14:38:00

BPOS BOARD/BRDID PRODUCT NUMBER REVISION STATUS ADD
/IND

003-0-70 NIU ROF 137 5396 R3A
CAA 1580001 R15A EXE
CAA 1580001 R2A OLD

<CNBIP:BPOS=1-0-70;

DEVICE BOARD REVISION INFORMATION
DATE: 20AUG04 TIME: 14:38:26

BPOS BOARD/BRDID PRODUCT NUMBER REVISION STATUS ADD
/IND

001-0-70 NIU ROF1375396/1 R3A
CAA 1580001 R9A EXE
CAA 1580001 R9A OLD

END
[/tt]

What do you think so about NIU firmwares?
Also what do you think about deleting some NIU cards from CCM application?(ACFUE)
DOes this help to PBX about load balancing or not? Also does this help to CTI perfomance using some NIU cards only for MML and CSTA application?
Rgrds.

 

[whosrdaddy]

most stable firmware up to date is R17A for NIU/1. don't know if this firmware is supported for BC10. anyway, if know some large (~stable) genesys installations and the md110's are all BC11SP13

--------------------------------------
What You See Is What You Get