CM variable "NO SPACE LEFT" ?

[ZeroZeroOne]

Here's a new one for me. Also on CM8 which makes me nervous.

     Operating system:  Linux 3.10.0-957.21.3.el7.x86_64 x86_64 x86_64
                Built:  Jun 14 02:54 2019

             Contains:  01.0.890.0
        CM Reports as:  R018x.01.0.890.0
    CM Release String:  vcm-018-01.0.890.0
          RTS Version:  CM 8.1.0.1.1.890.25517
     Publication Date:  10 April 2019
  VMwaretools version:  10.2.5.3619 (build-8068406)
       App Deployment:  Virtual Machine
       VM Environment:  VMware

UPDATES:
Update ID                         Status       Type  Update description
--------------------------------- ------------ ----- ---------------------------
01.0.890.0-25517                  activated    cold  25442 + CM28795

KERNEL-3.10.0-957.21.3.el7        activated    cold  kernel patch KERNEL-3.10.0-

Platform/Security ID              Status       Type  Update description
--------------------------------- ------------ ----- ---------------------------
PLAT-rhel7.6-0010                 activated    cold  RHEL7.6-SSP001


 CM Translation Saved:   2019-10-06 22:00:24

 CM License Installed:   2019-08-22 11:14:40

     CM Memory Config:   Large

Assigning values to variables, I find "NO SPACE LEFT" for some of the operations while reviewing the trace:

08:56:27   90 38 set BM = none ADD 89706
08:56:27   90 38      operand    = []
08:56:27   90 38      operand    = [89706]
08:56:27   90 38      ========= ADD =========
08:56:27   90 38      variable BM = [89706] collect local
...
08:56:27   90 40 set BO = none ADD 89708
08:56:27   90 40      operand    = []
08:56:27   90 40      operand    = [89708]
08:56:27   90 40      ========= ADD =========
08:56:27   90 40      variable BO = [NO SPACE LEFT]
08:56:27   90 40      variable BO = [] collect local

It seems to be random and can happen on the first or second variable just as often as the last.

Variables are set for 5 digits:

Var Description                 Type    Scope Length Start Assignment       VAC
BM  Opt 6 VDN                   collect P     5      1

I've used leading vectors to assign lots of values to persistent variables on other systems but this is the first time on CM8. It's a new system (not production) so perhaps there's a setting that's wrong?

Any suggestions on where to start?

Thanks!

 

[ZeroZeroOne]

Deleted the vector and entered it again. Variables assigned their values as normal so... problem solved?

Still don't know why it happened, however.

 

[dialtonebone]

Is this 8.0 or 8.1? If 8.1 it might be worth raising a ticket over. Seems to be a few easy corruption types of issues with the 8.x.

 

[ZeroZeroOne]

I've noticed lots of issues with System Manager V8 - Mostly interface issues but that disappearing Cover Path was a nightmare! - but this is the first (possible) corruption I've seen on Communication Manager V8. I'll be in this system a lot this week and next so I'll keep an eye on it.

There's lots of cooks with thumbs in this soup right now so perhaps we stepped on each others toes? Mixing my metaphors...

 

[kyle555]

Read the release notes. There's one where making a trunk group have >255 members (a new feature) can corrupt the trunk group.

There's also a clustered SM sig group feature where you can have 2 node names like SM1 and SM2 and make a cluster SM sig group including SM1 and 2 (like audio groups containing many VAL boards).

We didn't use that clustered SM sig group feature on a 7.1-->8.0 upgrade and yet the sig group got corrupted despite us not touching it. By building a new sig group on different ports we got around it. Avaya asked us if we were using that clustered SM feature and we said no because we weren't implementing that in any shape or form - it was just a straight upgrade with all the old config as-is. Yet somehow, CM thru the upgrade managed to trip that bug.

 

[ZeroZeroOne]

Sorry, I didn't mean to imply there were no issues with CM 8.1. I meant this is the first time I've seen "NO SPACE LEFT" when working with variables. It's also the first time I've used a ton of variables like this on a v8.1 so it's still a question of an issue with the software version or this specific installation.

The "NO SPACE LEFT" came back yesterday so it's time to open a ticket with Avaya.

 

[redphone]

This could simply be system limit being reached.

Run display events
You should see:
"355 Can't set, no lcl var xxx/xx"

Run list measurements summary
Under VARIABLE USAGE look for: "Local Variable Average Usage" and "Persistent Variable Average Usage"

The system limit is 24000 for variables, local and persistent. Add up the two values, if you are hovering close to this number during peak times, then it could have just maxed out.

 

[ZeroZeroOne]

Good thought!

Unfortunately, I'm not even close. This isn't a production system (yet) so the only calls right now are for testing purposes.

VECTOR VARIABLE USAGE

 Administered Global Vector Variables (702 Max): 1
           Total Local and Persistent Variables: 24000
                   Total Local Variables in Use: 0
           Local Variable Usage High-Water-Mark: 2
                   Local Variable Average Usage: 0
              Total Persistent Variables in Use: 0
      Persistent Variable Usage High-Water-Mark: 49
              Persistent Variable Average Usage: 0