Tek-Tips is the largest IT community on the Internet today!

Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!

Register Log in
  • Congratulations Mike Lewis on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!

UC Module 9.0 Memory Full ?! 10

Status
Not open for further replies.
Nuedgecom

Nuedgecom

Technical User
May 14, 2013
47
CA
I have an Avaya IPO 9.0.X runing with a UC module installed about 2 weeks ago.
System is barely used since the company hasn't officially launched.
I got an email notification showing that the 25GB are all used on the UC.
logged in to it and found out that there's only 3 short voicemails in the mailboxes.
The Auto attendant is one small recording.
I don't know what's consuming all the memory.
when i try to login to webcontrol i get this error
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/opt/webcontrol/web/wsgiserver/__init__.py", line 1174, in communicate
req.respond()
File "/opt/webcontrol/web/wsgiserver/__init__.py", line 544, in respond
self._respond()
File "/opt/webcontrol/web/wsgiserver/__init__.py", line 556, in _respond
response = self.wsgi_app(self.environ, self.start_response)
File "/opt/webcontrol/web/httpserver.py", line 237, in __call__
return self.app(environ, xstart_response)
File "/opt/webcontrol/web/httpserver.py", line 212, in __call__
return self.app(environ, start_response)
File "/opt/webcontrol/web/application.py", line 293, in wsgi
start_resp(status, headers)
File "/opt/webcontrol/web/httpserver.py", line 234, in xstart_response
self.log(status, environ)
File "/opt/webcontrol/web/httpserver.py", line 250, in log
print >> outfile, utils.safestr(msg)
IOError: [Errno 28] No space left on device
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Does anyone know what could be causing that?
Did that happen with someone?
Help please

Telecommunication Specialist
APSS - APDS - ACIS
 
Sort by date Sort by votes
It is broken.
Replace it with a new one.


BAZINGA!

I'm not insane, my mother had me tested!

 
Upvote 0 Downvote
SOLN236726
Customer is viewing the System Status Application and has noticed the UC Module is showing over 90% full. They are seeing alarms that UC Module is over 90% full.

To confirm the issue, access the UC Module operating system using a ssh client such as PuTTY.
• Login using the Administrator login, then the Admin login and then finally as the root user.

There are several nohup.out files that are growing more than they should.

run the following command from ./
• find -iname nohup.out

This will return the location of all nohup.out files on the hard drive.
Location of the nohup.out files:
./nohup.out (3.3GB)
./opt/vmpro/nohup.out (1.8 GB)
./opt/Avaya/oneXportal/openfire/bin/nohup.out (<100KB)


File sizes will vary depending on system size and configuration.
PROBLEM CLARIFICATION
nohup.out files are being written to incorrectly which is causing the UC Module HDD to fill up unnecessarily.
CAUSE
software
SOLUTION
The nohup.out files can be deleted and a UC Module reboot will be required to recover the disk space.

**WARNING**

Please note that deleting system files should only be completed by knowledgeable personnel as deleting the wrong files can cause irrevocable system damage.

The nohup.out files will continue to be written to even after being deleted as they are recreated on boot. The deletion of these files is only intended to provide interim relief until the available Critical Patch or software solution can be applied to the UC Module.

A Critical Patch has been created to resolve this issue.
CP vmpro-8.1-9302.1_el6.i586.rpm
CP cli-8.1.99-50738.el6.i586.rpm
CP cli-commands-8.1.99-50738.el6.i586.rpm
All 3 critical patches are required to completely resolve the nohup.ot files from being written to the UCM HDD

Referencing:



See also: thread940-1726121


 
Upvote 0 Downvote
If you are doing automatic backups in Voicemail Pro, those backups have a unique file name, for every file. If you are doing daily backups, the file name has the date of the backup in the name.

The problem here, is that the backup does not overwrite itself, and you will soon have a bunch of backups. Normally this isnt an issue, but with the UCM, and a more limited storage, can take up a of space.

So, lets see how to remove some of these files. Luckily Avaya has a perl script built into the UCM, so we just need to point to that and BOOM! WE GOOD!

Log into the UCM with a keyboard and monitor (obviously and HDMI monitor)
Log into root, press Enter
Enter root password, then press Enter
Enter this command, without the quotes, case sensitive:
“perl /opt/vmpro/VMProBackupManager.pl –remove“, then press Enter
Then enter the correlating number to delete that backup, and press Enter.

This will delete the backup file that correlates to the number you enter. Press the UP arrow to rerun the command, and delete another file(s).

 
Upvote 0 Downvote
I ended up replacing the UC the second day thinking that the UC was the problem.
I upgraded another new module i had in stock. (without doing the patch)
Here we are couple of weeks after i'm having the same problem the memory is already full and i can't access the webcontrol (same message appearing).
Now i'm able to login with PuTTY but cant find any of the files taht @Holdmusic34 suggested to delete.

run the following command from ./
• find -iname nohup.out
returns: nothing happens

when i type:
./nohup.out (3.3GB) (No such file or directory)
./opt/vmpro/nohup.out (1.8 GB) (No such file or directory)
./opt/Avaya/oneXportal/openfire/bin/nohup.out (<100KB) (No such file or directory)

I'm logged in as root. Never used SHSH commands but i go nothing to lose trying, the unit is already not functioning so maybe i get lucky if i can delete the files and make some temporary space to apply the patch.



Telecommunication Specialist
APSS - APDS - ACIS
 
Upvote 0 Downvote
Found the answer.
login
Administrator/your password
admin/password
root/rootpasword

cd .. (so now you'r in ./)
rm nohup.out
y

Done ! (deleted the first one)

cd ..
cd opt
cd vmpro
rm nohup.out
y

Done! (deleted the second one)

cd ..
cd opt
cd Avaya
cd oneXportal
cd openfire
cd bin
rm nohup.out
y

Done !

Now after deleting the files, the memory is still full and the web control still not working showing this message:
Traceback (most recent call last):
File "/opt/webcontrol/web/wsgiserver/__init__.py", line 1174, in communicate
req.respond()
File "/opt/webcontrol/web/wsgiserver/__init__.py", line 544, in respond
self._respond()
File "/opt/webcontrol/web/wsgiserver/__init__.py", line 556, in _respond
response = self.wsgi_app(self.environ, self.start_response)
File "/opt/webcontrol/web/httpserver.py", line 237, in __call__
return self.app(environ, xstart_response)
File "/opt/webcontrol/web/httpserver.py", line 212, in __call__
return self.app(environ, start_response)
File "/opt/webcontrol/web/application.py", line 293, in wsgi
start_resp(status, headers)
File "/opt/webcontrol/web/httpserver.py", line 234, in xstart_response
self.log(status, environ)
File "/opt/webcontrol/web/httpserver.py", line 250, in log
print >> outfile, utils.safestr(msg)
IOError: [Errno 28] No space left on device




Telecommunication Specialist
APSS - APDS - ACIS
 
Upvote 0 Downvote
Status update (note: I'm retyping this from memory, not a cut/paste - verify before using this info)

Problem: UCM reports disk usage at 100%

We want to:
1) Log into the UCM to perform a backup
2) Upgrade to the current patch revision
3) Restore the backup config and client voicemail

Because the disk is at 100% capacity, we are unable to perform a backup or even connect to webcontrol.

The problem we found was more than 1,400 11-megabyte log-files (approx 16GB) in the following directory - "/opt/Avaya/oneXportal/9.0.0_523/apache-tomcat/logs".

Steps:
- Log in and change working directory as above
- Run "ls -lh *2014*log | wc" (ours resulted in more than 1,400 files. Each file was 11MB)
- The filenames were something like "1XCSTAServiceRollingFile-2014-03-11-14_36_29.log". The system seemed to create 7 log files each day without removing old files.
- We ran "rm -f *2014*log". Immediately the disk usage returned to normal.

We were able to perform a backup, upgrade and restore without problem.

YAY!!!!!!!

Telecommunication Specialist
APSS - APDS - ACIS
 
Upvote 0 Downvote
It's going to keep doing that and filling up, unless you have a script running to delete the logs.

-Austin
ACE: Implement IP Office
qrcode.png
 
Upvote 0 Downvote
I apologize for bumping an old thread but I just found this answer for a customer issue we are having.

For those of you who are not incredibly comfortable navigating around via Linux (or like me just don't want to on a UC Mod), you can always connect to UC Modules via Filezilla (Freeware) and have a clean to move/transfer/backup files from and perform searches. I found a 10gb nohup.out file on a customer's UC module with a quick search based on filesize. You can search for files over/under/equal to X bytes and it helped immensely.

 
Upvote 0 Downvote
Jhengel make sure to upgrade the UC software. If you simply delete the nohup.out files it would work for a couple of weeks and the problem would come back.

Telecommunication Specialist
APSS - APDS - ACIS
 
Upvote 0 Downvote
Upvote 0 Downvote
I have a customer with this exact issue showing 95% used disk space. Their voice mail has stopped working and I can't web into it. I have followed the above mentioned post and used putty to find 3 nohup.out files which I have removed. After removing the files, shutting down and starting the UC Module back up, it still shows 95% used space. If I connect to the UC Module using FileZilla, I still see a 14GB nohup.out file in the / directory, but I can't delete it using FileZilla because you can only log in as Administrator and not root. If I use putty to get back in, none of the nohup.out files exist!?

Any assistance will be greatly appreciated! I am fairly new to Avaya and am learning a lot on the fly.

Thanks!
 
Upvote 0 Downvote
Yes, I did the reboot through putty and the System Status software.
 
Upvote 0 Downvote
after the reboot mine starting working fine... im still not selling any more of these though

thanks nuedgecom star given
 
Upvote 0 Downvote
Yes, they're a bit of a maintenance liability really, they've got a bit better but I still wouldn't touch them :)

 
Upvote 0 Downvote
We have a couple installed without issues so far.

BAZINGA!

I'm not insane, my mother had me tested!

 
Upvote 0 Downvote
We have had issues woth only one UC module. But since the last update it is running well...


 
Upvote 0 Downvote
Hello All - this bug was addressed in IPO 8.1 SP9 patch which launched May 20th, 2014.
Patching instructions below...

Upgrade your software to 8.1.85 SP9 release on the IP Office and UCM to correct these issues.
UCM 8.1.102-2_upgrade link: Use SSH to remove the files manually.
Manually delete the nohup.out files before load the upgrade for it.
 
Upvote 0 Downvote
Status
Not open for further replies.

Similar threads

Zero6
  • Locked
  • Question
UC module status shutting down 1
Replies
7
Views
338
gkittelson
gkittelson
Jeffnortel
  • Locked
  • Question
11.1 UC Module database restore
Replies
7
Views
192
budbyrd
budbyrd
smrikard
  • Locked
  • Question
C110 UC Module for Voicemail pro 1
Replies
7
Views
217
smrikard
smrikard
ScotttB
  • Locked
  • Question
UC Module missing Gateway
Replies
0
Views
85
ScotttB
ScotttB
mscontinuant
  • Locked
  • Question
UC Module 11.1.1 Disk Space at 93%
Replies
2
Views
204
IPOTS
I

Part and Inventory Search

Sponsor

Back
Top