Micollab MSL I/O problems detected 1

[persi01]

Hi All,

One of our client have mitel MiVB node around 20 in cluster plus one micollab server for voicemail.
Recently on micollab alarms getting Major alarms (I/O problems detected).
Kindly suggest what could be issue?
Also can One login MSL Linux server via winscp?

 

[techymitel]

I am going to assume this is a virtual server?

Have you checked the vCPU count, memory and disk size are the same as initially set? On MSL, under Administration / Virtualization will show the current settings and reservations.

KB article QA1322 will show you where to check that.

I/O problems detected normally happens when the virtual machine struggles to gain resource it needs from the virtual environment - CPU / Network / Disk read/write. It could be happening when your virtual environment takes snapshots or backups of any kind overnight? Or some other work was happening on the virtual environment. It would be worth asking whoever looks after the virtual environment if there were any scheduled tasks or work happening when the alarms started.


Yes, you can access MSL using winscp. Just use the username root and the root (same as admin) password on port 22.

You will need to enable SSH (Security / Remote Access / Set 'Secure Shell Access' to 'Allow access only from trusted and remote management networks')
The source subnet or IP address will need to be already set or added as a trusted network (Configuration / Networks)
OR
The source subnet or IP address will need to added to remote access networks (Security / Remote Access / Add a new remote manangement network)

 

[Billz66]

there was a bug with the MVF blade where it would generate lots of alarms
we mainly noticed it on micloud servers (they get auto snapshot veeam backed up daily)

the latest mvf blade version fixed it up
5.0.25.0 is the version with the fix in it
we also tend to disable the File system monitoring flag under virtualisation




If I never did anything I'd never done before , I'd never do anything.....

 

[TLDuk]

We monitor hundreds of MSLs with MPA and being the early shift engineer got sick of clearing nonsense MVF monitoring alarms every morning..

I spent some time digging into what the MVF alarm actually does and found it's a script that writes to a temporary file then reads it back again, this process is timed and if it takes over 5ms (default) an alarm in generated, the same process is used to recognize a file system locks, rare but it does happen, this is why we do not just turn file system monitoring off.

If you want to see how high the I/O latency was, use this command in the MSL shell

 grep High I/O latency! /var/log/mvf/mvf-mon-tool/current]

The output will look something like this..

The highest numbered test will be the one that triggered the last alarm..

The following command will update the timeout to 30ms and also restart the service, we have found that 30ms seem to cover most overnight slowdowns covered by overnight backups migration etc..

 sed -i 's/^timeout: [0-9]*$/timeout: 30/' /usr/mvf/conf/mvf-mon-tool-cfg.yml && signal-event mvf-mon-tool-update]

Just bear in mind I have had to repair a Micollab with a OVA redeploy that was corrupted, It was generating MVF alarms constantly with 300+ ms time after a storage server failure, so this monitoring is usefull.

If its not broke tweak it..

 

[techymitel]

This log shows times as TAI64N - for an explanation see [URL unfurl="true"]https://cr.yp.to/libtai/tai64.html[/url]

To see real dates and times, pipe the logfile to tai64nlocal

grep -i 'high i/o' /var/log/mvf/mvf-mon-tool/current | tai64nlocal

This will show a date and time that will match up with your event viewer log.

This will work on any other logfile showing the long TAI64N date stamp.

 

[SXWizard]

TLDuk - Tried the command to set to 30ms on our Micollab and got this error:
Bad event name on input: mvf-mon-tool-update] at /sbin/e-smith/signal-event.perl line 31.

 

[techymitel]

Looks like TLDuk's commands have an extra ']' on the end.

Try:

sed -i 's/^timeout: [0-9]*$/timeout: 30/' /usr/mvf/conf/mvf-mon-tool-cfg.yml && signal-event mvf-mon-tool-update