Mitel Virtualization Diagnostics Tool

The intended use of the Virtualization Diagnostic tool is to pinpoint performance and voice quality issues found when running Mitel applications in a virtual environment. The tool is especially helpful for customers who do not have control of the underlying infrastructure but are interested in determining the cause of problems.  

The Diagnostic tool is a component of the Mitel Virtualization Framework (MVF) and includes a "Mitel Virtualization" screen that appears within the MSL Server Manager. The screen enables you to obtain an overview of the virtual machine and MVF properties, manage storage monitoring, receive a diagnostic overview, configure a connection to the vCenter server or ESXi hypervisor, and run the diagnostic tool to generate a variety of log files containing statistical, performance and configuration data.

Note: The recommended method to monitor disk latency for a VM is to set up a monitoring alarm in vCenter. To do this, right-click the VM, select Alarm, New Alarm; give the alarm a name that contains the VM name, assign ‘VM Max Total Disk Latency’ as the key of the IF condition and finish defining the alarm as needed. Some recommended parameters from VMware is defining the alarm with 25ms threshold and violation duration of 30 seconds.  Email action can be specified so the appropriate contacts can be notified when problem is detected.
Disk latency monitoring is a best-effort task within the guest operating systems because the correct measurement information is not available in guest OS and therefore it's impossible for MVF to monitor disk latency the same way VMware could.
MVF does monitor various symptoms associated with disk performance issues and creates an MSL alarm whenever MVF detects slow disk responses. Slow disk response is what MVF can tell at best.  The root cause for lack of response may be actual disk latency, heavy cpu load on the vSphere host, heavy cpu load on the VM itself and many other reasons combined.

Supported Applications

To employ the Diagnostics tool, you require the following:

Reviewing the Virtual Machine Properties

The "Virtual Machine Properties" table displays information concerning the Virtual Machine and Mitel Virtual Framework. The information is presented in two columns:   

To review the virtual machine properties:

  1. Under Administration, click Virtualization.

  2. Under Virtual Machine Properties, review the following settings:

 

Setting

Description

MVF Version

The version number of the Mitel Virtualization Framework (MVF), a software package that enable Mitel applications to run in a virtual infrastructure. MVF has the capacity to support multiple operating systems and hypervisor products.

Virtualization Agent Version (VMware Tools)

The version number of VMware Tools, a suite of utilities that enhances the performance of the virtual machine's guest operating system and improves management of the virtual machine.

Hypervisor Version

The version number of the VMware ESXi hypervisor that hosts one or more virtual machines and their "guest" operating systems.

vCPU count

The number of virtual Central Processing Units (vCPUs) configured on this virtual machine.

Memory (MB)

The amount of virtual physical memory available for use by the operating system on this virtual machine.

Disk size (GB)

The virtual disk size available for use by the operating system on this virtual machine.

NIC count

The number of virtual network interface cards configured on this virtual machine.

CPU Reservation (MHz)

The guaranteed minimum allocation of CPU resources for this virtual machine.

Memory Reservation (MB)

The guaranteed minimum allocation of memory resources for this virtual machine.

CPU Limit (MHz)

The upper limit of CPU resources that can be allocated to this virtual machine. This limit is expressed in concrete units (Megahertz) and cannot be exceeded.

Memory Limit (MB)

The upper limit of for memory resources that can be allocated to this virtual machine. This limit is expressed in concrete units (Megabytes) and cannot be exceeded.

vCPU Speed (MHz)

The speed of the virtual CPU, which is dependant on the speed of your underlying processor. So if you have a 12 cores and a processor speed of 3.36GHz, that means a virtual machine with a single vCPU running a single threaded application can consume 3.36GHz.

The setting defines what a single vCPU will consume, not the aggregated amount among multiple vCPUs on a single virtual machine. Accordingly, if you have two vCPUs this figure should be doubled.

Managing Storage Monitoring

Use this tool to detect degrading storage conditions and take corrective actions.

To manage the storage monitoring settings:

  1. Under Administration, click Virtualization.

  2. Under Storage Monitoring, enter the following settings:   

 

Setting

Description

File System Monitoring

Use this setting to specify whether file system monitoring is enabled or disabled. If the feature is enabled (the default), the system will check for disk I/O errors every five seconds. If any errors are detected, a warning notification is sent to the "admin" email address configured on the Email Settings screen.  

The following errors are monitored:

  • File system errors: Errors related to storage degradation

  • CPU Starvation: When the monitoring process is not dispatched within a specified time (default is 5 seconds).

  • High I/O Latency: When I/O operations exceed the pre-determined duration of  5 seconds.

Reboot on Read-Only State

If this setting is enabled (the default), the system will automatically reboot whenever it enters read-only state. After the system reboots, all disk I/O errors will be cleared and the system will be in read-write state.

Notes:

  • File System Monitoring must be enabled before this feature can be employed.

  • Read-only state occurs when there are I/O errors on the virtual machine disk drives, and is intended to protect the file system from damage.

  1. Click Save.

Reviewing the Diagnostic Overview

The Virtualization Diagnostic tool constantly monitors the system in order to report on three alarm conditions and the state of the last nightly analysis.

To review the virtualization diagnostics overview:

  1. Under Administration, click Virtualization.

  2. Under Diagnostic Overview, review the following settings:  

 

Condition

Description

States

Hypervisor Version

Indicates whether or not the version of VMware ESXi Hypervisor is supported. The Hypervisor is also known as the Virtual Machine Monitor (VMM).

  • Supported - Your ESXi version is supported and no changes are required.

  • Unsupported - Your ESXi version is not supported and you must switch to a supported version in order to restore monitoring functionality. For example, if you are running ESXi 4.0 or earlier, you must upgrade to version 4.1 or later.

Current Dimensions

Indicates whether the currently configured application  resource dimensions are supported.

  • Supported - Your configuration is supported and no changes are required.

  • Unsupported - Your configuration is not supported due to a setting (vCPU count, Memory, Disk size, or NIC count) being out of boundaries. To resolve any performance issues, do the following:

  1. Revert to the default configuration for your deployment. For details, see Default Configurations.

  2. Contact Mitel Product Support for assistance.

AMC Connectivity

Indicates whether the Virtual Machine can connect to the Mitel Application Management Center (AMC) for licensing purposes.

  • Connected - Your Virtual Machine can connect to the AMC.

  • Error - Your Virtual Machine cannot connect to the AMC. Check the networking configuration and Application Resource ID (ARID). See the Mitel Standard Linux Installation and Maintenance Guide for more information.

Last Nightly Analysis

Indicates the date and time that the last nightly analysis was completed, and whether any problems occurred while it was being run. Upon successful completion, the nightly analysis generates the following log file: NIGHTLY-REPORT-YYYY-MM-DD.txt

YYYY/MM/DD & Problems (if any)

Configuring the Virtualization Diagnostics Credentials

To enable the Virtualization Diagnostics tool to collect statistics for the virtual machine and the host, and then use the statistics to generate log files, you must enter credentials for the vCenter server or ESXi hypervisor.

The information collected depends on the credentials entered:

Note: For optimum results, enter credentials for the vCenter. Entering credentials for the ESXi may result in connectivity problems if settings are changed on the hypervisor.

Enter New Credentials

To enter the virtualization diagnostics credentials:

  1. Under Administration, click Virtualization.

  2. Under Virtualization Diagnostics, enter the following settings:   

 

Setting

Description

FQDN or IP address

Enter the Fully Qualified Domain Name or IP address of the vCenter or ESXi hypervisor.

Username

Enter the username required to access the vCenter or ESXi hypervisor.

Password

Enter the password required to access vCenter or ESXi hypervisor.

Nightly Analysis Time

Specify the one-hour period during which the nightly analysis will be run each day. Select hours between 0-1 and 23-24.

Upon successful completion, the nightly analysis generates the following log file: NIGHTLY-REPORT-YYYY-MM-DD.txt.

  1. Click Save.

Once a connection is established, the system will obtain performance statistics for the virtual machine and the host, and you may click the Run Diagnostics button in order to manually generate log files and an online report. For more information, see Manually Generated Log Files.

Note: For a newly installed system, wait for it to collect statistics for at least 15 minutes before clicking the Run Diagnostics button.  

Remove Current Credentials

To remove the virtualization diagnostics credentials:

  1. Under Administration, click Virtualization.

  2. Under Virtualization Diagnostics, click Remove.

You may now enter new credentials.

Note: Without credentials, the system will not collect statistics or generate log files for virtualization diagnostics.

Reviewing the Log Files

The system generates log files containing performance and configuration data plus statistical events.

To view and/or download the log files:

Automatically Generated Log Files

The following log files are generated automatically by the system on a periodic basis.

 

Report Name

Description

NIGHTLY-REPORT-YYYY-MM-DD.txt

This report contains the previous day's detailed performance and configuration information, and is generated daily in the Nightly Analysis Time you have specified. The system retains seven reports, deleting the oldest file after seven days.

VM-STATS-YYYY-MM-DD.csv

This report contains virtual machine statistics for the previous week. The system polls for new data every 15 minutes and deletes existing data after seven days.

HOST-STATS-YYYY-MM-DD.csv

This report contains host system statistics for the previous week. The system polls for new data every 15 minutes and deletes existing data after seven days.

ALL-CONFIG-YYYY-MM-DD.csv

This report contains all CPU, performance and network configuration statistics concerning the host and virtual machine for the previous week. The system polls for new data every 15 minutes and deletes existing data after seven days.

Manually Generated Log Files

A number of log files are created when you request them.

To manually generate the log files and an online report:

  1. Under Administration, click Virtualization.

  2. Under Virtualization Diagnostics, click Run Diagnostics.

Notes:

 

Report Name

Description

USER-REPORT-YYYY-MM-DD-HH-MM-SS.txt

This report is similar to the NIGHTLY-REPORT-YYYY-MM-DD.txt report, but contains detailed performance and configuration information for the previous week (rather than a single day), collected from the moment you click the Run Diagnostics button. The report file is retained for seven days and then deleted.

In the event you cannot resolve a problem by yourself, Mitel Product Support will request that you obtain this log file and send it to them. For details, see View/Download Log Files

USER-SUMMARY.tmp

This report is an abbreviated version of USER-REPORT-YYYY-MM-DD-HH-MM-SS.txt report. It contains performance and configuration overviews for each day of the previous week.

This report is presented in two formats:

  • Displayed on the Mitel Virtualization screen. This report is retained until you navigate away from the screen.

  • Recorded in the log files. This file is retained until the Run Diagnostics button is clicked again.

VM-EVENTS-YYYY-MM-DD.csv

The report contains 15 days' activity regarding the operation of the Virtual Machine. This file is retained until the Run Diagnostics button is clicked again.

Log File Contents

Although the log files are primarily intended for use by Mitel Product support, you may use them to troubleshoot basic issues with the following issues:

See Analysis Tuning Parameters for detailed information concerning the system settings which control the generation of log file problems.

 

Performance Problems

Description

CPU Ready (seconds)

The virtual machine has exceeded the maximum amount of time that it can wait to be run on the physical CPU(s). The default is 2 seconds.

CPU Usage (percent)

The virtual machine has exceeded its CPU capacity limit, which is expressed as a percentage of the total amount available. For example, with a limit of 50%, if the virtual machine has four CPUs with 2 GHz processors, and you are running an application that requires 6 GHz (75% of capacity), the limit has been exceeded by 25%. The default is 50%.

Disk Latency (seconds)

The virtual machine has exceeded the maximum amount of time permitted for a SCSI command to be issued by the guest operating system to the virtual machine hard disk. The default is 0.02.

Network Usage (MB)

The virtual machine has exceeded the maximum network utilization (combined transmit and receive rates, in Megabytes per second). The default is 50.0 MB.

Memory Swapped (MB)

The virtual machine has exceeded the maximum amount of memory, in Megabytes, that can be swapped into memory from disk. The default is 0 MB.

Memory Use (MB)

The virtual machine has exceeded the maximum amount of memory capacity that it can use, expressed as a percentage of the total amount available. For example, if the virtual machine has 4 GHz of memory, and you are running an application that requires 3 GHz (75% of capacity), an event will be registered. The default value is 50%.

Number of Packets Dropped (average)

The virtual machine has exceeded the maximum number of received packets that can be dropped at the network interface. The default value is 0.

Disk Usage (MB)

The virtual machine has exceeded the maximum amount of data, in Megabytes per second, that can be read from the virtual machine hard disk. The default value is 30 MB.

Configuration Detections

Description (Yes/No)

High VM-to-host CPU ratio

If "Yes" is displayed, the ESXi host has exceeded the virtual CPU to host CPU ratio, which is 0.79 by default. For example, if five virtual machines with 4 GHz vCPUs are powered on, and the host has 8 physical/16 logical cores, then the ratio is 4 + 4 + 4 + 4 + 4 ÷ 16 = 1.25. Since 1.25 exceeds 0.79, a potential configuration issue is detected.

High VM-to-host Memory ratio

If "Yes" is displayed, the ESXi host has exceeded the virtual memory to host memory ratio, which is 1.20 by default. For example, if five VMs are powered on, each using 2 GHz of memory, and the host has 8 GHz of physical memory, then the ratio is 2 + 2 + 2 + 2 + 2 ÷ 8 = 1.25, which will cause an event to be registered.

Snapshots Present

If "Yes" is displayed, the system checks to determine if snapshots are supported on the virtual machine. Because snapshots create considerable disk I/O load, use of this feature may degrade the voice quality of calls.

Low CPU Speed (MHz)

If "Yes" is displayed, the maximum speed of the virtual CPU, which is dependant on the speed of the underlying processor on the ESXi host, has been exceeded.

No Hyperthreading (Ignore if running on non-Intel processor)

If "Yes" is displayed, the system checks to determine if hyperthreading is enabled on the ESXi host.

Note: This parameter can only report on Intel processors that support hyperthreading. It cannot report on AMD or other non-Intel processors.

vMotion occurred

If "Yes" is displayed, the system checks to determine if vMotion is enabled on the ESXi host.

Low CPU Reservation (MHz)

If "Yes" is displayed, the guaranteed minimum allocation of CPU resources for this virtual machine has been exceeded.

Low Memory Reservation (MB)

If "Yes" is displayed, the guaranteed minimum allocation of memory resources for this virtual machine has been exceeded.