Resource Pools – ESX Test

Resource pools can be used to hierarchically partition available CPU and memory resources. Each standalone host has an (invisible) root resource pool that groups the resources of that host. The root resource pool is not displayed because the resources of the host and the root resource pool are always the same. If you do not create child resource pools, only the root resource pools exist. Users can create child resource pools of the root resource pool or of any user-created child resource pool. Each child resource pool owns some of the parent’s resources and can, in turn, have a hierarchy of child resource pools to represent successively smaller units of computational capability. A resource pool can contain child resource pools, virtual machines, or both. You can therefore create a hierarchy of shared resources. The resource pools at a higher level are called parent resource pools. Resource pools and virtual machines that are at the same level are called siblings.

This test monitors the amount of the physical server’s resources that each resource pool on an vSphere/ESXi server is taking up.

Target of the test : An ESX server host

Agent deploying the test : An internal/remote agent

Outputs of the test : One set of results for each resource pool configured on the ESX server host that is monitored.

Configurable parameters for the test:
Parameter Description

Test Period

How often should the test be executed

Host

The host for which the test is to be configured

Port

The port at which the specified HOST listens. By default, this is NULL.

ESX User and ESX Password

In order to enable the test to extract the desired metrics from a target ESX server, you need to configure the test with an ESX USER and ESX PASSWORD. The user credentials to be passed here depend upon the mechanism used by the eG agent for auto-discovering the VMs on the target vSphere server and monitoring the server and its VMs. These discovery/monitoring methodologies and their corresponding configuration requirements have been discussed hereunder:

  • Discovering and monitoring by directly connecting to the target vSphere server: Starting with ESX server 3.0, a VMware ESX server offers a web service interface using which the eG agent discovers the guest operating systems on a physical ESX host. The VMware VI SDK is used by the agent to implement the web services interface. To use this interface for discovering the VMs and for monitoring, the eG agent should directly connect to the monitored vSphere/ESX server as an ESX USER with root privileges. However, if, owing to security constraints, you cannot use root user permissions, you can alternatively configure the tests with the credentials of a user who has been assigned the following permissions:

    • Diagnostics

    • TerminateSession

    To see how you can create such a user on the ESX server, refer to theCreating a Special Role on an ESX Server and Assigning the Role to a New User topic

  • Discovering and monitoring using vCenter: By default, the eG agent connects to each ESX server and discovers the VMs executing on it. While this approach scales well, it requires additional configuration for each server being monitored. For example, separate user accounts may need to be created on each server for accessing VM details. While monitoring large virtualized installations however, the agents can be optionally configured to perform guest discovery using the VM information already available in vCenter. The same vCenter can also be used to monitor the vSphere server and its VMs. In this case therefore, the ESX USER and ESX PASSWORD that you specify should be that of an Administrator or Virtual Machine Administrator in vCenter. However, if, owing to security constraints, you prefer not to use the credentials of such users, then, you can create a special role on vCenter with the following privileges:

    • Diagnostics

    • Change settings

    • View and stop sessions

    To know how to grant the above-mentioned permissions to a vCenter user, refer to Creating a Special Role on vCenter and Assigning the Role to a New User .

    If the ESX server for which this test is being configured had been discovered via vCenter, then the eG manager automatically populates the ESX USERand ESX PASSWORD text boxes with the vCenter user credentials using which the ESX discovery was performed.

Confirm Password

Confirm the specified ESX PASSWORD by retyping it here.

SSL

By default, the ESX server is SSL-enabled. Accordingly, the SSL flag is set to Yes by default. This indicates that the eG agent will communicate with the ESX server via HTTPS by default. On the other hand, if the eG agent has been configured to use the VMPerl API or CLI for monitoring (i.e., if the ESX USER parameter is set to none), then the status of the SSL flag is irrelevant.

Like the ESX sever, the vCenter is also SSL-enabled by default. If you have chosen to use the vCenter for monitoring all the ESX servers in your environment, then you have to set the SSL flag to Yes.

Webport

By default, in most virtualized environments, the ESX server and vCenter listen on port 80 (if not SSL-enabled) or on port 443 (if SSL-enabled). This implies that while monitoring an SSL-enabled ESX server directly, the eG agent, by default, connects to port 443 of the ESX server to pull out metrics, and while monitoring a non-SSL-enabled ESX server, the eG agent connects to port 80. Similarly, while monitoring an ESX server via an SSL-enabled vCenter, the eG agent connects to port 443 of vCenter to pull out the metrics, and while monitoring via a non-SSL-enabled vCenter, the eG agent connects to port 80 of vCenter. Accordingly, the WEBPORT parameter is set to 80 or 443 depending upon the status of the SSL flag.  In some environments however, the default ports 80 or 443 might not apply. In such a case, against the WEBPORT parameter, you can specify the exact port at which the ESX server or vCenter in your environment listens so that the eG agent communicates with that port.

Virtual Center

If the eG manager had discovered the target ESX server by connecting to vCenter, then the IP address of the vCenter server used for discovering this ESX server would be automatically displayed against the VIRTUAL CENTER parameter; similarly, the ESX USER and ESX PASSWORD text boxes will be automatically populated with the vCenter user credentials, using which ESX discovery was performed.

If this ESX server has not been discovered using vCenter, but you still want to discover the guests on the ESX server via vCenter, then select the IP address of the vCenter host that you wish to use for guest discovery from the VIRTUAL CENTER list. By default, this list is populated with the IP address of all vCenter hosts that were added to the eG Enterprise system at the time of discovery. Upon selection, the ESX USERand ESX PASSWORD that were pre-configured for that vCenter server will be automatically displayed against the respective text boxes.

On the other hand, if the IP address of the vCenter server of interest to you is not available in the list, then, you can add the details of the vCenter server on-the-fly, by selecting the Other option from the VIRTUAL CENTER list. This will invoke the ADD VCENTER SERVER DETAILS page. Refer to Adding the Details of a vCenter Server for VM Discoverysection to know how to add a vCenter server using this page. Once the vCenter server is added, its IP address, ESX USER, and ESX PASSWORD will be displayed against the corresponding text boxes.

On the other hand, if you want the eG agent to behave in the default manner -i.e., communicate with each ESX server for monitoring and VM information - then set the VIRTUAL CENTER parameter to ‘none’.

List All Pools

By default, resource pools are discovered by first discovering the VMs and then determining which resource pools they are a part of. According to this default behavior, the LIST ALL POOLSflag is set to No by default. In this case, if a resource pool does not have any VMs, such resource pools will not be discovered. Note that there is not much benefit in having a resource pool without any VMs assigned to it. If you have such “empty” resource pools that have no VMs, but you want these discovered as well, then you will have to set the LIST ALL POOLSflag to Yes. This will force the agent to change its discovery process - to list resource pools by parent child relationships first and then to identify which resource pools have VMs.

Report Cluster Resource Pools

A VMware ESX server can be a part of a cluster or it may not.

If a VMware ESX server is not part of a cluster, its resource pools will be discovered by the agent and metrics reported for the resource pools. These metrics will appear in the VMware vSphere ESX model of eG Enterprise.

If a VMware ESX server is part of a cluster, its resource pools and its resources will be managed by vCenter, and these clusters and resource pools will show up in the VMware vCenter model offered by eG Enterprise. If a VMware ESX server is not part of a cluster, its resource pools will not show up in the VMware vCenter model.

Therefore, when managing an ESX server, if that server is found to be a part of a cluster, then, by default, the agent will not report metrics for its resource pools (since this information will already be available in the VMware vCenter model). This is the default behavior and corresponds to the REPORT CLUSTER RESOURCE POOLS  flag being set to No by default. If this flag is changed to Yes, metrics for resource pools will be reported in the VMware vSphere ESX model irrespective of whether the ESX server is part of a cluster or not.

Detailed Diagnosis

To make diagnosis more efficient and accurate, eG Enterprise embeds an optional detailed diagnostic capability. With this capability, the eG agents can be configured to run detailed, more elaborate tests as and when specific problems are detected. To enable the detailed diagnosis capability of this test for a particular server, choose the On option. To disable the capability, click on the Off option.

The option to selectively enable/disable the detailed diagnosis capability will be available only if the following conditions are fulfilled:

  • The eG manager license should allow the detailed diagnosis capability

  • Both the normal and abnormal frequencies configured for the detailed diagnosis measures should not be 0.

Measurements made by the test
Measurement Description Measurement Unit Interpretation

Child resource pools:

Indicates the total number of child resource pools under this resource pool.

Number

 

VMs directly assigned to pool:

Indicates the number of virtual machines that were directly assigned to this resource pool.

Number

 

VMs powered on:

Indicates the number of virtual machines under this resource pool that is currently powered on.

Number

If the value of this measure is equal to the value of the VMs directly assigned to pool measure, it indicates that all VMs in the pool are in a powered-on state. If the values do not match, it indicates that one/more VMs are currently in a powered-off state.

Physical CPU utilization:

Indicates the percentage of physical CPU resources used from this pool.

Percent

A high value of this measure could indicate excessive CPU usage by the child resource pools/virtual machines configured under this resource pool. By comparing the value of this measure across resource pools, you can quickly identify that resource pool which is draining the CPU resources of the ESX host.

Physical CPU used:

Indicates the CPU usage of the resource pool in Mhz

Mhz

The percent CPU usage measure serves as an effective indicator of how resource-intensive a particular resource pool is on a specific ESX server. However, for performing capacity planning or what-if analysis, the CPU usage of a pool measured in absolute terms would be more useful.  For instance, the physical CPU usage of a pool could be 30% on a particular ESX server - this means that the resource pool is consuming 30% of the total physical CPU capacity of that ESX server. If you are planning to migrate the resource pool to another ESX server, then it would be unwise to assume that the pool will only consume 30% of CPU on the other ESX server as well, as the percentage will vary depending upon the physical CPU resources that are available to the other ESX server. The absolute measure however will remain unchanged across ESX server. Therefore, to decide which ESX server a VM/resource pool is to be moved to, and to analyze the impact of this movement on the CPU resources of the new ESX host, you would require an absolute measure of CPU usage.    

CPU reservation:

Indicates the amount of CPU resources that is guaranteed available to the resource pool.

Mhz

Typically, the Reservation setting for a resource pool specifies the minimum acceptable amount of CPU or memory - not the amount you would like to have available.

Follow the guidelines mentioned below to define the Reservation setting:

  • Do not set the Reservation too high. A very high reservation can limit the number of virtual machines you can power on in a resource pool.
  • Always leave some headroom while reserving resource. Do not commit all resources. As you move closer to fully reserving all capacity in the system, it becomes increasingly difficult to make changes to reservations and to the resource pool hierarchy without violating admission control.
  • If you expect frequent changes to the total available resources, use Shares, not Reservation, to allocate resources fairly across resource pools.

CPU reservation used:

Indicates the total amount of CPU resources that have been used to satisfy the reservation requirements of all descendants of this resource pool.

Mhz

CPU resources are typically reserved for powered-on virtual machines in a resource pool only. Virtual machines that are not currently running do not use any CPU resources.

Reserved resources are not wasted if they are not used. If the utilization is less than the reservation, the resources can be utilized by other running virtual machines.

CPU unreserved:

Indicates the total amount of resources available to satisfy a reservation for a child resource pool.

Mhz

In the undercommitted state, this value is limited by the capacity at the root node. In the overcommitted case, this could be higher since we do not perform the dynamic capacity checks.

Consider a resource pool with reservation=2GHz that is

totally idle. It has 2GHz reserved, but it is not actually using any of its reservation. In such a case:

  • Other resource pools cannot reserve these 2GHz.
  • Other resource can use these 2GHz, that is, idle CPU reservations are not wasted.

CPU allocation used:

Indicates the percentage of allocated CPU resources that are being used by this resource pool.

Percent

A high value of this measure indicates that one/more VMs in the pool are consuming the allocated CPU resources excessively.

CPU limit:

Indicates the limit beyond which this resource pool should not use CPU resources, even if there are resources available.

Mhz

If the UNLIMITED flag is enabled for a resource pool, then this measure will report the value Unlimited. This setting helps avoid wastage of idle resources.

Memory used:

Indicates the amount of memory used by this resource pool.

 

MB

 

Memory reservation:

Indicates the amount of memory resources that is guaranteed available to the resource pool.

MB

Typically, the Reservation setting for a resource pool specifies the minimum acceptable amount of CPU or memory - not the amount you would like to have available.

Follow the guidelines mentioned below to define the Reservation setting:

  • Do not set the Reservation too high. A very high reservation can limit the number of virtual machines you can power on in a resource pool.
  • Always leave some headroom while reserving resource. Do not commit all resources. As you move closer to fully reserving all capacity in the system, it becomes increasingly difficult to make changes to reservations and to the resource pool hierarchy without violating admission control.
  • If you expect frequent changes to the total available resources, use Shares, not Reservation, to allocate resources fairly across resource pools.

Memory reservation used:

Indicates the total amount of memory resources that have been used to satisfy the reservation requirements of all descendants of this resource pool.

MB

Memory resources are typically reserved for powered-on virtual machines in a resource pool only. Virtual machines that are not currently running do not use any memory resources.

Reserved resources are not wasted if they are not used. If the utilization is less than the reservation, the resources can be utilized by other running virtual machines.

Memory unreserved:

Indicates the total amount of resources available to satisfy a reservation for a child resource pool.

MB

In the undercommitted state, this value is limited by the capacity at the root node. In the overcommitted case, this could be higher since we do not perform the dynamic capacity checks.

If the resource pool does not use the reserved memory, then remember the following:

  • Other resource pools cannot reserve these memory resources
  • Other resource pools can use these memory resources, that is, idle memory reservations are not wasted.

Memory limit:

Indicates the limit beyond which this resource pool should not use memory resources, even if there are resources available.

MB

If the UNLIMITED flag is enabled for a resource pool, then this measure will report the value Unlimited. This setting helps avoid wastage of idle resources.

Active memory:

Indicates the amount of memory resources actively used from this pool.

MB

High memory consumption by a resource pool, besides eroding the physical memory resources of the host, can also impact the memory allocations to other resource pools/virtual machines; the lack of adequate memory to a VM can significantly strain its performance. 

Active memory used:

Indicates the percentage of total configured or available memory in this pool.

Percent

Zero memory:

Indicates the amount of memory that is zeroed out.

MB

 

Swapped memory:

Indicates the amount of memory that is swapped.

MB

ESX Server hosts use swapping to forcibly reclaim memory from a virtual machine when no vmmemctl driver is available because the vmmemctl driver:

  • Was never installed
  • Has been explicitly disabled
  • Is not running (for example, while the guest operating system is booting)
  • Is temporarily unable to reclaim memory quickly enough to satisfy current system demands

Standard demand-paging techniques swap pages back in when the virtual machine needs them.

Swap space must be reserved on disk for any unreserved virtual machine memory. This swap reservation is required to ensure the system is able to preserve virtual machine memory under any circumstances. In practice, only a small fraction of the swap space may actually be used.

Typically, swap space usage for each VM should be low. Since access from RAM is much faster than access from physical disk, excessive usage of swap memory will slow down the performance of a VM. Watch for VMs that are seeing higher swap usage and more swap reads and writes.

Memory overhead:

Indicates the memory overhead incurred by this resource pool.

MB

ESX Server virtual machines can incur two kinds of memory overhead:

  • The additional time to access memory within a virtual machine.
  • The extra space needed by the ESX Server host for its own code and data structures, beyond the memory allocated to each virtual machine.

The ESX server’s memory virtualization ensures that the time overhead to memory accesses is minimal.

The memory space overhead, on the other hand, is composed of two other components:

  • A fixed system-wide overhead for the service console (in the case of ESX 3/3.5) and the VMkernel.
  • Additional overhead for each virtual machine

In addition, the space reserved for the virtual machine frame buffer and various virtualization data structures, also add to the memory overhead.

Addition of more virtual CPUs, allocation of more memory to the guests, and choosing to configure 32-bit guest operating systems instead of 64-bit ones, can go a long way in reducing this overhead. The ESX server also provides optimizations such as memory sharing to save up memory.

Balloon memory:

Indicates the amount of memory used by memory control.

MB

The vmmectl driver that is installed on a virtual machine, emulates an increase or decrease in memory pressure on the guest operating system; this way, it forces the guest OS to place memory pages into its local swap file. This driver differs from the VMware swap file method as it forces the operating system to determine what memory it wishes to page. Once the memory is paged locally on the guest operating system, the free physical pages of memory may be reallocated to other guests. As the ESX hosts sees that memory demand has been reduced, it will instruct vmmemctl to “deflate” the balloon and reduce pressure on the guest OS to page memory.

The maximum amount of memory that can be reclaimed from a guest may be configured by modifying the “sched.mem.maxmemctl” advanced option.

If the memory reclaimed from a guest (i.e., the value of this measure) is very low, it indicates excessive memory usage by the guest. Under such circumstances, you might want to consider allocating more memory to the guest.

Shared memory:

Indicates the amount of memory in the resource pool that is shared.

MB

 

Memory granted:

Indicates the amount of memory that is granted to this resource pool.

MB