Monitor and investigate AVD Broker issues

What is the AVD Connection Broker and What is its Role in an AVD deployment?

A legacy way to deploy applications and desktops on Azure was often to put some Server 2022 VMs (Virtual Machines) in Azure running the standard RDS (Remote Desktop Services) roles – session hosts, brokers, gateway etc. and then pay the compute costs for your session hosts, brokers, and gateways, set up a public IP address and open ports. With Microsoft AVD (Azure Virtual Desktop) the broker and gateway are provided on Azure, and they are free – all you pay is the cost of the Session Host VMs. This has the additional benefits that IP addresses can be hidden and less secure architectures using open ports are avoided using built-in technologies such as “Reverse Connect”.

When a user tries to log in into Azure Virtual Desktop (AVD), it is the broker and gateway that handles the authentication of the user and their assignment to a particular session host. Once a session is established certain information that is readily available in less secure on-premises architectures will be unavailable (e.g., the client IP of the end-user – see details) and as such it is essential to continually monitor each logon if you wish to have this information available.

The AVD Broker is to a large extent a black box to most users, eG Enterprise will automatically capture all the necessary information from the AVD Broker for you and avoids this being a blind spot in your overall observability. Session host monitoring alone will fail to detect many issues that can impact users’ ability to logon or that can cause expensive resource wastage and excess Azure costs. Automated alerting from eG Enterprise with AVD Broker insights allows visibility on hosts that are idle, available, in drain mode, have failed upgrade and other issues.

What is a Connection Broker?

A connection broker is a resource manager that manages a pool of connections to connection-based resources such as remote desktops or databases, enabling rapid reuse of these connections by short-lived processes without the overhead of setting up a new connection each time. Remote desktop connection brokers perform a variety of tasks, including:

Checking user credentials.
Assigning users to remote desktops.
Turning remote desktops on and off as needed.
Load balancing the servers that host the desktops.
Managing desktop images.
Redirecting multimedia processing to the client.

Typically, a pool of AVD hosts will be available and rules on the broker allocate incoming requests for connections to the hosts available based on a set of pre-configured preferences and rules. Often the pool of hosts available are configured to offer resilience in the event of host failure, for example two hosts may be available in different Azure regions, in normal operation users would be connected to the geographically closest but if one host fails all incoming requests may be allocated to the remaining live host. Whilst users will be unaware of the failure in the system it is likely that the administrator will want to resolve this situation rapidly before any users are impacted and proactive AVD broker alerting enables such workflows.

The AVD Broker is a critical node in the logon process to monitor for information such as:

Failed logins from legitimate users
Signs of malicious attacks intended to compromise and access AVD deployments
Whether users are being allocated efficiently to hosts to minimize Azure costs
Have any upgrades to available hosts and golden images failed

How the AVD Broker Fits into the AVD End-to-end Architecture

Figure 1: The AVD Broker is only one component of the AVD Logon process and as such other key components need to be monitored and events correlated across the entire end-to-end AVD infrastructure. Auto-discovery and topology maps (such as this one within eG Enterprise) aid help desk operators understand the connection between components and auto-correlate and filter alerts so that a root-cause failure as shown in the Azure AD Connector does not trigger secondary alarms on the AVD Broker.

Session Host Communication Channel

Upon startup of the Azure Virtual Desktop session host, the Remote Desktop Agent Loader service establishes the Azure Virtual Desktop broker’s persistent communication channel. This communication channel is layered on top of a secure Transport Layer Security (TLS) connection and serves as a bus for service message exchange between session host and Azure Virtual Desktop infrastructure.

Reference: Understanding Azure Virtual Desktop network connectivity – Azure | Microsoft Docs

Monitoring AVD Broker Availability

The AVD Broker runs as an Azure Web Service, within eG Enterprise the underlying service is automatically monitored for events, availability, and performance. Key metrics around TCP connect time and availability, DNS availability and data transfer time are response times are all monitored.

Figure 2: Monitoring the underlying web service supporting the AVD Broker.

Key AVD Broker Metrics, log messages and errors to proactively monitor

The eG Enterprise AVD Broker module provides continual monitoring, by host, of the following metrics and events:

Host Metrics	Status Total session hosts (Number) Powered off hosts (Number) Drain mode hosts (Number) No heartbeat hosts (Number) Idle hosts (Number) Disconnected hosts (Number) Unavailable hosts (Number) Upgrade failed hosts (Number) Upgrading hosts (Number) Available hosts (Number) Healthy heartbeat hosts (Number) Connection allowed hosts (Number)
Session Metrics	User Session Details Total sessions (Number) Active sessions (Number) Unknown sessions (Number) Disconnected sessions (Number) Pending sessions (Number) Logoff sessions (Number) User profile disk mount sessions (Number)
Connection Metrics	Total UDP connections (Number) Total failed connections (Number) Total desktop connections (Number) Total remote application connections (Number) Total unique user sessions (Number) Total active connections (Number) Total connected connections (Number) Total completed connections (Number)
Error Metrics	Total management errors (Number) Total connection errors (Number) Total service errors (Number) Total feed errors (Number)
RTT and Bandwidth Metrics	Average estimated round trip time of host pool (Seconds) Average estimated bandwidth of host pool (Kbps)
Session Host Health Metrics	Session host health status (Number)
Client Device Metrics	Total unique client os (Number) Total unique client versions (Number) Total unique client types (Number)

Out-of-the-box eG Enterprise proactively monitors certain key AVD Broker metrics, applying alerting thresholds on which alerts are raised. If you wish to replicate this functionality manually, we recommend, at a minimum, setting up automated alerting on the following key metrics:

Status
Total Sessions
Drain Mode Hosts
Idle Hosts
Unavailable Hosts
Upgrade Failed Hosts
No heartbeat hosts
Total failed connections
Total management errors
Total service errors
Total connection errors
Average estimated round trip time of host pool
Average estimated bandwidth of host pool

eG Enterprise includes an AIOps (Artificial Intelligence for Operations) engine that leverages machine learning technologies to learn about time-of-day, day-of-week, and other seasonal variations, to apply dynamic thresholds which are triggered when systems’ behavior significantly deviates from normal behavior (anomaly detection). The “total sessions” is one metric we would certainly recommend applying dynamic threshold alerting to if using Azure Monitor or another third-party monitoring tool.

Figure 3: Out-of-the-box thresholds combining static and dynamic thresholds can be manually adjusted or auto-tuned to ensure multi-level automated alerting on key metrics.

Idle hosts are resources that will cost you money in Azure fees and so both long term and short-term monitoring should be implemented to ensure your enterprise optimizes and right-sizes their systems. Proactive alerting in eG Enterprise ensures you are alerted of idle hosts and can investigate. Here, eG Enterprise has noticed there are 3 idle hosts.

Figure 4: shows 3 idle hosts have been detected

When issues are detected the detailed diagnosis icon (A magnifying glass) appears to the right of the metric, allowing the operator to click through to detailed root-cause diagnostic information about the alert.

Figure 5: shows a detailed diagnosis of the (in this case) 3 idle hosts. Historical data is also available to evaluate if this is a continual or anomalous event. Notice one host is in drain mode and maintenance mode information can be used to assess if this was by design.

Rich dashboard overviews, include overviews of session hosts, including information such as – availability, whether idle and so on:

Figure 6: Clickable drilldowns on the dashboards allow administrator to drill down into the details of idle hosts and so on.

AVD Error Codes and Logs

eG Enterprise proactively captures and monitors key errors and audit logs and automatically watches for significant error codes, automatically raising alerts if appropriate. To manually access this data Azure Monitor users can capture and manually parse logs for error codes. Information on how to manually parse logs is available, here: Parse text data in Azure Monitor logs – Azure Monitor | Microsoft Docs.

Within the AVD Broker layered module, error messages and other key service and resource logs and events are automatically captured within a simple user-friendly GUI that requires no log extraction or parsing or KQL (Kusto Query Language) queries.

Figure 7: Error logs are continually trawled for AVD Broker errors alongside error logs for other Azure components and infrastructure.

Linking the AVD Broker data to the User Experience and Logon Performance

Rich UI (User Interface) dashboards and drilldowns allow the helpdesk operator and administrator to link the AVD Broker’s role and data collected to the overall user logon experience. Slow degradations can be monitored, and performance analyzed within the context of the individual users’ sessions and configurations.

Figure 8: User Experience drilldowns link the AVD Broker with other key components associated with a user and session allowing Digital Employee Experience (DEX) to be continually monitored. Colored alert overlays allow root-cause problem diagnosis – the yellow “i” on the AVD Broker has been triggered by an excessive number of idle hosts available to the broker.

Figure 9: logon data from the AVD Broker is associated with the individual user for every logon alongside key information such as client version and protocol in use. GPO (Group Policy Object) and CSE Extension breakdown times are also available.

If you are experiencing slow logons, logon slowdown or logon failures with AVD, you may find this troubleshooting guide of use, see: Troubleshoot Slow Azure Virtual Desktop Logons. This guide will also give you more context around how brokering works within the end-to-end logon process for AVD.

Using Azure Native Tools to Investigate Broker Issues

If you are relying on native Azure functionality to troubleshoot broker issues a reliable source of information is: Troubleshoot Azure Virtual Desktop Agent Issues – Azure | Microsoft Docs. The Azure Virtual Desktop Agent can cause connection issues because of multiple factors including an error on the broker that makes the agent stop the service.

Capacity Planning and Right-sizing

Live and historical reporting are included within the core eG Enterprise product to ensure you can plan for growth and optimize your deployments to reduce costs whilst eliminating resource bottlenecks and understanding user demand patterns.

These reports can also assist IT departments in implementing internal SLAs (Service Level Agreements) and communicating their KPIs (Key Performance Indicators) and achievements to management automatically.