Citrix Troubleshooting Steps

1. Firstly, on Citrix ADC (formerly NetScaler), bind the TCP profile “nstcp_default_XA_XD_profile” to your Gateway virtual server.
2. Secondly, edit profile “nstcp_default_XA_XD_profile” on ADC and uncheck “Use Nagle’s algorithm”.
3. Take a look at the “Optimized for WAN” Citrix policy template within Citrix Studio, which will give pointers to configuring policy settings that can help improve performance over WAN.
4. Consider preparing your ADC Gateway virtual servers and end-user devices to support Adaptive Transport. You can read more here: https://www.jgspiers.com/hdx-enlightened-data-transport/

No, currently, Citrix Director’s application probing only supports published applications. You may want to consider logon simulators and full session simulators available in the market. See the following links:

• https://www.jgspiers.com/app-probing-vs-logon-simulation/
• https://www.eginnovations.com/citrix-monitoring/full-session-simulation

The next step would be to log on to the VDA and attempt to end the user’s session from there. That may involve you killing hung processes. If that does not work, see https://www.jgspiers.com/user-stuck-citrix-desktop-force-log-off/

I haven’t personally seen any significant differences, nor have I come across any Citrix publication regarding this.

To improve logon times on Citrix Virtual Apps and Desktops, you can use several optimization scripts for Windows server and desktop operating systems which I have created, See https://www.jgspiers.com/category/scripts/ Besides this, other common practices for reducing logon times include Group Policy housekeeping, profile management best practice configuration, Write Cache best practice configuration, auto-logon and so on.

You can refer to this webinar “How to Make Citrix Logons 75% Faster” for additional details.

30 seconds and below is what I like to achieve in all my deployments. I can accept 40 seconds or less, but 30 seconds downwards is the real goal.

WEM and CPM are different products and they both have different uses. It can actually be beneficial to run both, as they work together well. Citrix WEM applies printers, mapped drives, registry settings and other actions to a user’s desktop session. Profile Management captures and roams the user profile between desktop or virtual application sessions.

WEM is available with XenDesktop Enterprise (now Virtual Apps & Desktops Advanced) and above subscriptions.

One of the reasons for this could be that you have not optimized the Windows Server 2016 image? Default settings in the operating system are not the best. Please refer to my optimization script: https://www.jgspiers.com/windows-server-2016-optimisation-script/ Also keep in mind that out of the box Windows Server 2016 will require more resource than Windows Server 2008. So, you should assign an extra 1-2GB RAM and another 1-2vCPU and see if there is much difference in performance between the two environments.

“Session interrupted” notifications generally occur when there is a network issue between the Virtual Apps server or desktop and the client terminal. I would run through a process of elimination to see if the issue only happens at a specific user location, with specific endpoint clients, with specific Receiver versions and so on.

You could also monitor the VDAs and check to see if there are TCP connection drops being reported. Have your network team run tests on the networking devices that are client-side to see if there is any packet loss.

This can be caused by network blips such as spikes in latency/packet loss. A slow performing/saturated storage array where the Target Device is stored or the PVS vDisks are stored can also be the cause of retries.

I only recommend rebooting XenServer hypervisors either during disaster recovery testing phases, or when applying hotfixes to XenServer.

I typically like to reboot my virtual apps workers at least every 1-3 days. However, it again depends on how often the VDAs are used and how much resource is assigned to them.

Third-party products can monitor print servers, VDAs, and the network to inform you if there are problems. Often slow printing can be the result of bad printer routing e.g. a printer and print server with a lot of latency between them or too many hops in the communication path. For this, take a look at the Citrix policy setting “Direct connections to print servers” which is explained in detail here: https://www.jgspiers.com/citrix-universal-printing/.

Other reasons for slow printing can be due to outdated/problematic print drivers in use, or lack of bandwidth/prioritization of the printing virtual channel (ICA).

UDP ports 1494 and 2598 are required. If you are providing EDT access via Citrix Gateway, then only UDP 2598 is required to be open from the Internet to Gateway.

The “Packet CPU Usage” counter on the Dashboard of Citrix ADC will show you if the ADC device is reaching its bandwidth limit or not. You can run reports from the Reporting tab. A built-in report named “CPU vs. Memory Usage and HTTP Requests Rate” can help.

Likely the Citrix Gateway is not the bottleneck though. If you are connecting in from a high-speed, broadband link and you still see latency, which would be cause for concern and potentially point towards it being a Gateway or DMZ issue.

Citrix ADM (Application Delivery Management – formerly known as NetScaler Management and Analytics System/MAS) can help track HDX Insight data and get reports on WAN latency, ICA RTT, datacenter latency and so on.

See https://www.jgspiers.com/citrix-netscaler-management-analytics-system/

There are procedures set out by Citrix on how to enable logging for Windows and Linux etc. Workspace app (Receiver) editions. You should consult with your thin client vendor on how these procedures can be carried out on the thin client.

Determine if this issue only happens to particular VDAs, particular VDA versions etc.

Also, check how many users are currently connected and if you have enough VDAs/resource to handle more users.

If this issue is only experienced during logon storms such as in the morning, then there might be a lack of VDAs to handle the concurrent logon rate (which can be adjusted via policy).

You can capture baselines yourself at the beginning of a deployment which helps when comparing the same once users have been loaded on to the environment. However, you really need third-party monitoring solutions that can alert you when parts of the infrastructure are under stress or down. You can configure alerts when metrics breach defined thresholds. For example, logon times. Citrix Director has some of this capability, but the capability is dictated by the license you own and is limited more to monitoring Citrix VDAs and sessions, and not so much the supporting infrastructure.

It is a problem that could be caused by many things. I would try to eliminate possibilities of it being the image, high CPU/RAM consumption on VDAs, Workspace app version, the client used, network location used, VDA version used and so on. If it happens in the cloud, I assume you have VDAs in Azure or AWS. The degraded performance and disconnects typically relate to network issues but it could be the VDA itself hanging. You will have to start troubleshooting from a high level and work your way down as you rule factors out one by one.

I would look at what has changed in the environment. Sometimes it is quicker to build a fresh new image considering the problems you face and the time it may take to troubleshoot them.

You have a XenDesktop 7.6 VDI site running Windows 7 VDAs, and those guys launch published apps from a XenApp 7.15 CU3 site. Now some of the features inside the published apps no longer work, whereas they used to work when running XenApp 7.6.

Has anything else changed on the Windows 7 image such as an upgrade of Receiver for Windows?

Have you tried updating to the latest Workspace app, or tried the same scenario using a newer VDA version? If that does not work, I would suggest you contact Citrix support.

Monitoring tools such as eG Innovations can give detailed insight into the Citrix logon process and what is causing the various steps to take so long.

You should put one of the affected VDAs into an isolated Active Directory Organizational Unit with no Group Policies applying.

Other things to try are testing the logon time through a console session rather than ICA, disabling profile management (if in use) and so on.

The process of elimination will help find the root cause quicker.

Profile Management can auto-delete profiles from a VDA using policy setting “Delete locally cached profiles on logoff”.

When it comes to deleting profiles from a profile server automatically, there isn’t any script out there to do that. The script wouldn’t know which profile to delete and when.

The answer is both yes and no! You have less to troubleshoot because management of the control plane (Delivery Controllers, SQL servers, etc.) is done by Citrix. This said:

• You are still responsible for monitoring and managing the virtual apps servers and virtual desktops;
• And you still have ownership of the overall service performance. When there is slowness, you will still need to understand how to pinpoint where an issue lies. If the issue is with Citrix Cloud, then you must depend on Citrix to fix it.

The eG Innovations webinar “Does Using Citrix Cloud services make performance monitoring easier?” may be something you want to review for more details.

You should remove these applications from the OS Layer.

Antivirus agents should be installed on the gold image and all other infrastructure components such as your Delivery Controllers and StoreFront servers.

Hypervisor introspection is a technology that allows for lightweight agents or no agents at all to be placed on the VDA to reduce footprint. This technology can help with scalability.

There is no ideal specification as it depends on the workloads of your users. Typical “Task Worker”, “Knowledge Worker”, and “Power User” Windows 10 workloads may be able to use “2vCPU/2GB RAM”, “2vCPU/4GB RAM”, and “4vCPU/8GB RAM” configurations respectively, but you need to test these numbers in your own environment.

A Delivery Controller can support up to 5000 VDAs. If you have 10,000 VDAs for example, deploy 3 DDCs minimum. You should always follow the N+1 model. This allows you to endure a Delivery Controller failure without impact.

An optimized image is leaner, so there is less going on. That is the reality. As a result, the Write Cache should not be used as much as a bulky image with everything turned on would use it. I suggest regularly performing disk defragmentation on your vDisks as that also drives down Write Cache usage.

Yes. Some processes on the VDA will be running, ultimately consuming resources. However, the resource utilization should be low given the session will be idle.

For HDX Insight, a Premium license offers historical capturing of this data. An Advanced license only provides 1 hour, so basically real-time capturing. Web Insight is different and does not have a licensing requirement.

eG Innovations and other Citrix Ready monitoring partners offer AppFlow monitoring capabilities that will work if you have an advanced ADC license.

Citrix ADM (Application Delivery Management) is useful if the affected users are remote workers and your bottleneck is in the network.

Citrix Director provides visibility into specific parts of the infrastructure. To get complete end-to-end visibility into the Citrix tiers (StoreFront, Virtual app servers, license servers, Delivery Controllers, ADCs, PVS, WEM, etc.) and the supporting infrastructure, you can look at Citrix monitoring tools like eG Innovations and other third-party monitoring solution vendors.

Yes, eG Enterprise monitors Citrix ADC/NetScaler in-depth.

See https://www.eginnovations.com/citrix-monitoring/netscaler-monitoring. All the key metrics of NetScaler can be monitored agentless. AppFlow data from NetScalers can be exported to eG Enterprise and analyses as well.

Yes, eG Enterprise offers an on-premises solution. The management server, reporting engine and agents can all be deployed on-premises and no data is sent to the cloud.

Smart Tools is available to Citrix cloud customers and on-premises Virtual Apps and Desktops customers that hold a “Customer Success Services – Select” agreement. See https://www.citrix.co.uk/products/smart-tools/feature-matrix.html

Citrix Director 7.18 can provide statistics for Profile Load, Brokering time, GPO Processing time and so on. A further enhancement was made to the product in 7.18 that breaks Interactive Session time down into three sub-sections.

Third-party monitoring tools such as eG Enterprise have been providing breakups of Citrix logon time including details of interactive session time. This is useful for administrators to quickly determine which logons are slow because of profile loading and which GPOs are slowing down logons.

See https://www.eginnovations.com/citrix-monitoring/citrix-logon-monitoring

eG Enterprise is compatible with all third-party Citrix profile management solutions.

There are many ways in which eG Enterprise brings value to Citrix customers. These include:

• Ability to monitor the user experience using synthetic monitoring (logon simulation and full session simulation)
• More granular insights into why Citrix logons are slow.
• Real-time monitoring of application launch times and proactive alerting through auto-baselining.
• Unified visibility into all the Citrix tiers – StoreFront, WEM, PVS, ADC, License server, Delivery Controllers, Virtual App servers and Virtual Desktops
• Integrated monitoring of all the supporting tiers including network, virtualization, cloud, storage, Active Directory and so on.
• Embedded auto-correlation and root-cause diagnosis technology that helps easily determine if slowness is due to a Citrix problem or not.

Refer to this blog post for a detailed comparison of Citrix Director and eG Enterprise: https://www.eginnovations.com/blog/citrix-director-end-to-end-citrix-workspace-monitoring/

Yes, eG Enterprise monitors ICA round trip time (ICA RTT) which is the latency that the user perceives. In addition, it can report the network latency between the user terminal and the server farm. By comparing these two values, administrators can easily identify if there is a network issue that is affecting Citrix performance. See this short video on how eG Enterprise makes Citrix troubleshooting simple.

Third-party solutions such as eG Enterprise will provide you with more detail around the logon steps and what is causing them to take a long time. Whilst you say nothing has changed, it would be worth reviewing if any Group Policy settings have been added, or what Windows updates have been installed since December. Other things to investigate is drive maps and printer mapping via Group Policy. Are the printers/drive map locations still available? Does Event Viewer on the VDAs give any hints?

Check that the connection strings have been correctly updated. You may refer to the following blog for assistance: https://www.citrix.com/blogs/2014/02/05/xendesktop-7-x-database-migration/

If you have any further questions on the topic of Citrix troubleshooting, or you’d like to let us know how some of the tips shared in this webinar had a positive effect on your ability to troubleshoot, send us an email at info@eginnovations.com.

You can watch the recording of the Citrix Troubleshooting 101 webinar at your convenience: https://www.eginnovations.com/webinar/citrix-troubleshooting.