Thursday, September 19, 2013

Troubleshooting Terminal Server Performance Issues

On the Terminal Server

General
Look for processes with high CPU
Look for processes with high disk I/O
Disable “real time” virus scan
Uninstall virus scan software completely and reboot the server
Verify JP2SSV.DLL and SSV.DLL are “disabled” (Everyone: Deny NTFS permission)
Check for excessive queries to the registry (user Process Monitor)
Check for “startup” or “logon” processes (HKLM\...\Run and RunOnce keys)
Check for excessive / unnecessary processes running for all users (like besclientui.exe)

Memory
Available memory
Paged pool
Non-paged pool
Free system Page Table Entries (PTEs)
Registry size (Can cause “out of memory” errors)

Disk
Processes with high disk IO (read bytes and write bytes in Task Manager)
Disk queue length

Network
Outbound queue length
Using HP Network statistics, check interfaces for errors and retransmits
Fail over to backup/primary NIC: Verify behavior exists on both NICs
Dissolve and re-create NIC teams

Slow Logon
Disable mapping drives back to the TS Client
Determine if user’s home drive is mapping to a file server that is not on the same local network as the Terminal Server
Implement HOMEDRIVE\HOMEPATH redirection to C:\TSHOME


Physical Network (Data Center)
Switch NIC between primary and secondary team members and perform a test. Tests two separate data paths to identify issues with either of the networking cores

Switch port
- Duplex setting
- Errors
- Dropped packets

Switch health:
- CPU utilization
- Available memory
- Utilization (not flooded)
- Dropped packets
- Errors

Router health:
- CPU utilization
- Available memory
- Utilization (not flooded)
- Dropped packets
- Errors

Internet-facing firewalls to DMZ
- CPU utilization
- Available memory
- Utilization (not flooded)
- Dropped packets
- Errors

DMZ to internal network firewalls:
- CPU utilization
- Available memory
- Utilization (not flooded)
- Dropped packets
- Errors

F5:
- CPU utilization
- Available memory
- Utilization (not flooded)
- Dropped packets
- Errors

Perform a tracert; identify any routing loops

Understand if QOS is in use. If it is, understand when it takes affect (in some organizations, it only takes affect if the circuit utilization is over 80%)

PING is not an accurate performance test. PING depends upon ICMP. ICMP packets are typically discarded by QOS



Terminal Server Client
Disable virus scan, specifically any “real time” scanners
Remove virus scan and reboot
Virus scan updates or DAT updates recently? Reboot?
Virus scan software very old / out of date?

Physical Network (Client Site)
HSIA utilization

HSIA dropped packets, errors on interfaces

HSIA router health
CPU
Memory
Circuit utilization
Dropped packets
Errors on interfaces

HSIA firewall health
CPU
Memory
Circuit utilization
Dropped packets
Errors on interfaces

HSIA connection really is a high-speed link, and not a low bandwidth backup link? We had one customer that had failed over to a backup 128K ISDN line and were not aware of it.

Check packet round trip time (Wireshark report)

Check MTU size, look for packet fragmentation (Network Monitor / Wire Shark)

Look for excessive SYN and Reset packets (Network Monitor / Wire Shark)