Comparing Office 2010 performance – XenApp 6.5 (2K8R2) vs XenApp 7.5 (2K12R2)

Microsoft Office is the most popular office productivity suite in corporate IT environments. Citrix XenApp, a popular application virtualization platform in corporate environments is commonly used to deliver Microsoft Office applications to end users. The Citrix XenApp server hosts applications, such as MS Office, on a centralized server and allows users to connect and work with the applications remotely from any device via any network connection. When virtual Windows applications are hosted on shared hardware in a datacenter, it is important to carefully manage performance and capacity. Because the hardware is shared by multiple users, and the available resources are limited, sizing and user experience become important topics of discussion.

With the availability of the new Citrix XenApp 7.5 (or XenApp 7.6) platform, enterprises and organizations today are planning upgrades from legacy XenApp 6.5 farms to the new 7.x platform. However, how does this upgrade impact the performance of existing applications, such as MS Office 2010?

Companies and organizations that consider upgrading their XenApp farm must test their environments in order to learn how existing workloads will behave and evaluate the changes made to the underlying infrastructure. Obviously this evaluation must take place before the actual upgrade is made in production.

In this article, we will test and compare how MS Office 2010 performs on Windows 2008 R2 with XenApp 6.5 vs. Windows 2012 R2 with XenApp 7.5. To do this, we simulated a continuous `growing load of virtual users while monitoring the underlying hardware resources’ utilization, including CPU load, memory consumption and I/O usage.

Our main results showed that Office 2010 has lower resource utilization when running on XenApp 7.5 (Windows 2012 R2)  in comparison to XenApp 6.5 (Windows 2008 R2):  

• ~10% less memory consumption
• A reduction from 4 IOPS (XenApp 6.5) to 1 IOPS (XenApp 7.5) for a single user running all Office 2010 apps

Below, you will find our test configuration, detailed results analysis and conclusions.

The Test Environment

This section provides details regarding the test environment, hardware configuration, software, testing methods, and tools.

The test environment (as seen below) consists of the following:

• 2 m3.xlarge AWS instances (for running the XenApp 6.5 and 7.5 workloads)
• 2 m3.large AWS Instances (for running the infrastructure servers)
• An Active Directory Domain Controller and DNS server
• A 2008 R2 XenApp 6.5 server and 2012 R2 XenApp 7.5 VM’s
• Microsoft Office Professional 2010 SP1 (32 Bit)
• Login VSI 4.1 Express (which is used to generate a load of MS Office 2010 users)
• ControlUp 4.0 (to monitor and capture real time performance as well as generate reports based on the captured data)

Simulating the Load

We used Login VSI to simulate 10 users that run simultaneously on a system. Login VSI was configured with the Knowledge Worker workload to simulate real users working with applications such as Office, IE, PDF, and others. The screenshot below illustrates the Login VSI configuration:

Monitoring and Results Reporting

In order to learn the system’s behavior in each case to run the comparison analysis, we used ControlUp,which is an IT operations platform that enables real-time and historical monitoring on RDS and VDI workloads. Accordingly, we used both the real-time views and historical data to analyze the Office 2010 resource consumption on XenApp 6.5 and XenApp 7.5 in our test.

As shown in the screenshot below, ControlUp provided a good amount of live metric data across the environment, such as static data (OS version, installed CPU, installed RAM, and Disk Space), performance data (%CPU utilization, disk queue, and user sessions) and other metrics that are especially useful for monitoring XenApp deployments.

Learn more about ControlUp

Comparison Results and Analysis

Below, you can find results for both the XenApp 6.5 and XenApp 7.5 server performances. The test ran for around an hour, simulating a load of 10 virtual users.

Real Time Views

The screenshots below were taken from ControlUp’s real-time views. As you can see in the graphs, it is clear that running the MS Office work load with XenApp 7.5 reduces memory consumption in comparison to running it on XenApp 6.5.

MS Office Real-Time Performance with XenApp 6.5 (Windows 2008 R2)

MS Office Real Time Performance with XenApp 7.5 (Windows 2012 R2)

As seen above, there is lower memory utilization with XenApp 7.5. It is also important to note that the screenshots above illustrate a single point in time. In order to get an overall and more complete picture of the utilization differences, we used ControlUp’s historical reports.

Comparison Reports and Analysis

In order to track the environments’ historical performance, we aggregated the ControlUp exported (CSV) report results and generated the following performance graphs.

As seen in the comparison results below (graphs 1-4), there is almost no CPU utilization in all four Office applications, memory is the most used resource and IOPS seems relevant only in Outlook and Word. After testing the same load with both XenApp versions, the results in each Office application demonstrate that the XenApp 7.5 application performed better (red bar) than 6.5 (blue bar).

 

 

 

 

The following graph is based on the aggregated historical metrics report and shows the average server’s resource consumption throughout the life of the test.

 

 

Explaining the Results:

1. CPU usage on XenApp 6.5 was much higher (30%) than on XenApp 7.5 (2%). We found that it is mostly due to iexplore.exe and winlogon.exe. When neutralizing these processes (which are not related to Office), XenApp 6.5’s CPU usage was 10% compared to XenApp 7.5’s, which was 2%.
2. Memory utilization was almost identical (XenApp 6.5 = 33% and XenApp 7.5 = 37%)
3. Disk queue was low on both servers (so there is no major disk performance issue)
4. IOPS usage was higher on XenApp 6.5 (25 IOPS) compared to XenApp 7.5 (4 IOPS)

It seems that in general Windows 2012 R2 handles memory management and IO caching more efficiently than 2008 R2 which explains the lower resource consumption.

Final Notes

As mentioned at the beginning of the article, and as shown by the graphs above, Office 2010’s memory usage is higher on XenApp 6.5 than on XenApp 7.5 by nearly 10%. In addition, CPU usage is slightly higher on XenApp 6.5 and I/O reads/writes for Outlook and Word are slightly higher on XenApp 6.5, as well. Fortunately, based on our results and analysis, companies and organizations that migrate their Office 2010 suite from Windows 2008 R2 XA 6.5 to Windows 2012 R2 XA 7.5 can expect lower resource consumption.

Appendix – How Did We Use the ControlUp Reporter

ControlUp Reporter is a utility that enables users to analyze log files exported from ControlUp using the “Scheduled Export” feature. The resulting CSV files (as you can see below) are aggregated by ControlUp Reporter into easily readable and customizable Excel reports which can be used for the following purposes:

• Historical reporting and trending queries (What was the user session count on the terminal server farm over the past week?)
• Incident investigation (What was the CPU utilization of server X on Friday at 4pm?)
• Auditing and usage tracking (Which published applications did user X run on the server farm yesterday?)

In both test cases, we configured ControlUp to export the historical performance data every minute. Then, we used the data from our ControlUp Reporter reports to perform the comparison.

In order to create the performance graphs for each Office application, we used “Computer Processes by Sample”, which contains utilization samples that were captured every minute. We aggregated the results and processed the average for each component (CPU, Memory and I/O).

The following ControlUp graphs were generated based on the historical samples of the servers’ CPU utilization. As clearly seen below, there is a decrease in XenApp 7.5 (Windows 2012 R2)CPU utilization compared to XenApp 6.5 (Windows 2008 R2).

Total CPU Usage (Xenapp 6.5)

Total CPU Usage (Xenapp 7.5)