Logon Duration Research: AppX Packages

In the last installment in this series, we explored the way Microsoft has designed Windows to have its AppX packages operate in different ways, even if the user context—logging on to the system and the differing logon durations amongst the operating systems—is the same.

The wildly different configurations of Windows have led to some intriguing questions for Citrix, VMware, and Microsoft EUC administrators, chiefly, “Do I need what this feature is offering?”

Multi-session support is new to Windows 10, but has been on Server operating systems for a long time. Are your multi-session users happy with the experience they get on the Windows Server OS? If they are happy or productive, why would a full-fledged Windows Desktop need all the other stuff? What would the drawbacks be if you removed all the superfluous features from the Windows Desktop OSes?

Before we get out over the tips of our skis, though, let’s start by establishing a baseline. 

As an experiment, I installed six different versions of Windows:

  • Windows 10 1809 Enterprise
  • Windows 10 1809 Enterprise—Long Term Service Channel
  • Windows 10 2004 Enterprise
  • Windows 10 2004 Enterprise N
  • Windows 10 2004 Enterprise for Virtual Desktops
  • Windows 2019 Server Standard

I chose these operating systems for some specific reasons.

Windows 10 1809 Enterprise: Long-Term Service Channel

The LTSC version of Windows 10 (supposedly) comes with reduced AppX packages. I’ll validate if that is true, plus the differences between Win10 1809 LTSC and the other OSes.

Windows 10 1809 Enterprise

If the LTSC version has reduced AppX packages, we should be able to measure the impact by comparing it to the “SAC” 1809 version to see what additional packages it has and what they are contributing.

Windows 10 2004 Enterprise

Arguably the latest and greatest version of Windows (as of this writing). I profiled the AppX packages, but we can take a look at the additional packages it contains, compared to its peers.

Windows 10 2004 Enterprise N

This is the “Media Player Free” version of Windows 10. This version automatically comes with at least one less piece of software (compared to the non-N versions). Specifically, I’m curious to see if other AppX packages are removed here, as well, and about how Microsoft chose to handle the file-type associations.

Windows 10 2004 Enterprise for Virtual Desktops

As examined in my last post, Windows 10 Enterprise for Virtual Desktops makes some major changes to the way many AppX packages and file associations are made available. Microsoft significantly reduces the count on both when compared to its Enterprise peer. 

Windows 2019 Server Standard with RDSH enabled

The gold standard: server operating systems contain much-reduced AppX packages, since most are not useful in server environments. 

Since loading AppX packages is a CPU-intensive task, I decided to test with 1vCPU, 2vCPU, and 4vCPU configurations. Observations showed that AppX packages load asynchronously, so the more CPU you have, the less contention and the faster loading happens. Reducing the number of CPUs should increase the duration, as contention is higher; not all packages will get a time slice of the CPU while the operation is processing.

Finally, since CPU is such an important part of this test, it’s important to disclose the types of processors on which I ran the tests. In this case, I used 12-core, 24-thread Ryzen 9 3900X 3.8Ghz processors.

On each of the operating systems, I measured the following data:

  • Number of AppX packages
  • Number of AppX Provisioned Packages
  • Number of File Associations
  • Number of File Associations to AppX packages

For this test, I installed each operating system directly from its release ISO and applied Windows Updates. Each was joined to the domain in a OU with no group policy objects applied, since we just want to test the AppX package impact. Power settings were set to “High Performance” or “Ultimate Performance” both in Windows and on the Hypervisor. The Windows 2019 server operating system had Remote Desktop Session Host role installed.

Before I get into all that, though, I want to share some things I found intriguing during the setup phase.

Windows 10 EVD doesn’t have “Ultimate Performance”—only “High Performance”Windows 10 EVD doesn’t have Ultimate Performance — only High Performance

Windows 10 EVD will prompt you for a reason as to why you want to shut down or restart the machine.

Windows 10 EVD prompt to shut down or restart machine.

Memory Compression is enabled on all Windows 10 machines (including EVD), but is disabled on Server machines.

I observed that the Windows 10—non-EVD—machines generated CPU load when idling. This was the result of WinSAT.exe running and appeared ~25 mins after boot-up. As such, I waited an hour to run any/all tests to ensure no outside processes were interfering.

Additionally, machines set up for multi-session (Windows 10 EVD and Server 2019) had a process execute [at the ten uptime mark] that consumed significant CPU for an extended period. For Server 2019, it consumed CPU for seven minutes, and on Windows 10 EVD it consumed CPU for one minute.  The CPU consumption for that was mscorsw.exe, which is .NET Framework optimization. Optimizer utilities, such as Citrix Optimizer, VMware Operating System Optimization Tool (OSOT), and Base Image Script Framework (BISF), should remove this load. Since my setup has minimally configured operating systems, this was something I needed to consider, so it wouldn’t impact the performance of my test logons. I captured the early uptime process with ControlUp, so I could see this in real time. See this video:

In Windows 10 1809 Enterprise, the OneDriveSetup.exe utility launched on logon and consumed huge amounts of CPU for significant periods of time; the LTSC version of 1809 did not.

1vCPU Logon Duration:

1vCPU Logon Duration

 

Averaged out over three runs:

Operating SystemLogon DurationAppX Package Load Time
1809 Enterprise27 seconds21.3
1809 Enterprise – LTSC26 seconds21.2
2004 Enterprise29 seconds23.0
2004 Enterprise N27 seconds21.3
2004 Enterprise for Virtual Desktops26 seconds21.1
2019 Server Standard10 seconds6.8

 

2vCPU Logon Duration:

2vCPU Logon Duration

 

 

 

 

Averaged out over 3 runs:

Operating SystemLogon DurationAppX Package Load Time
1809 Enterprise25 seconds21.0
1809 Enterprise – LTSC24 seconds21.1
2004 Enterprise25 seconds21.2
2004 Enterprise N25 seconds21.0
2004 Enterprise for Virtual Desktops23 seconds19.6
2019 Server Standard7 seconds3.7

 

4vCPU Logon Duration:

4vCPU Logon Duration

 

 

 

 

Averaged out over 3 runs:

 

Operating SystemLogon DurationAppX Package Load Time
1809 Enterprise24 seconds20.9
1809 Enterprise – LTSC24 seconds21.0
2004 Enterprise24 seconds21.0
2004 Enterprise N24 seconds21.0
2004 Enterprise for Virtual Desktops22 seconds19.1
2019 Server Standard6 seconds3.0

 

Intriguingly, there appears to be an overhead that increasing the number of CPUs can’t push past. 

To gather more data, I logged in and queried for the number of provisioned packages, as well as the number of “AppX” packages, then looked at my AppX registry key AND the event viewer to see how many of each package was reported.

Looking at the counts of each operating system:

 

Operating SystemGet-AppxProvisionedPackage -onlineGet-AppXPackageAppX packages as counted by the registryAppX packages as counted event 327
1809 Enterprise411057777
1809 Enterprise – LTSC0333333
2004 Enterprise401157979
2004 Enterprise N291046868
2004 Enterprise for Virtual Desktops401277979
2019 Server Standard0282828

NOW we’re getting some interesting oddities! Windows 10 1809 E LTSC has close to the same number of packages as Server 2019 standard, but has significantly longer AppX package load times (each of these packages has their load time tracked in the Microsoft-Windows-AppReadiness/Admin log).

The list of packages [added to your “task list”] to execute is ID 240. Generating the list of tasks on my system spanned around ~0.5ms per task. For the 68 packages in 2004 Enterprise N, it took 28 ms to generate the list of packages to be installed (execution is measured by event 213). I created a script action to output my logon events to measure how long they took.

Windows 10 AppX packages installation time

My logon time was 11:00:43; installation of the packages started ~0.6 seconds after logon and finished ~7.4 seconds after that. Total CPU duration was 11.07 seconds for loading of the packages. Sorting by duration revealed some outliers that should be considered. A far cry from the ~68 packages that get loaded, the output here showed just 20 packages. That’s because Microsoft has deemed that some packages aren’t required to be processed at logon, but can be processed later or at app launch. Some (all?) of these packages are processed asynchronously, but I don’t know if that’s all at once or throttled (like 4 at a time).

The reported start/stop for this event was 21 seconds. We’ve now accounted for ~8 seconds, so where are the other 13?

AppX packages processed during logon

Before we get into that, let’s update our chart with the number of packages that were processed during logon.

 

Operating SystemGet-AppxProvisionedPackage -onlineGet-AppXPackageAppX packages as counted by the registryAppX packages as counted by eventid 327AppX packages actually processed at logon
1809 Enterprise41105777712
1809 Enterprise – LTSC03333336
2004 Enterprise40115797920
2004 Enterprise N29104686820
2004 Enterprise for Virtual Desktops40127797921
2019 Server Standard02828285

 

Let’s also update this chart with the package durations:

 

Operating SystemAppX Package Load TimeAppX Package processing time (seconds)
1809 Enterprise20.94.5
1809 Enterprise – LTSC21.00.7
2004 Enterprise21.06.1
2004 Enterprise N21.07.4
2004 Enterprise for Virtual Desktops19.17.5
2019 Server Standard3.00.6

 

Finally, let’s update the packages evaluated at logon for each OS

 

1809 E1809 LTSC2004 E
Microsoft.AAD.BrokerPlugin_cw5n1h2txyewyMicrosoft.AAD.BrokerPlugin_cw5n1h2txyewyMicrosoft.549981C3F5F10_8wekyb3d8bbwe
Microsoft.MicrosoftEdge_8wekyb3d8bbweMicrosoft.Windows.CloudExperienceHost_cw5n1h2txyewyMicrosoft.AAD.BrokerPlugin_cw5n1h2txyewy
Microsoft.Windows.CloudExperienceHost_cw5n1h2txyewyMicrosoft.Windows.ContentDeliveryManager_cw5n1h2txyewyMicrosoft.BioEnrollment_cw5n1h2txyewy
Microsoft.Windows.ContentDeliveryManager_cw5n1h2txyewyMicrosoft.Windows.Cortana_cw5n1h2txyewyMicrosoft.DesktopAppInstaller_8wekyb3d8bbwe
Microsoft.Windows.Cortana_cw5n1h2txyewyMicrosoft.Windows.ShellExperienceHost_cw5n1h2txyewyMicrosoft.MicrosoftEdge_8wekyb3d8bbwe
Microsoft.Windows.Photos_8wekyb3d8bbwewindows.immersivecontrolpanel_cw5n1h2txyewyMicrosoft.Windows.CloudExperienceHost_cw5n1h2txyewy
Microsoft.Windows.ShellExperienceHost_cw5n1h2txyewyMicrosoft.Windows.ContentDeliveryManager_cw5n1h2txyewy
Microsoft.WindowsCamera_8wekyb3d8bbweMicrosoft.Windows.OOBENetworkCaptivePortal_cw5n1h2txyewy
microsoft.windowscommunicationsapps_8wekyb3d8bbweMicrosoft.Windows.OOBENetworkConnectionFlow_cw5n1h2txyewy
Microsoft.WindowsStore_8wekyb3d8bbweMicrosoft.Windows.Photos_8wekyb3d8bbwe
Microsoft.XboxIdentityProvider_8wekyb3d8bbweMicrosoft.Windows.Search_cw5n1h2txyewy
windows.immersivecontrolpanel_cw5n1h2txyewyMicrosoft.Windows.ShellExperienceHost_cw5n1h2txyewy
Microsoft.Windows.StartMenuExperienceHost_cw5n1h2txyewy
Microsoft.WindowsCamera_8wekyb3d8bbwe
microsoft.windowscommunicationsapps_8wekyb3d8bbwe
Microsoft.WindowsStore_8wekyb3d8bbwe
Microsoft.XboxIdentityProvider_8wekyb3d8bbwe
MicrosoftWindows.Client.CBS_cw5n1h2txyewy
MicrosoftWindows.UndockedDevKit_cw5n1h2txyewy
windows.immersivecontrolpanel_cw5n1h2txyewy
2004 EN<2004 EVD2019 STD
Microsoft.549981C3F5F10_8wekyb3d8bbweMicrosoft.549981C3F5F10_8wekyb3d8bbweMicrosoft.AAD.BrokerPlugin_cw5n1h2txyewy
Microsoft.AAD.BrokerPlugin_cw5n1h2txyewyMicrosoft.AAD.BrokerPlugin_cw5n1h2txyewyMicrosoft.Windows.CloudExperienceHost_cw5n1h2txyewy
Microsoft.BioEnrollment_cw5n1h2txyewyMicrosoft.BioEnrollment_cw5n1h2txyewyMicrosoft.Windows.Cortana_cw5n1h2txyewy
Microsoft.DesktopAppInstaller_8wekyb3d8bbweMicrosoft.DesktopAppInstaller_8wekyb3d8bbweMicrosoft.Windows.ShellExperienceHost_cw5n1h2txyewy
Microsoft.MicrosoftEdge_8wekyb3d8bbweMicrosoft.MicrosoftEdge_8wekyb3d8bbwewindows.immersivecontrolpanel_cw5n1h2txyewy
Microsoft.Windows.CloudExperienceHost_cw5n1h2txyewyMicrosoft.VCLibs.140.00_8wekyb3d8bbwe
Microsoft.Windows.ContentDeliveryManager_cw5n1h2txyewyMicrosoft.Windows.CloudExperienceHost_cw5n1h2txyewy
Microsoft.Windows.OOBENetworkCaptivePortal_cw5n1h2txyewyMicrosoft.Windows.ContentDeliveryManager_cw5n1h2txyewy
Microsoft.Windows.OOBENetworkConnectionFlow_cw5n1h2txyewyMicrosoft.Windows.OOBENetworkCaptivePortal_cw5n1h2txyewy
Microsoft.Windows.Photos_8wekyb3d8bbweMicrosoft.Windows.OOBENetworkConnectionFlow_cw5n1h2txyewy
Microsoft.Windows.Search_cw5n1h2txyewyMicrosoft.Windows.Photos_8wekyb3d8bbwe
Microsoft.Windows.ShellExperienceHost_cw5n1h2txyewyMicrosoft.Windows.Search_cw5n1h2txyewy
Microsoft.Windows.StartMenuExperienceHost_cw5n1h2txyewyMicrosoft.Windows.ShellExperienceHost_cw5n1h2txyewy
Microsoft.WindowsCamera_8wekyb3d8bbweMicrosoft.Windows.StartMenuExperienceHost_cw5n1h2txyewy
microsoft.windowscommunicationsapps_8wekyb3d8bbweMicrosoft.WindowsCamera_8wekyb3d8bbwe
Microsoft.WindowsStore_8wekyb3d8bbwemicrosoft.windowscommunicationsapps_8wekyb3d8bbwe
Microsoft.XboxIdentityProvider_8wekyb3d8bbweMicrosoft.WindowsStore_8wekyb3d8bbwe
MicrosoftWindows.Client.CBS_cw5n1h2txyewyMicrosoft.XboxIdentityProvider_8wekyb3d8bbwe
MicrosoftWindows.UndockedDevKit_cw5n1h2txyewyMicrosoftWindows.Client.CBS_cw5n1h2txyewy
windows.immersivecontrolpanel_cw5n1h2txyewyMicrosoftWindows.UndockedDevKit_cw5n1h2txyewy
windows.immersivecontrolpanel_cw5n1h2txyewy

 

From just eye-balling it, Windows 1809 LTSC should have the same (or extremely close to) performance as Server 2019. The actual processing time differs by only 100 or so milliseconds, but the actual logon differs by 18 seconds!

So, now we have an idea what some out of the box Windows operating systems have for a baseline performance.

In our next article, we’ll dive further into the gaps of time that AppX reports and start on optimizations.