Do you know what AppX packages are? I don’t think most IT people do. 

AppX packages are applications installed on Windows, but the AppX (or MSIX) format is unique. It’s been designed as a container instead of the traditional MSI or EXE installers you might be used to. AppX has advantages, too, like being able to make your app available in the Microsoft Store, and having additional security, since the apps run as per user, instead of as per machine.

And this, cool as it is, appears to be where we run into problems.

Microsoft appears to enumerate all the AppX packages upon a new-user logon. Then, in an effort to facilitate a good user experience, each AppX package is evaluated for things like DefaultAssociations (e.g., http == MSEdge) and FileTypeAssociations (e.g., “.pdf” == MSEdge). To make this process faster, it seems Microsoft tries to operate this asynchronously. This is both good and bad. It’s good, because the whole phase operates super quickly. It’s bad because CPU contention from multiple threads can slow the whole process down.

The challenge with all this enumerating, associating, and configuring is that it is time-consuming. The more AppX packages you have, the longer it takes. Windows Registry operations are slow to begin with, and get slower the larger your hive. 

I highlighted this problem here, and the impact is more severe when you account for Speculative Execution patches. Add to that, the more AppX packages you have, the longer it takes to read/modify the hive, and so on. And this process happens TWICE.  Once during the “User Profile” phase, and then again during the “Preparing Windows” phase.

There is a definite lack of documentation on this whole process, so almost everything appears to be via user discovery (like this document).

The stress of work amid the COVID-19 pandemic is quite enough on its own, but these AppX difficulties aren’t making things any easier for people.  I can hear the frustration in the voices of our customers—and honestly in my own—when I don’t have an answer for their logons being slow because of AppX Packages. These people are onboarding thousands of users a week and this logon performance is, to say the least, not helping.

“Could it really be as bad as all that,” you ask? Let’s take a look.

AppX packages are applications installed on Windows, but the AppX (or MSIX) format is unique.

The example above shows a 54.7-second logon, 38.0 seconds of which is the shell starting. Of this, 36.7 seconds is AppX packages doing its thing. In short, with AppX, the time for this logon could literally be cut in half (or less!)

At ControlUp, we’re all about making our customers’ lives simpler. So, I set out to see if I could come up with a way to help our customers fix this.

For persistent machines, initial logons are usually slow, and subsequent logons are faster. During logon, the AppReadiness and AppXDeploymentServer processes seem to recognize that these packages are already staged and will skip the whole evaluation process above. When this process is skipped, you see the following in EventViewer:

During logon, the AppReadiness and AppXDeploymentServer processes recognize that these packages are already staged and will skip the evaluation process.

Whereas the first time you logon (or if a new AppX package was loaded) while you were logged off, you’ll get packages actually populated in this event. And it will look like this:

If your AppX profile solution is a profile container, then it should capture everything and subsequent logons to the same machine should be short.

If your profile solution is a profile container, then it should capture everything and subsequent logons to the same machine should be short. However, I have seen it get reevaluated and every logon has the AppX package delay. 

The reason for this appears to be non-persistent machines. There appears to be some AppX packages notifying the server that a user has been here before so check for changes and initialize those, otherwise just skip.

You can view users and who are set up on the server by looking at this registry path: HKLM\Software\Microsoft\Windows\CurrentVersion\AppReadiness

You’ll find this key contains SIDs of the users who’ve logged on to that box.

If you have Windows Virtual Desktop (WVD) and publish applications across different servers, you probably know this pain first hand, as each application has this delay on app launch.

One of things I was interested in was comparing the operating systems themselves. Server 2019 has AppX packages, too, but its logons are considerably faster than those on Windows 10 Enterprise or Windows 10 Enterprise for Virtual Desktops.

This made me wonder: what is missing from the Server 2019 experience? Or is it good enough that we can remove all the additional packages from the Windows 10 operating systems? If we remove all additional packages, would the performance be similar?

My Journey of AppX Discovery Had Begun

I did all the “plain Jane” Windows Operating System installs. I ran the following commands on all the operating systems:

 

The Results:

Command Server 2019 Windows 10 EVD 2004 Windows 10 E 2004
Get-AppxProvisionedPackage -online Microsoft.549981C3F5F10
Microsoft.BingWeather
Microsoft.DesktopAppInstaller
Microsoft.GetHelp
Microsoft.Getstarted
Microsoft.HEIFImageExtension
Microsoft.Microsoft3DViewer
Microsoft.MicrosoftEdge.Stable
Microsoft.MicrosoftOfficeHub
Microsoft.MicrosoftSolitaireCollection
Microsoft.MicrosoftStickyNotes
Microsoft.MixedReality.Portal
Microsoft.MSPaint
Microsoft.Office.OneNote
Microsoft.People
Microsoft.ScreenSketch
Microsoft.SkypeApp
Microsoft.StorePurchaseApp
Microsoft.VCLibs.140.00
Microsoft.VP9VideoExtensions
Microsoft.Wallet
Microsoft.WebMediaExtensions
Microsoft.WebpImageExtension
Microsoft.Windows.Photos
Microsoft.WindowsAlarms
Microsoft.WindowsCalculator
Microsoft.WindowsCamera
microsoft.windowscommunicationsapps
Microsoft.WindowsFeedbackHub
Microsoft.WindowsMaps
Microsoft.WindowsSoundRecorder
Microsoft.WindowsStore
Microsoft.WindowsTerminal
Microsoft.Xbox.TCUI
Microsoft.XboxApp
Microsoft.XboxGameOverlay
Microsoft.XboxGamingOverlay
Microsoft.XboxIdentityProvider
Microsoft.XboxSpeechToTextOverlay
Microsoft.YourPhone
Microsoft.ZuneMusic
Microsoft.ZuneVideo
Get-AppxPackage 1527c705-839a-4832-9118-54d4Bd6a0c89 1527c705-839a-4832-9118-54d4Bd6a0c89 1527c705-839a-4832-9118-54d4Bd6a0c89
c5e2524a-ea46-4f67-841f-6a9465d9d515 c5e2524a-ea46-4f67-841f-6a9465d9d515 c5e2524a-ea46-4f67-841f-6a9465d9d515
E2A4F912-2574-4A75-9BB0-0D023378592B E2A4F912-2574-4A75-9BB0-0D023378592B E2A4F912-2574-4A75-9BB0-0D023378592B
F46D4000-FD22-4DB4-AC8E-4E1DDDE828FE F46D4000-FD22-4DB4-AC8E-4E1DDDE828FE F46D4000-FD22-4DB4-AC8E-4E1DDDE828FE
Microsoft.AAD.BrokerPlugin Microsoft.AAD.BrokerPlugin Microsoft.AAD.BrokerPlugin
Microsoft.AccountsControl Microsoft.AccountsControl Microsoft.AccountsControl
Microsoft.AsyncTextService Microsoft.AsyncTextService Microsoft.AsyncTextService
Microsoft.BioEnrollment Microsoft.BioEnrollment Microsoft.BioEnrollment
Microsoft.CredDialogHost Microsoft.CredDialogHost Microsoft.CredDialogHost
Microsoft.ECApp Microsoft.ECApp Microsoft.ECApp
Microsoft.LockApp Microsoft.LockApp Microsoft.LockApp
Microsoft.Win32WebViewHost Microsoft.Win32WebViewHost Microsoft.Win32WebViewHost
Microsoft.Windows.Apprep.ChxApp Microsoft.Windows.Apprep.ChxApp Microsoft.Windows.Apprep.ChxApp
Microsoft.Windows.CapturePicker Microsoft.Windows.CapturePicker Microsoft.Windows.CapturePicker
Microsoft.Windows.CloudExperienceHost Microsoft.Windows.CloudExperienceHost Microsoft.Windows.CloudExperienceHost
Microsoft.Windows.NarratorQuickStart Microsoft.Windows.NarratorQuickStart Microsoft.Windows.NarratorQuickStart
Microsoft.Windows.OOBENetworkCaptivePortal Microsoft.Windows.OOBENetworkCaptivePortal Microsoft.Windows.OOBENetworkCaptivePortal
Microsoft.Windows.OOBENetworkConnectionFlow Microsoft.Windows.OOBENetworkConnectionFlow Microsoft.Windows.OOBENetworkConnectionFlow
Microsoft.Windows.PeopleExperienceHost Microsoft.Windows.PeopleExperienceHost Microsoft.Windows.PeopleExperienceHost
Microsoft.Windows.PinningConfirmationDialog Microsoft.Windows.PinningConfirmationDialog Microsoft.Windows.PinningConfirmationDialog
Microsoft.Windows.SecHealthUI Microsoft.Windows.SecHealthUI Microsoft.Windows.SecHealthUI
Microsoft.Windows.ShellExperienceHost Microsoft.Windows.ShellExperienceHost Microsoft.Windows.ShellExperienceHost
Microsoft.Windows.XGpuEjectDialog Microsoft.Windows.XGpuEjectDialog Microsoft.Windows.XGpuEjectDialog
Windows.CBSPreview Windows.CBSPreview Windows.CBSPreview
windows.immersivecontrolpanel windows.immersivecontrolpanel windows.immersivecontrolpanel
Windows.PrintDialog Windows.PrintDialog Windows.PrintDialog
InputApp
Microsoft.Windows.Cortana
Microsoft.549981C3F5F10 Microsoft.549981C3F5F10
Microsoft.Advertising.Xaml Microsoft.Advertising.Xaml
Microsoft.BingWeather Microsoft.BingWeather
Microsoft.DesktopAppInstaller Microsoft.DesktopAppInstaller
Microsoft.GetHelp Microsoft.GetHelp
Microsoft.Getstarted Microsoft.Getstarted
Microsoft.HEIFImageExtension Microsoft.HEIFImageExtension
Microsoft.Microsoft3DViewer Microsoft.Microsoft3DViewer
Microsoft.MicrosoftEdge Microsoft.MicrosoftEdge
Microsoft.MicrosoftEdgeDevToolsClient Microsoft.MicrosoftEdge.Stable
Microsoft.MicrosoftOfficeHub Microsoft.MicrosoftEdgeDevToolsClient
Microsoft.MicrosoftSolitaireCollection Microsoft.MicrosoftOfficeHub
Microsoft.MicrosoftStickyNotes Microsoft.MicrosoftSolitaireCollection
Microsoft.MixedReality.Portal Microsoft.MicrosoftStickyNotes
Microsoft.MsixPackagingTool Microsoft.MixedReality.Portal
Microsoft.MSPaint Microsoft.MSPaint
Microsoft.NET.Native.Framework.2.2 Microsoft.NET.Native.Framework.2.2
Microsoft.NET.Native.Runtime.2.2 Microsoft.NET.Native.Runtime.2.2
Microsoft.Office.OneNote Microsoft.Office.OneNote
Microsoft.People Microsoft.People
Microsoft.ScreenSketch Microsoft.ScreenSketch
Microsoft.SkypeApp Microsoft.SkypeApp
Microsoft.StorePurchaseApp Microsoft.StorePurchaseApp
Microsoft.VCLibs.140.00 Microsoft.VCLibs.140.00
Microsoft.VCLibs.140.00.UWPDesktop Microsoft.VCLibs.140.00.UWPDesktop
Microsoft.VP9VideoExtensions Microsoft.VP9VideoExtensions
Microsoft.Wallet Microsoft.Wallet
Microsoft.WebMediaExtensions Microsoft.WebMediaExtensions
Microsoft.WebpImageExtension Microsoft.WebpImageExtension
Microsoft.Windows.AssignedAccessLockApp Microsoft.Windows.AssignedAccessLockApp
Microsoft.Windows.CallingShellApp Microsoft.Windows.CallingShellApp
Microsoft.Windows.ContentDeliveryManager Microsoft.Windows.ContentDeliveryManager
Microsoft.Windows.ParentalControls Microsoft.Windows.ParentalControls
Microsoft.Windows.Photos Microsoft.Windows.Photos
Microsoft.Windows.Search Microsoft.Windows.Search
Microsoft.Windows.SecureAssessmentBrowser Microsoft.Windows.SecureAssessmentBrowser
Microsoft.Windows.StartMenuExperienceHost Microsoft.Windows.StartMenuExperienceHost
Microsoft.WindowsAlarms Microsoft.WindowsAlarms
Microsoft.WindowsCalculator Microsoft.WindowsCalculator
Microsoft.WindowsCamera Microsoft.WindowsCamera
microsoft.windowscommunicationsapps microsoft.windowscommunicationsapps
Microsoft.WindowsFeedbackHub Microsoft.WindowsFeedbackHub
Microsoft.WindowsMaps Microsoft.WindowsMaps
Microsoft.WindowsNotepad
Microsoft.WindowsSoundRecorder Microsoft.WindowsSoundRecorder
Microsoft.WindowsStore Microsoft.WindowsStore
Microsoft.WindowsTerminal Microsoft.WindowsTerminal
Microsoft.Xbox.TCUI Microsoft.Xbox.TCUI
Microsoft.XboxApp Microsoft.XboxApp
Microsoft.XboxGameCallableUI Microsoft.XboxGameCallableUI
Microsoft.XboxGameOverlay Microsoft.XboxGameOverlay
Microsoft.XboxGamingOverlay Microsoft.XboxGamingOverlay
Microsoft.XboxIdentityProvider Microsoft.XboxIdentityProvider
Microsoft.XboxSpeechToTextOverlay Microsoft.XboxSpeechToTextOverlay
Microsoft.YourPhone Microsoft.YourPhone
Microsoft.ZuneMusic Microsoft.ZuneMusic
Microsoft.ZuneVideo Microsoft.ZuneVideo
MicrosoftWindows.Client.CBS MicrosoftWindows.Client.CBS
MicrosoftWindows.UndockedDevKit MicrosoftWindows.UndockedDevKit
NcsiUwpApp NcsiUwpApp
Microsoft.NET.Native.Framework.1.7
Microsoft.NET.Native.Runtime.1.7
Microsoft.Services.Store.Engagement
Microsoft.Services.Store.Engagement
Microsoft.UI.Xaml.2.0

Interesting results. Windows 10 Enterprise for Virtual Desktops did not list anything for AppXProvisionedPackages whereas Windows 10 Enterprise showed an extensive list. Both Windows 10 operating systems have far more AppX packages overall compared to Server 2019 at 58 for Win10EVD and 62 for Win10E. Server 2019 has two packages that the desktop OS’s do not have: InputApp and Cortana.

But there was something else I wanted to check.  

I know from looking at procmon logs of Windows logging in that it seems to set file type associations and default program associations. From looking at the procmon log, it appears to iterate through every AppX package and if it finds these associations configured, it sets them up in the user’s profile. This is another process that is quite long, and Windows offers a way to pre-configure default associations that get processed for every user. I was curious about the differences between the operating systems.

The default association can be exported by executing the following command:

If exports an XML file in this format

 

I took this XML file and created a list to make it easier to parse.

The Windows 10EVD list was pretty short, Win10E was quite long, and Server 2019 looked like it might be able to do without the Media Player associations (if Windows 10EVD can do without them).

Windows 10EVD, Win10E, and Server 2019 lists with Media Player associations

The differences between the different -types- of operating systems are notable.

This journey is just beginning.

Next I’m going to explore, in more detail, the differences between Windows 10 1809, which should be analogous to Server 2019 and the newest Windows 10 2004—including EVD. Stay tuned for the next installment in this series as I explore AppX packages further.


Related Links:

 

Subscribe to Our Blog

Get tricks and tips to do your job better and the latest news about ControlUp product releases.