AppX Packages Slowing You Down?

Do you know what AppX packages are? I don’t think most IT people do. 

AppX packages are applications installed on Windows, but the AppX (or MSIX) format is unique. It’s been designed as a container instead of the traditional MSI or EXE installers you might be used to. AppX has advantages, too, like being able to make your app available in the Microsoft Store, and having additional security, since the apps run as per user, instead of as per machine.

And this, cool as it is, appears to be where we run into problems.

Microsoft appears to enumerate all the AppX packages upon a new-user logon. Then, in an effort to facilitate a good user experience, each AppX package is evaluated for things like DefaultAssociations (e.g., http == MSEdge) and FileTypeAssociations (e.g., “.pdf” == MSEdge). To make this process faster, it seems Microsoft tries to operate this asynchronously. This is both good and bad. It’s good, because the whole phase operates super quickly. It’s bad because CPU contention from multiple threads can slow the whole process down.

The challenge with all this enumerating, associating, and configuring is that it is time-consuming. The more AppX packages you have, the longer it takes. Windows Registry operations are slow to begin with, and get slower the larger your hive. 

I highlighted this problem here, and the impact is more severe when you account for Speculative Execution patches. Add to that, the more AppX packages you have, the longer it takes to read/modify the hive, and so on. And this process happens TWICE.  Once during the “User Profile” phase, and then again during the “Preparing Windows” phase.

There is a definite lack of documentation on this whole process, so almost everything appears to be via user discovery (like this document).

The stress of work amid the COVID-19 pandemic is quite enough on its own, but these AppX difficulties aren’t making things any easier for people.  I can hear the frustration in the voices of our customers—and honestly in my own—when I don’t have an answer for their logons being slow because of AppX Packages. These people are onboarding thousands of users a week and this logon performance is, to say the least, not helping.

“Could it really be as bad as all that,” you ask? Let’s take a look.

AppX packages are applications installed on Windows, but the AppX (or MSIX) format is unique.

The example above shows a 54.7-second logon, 38.0 seconds of which is the shell starting. Of this, 36.7 seconds is AppX packages doing its thing. In short, with AppX, the time for this logon could literally be cut in half (or less!)

At ControlUp, we’re all about making our customers’ lives simpler. So, I set out to see if I could come up with a way to help our customers fix this.

For persistent machines, initial logons are usually slow, and subsequent logons are faster. During logon, the AppReadiness and AppXDeploymentServer processes seem to recognize that these packages are already staged and will skip the whole evaluation process above. When this process is skipped, you see the following in EventViewer:

During logon, the AppReadiness and AppXDeploymentServer processes recognize that these packages are already staged and will skip the evaluation process.

Whereas the first time you logon (or if a new AppX package was loaded) while you were logged off, you’ll get packages actually populated in this event. And it will look like this:

If your AppX profile solution is a profile container, then it should capture everything and subsequent logons to the same machine should be short.

If your profile solution is a profile container, then it should capture everything and subsequent logons to the same machine should be short. However, I have seen it get reevaluated and every logon has the AppX package delay. 

The reason for this appears to be non-persistent machines. There appears to be some AppX packages notifying the server that a user has been here before so check for changes and initialize those, otherwise just skip.

You can view users and who are set up on the server by looking at this registry path: HKLMSoftwareMicrosoftWindowsCurrentVersionAppReadiness

You’ll find this key contains SIDs of the users who’ve logged on to that box.

If you have Windows Virtual Desktop (WVD) and publish applications across different servers, you probably know this pain first hand, as each application has this delay on app launch.

One of things I was interested in was comparing the operating systems themselves. Server 2019 has AppX packages, too, but its logons are considerably faster than those on Windows 10 Enterprise or Windows 10 Enterprise for Virtual Desktops.

This made me wonder: what is missing from the Server 2019 experience? Or is it good enough that we can remove all the additional packages from the Windows 10 operating systems? If we remove all additional packages, would the performance be similar?

My Journey of AppX Discovery Had Begun

I did all the “plain Jane” Windows Operating System installs. I ran the following commands on all the operating systems:

Get-AppxProvisionedPackage -online

Get-AppxPackage

 

The Results:

CommandServer 2019Windows 10 EVD 2004Windows 10 E 2004
Get-AppxProvisionedPackage -onlineMicrosoft.549981C3F5F10
Microsoft.BingWeather
Microsoft.DesktopAppInstaller
Microsoft.GetHelp
Microsoft.Getstarted
Microsoft.HEIFImageExtension
Microsoft.Microsoft3DViewer
Microsoft.MicrosoftEdge.Stable
Microsoft.MicrosoftOfficeHub
Microsoft.MicrosoftSolitaireCollection
Microsoft.MicrosoftStickyNotes
Microsoft.MixedReality.Portal
Microsoft.MSPaint
Microsoft.Office.OneNote
Microsoft.People
Microsoft.ScreenSketch
Microsoft.SkypeApp
Microsoft.StorePurchaseApp
Microsoft.VCLibs.140.00
Microsoft.VP9VideoExtensions
Microsoft.Wallet
Microsoft.WebMediaExtensions
Microsoft.WebpImageExtension
Microsoft.Windows.Photos
Microsoft.WindowsAlarms
Microsoft.WindowsCalculator
Microsoft.WindowsCamera
microsoft.windowscommunicationsapps
Microsoft.WindowsFeedbackHub
Microsoft.WindowsMaps
Microsoft.WindowsSoundRecorder
Microsoft.WindowsStore
Microsoft.WindowsTerminal
Microsoft.Xbox.TCUI
Microsoft.XboxApp
Microsoft.XboxGameOverlay
Microsoft.XboxGamingOverlay
Microsoft.XboxIdentityProvider
Microsoft.XboxSpeechToTextOverlay
Microsoft.YourPhone
Microsoft.ZuneMusic
Microsoft.ZuneVideo
Get-AppxPackage1527c705-839a-4832-9118-54d4Bd6a0c891527c705-839a-4832-9118-54d4Bd6a0c891527c705-839a-4832-9118-54d4Bd6a0c89
c5e2524a-ea46-4f67-841f-6a9465d9d515c5e2524a-ea46-4f67-841f-6a9465d9d515c5e2524a-ea46-4f67-841f-6a9465d9d515
E2A4F912-2574-4A75-9BB0-0D023378592BE2A4F912-2574-4A75-9BB0-0D023378592BE2A4F912-2574-4A75-9BB0-0D023378592B
F46D4000-FD22-4DB4-AC8E-4E1DDDE828FEF46D4000-FD22-4DB4-AC8E-4E1DDDE828FEF46D4000-FD22-4DB4-AC8E-4E1DDDE828FE
Microsoft.AAD.BrokerPluginMicrosoft.AAD.BrokerPluginMicrosoft.AAD.BrokerPlugin
Microsoft.AccountsControlMicrosoft.AccountsControlMicrosoft.AccountsControl
Microsoft.AsyncTextServiceMicrosoft.AsyncTextServiceMicrosoft.AsyncTextService
Microsoft.BioEnrollmentMicrosoft.BioEnrollmentMicrosoft.BioEnrollment
Microsoft.CredDialogHostMicrosoft.CredDialogHostMicrosoft.CredDialogHost
Microsoft.ECAppMicrosoft.ECAppMicrosoft.ECApp
Microsoft.LockAppMicrosoft.LockAppMicrosoft.LockApp
Microsoft.Win32WebViewHostMicrosoft.Win32WebViewHostMicrosoft.Win32WebViewHost
Microsoft.Windows.Apprep.ChxAppMicrosoft.Windows.Apprep.ChxAppMicrosoft.Windows.Apprep.ChxApp
Microsoft.Windows.CapturePickerMicrosoft.Windows.CapturePickerMicrosoft.Windows.CapturePicker
Microsoft.Windows.CloudExperienceHostMicrosoft.Windows.CloudExperienceHostMicrosoft.Windows.CloudExperienceHost
Microsoft.Windows.NarratorQuickStartMicrosoft.Windows.NarratorQuickStartMicrosoft.Windows.NarratorQuickStart
Microsoft.Windows.OOBENetworkCaptivePortalMicrosoft.Windows.OOBENetworkCaptivePortalMicrosoft.Windows.OOBENetworkCaptivePortal
Microsoft.Windows.OOBENetworkConnectionFlowMicrosoft.Windows.OOBENetworkConnectionFlowMicrosoft.Windows.OOBENetworkConnectionFlow
Microsoft.Windows.PeopleExperienceHostMicrosoft.Windows.PeopleExperienceHostMicrosoft.Windows.PeopleExperienceHost
Microsoft.Windows.PinningConfirmationDialogMicrosoft.Windows.PinningConfirmationDialogMicrosoft.Windows.PinningConfirmationDialog
Microsoft.Windows.SecHealthUIMicrosoft.Windows.SecHealthUIMicrosoft.Windows.SecHealthUI
Microsoft.Windows.ShellExperienceHostMicrosoft.Windows.ShellExperienceHostMicrosoft.Windows.ShellExperienceHost
Microsoft.Windows.XGpuEjectDialogMicrosoft.Windows.XGpuEjectDialogMicrosoft.Windows.XGpuEjectDialog
Windows.CBSPreviewWindows.CBSPreviewWindows.CBSPreview
windows.immersivecontrolpanelwindows.immersivecontrolpanelwindows.immersivecontrolpanel
Windows.PrintDialogWindows.PrintDialogWindows.PrintDialog
InputApp
Microsoft.Windows.Cortana
Microsoft.549981C3F5F10Microsoft.549981C3F5F10
Microsoft.Advertising.XamlMicrosoft.Advertising.Xaml
Microsoft.BingWeatherMicrosoft.BingWeather
Microsoft.DesktopAppInstallerMicrosoft.DesktopAppInstaller
Microsoft.GetHelpMicrosoft.GetHelp
Microsoft.GetstartedMicrosoft.Getstarted
Microsoft.HEIFImageExtensionMicrosoft.HEIFImageExtension
Microsoft.Microsoft3DViewerMicrosoft.Microsoft3DViewer
Microsoft.MicrosoftEdgeMicrosoft.MicrosoftEdge
Microsoft.MicrosoftEdgeDevToolsClientMicrosoft.MicrosoftEdge.Stable
Microsoft.MicrosoftOfficeHubMicrosoft.MicrosoftEdgeDevToolsClient
Microsoft.MicrosoftSolitaireCollectionMicrosoft.MicrosoftOfficeHub
Microsoft.MicrosoftStickyNotesMicrosoft.MicrosoftSolitaireCollection
Microsoft.MixedReality.PortalMicrosoft.MicrosoftStickyNotes
Microsoft.MsixPackagingToolMicrosoft.MixedReality.Portal
Microsoft.MSPaintMicrosoft.MSPaint
Microsoft.NET.Native.Framework.2.2Microsoft.NET.Native.Framework.2.2
Microsoft.NET.Native.Runtime.2.2Microsoft.NET.Native.Runtime.2.2
Microsoft.Office.OneNoteMicrosoft.Office.OneNote
Microsoft.PeopleMicrosoft.People
Microsoft.ScreenSketchMicrosoft.ScreenSketch
Microsoft.SkypeAppMicrosoft.SkypeApp
Microsoft.StorePurchaseAppMicrosoft.StorePurchaseApp
Microsoft.VCLibs.140.00Microsoft.VCLibs.140.00
Microsoft.VCLibs.140.00.UWPDesktopMicrosoft.VCLibs.140.00.UWPDesktop
Microsoft.VP9VideoExtensionsMicrosoft.VP9VideoExtensions
Microsoft.WalletMicrosoft.Wallet
Microsoft.WebMediaExtensionsMicrosoft.WebMediaExtensions
Microsoft.WebpImageExtensionMicrosoft.WebpImageExtension
Microsoft.Windows.AssignedAccessLockAppMicrosoft.Windows.AssignedAccessLockApp
Microsoft.Windows.CallingShellAppMicrosoft.Windows.CallingShellApp
Microsoft.Windows.ContentDeliveryManagerMicrosoft.Windows.ContentDeliveryManager
Microsoft.Windows.ParentalControlsMicrosoft.Windows.ParentalControls
Microsoft.Windows.PhotosMicrosoft.Windows.Photos
Microsoft.Windows.SearchMicrosoft.Windows.Search
Microsoft.Windows.SecureAssessmentBrowserMicrosoft.Windows.SecureAssessmentBrowser
Microsoft.Windows.StartMenuExperienceHostMicrosoft.Windows.StartMenuExperienceHost
Microsoft.WindowsAlarmsMicrosoft.WindowsAlarms
Microsoft.WindowsCalculatorMicrosoft.WindowsCalculator
Microsoft.WindowsCameraMicrosoft.WindowsCamera
microsoft.windowscommunicationsappsmicrosoft.windowscommunicationsapps
Microsoft.WindowsFeedbackHubMicrosoft.WindowsFeedbackHub
Microsoft.WindowsMapsMicrosoft.WindowsMaps
Microsoft.WindowsNotepad
Microsoft.WindowsSoundRecorderMicrosoft.WindowsSoundRecorder
Microsoft.WindowsStoreMicrosoft.WindowsStore
Microsoft.WindowsTerminalMicrosoft.WindowsTerminal
Microsoft.Xbox.TCUIMicrosoft.Xbox.TCUI
Microsoft.XboxAppMicrosoft.XboxApp
Microsoft.XboxGameCallableUIMicrosoft.XboxGameCallableUI
Microsoft.XboxGameOverlayMicrosoft.XboxGameOverlay
Microsoft.XboxGamingOverlayMicrosoft.XboxGamingOverlay
Microsoft.XboxIdentityProviderMicrosoft.XboxIdentityProvider
Microsoft.XboxSpeechToTextOverlayMicrosoft.XboxSpeechToTextOverlay
Microsoft.YourPhoneMicrosoft.YourPhone
Microsoft.ZuneMusicMicrosoft.ZuneMusic
Microsoft.ZuneVideoMicrosoft.ZuneVideo
MicrosoftWindows.Client.CBSMicrosoftWindows.Client.CBS
MicrosoftWindows.UndockedDevKitMicrosoftWindows.UndockedDevKit
NcsiUwpAppNcsiUwpApp
Microsoft.NET.Native.Framework.1.7
Microsoft.NET.Native.Runtime.1.7
Microsoft.Services.Store.Engagement
Microsoft.Services.Store.Engagement
Microsoft.UI.Xaml.2.0

Interesting results. Windows 10 Enterprise for Virtual Desktops did not list anything for AppXProvisionedPackages whereas Windows 10 Enterprise showed an extensive list. Both Windows 10 operating systems have far more AppX packages overall compared to Server 2019 at 58 for Win10EVD and 62 for Win10E. Server 2019 has two packages that the desktop OS’s do not have: InputApp and Cortana.

But there was something else I wanted to check.  

I know from looking at procmon logs of Windows logging in that it seems to set file type associations and default program associations. From looking at the procmon log, it appears to iterate through every AppX package and if it finds these associations configured, it sets them up in the user’s profile. This is another process that is quite long, and Windows offers a way to pre-configure default associations that get processed for every user. I was curious about the differences between the operating systems.

The default association can be exported by executing the following command:

Dism /Online /Export-DefaultAppAssociations:"F:AppAssociations.xml"

If exports an XML file in this format

<?xml version="1.0" encoding="UTF-8"?>

<DefaultAssociations>

  <Association Identifier=".3g2" ProgId="WMP11.AssocFile.3G2" ApplicationName="Windows Media Player" />

</DefaultAssociations>

 

I took this XML file and created a list to make it easier to parse.

The Windows 10EVD list was pretty short, Win10E was quite long, and Server 2019 looked like it might be able to do without the Media Player associations (if Windows 10EVD can do without them).

Windows 10EVD, Win10E, and Server 2019 lists with Media Player associations

The differences between the different -types- of operating systems are notable.

This journey is just beginning.

Next I’m going to explore, in more detail, the differences between Windows 10 1809, which should be analogous to Server 2019 and the newest Windows 10 2004—including EVD. Stay tuned for the next installment in this series as I explore AppX packages further.


Related Links: