5 Key Updates to ControlUp Analyze Logon Duration, Part 3: RSoP Logging

Among the trickier things for IT administrators to nail down are the causes of slow logons and how to fix those problems quickly. That’s why ControlUp created the “Analyze Logon Duration” (ALD) script. And now, we’ve made five key changes to the ALD script that will make spotting problems—and remedying them quickly—simple. In this series, we’ll go over the enhancements and what additional information we can glean from them.

The five changes:

  • Citrix Resultant Set of Policies (RSoP) analysis

  • Resultant Set of Policies Logging (RSoP Logging)

  • Windows Management Instrumentation (WMI) Filter duration analysis

  • Individual AppX Package Load Times

  • Loopback Processing Mode state

 

About Logon Duration Analysis

Out of the box, ControlUp offers some amazing technology for reporting on logon durations for users. Overall Logon Duration, Profile Load Time, Group Policy Load Time, Desktop Load Time and a column for unknown logon duration contributors are all calculated and displayed—in real time—right after a user has logged on.

ControlUp Analyze Logon Duration

 

This is extraordinarily helpful for identifying low-hanging fruit optimization opportunities.

Originally introduced in 2015, ControlUp (along with help from our community members—particularly Guy Leech) has been continually improving and increasing the functionality of the Analyze Logon Duration script, which can break logons down into granular phases, based on technology or logon processes.

About ControlUp Analyze Logon Duration

But ALD can only report back on the things it understands.

See those two large gaps of time on the far right of the image? 33.9 seconds and 20.9 seconds?

When ControlUp customers have unexplained logon duration gaps, they can reach out to us to help them assist in troubleshooting the root cause of these gaps. From here, ControlUp Professional Services can assist in identifying the technology or reason for the gaps. This article will go into the technical details of the gap in this screenshot, what it is we discovered, and how ALD reports it after we added this technology into the script.

About Resultant Set of Policies

Resultant Set of Policies (RSoP) logging is a feature available in Windows that shows you what Group Policy settings the Group Policy Engine decided to apply against a user or computer.

The most common view of RSoP is the “rsop.msc” to view the applied settings:

ControlUp Analyze Logon Duration

 

This is great for troubleshooting!

But what’s the catch (side note: there is always a catch)?

 

The Problem

Resultant Set of Policies is a cool feature! You can see the end state of a user’s or machine’s group policy. And with security filtering, WMI filtering, OU nesting, blocking of inheritance, different loopback processing modes, and other ways of manipulating the  settings that get applied, RSoP is a powerful tool in understanding what’s happening in your environment.

So, what’s the problem?

RSoP can slow your logons down… substantially.

But how does it slow down your logons?

To answer that, we need to examine how RSoP works. There are numerous articles that touch on the ways RSoP operates, but they are usually limited in context. Here’s the “RSoP Architecture” from Microsoft. Another article, this one by Ned Pyle, dives into some nuggets while talking about how to script RSoP.  I’ll summarize my findings as best as I can.

RSoP stores information in the WMI repository. It has interfaces you can implement for writing and querying data. This doesn’t sound too bad. So, what’s bad about this?

Windows Management Instrumentation (WMI)

WMI has some well-known performance issues. The worst is a O(n^2) performance bug related to the size of the repository.

Next: you can lock WMI, thereby preventing it from being written to or read from until the lock is released. This can stall operations in a major way. This impact was quantified, back in 2019, by Bruce Dawson in his blog post, “O(n^2), again, now in WMI.”.

Last, each CSE can store its RSoP to WMI as it sees fit. They can record each value as it’s evaluated, or as a whole in the end. Recording each as it’s evaluated is bad because writing to WMI isn’t fast. It’s up to the CSE makers and not all CSEs do this in an optimal fashion.

Here’s where things start to get a bit crazy.

RSoP stores its group policy evaluation results in the WMI repository. The results can be viewed with WMI Explorer.

ControlUp Analyze Logon Duration

Each time it stores a RSoP result, the WMI repository gets a little bigger (and also makes it that much slower). Now, imagine a multi-session system with users constantly logging on and off. As more users log on and off, it adds more and more bloat. Also, when group policy refreshes for the user it deletes the current results and stores the results of the refresh! If you have 50 users, this operation will execute 50 times. To make matters worse, as each user re-runs their RSoP in turn, they consume increasingly more CPU, making the machine slower for everyone. Imagine those 50 users spawning WMI tasks at the same time because the Group Policy refresh interval triggered. It’s not a good thing.

“The server grinds to a halt for a minute or so every couple hours.” Ever heard that before? It might be WMI…

But what if you have a small number of users? No biggie, right?

There is a difference between concurrent users and total users. For RSoP and WMI, each plays a factor. One of the biggest challenges of RSoP, as it exists today, is that it stores data for all users that have logged onto the machine. I’m not aware of any purging of data (I’ve never observed RSoP data getting purged), so this is just a cumulative process. The more users that cycle through a machine, the more that data is stored, the larger the WMI Repository grows, the slower it operates. Good times.

ControlUp Analyze Logon Duration

Only one user is logged on, but RSoP data from another user persists in the repository.

ControlUp Analyze Logon Duration

You can kind of observe WMI in action using the event viewer, but the event viewer is cumbersome if you’re trying to see events as they are generated. You may need to check the WMI Analytic and Debug logs to see them, but they are there.

What’s the solution?

Microsoft has some interesting notes and guidance for RSoP.

RSOP logging to the WMI repository delays console and especially RDP logons

WMI impact by Group Policy RSOP logging could be tested by temporarily disabling RSOP logging as described in KB 2020286

In short, turn off RSoP!

Since the Group Policy setting is a negative with a positive application, you need to enable the “Turn Off RSoP” setting. If you Disable “Turn off RSoP” then it will be Enabled. So be careful out there! ALD will report on the current state of RSoP on your machines:

ControlUp Analyze Logon Duration

How does this impact logon duration?

The answer to this question is, “it depends.” It can impact a little, not at all, or significantly. This is pretty much the same answer for every technology that touches the logon process. 😊

ControlUp and ALD makes it incredibly easy to analyze it, though, and determine the impact and its impactor. To analyze sessions in your environment, right-click on a session, select “Script Actions,” “Analyze Logon Duration” and then “OK.”

About ControlUp Analyze Logon Duration

Or use the ControlUp Virtual Expert™️, click on the “menu” icon next to “Logon Duration,” then select “Analyze Logon Duration.”

ControlUp Analyze Logon Duration

Easy, right?

Resultant Set of Policies—or users—can be turned on and off, on demand. Just set Turn Off Resultant Set of Policies to “Enabled” and RSoP won’t be collected. But if you need someone’s RSoP results, you can disable “Turn Off Resultant Set of Policies” and either have them log back onto that box or have the user do a gpupdate.

Analyze Your Environment!

To get insights like these in your environment, start a trial of ControlUp today (it’s FREE; there’s no sales call required)! Just to one of your machines and analyze one of your recent logons. If you need further assistance getting the proper data, reach out to sales@controlup.com and one of our pre-sales engineers can help get your environment set up.

For additional information on what can impact logons check out the ControlUp blogs and check out the rest of our Analyze Logon Duration deep dive series:

 

About the author

Trentent Tye

Trentent Tye, a Tech Person of Interest, is based out of Canada and its many, many feet of snow. FUN FACT: Trentent came to ControlUp because, as a former customer, the product impacted his life in so many positive ways—from reducing stress, time to remediation, increased job satisfaction, and more—he had to be our evangelist. Now an integral part of ControlUp’s Product Marketing Team, he educates our customers, pours his heart and soul into the product, and generally makes ControlUp a better place. Trentent recently moved to be closer to family. He does not recommend moving during a pandemic.