ControlUp Automation – Faster than human – Augmented Troubleshooting

The challenges of troubleshooting intermittent problems  

the connection between the Citrix Broker Service and the database has been lost

I’m working with a customer who’s having an intermittent issue. The database that Citrix connects to occasionally becomes unavailable — why, no one knows. Since Citrix has a 90 second delay from the database outage to the local host cache (LHC) activating, this outage is really disruptive.  If it happens during the morning logon rush or a shift change, you could be preventing hundreds or thousands of end users from working.

Citrix has asked for some CDF traces. However, this problem might only last for a few minutes and resolve itself by the time the ticket comes. 

The ControlUp Automation Platform can assist with intermittent problems in two ways: augmented troubleshooting and automatic remediation. This post will cover augmented troubleshooting.

Augmented Troubleshooting

ControlUp can pick up on the events (like the one described above) as they are detected by the broker(s) and immediately start the information capture. Since this issue is intermittent, the capture will stop when the issue resolves itself.

Typically, the ask is to get multiple traces. This means letting the environment exist in the database down state until enough CDF trace files are captured for Citrix to analyze. 

Even though Citrix has asked only for CDF traces, we can do better. I’ve configured four additional scripts to provide us even more context for what the issue might be.

The first script is a continuous ping test to validate connectivity.
The second script leverages the Citrix Test-BrokerDBConnection cmdlet.
The third script executes a packet capture of network traffic.
The fourth executes a CDF trace.  

CDF Trace Setup

The first thing you need to do is set up our environment so you can execute CDF traces.

Download CDFControl and extract it somewhere in your system. For the purposes of this post, I’ve set it up in this path: C:\Swinst\CDFControl

Open CDFControl on your Citrix Desktop Delivery Controller (DDC). Under the “Trace Categories” menu, select “All Modules”.

CDFControl CDF Control

Under “Tools” select “Create CTL File”.

Create CTL File

Save the file in your CDFControl folder.

In my examples, I’ll be storing the CDFControl.exe here: “C:\Swinst\CDFControl\CDFControl.exe”.

Copy the whole folder with CDFControl and your CTL file to all other Citrix Desktop Delivery Controllers.

You are now set up to gather additional information.

Script Actions

The first script will start a continuous ping against our database with timestamps.
The second script will use the Citrix cmdlet’s to execute a database health check.
The third script starts a network packet capture.
The fourth script will start CDF Trace.

Script 1 – Continuous Ping until the database responds

 

Script 2 – Test the Citrix Broker Database using the Citrix cmdlets

 

Script 3 – Capture network traffic

 

Script 4 – Execute a CDF Trace during the outage

Configuring the Trigger for Automation

Now you need to configure the trigger. For Citrix database outages you can key in on a specific Citrix event. In this case, that’s Event ID 1201 from the Citrix Broker Service:

Event id equals to 1201 Log is Application Source is Citrix Broker Service Type equals to Warning

You then can configure an Event-based trigger with the following properties:

Now, you attach multiple script actions to the trigger.  

By default, ControlUp will execute automation sequentially, with each script waiting until the other has completed before starting the next. For this scenario you need to set up the automation for simultaneous script execution. Simply add a registry value to the ControlUp Monitor:

Registry Key: HKEY_USERS\S-1-5-20\Software\Smart-X\ControlUp\MonitorSvc
Name: MaxAutoExecJobsTarget
Type: REG_DWORD
Value: (in decimal) number of allowed simultaneous scripts

Let’s see it in action:

 

In my next blog I’ll cover how to use ControlUp Automation to automatically remediate the problem. 

 

Leave a Reply

Your email address will not be published. Required fields are marked *