COVID-19 and Healthcare IT Business Continuity

I worked for Alberta Health Services for seven years as a Citrix Administrator. In that role, I learned the differences between working in IT for a government entity versus a business, how to operate as efficiently as possible in a bureaucracy, and how to drive initiatives that most benefited the organization and its users. During this time, I experienced the Calgary Flood of 2013, which actually flooded the building I was working in, forcing me to work from home for a few weeks while my employer sorted out space for IT to work.

But since I am not currently working in IT, I haven’t experienced anything like the effects that COVID-19 (aka the Coronavirus) has had on workplaces in the past few turbulent weeks.

To learn more, I reached out to a few members of the ControlUp Community to talk to them about what their experience has been like so far. It’s been a week since Work From Home has been enforced in my hometown, and in other areas it’s been a bit longer or shorter.

In part one of this two-part series I spoke to Rory Monaghan of RoryMon.com.

Rory works for a healthcare organization based in Arizona.

Trentent: Are you working from home full time now?

Rory: Yes, I’m working from home.

Trentent: Everyone except “essential personnel” are working from home now. Everyone has their kids at home while they’re trying to work.

Rory: Yeah, I was talking to my sister today and a lot of companies that don’t have the work from home thing are forcing people to work from home now. In one mind employers are thinking that this is awesome, maybe we’ll be able to do this in the future… But then they might look at employees’ work from home productivity being less than when working in an office. But that’s not really fair because people will have their kids at home, or maybe are dealing with elderly parents or immunocompromised family members, so this wouldn’t be the best time to measure productivity when working from home.

Trentent: My wife and I will be working alternate days or times so one of us can watch our child. For sure, I expect this to impact my productivity.

Rory: My sister’s boss mentioned that they understand and hope the employees will just do their best — that’s all they can ask for.

Trentent: Did you have remote workers before COVID-19?

Rory: Yes we have. As luck would have it, we’ve had enabling remote work as a business initiative over the last six months or so.

It seems that the company is growing and there was demand to expand the physical offices people work in, but I believe when considering the cost of prime office real estate, particularly during that boom time, you would have been pressed to try to find somewhere more affordable. Where is more affordable? The answer was to try having them work from home. The project kicked off a year ago, but the wheels only really started turning about six months ago.

We’ve been building out a Windows 10 virtual desktop. The desktop is very performant, it has all of our core apps, it’s been tested. This was going to be one of our options. Could we move forward with it? No. Unfortunately, we can’t. The desktop is only available in our non-prod environment. And we lacked the storage to make it work. We need to expand our storage to be able to support the number of users we are expecting to take on. We have the storage coming, but it’s coming from China — which, as you know, has delayed shipments for 8-10 weeks. We lost this option due to circumstance and logistics.

What our team recommended was Citrix RemotePC. These end-users already have physical workstations in the office, they work in the office today, and they are just going remote. They have everything they need today on those desktops, they know it works and we already have 1500 people using RemotePC everyday. We know RemotePC from a Citrix Policy perspective, we know it’s been fine tuned and we know it works fine.

But…

We got pushback for that idea. The concern was around relying on the physical devices themselves. If a user shut down the machine, or if a power outage occurred, that would shut these machines down, bringing down the ability for people to work.

We do have a last option.

We have a shared desktop based on Windows Server 2012R2. It was originally built for repetitive task workers, but now the decision is to expand this out to put up to 3,000 people on to. So we’re moving forward with this option.

Trentent: And when are you cutting these users over to the remote solution?

Rory: That’s the million dollar question. We were told it was happening this week on Wednesday, but last Friday we were told to have the capacity for our remote users built out and ready for Monday.

Trentent: Wow! That’s short notice, which I guess everyone is dealing with. What are you doing about your hardware capacity for all these users? Do you even have hardware to accommodate this many remote workers?

Rory: As luck would have it, since December we’ve been upgrading our EHR environment to new servers with faster processors. We’ve been adding these new servers to our EHR Citrix Prod cluster. Because the new hardware is so much faster, we put a Citrix policy in place to direct the majority of the users to the newer hardware. This has left the older hardware with very few sessions per VDA. So we started to take the VDA’s off the older hardware which freed up our hosts.

With these hosts now available, we started to move them over to support the 2012R2 shared desktops.

Trentent: How did you get around the storage issue with these servers?

Rory: These servers already have storage assigned to them, but it’s tier 3 storage. To get this solution to work we need to use what we have. If the users coming on demand better performance, they will just have to live with it. It is what it is — for now. Fortunately, most of the users’ applications are published apps, so they won’t rely on the shared desktops’ own storage for much.

Trentent: So it sounds like you got pretty lucky using older hardware and repurposing it. Have you done any scalability testing or is it going to be “it is what it is”?

Rory: Yeah, “it is what it is” will be it for right now. We did some scalability testing with some apps like WebEx or Jabber. That was interesting. WebEx was not that heavy for someone doing screen sharing, but for the people who got the screenshare, it was very heavy chomping through the CPU. Jabber supports offsetting down to the client and supports all the EUC infrastructure so we’re pretty lucky there.

Trentent: Any other interesting challenges?

Rory: I expect a lot of challenges. We have a project to move our multi-factor authentication (MFA) to another vendor and this is supposed to happen at the end of March. So as people start working from home they are going to work with one MFA and then a week later be told to use another MFA. I bet we’ll get hammered with calls on just that. It’s just unfortunate timing.

Our server team is also upgrading to vCenter and ESXi 6.7. Which would normally be fine, but one of our farms is on Citrix Virtual Apps and Desktops (CVAD) 7.15 CU1 — because it hasn’t been used very much — which doesn’t support ESXi 6.7. We’ve been building a CVAD 1912 farm since December, but it’s been a lower priority due to some large application upgrades that occurred in January/February. So it was a low priority while we got that work done, and everything with that farm was working just fine. We quickly had to upgrade the farm to a newer CU so the ESXi upgrade could get done. The server team had to quickly corral all of that older hardware and get them ready and available for use with these shared desktops.

It was just a perfect storm of stuff changing, right before we need to put a whole bunch of people on these servers.

So with setting up the shared desktops, the MFA, upgrades and hardware refreshes — it’s not an ideal situation. But we are working hard under the gun to try and deliver answers.

Is this going to work?

We’re going to try and anticipate the stumbling blocks when people start migrating. For instance, we have logon scripts to map network drives. We had a project to migrate these to Citrix Workspace Environment Manager (WEM), but it’s not done. So these users will be logging in without their mapped network drives. I’m hoping as we onboard people we get them to tell us what mapped network drives they need. I know some users have manually mapped network drives, so when they get their non-persistent desktop, those will be gone too. That’s going to be a tricky one to try and make available for them.

My guess is, for the next two months we’ll be in complete operations mode to try and fix and play catchup to get missing applications to the desktop.

We do have some technologies which will help us. We have App-V Scheduler and App-V which will help deliver applications. We also have FSLogix for AppMasking just in case we do need to put something in the image.

Trentent: So you’re going to be very reactive? Especially initially.

Rory: Yup.

Trentent: Sounds absolutely hectic. Can you give an example from the last two weeks of when you had to pivot on a dime to get ready for this?

Rory: We published our shared desktop out for two users to try and they came back with a requirement to print to PDF. We didn’t have a PDF printer in our image, so we had to get one. Which meant going through Security to get it approved because of patient health information (PHI) concerns, which meant it could take awhile, but it was escalated and it went through quickly. But Security made a requirement, only users who need to print to PDF should be able to and no one else.

Well, the software is in a shared desktop! This was just one of the applications that was not a good candidate for App-V but also not good to put in the image for access to everyone. One of the downsides of a published desktop vs a Windows 10 virtual desktop.

So in less than a week, we got the PDF software in the image and then we had to come up with a solution to restrict it. Just a few days ago, I installed FSLogix AppMasking into the image. But now we need to deliver the AppMasking rules, and we want to do this dynamically because cracking open the image just to change the rule file is painful.

I set up a Group Policy Preferences that just did a copy of the AppMasking rules file from a share to the rules directory. Pretty easy!

And it worked! But then it didn’t work!

It copied the rules file but the rules did not seem to apply properly. What I was experiencing was that the PDF printer was hidden for everyone, including people in the group that was supposed to enable it for them. I thought, let’s get it done, so I cracked open the image and added the rules and it still wasn’t working.

I thought, what if I remove all rules, let the desktop start, and then add the rules after.
Since I didn’t want to do a copy for each machine, I created a ControlUp Script Action that would do the copy for me. I ran it and it worked!

I don’t know why yet! But the end result is users who need the PDF printer, now see it. And those who don’t need the PDF printer, don’t see it.

Trentent: When the situation changes and you can get back to building a better planned and maybe more optimal solution, what do you intend to do?

Rory: We would like to introduce a modern application delivery solution like Cloudpaging to provide better agility for deploying applications to published desktops, physical desktops and virtual desktops with less restrictions than App-V and or other current tools.

When the delays for storage clear up, we’d like to get a few more NVME drives and get our Windows 10 virtual desktops ramped up on Datrium. In that environment, we also have NVIDIA vGPUs in play so we hope the performance on that desktop will blow away the current published desktop and some of our aging legacy solutions.

We hope to also use ControlUp to show our performance on the Published Desktops during this time of demand and performance on our other legacy systems to compare to the fully built out Windows 10 desktops.

Trentent: Amazing! Thank you for your time Rory and sharing with the community!

Editors Note: Rory also shared more thoughts on his personal blog.

About the author

Trentent Tye

Trentent Tye, a Tech Person of Interest, is based out of Canada and its many, many feet of snow. FUN FACT: Trentent came to ControlUp because, as a former customer, the product impacted his life in so many positive ways—from reducing stress, time to remediation, increased job satisfaction, and more—he had to be our evangelist. Now an integral part of ControlUp’s Product Marketing Team, he educates our customers, pours his heart and soul into the product, and generally makes ControlUp a better place. Trentent recently moved to be closer to family. He does not recommend moving during a pandemic.