AI assistants are part of daily work for most of us, but few people know what’s going on behind the chat window. Chris Twiest, my colleague and ControlUp’s Solution Innovation Manager, built a tool to show what’s happening behind the scenes. He starts with a raw language model and adds one layer at a time until it’s working on real IT data.
It Starts with Tokens
An AI doesn’t read words the way we do. It breaks text into tokens, small chunks that each map to a number, and works only with those. Picture a sentence cut into puzzle pieces, with the model’s whole job being to guess which piece fits next. Everything an assistant writes comes from repeating that guess, over and over.
The Raw Model Knows Very Little
On its own, a model has no idea what day it is, who it’s talking to, or what’s in your environment. It has no memory of what you said a moment ago, either. Chat apps get around that by sending the entire conversation back to the model every time you press Enter.
Each Layer Adds Something
- Instructions. Every AI product hands the model a set of written ground rules, called a system prompt, that cover how to behave and basics like the current date.
- Memory. When an assistant seems to know you, it’s usually working from a handful of saved notes slipped into each request. That means they take up space and can be outdated or incorrect.
- Tools. A model can’t do anything in the real world. It can only request that something be done. With the Model Context Protocol (MCP), the app does the actual work: it executes the request and returns the result as text. Think of MCP as a standard plug that lets an AI connect to a system like ControlUp without custom wiring. In the demo, a single question about endpoints results in several calls to the ControlUp MCP server.
- Skills. This is the part we found most useful. In the demo, the model pulled the right data but still got the answer wrong because it didn’t understand how that data was stored. Some values use a different scale, and placeholder “no data” entries pulled the average off. A skill file is a short set of written guidance that teaches the model those quirks. Once it was loaded, the same model, using the same tools, answered correctly.
Think of it like hiring a very smart new analyst. Giving them access to your dashboards isn’t enough. They also need the unwritten knowledge about what the numbers mean.
Why This Matters for IT Teams
The model is just one part of the picture. What makes an AI assistant useful is the real-time data it can see, the tools it can use, and the expertise written into its instructions. That’s the thinking behind how we approach autonomous IT at ControlUp: decisions are only as good as the data and context behind them, and action only helps when it’s grounded in both.
IT teams can try this themselves. With ideas from the video and ControlUp’s MCP server and tools, customers can build their own AI assistant that works with their ControlUp data.
Watch & Read More
Watch the full walkthrough above, or read Chris’s own write-up on Workspace Guru. His article explores how to run multiple AI models and key privacy considerations before connecting AI to your data.
About the ControlUp Innovation Guild
The ControlUp Innovation Guild is our fast-track to solving real-world IT problems, driven by a team of technical experts in scripting, automation, and solution design. They build lightweight, practical tools that extend the ControlUp ONE platform, shaped by customer feedback and community insight. Chris is one of the people behind it. The Guild shares early-access tools and scripts through the Download Center, and select community members can help shape what comes next. To get involved, join our Slack community.
