How to Use AI to Get Mission Work Done

The best first AI task is one where the person doing the work stays in the driver’s seat. Mahoney described working with operations officers during a field exercise. Automating pieces of their seven-step planning process left them as passengers who had to direct and correct a new participant. One officer summed it up: “The problem is not the process. The problem is in the process.” His test for a good fit is whether AI speeds up the work a person already drives or gives them more to manage. He offered a second check for value: do the AI’s responses move the task forward, or do they keep asking the user for clarification?

People and operational context are the two biggest barriers to moving AI from pilots into daily work, according to Mahoney. Users arrive with very different experience of AI, and handing them a tool they must learn to manage adds to their workload. Real conditions then test the tool: information that changes, disconnected networks, and smaller models than the prototype used. When it proves unreliable, people go back to their old methods. His advice for reliability is to narrow the data. At one maintenance customer, Legion scopes each use case to its own manuals and system of record, such as IBM Maximo for facilities maintenance. A narrower scope improves reliability and repeatability, and it can allow a smaller model, which lowers cost.

Protected and classified data can be handled with attribute-based access controls that give AI the same permissions as the person using it. Mahoney explained that Legion runs incoming data through an ingestion step that detects portion markings and tags each document. Only users and groups with matching attributes can reach it, and agents inherit those limits instead of acting with privileged access of their own. To scale a workflow that works, he recommended three moves:

• Rely on the people who recognize the same shape of work elsewhere.
• Reuse proven capabilities.
• Expand within the same domain using the integrations already in place.

At the Department of Energy, Legion’s work plan control workflow grew into permitting and inventory.

Legion Intelligence builds Legion Packs, role-based apps that connect to the systems a team already uses and keep people in charge of every step that matters. The aviation maintenance example from this session is one of them, and Mission Builder is how teams build new Packs and reshape existing ones as the work changes.

Transcript

This transcript was produced through automated speech recognition and has been lightly edited for readability.

Opening and Introductions

Emily Jarvis: Hello, everyone, and welcome to our session, How to Use AI to Get Mission-Critical Work Done. By way of introductions, I'm Emily Jarvis, and I'm the Senior Manager for Events here at GovLoop. Today I have the added pleasure of also being your moderator, so thanks so much for joining me. And a special thank you to our sponsors at Legion Intelligence for helping us put on this session today.

Let's get right into it, because many agencies have access to AI tools, but few are using them to complete mission-critical work from start to finish. The challenge in getting AI into the workplace is identifying the right tasks, connecting AI to existing workflows, and of course, maintaining human oversight.

So today, our expert will walk through how to identify the best first use case, distinguish between general-purpose AI and task-focused agents, maintain accountability and traceability, and scale success across teams without starting from scratch. It's gonna be a great conversation, but before we get to that, we do have a few housekeeping items to go over.

We've got some great resources available for you on the console under the Resources button along the bottom panel. This is also where you can see our speaker and learn a little bit more about him, so make sure you check that out. If you need the closed captioning feature, just look for the Show Captions icon along the bottom right-hand side of your screen. That's where you'll find that added accessibility.

Of course, we hope you don't have any technical difficulties, but if you do, send us a message through the Q&A tab, and my GovLoop colleague, Maddie, will get you all sorted out. You can use that same tab to submit questions to our speaker, and we want your questions, so don't you dare be shy.

You can also chat with your fellow government employees using the chat button along the bottom panel. It's Halloween week, so let us know any spooky use cases you're using for AI, of course. And finally, we are recording this training, so check your inboxes tomorrow, because we'll be emailing you a link so you can watch the entire session on demand.

All right, with all of that housekeeping out of the way, I want to introduce you to our speaker. Joining us today is Steve Mahoney, who is the Senior Vice President for Product at Legion Intelligence. Steve, thanks so much for being with us today!

Steve Mahoney: Thanks for having me.

Moving AI From Experiments to Work People Depend On

Emily Jarvis: Of course! All right, so, AI. This is the topic that is really resonating with folks right now. It's something that I think we've talked about for years, and now we're seeing agencies really move from experimentation and pilot phases into looking at where AI could really have mission impact.

My first question for you here, Steve: when we talk about that evolution of moving AI from experimentation to mission-critical work, what does that actually look like in practice?

Steve Mahoney: That is the key question that every organization inside of the government and outside of the government is asking themselves. We talked a lot internally at Legion about 2026 being the year when folks got to the end of their experiments and the end of their trials, and maybe even the end of some of the purchases that they first made.

They had to take stock and say, is this delivering value? Is this something that's real for us, and are we going to double down on it? I think experimentation is important. It's what proves that AI has the capabilities that are necessary for you to be able to take advantage of it. Can it actually perform some sort of task?

But when we move into mission-critical work, that's when we go to prove: can people actually depend on it? Is it something that's reliable? We might play around with things like chatbots, or upload manuals, or generate a SITREP. But at the end of the day, humans are still doing all of the glue, the connective tissue, where they're taking that work and moving it from system to system, and leveraging it in that way.

The opportunity to take advantage of AI for mission-critical work is to move from just getting answers to actually completing meaningful work, meaningful tasks, and having humans be accelerated because of that.

When I think of mission-critical work and shifting away from experimentation, that's the goalpost for me: how can I actually move into work that can be reliable and that humans can actually take advantage of to complete real work?

Choosing Work Where Humans Keep Driving

Emily Jarvis: In that mindset, then, of real work, how should agencies really determine what tasks or workflows are a good fit for AI, and which ones should remain primarily human-led?

Steve Mahoney: I like the human-led component of that. The reason I have this perspective on things is that I've had to eat a lot of humble pie. I'm a platform-focused person. I've worked in the enterprise quite a bit. I've worked in public health, but I've never worked in the national security industry writ large.

At an exercise that I was a part of a few months ago, we got to work really closely with a series of ops officers, and MDMP is the planning process that they leverage. We came in thinking, okay, let's bring in these agents that we can leverage. They can take the process, it's like seven different steps, we can automate a bunch of it, and we can deliver some real value to these folks in the field, in this exercise.

It was not a very big success. And it wasn't the technology. An ops officer pulled me to the side, she actually works at Legion Intelligence, and she had the best phrase. It was, "Steve, the problem is not the process. The problem is in the process."

When you shove AI into trying to pull apart the process, you're suddenly making humans no longer the drivers. They're still in the car, but as a passenger. So they're stuck on this journey, and now they have to direct yet another person, they have to coordinate, and they have this intuition and this experience that they can no longer leverage in the same way.

When I think about that, and I think about how I find opportunities for a great fit for leveraging AI, I'm thinking about the places where I can have humans continue to drive. If I can't automate the whole process, then I need to accelerate the work that is being done, and not force a human to suddenly be a participant in their own process that they were driving before.

That might seem a little bit vague, but you can imagine all of those processes where you are the glue, where you are the one ushering the work. If all of a sudden you had somebody jump in that doesn't have the same level of intuition and experience you do, you're now managing them and the work in your hands.

So that, for me, ends up being the litmus test: have we chosen a program or an opportunity where we can actually have meaningful impact on the work, or are we just going to create more work for the people that are driving this?

Emily Jarvis: That's so interesting, because I don't think I've ever heard someone phrase it in that way before. It almost made me feel like, in the scenario you were describing, you were having AI come in and be micromanaged by the human, and no one in the history of the world has ever wanted to be micromanaged.

So to then have to do that for an AI tool that is supposed to make your job better and easier, I could see how that could create such friction. You're hearing from leadership and executives, "Use AI, it's gonna make your life easier," but your practical experience is, no, it's not. It's adding friction into a process that was already working all right as it was.

Steve Mahoney: Yeah, we don't wake up and go, how can I automate myself today? We don't think in those terms as human beings. We think of the work that we are doing. So I think framing things around the work to be done, the mission to be accomplished, and the structure of that is the opportunity that's in front of you.

Checking That AI Responses Are Valid and Add Value

Emily Jarvis: All right, we're already getting some great questions into the chat, which I love to see. I'm going to encourage folks: Steve and I are purposely going to be driving most of this chat today through your questions, so put them in early and often. There are no bad questions.

Steve, we got one in here from our friend Richard, and it says, what restrictions or checks should be made to ensure the AI you're providing gives you valid responses that are value-added? What kind of checks do you see being put in place here?

Steve Mahoney: I think we have all become a bit accustomed to citations and sourcing. There are different ways in which different products and AI tools will produce this, whether it's just saying, hey, here are the sources where I got my information. That's the bare minimum; we're trying our best.

Something that is meaningful, though, is being able to actually see the citation attached to the response directly. For instance, we had to incorporate portion marking and understanding classification. That means that in the different passages of information, we actually link directly: this specific part of the response is cited and portion marked based on this resource. That is a grounding mechanism. That is something we always want to look for in a lot of the responses.

But there is a second part of that question that I think is really important, which is the value added. My favorite example of this: we worked with the Department of Energy, and they use what's called WPC, or Work Plan Control, for all of their projects. It's quite a tedious process. It feels red-tapey and bureaucratic, but at the same time, it's important.

It helps them with hazards and safety, and with how they think about whether or not they're going to need to break ground, or if there's electrical work. For this lengthy process, we leverage a series of agents that do things like templated writing and research to pull together the standard operating procedures.

When it comes to value added, they handed us, like a gift, another phrase: "I don't want to talk with a dumb chatbot. I don't want to just talk with an AI and have me, again, holding the hand of AI and making it work with me to do this process. I want it to understand WPC. I want it to understand the process for permitting and for moving me forward."

So a good litmus test is: are the responses that the AI model is giving you driving things forward, or are they just clarification statements? For me, that is always the check. Whenever I'm using our product or any other product, is what I'm using driving things forward? Is it carrying the work?

Or is it making me do the clarification, the carry of the work, while I'm just updating the AI intern that I now have? I think that's really the way that I think about valuing and measuring.

From Assistants to Agents to the Command Layer

Emily Jarvis: I appreciate that, and Richard, I hope that answers your question. If it didn't, let us know, and we'll continue to follow up. I also wanted to note that we have our first polling question up right now. We're trying to figure out from you all how you would describe your organization's current use of AI, so go ahead and fill that out for me.

While they do, Steve, I think we're seeing agencies move, and you were just alluding to this a little bit, from general-purpose AI assistants to building actual agents. But are agents enough to support the mission work, or is there a next frontier or advancements that we're seeing agencies adopting?

Steve Mahoney: I did hint at this in my last response a bit. When we got started, five-ish years ago, we were watching folks in the Department of War use AI tools in their earliest of days to just answer questions, get information, and generate SITREPs and things of that nature as ARs. The challenge we were seeing is that they were still managing these things.

Then we saw the year of the agents a few years ago, where all of a sudden these things could take on more tasks, and AI models, given the right direction, could go accomplish more work. That's all fine and good, but it's still a place where humans are having to drive a lot of the work and be that connective tissue.

Individual agents are really great at completing tasks, but mission work requires coordination across people and information and decisions and systems. So we see agents as essentially a building block, but not necessarily the destination.

The next frontier for us, or the way that we see the world, is starting to see software fundamentally change around human intent, and humans operating at what we refer to as the command layer. I can put the objectives and the intent down into the work that I am doing, and agents and software can coordinate a lot of that work for us.

I can still retain authority. I can still be on the loop as a human, but I don't necessarily have to be that glue, that tissue that connects together all the different systems of record. Because then I'm not really saving much work if I have to maintain all of that state in my head.

That, for me, is where I see things going. As we start to see what I would consider truly agentic software, software that shifts the burden of the work down and allows me to operate at the command layer, that's really what I want to see as the frontier coming next.

Poll Results: How Agencies Use AI Today

Emily Jarvis: Okay, we're gonna go ahead and end the poll here and get the results. Topping out our poll, when asked how you would describe your organization's current use of AI, 35% said using AI for specific workflows or tasks. 25% said they were exploring, and another 21% said they were testing a few pilots or proofs of concept.

Then you've got a very small group saying not really using AI, or scaling AI across multiple teams or programs. What I'm gathering this poll is telling us is that the majority of our audience is exploring AI, but they're not necessarily to the point we were talking about at the beginning, where you're using AI from the beginning to the end of the workflow. Is that your interpretation here, too?

Steve Mahoney: Yeah, I think so, and this is interesting, too. We've done this poll for a number of years, a number of quarters, and it's been really great to see the evolution. Things are definitely skewing more towards how people are exploring how they can roll it out.

I especially love to see scaling AI across multiple teams and programs, because that tells you that there is a measurement of value and outcomes that are being delivered. I don't know if people know who else is participating, but having roundtables and being able to be a part of those kinds of discussions is great. I have found the most value in being able to sit and watch how they have bumped into some of those challenges.

I think that's the next segment, but for me, seeing how folks are thinking about where they're exploring, and the things that they're looking to identify in terms of opportunities, I'm very excited for some of the things that we're gonna chat about. I think there's some real opportunity for you to measure that and take advantage of it really quickly.

People and Operational Context as the Biggest Barriers

Emily Jarvis: Agreed. Alright, next question here for you, Steve. Barriers, they exist. There are probably a few of them that agencies could run into as they try to move from a pilot program to something that employees can really rely on for mission work. What are some of those barriers, and when and where do agencies come in contact with them?

Steve Mahoney: This is probably reflected in the poll. When we did this same poll two-ish years ago, at a broader enterprise conference that we were a part of, the biggest barrier that folks talked about was people. It is change management. Even if you're in an agency where it's a stick more than a carrot, it's still a difficult process, because people have a varied experience that comes with AI.

Actually, maybe I do have a slide. It's a visual that I sometimes use. I've done some talks on thinking about building for people, and I talk about this spectrum, wherein people have had a variety of experience with AI in their personal lives, not even just with the tools that they use at work.

It goes from the folks that have never touched it to where most people sit, which is in the "I use AI instead of Google" or "I use AI as an assistant or co-pilot that's attached to the work that I do." They're not often over in the "I look to AI to orchestrate and take advantage of a lot of the work that I do," because that requires a lot of human faculty and a lot of investment in managing that solution, that product.

You've now given somebody more work than just, "Here's AI, go take advantage of it." You've actually said, "Here's this brand new piece of technology that you now need to learn and figure out how to manage. And by the way, we also haven't figured out how to manage it, so if you could help us all along for the ride, use the art of the possible, figure out the magic."

There's a lot of promise behind that, but I think that becomes the biggest human challenge that people face.

The second component of this is, say you get past that human challenge, you now have the challenge of the operational context in which this is going to be deployed. For me, it's the real mission environments. Sure, in this really great experiment, I can add documents and generate SITREPs and things. That's all great. I can interrogate a manual and find the instructions.

Now I have to do that where the information is changing. I'm now in a disconnected environment. I now have an unexpected failure of some sort, or maybe I ran out of tokens, or maybe I'm using a smaller model than in the prototype that we ran. In that new operational context, trust becomes an issue. I deliver this to people, it's a new technology, and now I have an unreliable technology, and I've lost trust.

That means that people will default back to working the same way that they did before. They'll use the same tools, because those are faster in their heads, because that's how they've been operating. They can fit the full context in their brains. They don't have to offload it to another system, and that's how they can operate.

That truly ends up being the biggest barrier: everything that can affect trust in how you deploy these solutions. That is what you need to get past when it comes to defeating some of these barriers. So people and the operational context are probably the two biggest ones.

Building Trust by Fitting the Job

Emily Jarvis: It's so interesting, because we've done hundreds of trainings over the course of my time here at GovLoop, and it never is the technology that is the challenge. It is the people and the process.

But I think what you're articulating here is a feeling that a lot of folks have, because there's so much energy. There are a bunch of executive orders out right now, and there are a bunch of state- and local-led goals and objectives about using AI. I think we've seen it in the private sector, too. There's been such a focus on AI, and now we're learning that in some pockets they actually have to bring more people back into the organization.

They didn't think through the people and process part of it first; they just went all in on the technology. So, with that in mind, Steve, what can agencies do to build that trust with these tools and with their employees, so that they can see some of those operational efficiencies that have been promised, or noted, that the art of the possible is there?

Steve Mahoney: I feel like, even just through the few questions in our conversation, we could have titled this "How to Build for People" or "How to Build for Humans," because that fundamentally is the challenge.

Yes, it's a Millennium Falcon, and I have a 10-month-old daughter. I will, one day, get my hands on that LEGO.

But the human part of this really is the biggest hurdle for a lot of folks to unlock, and that means not just building technology and throwing technology at people. You nailed it. We do all of these trainings, and that feels like it's already off on the wrong foot. The fact that we have to introduce a new technology and retrain people to rethink their brains means they're already going to be in a defensive position.

They're thinking, "Okay, I am a human who knows how to do my job. I've been doing it for years." Or, "I am somebody who's fresh into this new world, and I am trying to figure out how to navigate it, and you all don't even have expertise in this, and you're training me on it."

What I end up thinking about is that it's not a lack of software or a lack of technology. It's that people are still the connective tissue. People are the ones gathering information, triaging it, and manually moving it between applications.

So even if we give them an LLM, or even if we give them something that could generate agents, like a lot of the tools that are out there, GenAI.mil, and even some of the frontier models that you have access to in the broader public sector, the challenge is that those things are not connected to all of your systems. They don't do all of the handoffs. They aren't governed and repeatable, so what you create as a solution is different from what I create as a solution.

The way that we think about this is that the models and the tools should work with the systems of record. They should be connected to all of those things, and it should be repeatable, in the sense that when I log into that agent, or build that agent or tool, the way that I'm connecting to it should be the same way that you, Emily, connect to it, or the way that somebody else connects to it.

When I go to build an application, or an agent, or some sort of workflow that can sit across those different things and maybe triages some of the work that I'm doing, it should be the same way that you do the triage. Maybe there is nuance in that. I'm a Tier 1 and you're a Tier 2, so you have all of the higher-order operating procedures and things like that.

But at the end of the day, if it's not repeatable, if it is just open-ended and there are no boundaries, then we end up being that connective tissue. We haven't really accelerated any of the work. We've just changed the shape of it, and that's really not what this is all about.

I hope that makes sense. The experience that you build and deliver needs to fit the job. It needs to be about the work that people are doing, and not about how to put technology first and then let the humans figure out how to leverage the technology.

Connecting AI to Existing Systems: An Aviation Maintenance Example

Emily Jarvis: Alright, Richard is over here throwing tons of questions into the chat, and I love that engagement, Richard. We are going to get to all of your questions, I promise. But Steve, I've got one more of my own first. Selfishly, I'm gonna make you answer my question first.

That's around this idea of that secondary step that you were talking about. How can agencies connect AI to existing systems and workflows without creating another disconnected tool or adding complexity for employees? What are the steps that you take, and what does that look like?

Steve Mahoney: There is no lack of ways to connect these systems. I'll throw out all of the acronyms: there are the MCPs, there are the APIs, there are the direct tools that we can use. There are myriad ways to do this, and this has been around for the longest time. The way that we have thought about this is actually turning the way that you build these around.

We joked about how humans don't wake up and think about automation. The same should be true when we approach using these solutions, or building these solutions. Don't approach it as a technology builder, where you go, "Okay, I'm gonna get a webhook, and I'm gonna send the data over here."

Instead, you should be thinking in terms of, "Okay, what is the work that these folks are gonna be doing?" Well, they're doing first-fix prioritization. They're in aviation maintenance. Actually, I have a screen. Gotta have a visual.

This is a product that we've built out that's referred to as Aviation Maintenance. We have a fleet of, no pun intended, maintenance experiences. Here, we're thinking from a human perspective. I don't want the human to come in and have to start from a chat box and say, "Okay, chat with me and tell me your intention." I want to start with information that's actionable straight away.

As you start to think from the human perspective, you ask: how does this person do their job, where in their job do they need to be connected to different systems, what information is the most important for them right now, and what's the biggest action that they can take, that they can bite off, that would be meaningful in moving things forward?

That's really where we think, "Great, if that's the case, you're gonna need access, in this particular case, to the inventory system, the asset management system, the work order system." You're gonna need to connect all of those different things so that the human doesn't have to be the one that comes in and does it themselves.

From there, you might end up with something like this, where it's a visual experience, and you're like, "Okay, well, where is AI in this?" That's actually part of the whole point. AI is a tool. It is a thing that enables us to accelerate the work that we're doing. It is not meant to be something that's shoved in your face as a piece of technology.

In this particular case, I'm looking at what aircraft are mission capable right now. This tells me that what I've got to go do right now is figure out how to move that number for the 21 craft. You get the idea. I have to now think about how I'm going to go and fix those craft to get them operational.

AI should be helping me move through that. That means that behind the scenes, I need to connect the data where it makes sense. But I'm working across that. I'm not having to move from one system to the next, take this data, and create the ticket myself. That's where AI can go and help me: "You go update those tickets. I'm gonna be the one that's taking on and putting the plan together to move this forward."

Emily Jarvis: I loved that example. I'm going to extrapolate here just a little bit and make some stuff up, but of the 75 or however many planes, 54 of them were ready to go, and 21 of them had some sort of maintenance issue, whether it was mechanical or they just need gas. Who the heck knows what's wrong with these planes?

But what your dashboard is showing you is where each of those planes is at any given moment. So instead of having to go in and manually look up each plane, that information is now in front of you, and that's what you're saying the AI is doing. Is that right?

Steve Mahoney: That's correct, and it's sorting them by the ones that need to be addressed first. It's taking advantage of the fact that there's a thing called cannibalization that happens for a lot of these aircraft as they maintain them: which ones would be the most appropriate for us to select and perform that with?

And the important part, which you touched on, is that I've come in here and this has presented this information to me straight away as something that is actionable, that I can take and jump on top of, instead of it being something where I have to go in and pull the information from a bunch of disparate sources.

Importantly, it's keeping me in the driver's seat. It's presenting this information for me to move this forward and accelerating this work, instead of deciding which ones it's going to triage, doing some of them, and then I have to coax it and move it into the right position if it's made the wrong choice.

It's keeping me in the driver's seat: here are my recommendations, here's how I say that you should be moving forward. That's really what we want to see from a lot of these agents today, as we build that trust and reliability for them to be able to take on larger tasks.

Emily Jarvis: Okay, great.

Narrowing the Data Scope for Reliability

Emily Jarvis: Alright, we have quite a few questions in here, but before we get to those, I'm going to throw up our second poll question, so folks can be answering a question we have for them while you answer their question. I love that.

Trust is a topic we were just focusing on, and Richard's question says, in developing trust, is a repository of examples that demonstrated a proven solution, so the examples you can ask of it. I think this is getting to the heart of where you are getting your data and your information from, and how you are verifying that that information is accurate.

So, what advice do you have for culling through this data? Because we know AI is only as good as the data it's using.

Steve Mahoney: That's right. I think this is going to feel counterintuitive, but the way that I tend to operate is I often create a narrower scope and use case for the data set that I'm working on. We talk about how it's only as good as the data that we feed it. Well, the more data you feed it, the more opportunity you have for challenges and issues in that.

I'll use this aviation maintenance example. There are hundreds of different aircraft, and the facility that we work with is actually also doing facilities maintenance and ground vehicle maintenance, and the number of assets that they have is absolutely ginormous. They get hundreds of thousands of work orders on a yearly basis.

If we just took all of that data and all the manuals and assets, dunked it all together and said, "Here you go, work with this information, you can do it," that would actually pose a challenge. You'd have overlapping information, maybe some of the information is actually conflicting, or you just get confusion. You have large, vast quantities of data.

So, intelligently separating out those different use cases and the data sets while still maintaining them as available to the models is the way in which we've been able to create these experiences that are incredibly efficient and reliable. In the aviation case, we're only providing manuals and connectivity to the systems of record, and the scope in that system of record, for just the aviation use cases.

In the facilities maintenance use cases, we're connected to IBM Maximo for their operations, and that's also where they store their assets and the manuals for those. So instead of saying, "Here's all of the SOPs and all of the manuals," we have a smaller repository of data at any given time, even though the total potential repository of data would be large.

That keeps things much more efficient. It also means that you can reduce the size of the model that you're working with, and it means that things can be much more repeatable. It's great for cost, great for reliability, great for repeatability. Those are all wins in my book.

Emily Jarvis: Interesting. If what I'm hearing you say is correct, instead of taking an encyclopedia perspective, where you are dumping in every piece of information, you're taking the CliffsNotes version, but the CliffsNotes of the particular part you want to learn.

So, if you're looking at Encyclopedia Britannica, you're just focused on the oceans, not on the whole world at the same time, so that you can keep that information accurate and usable, and not confuse it with other parts of the information or data that exists there.

Steve Mahoney: Maybe, to stay on the encyclopedia reference, I don't know if you grew up with the suite of encyclopedias. Although they are obviously arranged in alphabetical order, think about them being arranged by subject matter, and then breaking off a portion of subjects and saying, "Actually, this is the only chunk of the encyclopedia we need for the task at hand. We are going to reference the things in here."

That's not restricting access to the other data, unless it requires restriction, but it makes it so that you can actually have those references be locked into just that portion of things.

A great example of this is a test that I often run. We call it the ums and ahs test. We transcribe a lot of the meetings that we have internally, and we capture a lot of that information. It falls into our repositories, where we can look through it, and we have meeting notes, transcriptions, and summarizers.

If I leave all of that data in with all of the rest of the encyclopedic knowledge of our organization, and I just go in and search blindly across everything and say, "Tell me what you know about Steve Mahoney," it would probably say he says "um" and "ah" a lot. That's because the vast majority of data that we have on Steve is transcriptions of him saying "um" and "ah" in meetings.

So the value of that data starts to go down, unless you are putting the right context around why you're consuming it and how you're consuming it. It's a bit of a management piece and overhead. That's why we use platforms like Legion Intelligence, to avoid having to do that pinpoint management.

Emily Jarvis: Interesting. I would be horrified to know how many times I say "um" or "ah" in a single day, probably on this recording alone. It would terrify me, but that is a great example.

Poll Results: Security, Privacy, Governance, and Shadow AI

Emily Jarvis: Alright, let's see, there we go. Let's go ahead and get the results of the poll. When we asked our audience what is the biggest barrier to AI for mission-critical work, coming in at the top is security, privacy, and governance concerns.

I think that makes total sense, especially in a government space. The data that they are protecting is obviously essential. Steve, anything you want to note here on this poll?

Steve Mahoney: I unfortunately didn't get to see the results.

Emily Jarvis: That's because I haven't shared them yet. There you go. Alright.

Steve Mahoney: It makes perfect sense that security, privacy, and governance would be some of the challenges. There's actually quite a bit of nuance, even in that response alone, because many agencies especially deal with different information levels, and even moving data between information levels can be problematic. That ends up being the most daunting task.

A bigger challenge that you probably also experience, for those of you who are scaling programs and looking at different ways to do this, and I saw that in the responses, is what we call shadow AI. You have people that are pulling up models and other technologies, dropping in their own documents, and sharing them out to cloud-based AI systems.

Now you're like, "Great, we've got all of our private information that we have worked so hard to maintain being sent out to all of these different systems." That really ends up being a lot of the biggest barrier for the work that's being done, and it's one of the things that we've had to work hard to secure. So, that makes perfect sense.

I think identifying the right use cases is also related to that first poll pretty heavily, and to the fact that folks are looking for the opportunities to get in front of it. When we get to the next section of questions, I think we'll cover a few of those topics pretty well.

Handling Protected Data With Attribute-Based Access Controls

Emily Jarvis: For sure. I did just want to add on, real quick, that Jay put into the chat: "What are your thoughts when critical mission data includes data that is protected or needs to be handled in a restrictive manner? How do you build operational context using an existing LLM or building your own?" This is just what we were talking about. Real quickly, what advice do you have there?

Steve Mahoney: Quick plug that Legion Intelligence has had to solve this problem and thinks a lot about this. We do have some posts online that talk about this, but I will say there are ways to approach this problem. I don't think it's intractable.

When I hear or read about data that is protected or needs to be handled in a restrictive manner, I immediately go to different classification levels of data and portion marking and things of that nature. There's a place to start, which is the most restrictive place to start: even just role-based access controls and managing access to the data. That is one place that you can operate if you're just using language models and attaching them to the data that you have access to.

LLMs themselves don't have agency to break their barriers or their guardrails in that sense. It's when you start to get into workflows and agents that can make decisions and plan that you want to introduce a lot of that additional thinking.

But when it comes to working with portion marking and classifications, we've introduced an additional layer into our platform: ABAC, or attribute-based access controls. The way that we do that is, as the data comes into the system, it first takes an ingestion path that runs a model against that data to ask, "Are there portion markings in here?"

It's well documented what the different portion marking stylings are, so you can create those rules and actually have that data be tagged in such a way that it says this is classified, or this is top secret, or whatever the case may be. Then only folks that also have those attributes attached to their users or their groups would be allowed to have access to that data.

That is effectively how we keep the walls between different pieces of data. And because the humans don't have access to that data, the LLMs and the agents within our platform don't have access to that data. You can't say, "Hey, agent, go use your privileged access and go access data."

It should very much be the case that any time a human is interacting with AI in any way, they're sharing the same level of permissions. That needs to be pushed all the way down. That's a principle that we operate with as well.

Task-Focused AI Use Cases in Federal and Defense Work

Emily Jarvis: Love that. Alright, since we are rapidly running out of time here, and we have quite a bit more to get through, I'm going to go ahead and throw up the third poll question for folks to answer while things are going.

Steve, I'm wondering if you could share some use cases you're seeing for task-focused AI in federal and defense environments today. I know you shared the maintenance example, but are there any other examples you can share that articulate what we've been talking about so far?

Steve Mahoney: Yeah, and maybe I'll tie this answer together with some of the other opportunities, just looking at the poll responses and how people are thinking about the people part of things and rolling it out.

Again, I'm a platform person, and the way my brain is wired is I'm often looking at repeatable capabilities. I'm not looking at whether that use case or that opportunity is the exact same thing. I'm often looking and asking, how is this work, and the way that they do this work, similar?

If I were going to break down some of the use cases that we engage with very frequently right now, at the top of the list, of course, is writing and operational coordination. It started in the early days with SITREPs, FRAGOs, or OPORDs, the common things. But as we moved into work that is more coordinated across a process, even that simple stuff becomes really impactful.

For the Department of Energy, we do this work plan control process, where we can hand off approvals and assignments. As you are putting together a project, it can assign that work out and say, "Hey, there's a permitting job opportunity that we need to do here before we break ground."

Moving over from writing is investigation and analysis: preparing data, analyzing it, and building reports so that somebody in command can actually make decisions and then disseminate that information out. There is a lot of prep work that goes into that. Again, it's about preventing the human from having to be the connective tissue across all those systems, and having an agent run and do that investigation and analysis.

Then my recent particular favorite is this idea of triage. Triage is something that I didn't realize we all just do all the time. What are you doing when you open your inbox in the morning, besides panicking and drinking coffee? You're triaging. How am I going to respond to all of these? What's the top priority? What's the next step that I'm going to take? Marking that one as unread, not going to bother with it right now.

That is actually something that you can apply to so many different things. Beyond the maintenance one that I shared, we work with security operations controllers in Tier 1 security. You can think of myriad different use cases for that kind of operation.

To roll it into the next way of thinking about how to leverage that technology, that's how you think about these programs and rolling them out. Look at the opportunity where you can bite off one of those things with a capability. Look at preparation for reporting and operational coordination. Yes, how do I prepare the report? But also, how do I make sure that it gets handed to the right person, where approvals and assignment will happen?

Cool. What's the next use case that is tied to that in some way? What is the next way that I can leverage this same capability that I've created trust and reliability in, and attach to it?

I apologize, I'll share my screen again, but going back to this aviation maintenance example, that's how we think about expanding from just understanding fleet health to: how do I actually give the techs the ability to go interrogate the manuals? How do I help them build CAR responses?

So, you start to see that you begin with one part of the use case. Then, as you expand out, it's based on those capabilities and being able to expand into the work being done, so that they can keep the folks that are doing that work in their head.

Scaling a Successful Workflow Without Starting Over

Emily Jarvis: Once you've got this pilot, or this workflow that is working for you and maybe your small team. I love how you phrased it: the work doesn't have to be the same, but the way they work is going to be helpful to create these repeatable AI workflows.

But once an AI workflow proves successful, what are the best practices that you've seen for scaling it to other teams, programs, or mission areas without rebuilding everything from scratch, but knowing every team is unique in their own way?

Steve Mahoney: We're gonna stick with our theme. A lot of it is people. People are your champions, and it is people that are gonna understand the shape of that work. People can recognize the shape of the same work that they're doing in other work.

So bring them into that next piece of work, where they can say, "Hey, that is interesting. It looks very similar to the reporting that I was doing before, but it takes a slightly different shape." That's something that you can leverage from them.

And again, reusing those capabilities. I think that is important to how you think about expanding it. And then finally, build on the integrations and that connective tissue that you already have.

So instead of thinking about going and attacking an entirely new program, or an entirely new suite of use cases that maybe uses that triage or that writing and reporting but is fully outside of this domain, where you've got to start from square one again, maybe consider expansion inside of that same use case.

In the work plan controls, for us, we moved from that into permitting, and from permitting into inventory, and building the deep scientific research, and how they're actually performing the experiments. So think about connecting all of that work together, because now you have champions and you have context that people have built up.

And you already have connection across all of those different systems of record. That's really how you can start to expand and feel meaningful impact. And you can take stock and say, "Look, we've actually built tools around this whole big thing."

That growth is really important, more so than just "we cover the gamut of LLM-generated documents," or whatever the case may be. I think you'll find a lot more value in that.

Poll Results: Document Preparation, Reporting, and Review

Emily Jarvis: Yeah, no, I agree. All right, the results for our third poll are up right now, and I hope you and our attendees can see these as well. We asked: which type of mission work offers the greatest near-term opportunity for AI in your organization? And we have a clear winner here in document preparation, reporting, and review.

Steve Mahoney: Yeah, and I think, fundamentally, when we build applications, we have something like 84 different mission applications that we have built or have emerging concepts around from ideas. Universally, they all have some sort of preparation, reporting, writing, or review that comes with it, because that is the artifact that humans end up being attached to most frequently.

There's an agentic future where some of that won't be as necessary. But if we want to get started and we want to bring people along for the ride, I do think that is one of the biggest opportunities that we have.

I'm very excited to see that investigation and analysis is also there. I have one more slide that I'd love to share that talks about attaching the type of work that we do to the solutions themselves, and that is this idea that different solutions fit different types of work.

When we think about folks that are in execution, there's a prescribed task or resolution, and they're doing some sort of triage. You can imagine the ease with which we could spin up assistants and co-pilots that could help them get started. And that's why we jump on those things so frequently, because they're also the most numerous tasks that we have.

But as we start to shift to more open-ended command and things like that, that's when we need to look at more sophisticated solutions that can help prepare all of that ambiguous information and provide the background so that you can go and answer those unclear questions.

To me, what we're seeing in this data is that the things we know are the highest volume are on the left-hand side of the spectrum. But the stuff that's on the right is hard, and it's gonna be crazy valuable when we can start to take it on. We need to build the trust to get there, and that's what a lot of people are doing today.

Turning Agent Work Into a Repeatable Recipe

Emily Jarvis: Love that. All right, two more quick questions for you. The first one here comes from Richard, and he said: after an AI tool or agent has been developed, can AI provide you the recipe for how it was created, i.e., a user manual to summarize the lessons learned for repeating it and suggesting it for other examples and use cases?

Steve Mahoney: Yeah, I love this. So there's the thinking process. You can actually see, oftentimes, the thinking of the AI model or agent as it's going through it.

But this is exactly how our platform works, and the prescriptive way that we've decided to build what we all refer to as a harness for working with these models. As AI goes through the process of, say, planning and deciding how it's going to triage, we don't just do that as a one-off.

I mean, you could come in and say, "Hey, I'd like you to triage my inbox, I don't want to deal with my email right now." But on the other side of that, you want it to be repeatable. I like how it's described as a recipe. We describe it as a workflow and the tasks that are attached to the workflow.

That is actually how you can think about, in the very short term, being able to take open-ended work that people do and decompose it into deterministic pieces that are really inexpensive, or stuff that requires less of the power of the model. So: give me frontier intelligence, give me the most powerful model for me to plan and build out a really great way of approaching these problems.

Let's take the triage of the email. It's gonna open your email, it's gonna go through the top 10, and it's gonna build a rubric for how it assesses what's a good email to look at or not. It's gonna draft some responses.

After you've done that, you can actually have the model take that plan and say, "Cool, what's the deterministic part?" The rubric. Always use the rubric this way, but assess the email in a different way every time.

And as you think about decomposing that work, now you have a repeatable process. And that's a thing that you could use fewer tokens for every time. It's faster, and you can actually hand that off to somebody else, and they get a repeatable way of doing it.

That is, in fact, exactly how the Legion Intelligence platform works, and that is how we would propose anybody approach the same set of problems.

Emily Jarvis: Love that. And as someone who can't cook, recipes saved my life, so I have to imagine that with AI, recipes would do much the same here. I love that approach. All right.

How AI Will Change the Way Government Gets Work Done

Emily Jarvis: Final question here for you, Steve. We've talked about so much on today's session, but specifically about AI performing real tasks. Looking ahead, how do you see AI fundamentally changing the way government gets work done, and what should agencies be doing today to prepare for that shift?

Steve Mahoney: Yeah. I love the way that you phrased it around the getting of the work done, because, fundamentally, right now we are in a place where agents and AI are accelerating a lot of the things that we're doing.

But what we should be talking about in the shift that's coming is what we think of as truly agentic software: software where humans and agents are collaborating. Collaborating, but it's very much one where humans can very much be in the driver's seat, effectively operating at what we call the command layer.

Being able to push their intent, or the objectives, down, and then the agents are able to operate as the connective tissue. So a future is one where, instead of us having to be the connective tissue across all of these different systems in order to grab the data, build the thing, and put it over into this next thing, agents can handle that task.

And we can operate at the decision layer. We can operate in the way that allows us to shape the work that's being done, and then continue on to the next bit of work.

The anecdote that I'll give is from the maintenance customer that we work with. They have something like 100,000 work orders a year, and they have humans just trying to chug through and process all of those work orders.

The shift for them is to go from that to being able to just push down the command and the objectives, and then focus on procurement and inventory management and other things that are such high-value tasks for them, and the triage work can actually be pushed down into the stack. That is what we think of as really, truly agentic software, and that's what we want.

So how do you prepare? Investing in accessible data, and thinking about the connected systems you have. You don't have to throw out the systems of record that you have.

But having an understanding of how you connect to all of those systems, if you were going to put something in place today, is one thing that you would want to think about. How would I grant access to all of these systems and connect them where it's not just a human that's doing it?

And then, thinking about the measurable outcomes that you're looking for. Whenever we come back and say, "Hey, we did a training program, we gave them an LLM, and we said, go nuts, tell us what you think," that's not actually giving somebody something and saying, "Here's the measurable outcome we want to get to."

Starting from the measurable outcome and thinking about what the things are that you want to accomplish. What's the work that could be done today that is not done because humans are stuck triaging other things? If that is a measurable outcome that you can set yourselves up for in terms of measuring success, that should be where we land in the future, for sure.

Closing

Emily Jarvis: Steve, I really appreciate your time joining us today and providing insights.

Steve Mahoney: Yeah, it's been fun.

Emily Jarvis: Those slides that Steve pulled up are over there in the Resource Center, so make sure you check those out. I think they're a great visual to add to the discussion. But unfortunately, we are out of time today.

A few reminders before everyone logs off. There are some great resources, like those slides that I mentioned, available for you to download, so be sure to check those out. We'll also be sending you a link to the on-demand version of this session tomorrow, so look for that in your inboxes.

On behalf of GovLoop and Legion Intelligence, I hope you enjoyed this session. As always, we really appreciate and value the work that you do, so thanks for taking the time out of your busy schedules to join us this afternoon. Have a great rest of your day, and we'll see you in our next training. Bye!

Steve Mahoney: Thanks, everyone.